Metabolism Queue Reconciliation

metabolism_queue_reconciliation is a runnable model of a durable job queue. It stands up a synthetic SQLite store in a temporary directory, hands out work on a lease, recovers the job when the lease expires, and then runs a consistency check that flags impossible job/run/log states for a person to review. metabolism_queue_reconciliation is a runnable model of a durable job queue. It stands up a synthetic SQLite store in a temporary directory, hands out work on a lease, recovers the job when the lease expires, and then runs a consistency check that flags impossible job/run/log states for a person to review.

The component surfaces one Engine Room bundle, metabolism_runtime, as a first-class part of the public system. The component surfaces one Engine Room bundle, metabolism_runtime, as a first-class part of the public system. It runs four fixture scenarios: two clean ones that must compute end to end, and two deliberately broken ones that must be caught. Each scenario runs against its own temporary database that is created and torn down per case. It emits metadata-only result records and a CLI entry point, and it asserts nothing about the live private queue.

The honest unit here is the named reconciliation rule, not the word "healthy." A clean store should produce no findings; a contradictory store should produce a finding tagged with the rule that describes the contradiction. The honest unit here is the named reconciliation rule, not the word "healthy." A clean store should produce no findings; a contradictory store should produce a finding tagged with the rule that describes the contradiction.

Purpose

A job queue that drifts into contradictory state is hard to trust. Two states are the ones this component plants and catches: a job marked running with no run row recorded, and a run that finished while its job still says running. A job queue that drifts into contradictory state is hard to trust. Two states are the ones this component plants and catches: a job marked running with no run row recorded, and a run that finished while its job still says running. A queue that passes over these silently, or auto-repairs them without a record, hides exactly the situations an operator needs to see.

The component makes reconciliation explicit and rerunnable over a bounded store. It does not fix the contradiction. The component makes reconciliation explicit and rerunnable over a bounded store. It does not fix the contradiction. It recomputes the job/run/log relationship, names the rule that failed, and marks the store for review.

How it works

The component wrapper in src/microcosm_core/organs/metabolism_queue_reconciliation.py loads fixture cases and drives them through the bundle. The component wrapper in src/microcosm_core/organs/metabolism_queue_reconciliation.py loads fixture cases and drives them through the bundle. The bundle in src/microcosm_core/engine_room/metabolism_runtime.py owns the queue, the blackboard ledger, and the reconciler.

Per-case scratch-store evaluation

build_result reads the case files with _fixture_cases (a sorted glob over *.json in the input directory, or a single file), opens one temporary scratch directory, and evaluates every case inside it. build_result reads the case files with _fixture_cases (a sorted glob over *.json in the input directory, or a single file), opens one temporary scratch directory, and evaluates every case inside it. _evaluate_case forces the bundle's own expected_status to pass, calls the bundle's evaluate_case, and reads back the observed status and which reconciliation rules fired via _fired_reconciliation_rules. A case declared positive passes only when the bundle computed cleanly and no rule fired. A case declared negative passes only when the bundle ran cleanly, the expected rule id from EXPECTED_NEGATIVE_CASES fired, and the declared expectation was rejection. build_result then requires at least one positive and one negative case, every positive clean, every negative rejected, and both expected negative ids present, before it reports overall pass. run writes the result, board, and validation records with write_json_atomic, plus an sign-off record when an sign-off path is given. Every record sets body_in_receipt to false.

Inside the bundle, evaluate_case dispatches on the case kind to one of five runners. The queue path uses enqueue_job, which inserts a job under a partial unique index so a second job on the same idempotency key while an earlier one is still active is blocked. Inside the bundle, evaluate_case dispatches on the case kind to one of five runners. The queue path uses enqueue_job, which inserts a job under a partial unique index so a second job on the same idempotency key while an earlier one is still active is blocked. claim_next_job leases the next queued or recoverable job to a worker and sets a lease expiry. requeue_expired_jobs moves claimed or running jobs whose lease has passed back to recoverable and clears the owner. The blackboard path uses append_claim_event to record an assertion and then a contradiction, and build_blackboard_projection drops any assertion that a later contradiction, expiry, or supersession event invalidated, so a contradicted claim projects to zero active claims.

reconcile is the check. It walks the jobs currently in claimed or running state, fetches the latest run for each with latest_run_for_job, and builds a ReconciliationFinding when the run is missing, when a finished run sits under a still-running job, or when the run's log file is missing or stale. reconcile is the check. It walks the jobs currently in claimed or running state, fetches the latest run for each with latest_run_for_job, and builds a ReconciliationFinding when the run is missing, when a finished run sits under a still-running job, or when the run's log file is missing or stale. It counts findings by rule, sets status to needs_review when any finding exists and healthy otherwise, and every finding carries the action operator_review_required. It never mutates the jobs it inspects.

flowchart TD Cases["Load fixture cases _fixture_cases"] Scratch["Temp SQLite store per case"] Bundle["evaluate_case run one scenario"] Recon["reconcile job/run/log check"] Decide["_evaluate_case pass or reject"] Result["build_result + run metadata-only records"] Cases --> Scratch --> Bundle --> Recon --> Decide --> Result

Negative cases

Two scenarios plant a contradiction and confirm the reconciler catches it.

running_job_no_run_row_rejected enqueues a job, claims it, and forces its state to running without ever recording a run. running_job_no_run_row_rejected enqueues a job, claims it, and forces its state to running without ever recording a run. reconcile finds a running job with no run row and fires running_job_no_run_row. The case passes only because that rule fired and its declared expectation was rejection.

finalized_run_running_job_rejected enqueues a job, claims it, starts a run, and finishes the run without finalizing the job. finalized_run_running_job_rejected enqueues a job, claims it, starts a run, and finishes the run without finalizing the job. The job still says running while its run has a completion time. reconcile fires run_finalized_but_job_running. The two clean scenarios, queue_lease_recovery_ok and blackboard_projection_ok, must produce no findings at all; a rule firing there would fail the positive case.

Prior Art Grounding

This follows standard software-engineering practice for durable work queues and lease recovery, and for reconciliation passes that flag inconsistent records for human review rather than repairing them in place. This follows standard software-engineering practice for durable work queues and lease recovery, and for reconciliation passes that flag inconsistent records for human review rather than repairing them in place. The queue design uses a partial unique index for active-key idempotency and a claim lease with expiry recovery, both common patterns in SQLite-backed job systems. The local prior art is the Plectis paper-module coverage contract: a reader page states what source row and public fixtures can be checked, names the generated projections as navigation aids, and keeps its claims inside the evidence boundary. No external citation is claimed for the specific reconciliation taxonomy; it is named in the source constants and this page reads it back.

Validation Result record Path

Run the component over its public fixtures:

A pass means both clean scenarios computed with no findings and both planted scenarios were rejected by the expected rule. A pass means both clean scenarios computed with no findings and both planted scenarios were rejected by the expected rule. From the repository root, the coverage contract and corpus parity check confirm this page still matches its source record:

PYTHONPATH=src ./repo-pytest tests/test_plectis_paper_module_coverage_contract.py -q --tb=short
PYTHONPATH=src ./repo-python scripts/build_doctrine_projection.py --check-paper-module-corpus

Source-drift handling for later passes

If a later pass changes the bundle source, fixture manifest, runtime locus, or record authority, refresh this page and rerun both commands before treating the structured source record as current. If a later pass changes the bundle source, fixture manifest, runtime locus, or record authority, refresh this page and rerun both commands before treating the structured source record as current. These are the validation result records for this module.