Engine Room Metabolism Runtime

Long-running runtime's durable

A long-running runtime's durable record of jobs, leases, runs, and claims drifts, so this component reads that state alone and asks which rows are inconsistent and which a person should review first. A long-running agent runtime keeps a durable record of its work: jobs waiting to run, leases held by workers, runs in flight, and claims asserted on a shared blackboard. That record drifts. A worker dies mid-run and its lease is never released. A run finishes but the job it belonged to still reads running. A launch log goes stale because nothing writes to it. This component takes the durable state alone and answers one question: which rows are inconsistent, and which of those should a person look at before anything touches them.

Is synthetic sqlite

It is a synthetic SQLite exercise, not the live runtime: a single stdlib file that creates a real database, drives the queue, lease, run, blackboard, and reconciliation paths, and emits one JSON record. It is a synthetic SQLite exercise, not the live runtime. It ships fixtures and creates a real database file, drives the queue, lease, run, blackboard, and reconciliation paths over that file, and emits one JSON result record. The whole component is src/microcosm_core/engine_room/metabolism_runtime.py, a single file that uses only the Python standard library.

Touches no live

It touches no live runtime: it reads no private runtime database, dispatches no worker or provider, and does not stand in for distributed-database behaviour. It reads no private runtime database, dispatches no worker or provider, and does not stand in for distributed-database behaviour.

Purpose

What reconciliation refuses to repair

Reconciliation tags every finding operator_review_required and refuses to auto-repair ambiguous state, recovering only the expired lease that has a clean recovery path via requeue_expired_jobs. The interesting choice is what the reconciliation pass refuses to do. It reads the jobs and runs tables, applies its rules, and tags every finding operator_review_required. It does not auto-repair. An expired lease has a clean recovery path, so requeue_expired_jobs moves it back to recoverable on its own. But a running job with no run row, or a finished run whose job still reads running, is ambiguous. The safe move is to surface it, not to guess. A loose runtime that guesses here corrupts the record; this component draws the line and stays on the safe side of it.

The blackboard is append-only

Claims are never edited or deleted in place. An assertion is one event row. A contradiction, expiry, or supersession is a separate event that points back at the assertion it invalidates. State is reconstructed from an append-only history, so the reason a claim is no longer active stays on the record.

How it works

Mechanism lives in

The mechanism lives in one module. The stages below run in the order the fixture harness drives them.

Storage opens a WAL SQLite store

connect opens a WAL SQLite file with foreign keys on and ensure_schema creates the jobs, runs, and blackboard_claim_events tables plus a partial unique index over the active states. connect opens a SQLite file with PRAGMA journal_mode=WAL, normal sync, and foreign keys on, then calls ensure_schema. ensure_schema creates three tables, jobs, runs, and blackboard_claim_events, and a partial unique index idx_jobs_active_idempotency on idempotency_key restricted to the four active states in ACTIVE_STATES (queued, claimed, running, recoverable).

Queue insertion and lease recovery

enqueue_job rejects a duplicate idempotency key that is still active, claim_next_job leases the next queued or recoverable job, and requeue_expired_jobs recovers lapsed leases to recoverable. enqueue_job inserts one job row and returns (job, inserted). When a job with the same idempotency_key is still active, the partial unique index raises sqlite3.IntegrityError, which the function catches and reports as inserted=False. Once the earlier job reaches a terminal state the key is free again. claim_next_job selects the next queued or recoverable job by priority then creation time, stamps a claim_expires_at lease, and moves it to claimed. requeue_expired_jobs finds claimed or running rows whose lease has lapsed, clears the owner and expiry, moves them to recoverable, and returns how many it recovered.

Runs and the finalize-job flag

start_run moves a job to running and complete_run records a return code, propagating a terminal job state only when finalize_job=True, which makes the still-running-job defect reachable on purpose. start_run inserts a run row and moves its job to running through update_job_state. complete_run writes completed_at and a return code; it only propagates a terminal job state when the caller passes finalize_job=True. That flag is what makes the "finished run, still-running job" defect reachable on purpose.

Blackboard events and projection

append_claim_event appends one immutable event of a given kind, and build_blackboard_projection replays the whole log to return the assertions that were not invalidated, with active, event, and contradiction counts. append_claim_event appends one event of a given kind (claim_asserted, claim_contradicted, claim_expired, claim_superseded) and never mutates a prior row. build_blackboard_projection reads the whole event log, collects the assertion ids named by any invalidating event, and returns the assertions that were asserted and not invalidated, with counts for active claims, total events, and contradictions.

Reconciliation surfaces four defect rules

reconcile walks claimed and running jobs against their newest run and emits a ReconciliationFinding with operator_review_required for each of four defect rules, returning healthy or needs_review. reconcile walks the claimed and running jobs, pulls the newest run for each with latest_run_for_job, and emits a ReconciliationFinding for each defect it sees. Every finding carries an expected value, an observed value, and the action operator_review_required, and derives a stable finding_id from its rule and object. The four rules are running_job_no_run_row, run_finalized_but_job_running, running_job_stale_launch_log, and running_job_missing_launch_log. The pass returns status healthy when it finds nothing and needs_review when it does, plus per-rule counts, the findings, the CLAIM_CEILING string, and the scope boundary list.

Fixture harness and CLI

evaluate_case runs one named case in an isolated scratch directory, evaluate_fixture_dir passes only when every case met its expectation, and main exposes this as the evaluate-fixtures command. evaluate_case runs one named case against an isolated scratch directory and compares its status to the expected status. evaluate_fixture_dir reads every *.json file in the input directory, runs each case, and returns status: pass only when at least one case exists and every case met its expectation. build_parser and main expose this as the evaluate-fixtures command, printing the JSON record and returning a process exit code of 0 for pass and 1 for fail.

flowchart TD Fixtures["evaluate_fixture_dir reads *.json cases"] Store["connect / ensure_schema WAL SQLite: jobs, runs, blackboard_claim_events"] Queue["enqueue_job / claim_next_job requeue_expired_jobs"] Runs["start_run / complete_run"] Board["append_claim_event build_blackboard_projection"] Recon["reconcile ReconciliationFinding per defect"] Record["JSON record status, rule counts, findings, scope limit"] Fixtures --> Store Store --> Queue Queue --> Runs Store --> Board Runs --> Recon Board --> Recon Recon --> Record

Negative cases

Five fixture cases, three forcing findings

Five named fixture cases each expect pass, and three of them exist to force a reconciliation finding. The fixtures in fixtures/first_wave/engine_room_metabolism_runtime/input are five named cases, each expecting pass. Three of them exist to force a reconciliation finding.

Queue recovery and blackboard projection cases

queue_recovery rejects a duplicate key and recovers an already-expired lease to recoverable, while blackboard_projection asserts then contradicts a claim and checks zero active claims and one contradiction. queue_recovery enqueues the same idempotency key twice, expecting the second insert to be rejected, claims the job with a lease of -1 seconds so it is already expired, and checks that requeue_expired_jobs returns 1 and the job lands in recoverable. blackboard_projection asserts one claim then contradicts it, and checks the projection reports zero active claims and one contradiction.

Three cases that force reconciliation rules

running_job_no_run_row, finalized_run_running_job, and stale_log each provoke their named reconciliation rule, and any case whose observed status misses its expected status fails the whole record. running_job_no_run_row marks a job running with no run row and expects running_job_no_run_row in the rule counts. finalized_run_running_job starts a run, completes it with finalize_job=False, and expects run_finalized_but_job_running. stale_log writes a log file, back-dates its mtime by 1200 seconds, and expects running_job_stale_launch_log under a 60-second freshness threshold. A case whose observed status does not match its expected status makes the whole record fail.

Prior Art Grounding

Durable-runtime control-loop lineage

The component follows the durable-runtime control-loop lineage of autonomic computing and SRE monitoring over a WAL store, borrowing the self-management loop while keeping the jobs, leases, and findings synthetic. The component follows the durable-runtime control-loop lineage: keep the work state in a durable log, detect stale or inconsistent state, recover what has a clean recovery path, and keep that log separate from the acting dispatcher. Relevant anchors are IBM's autonomic-computing architecture and its monitor-analyze-plan-execute loop, SQLite write-ahead logging for the local durability and concurrency behaviour used here, and Google's SRE monitoring guidance for separating symptoms from causes. The borrowed idea is the self-management loop over a durable local store; the jobs, leases, claims, and findings are synthetic and public-safe.

Composition context

Admitted as a staged demo bundle

demo.py stages this component as one CapsuleExercise covering the metabolism_runtime and metabolism_reconciler targets, running its evaluate_fixture_dir in sequence as a demo, not registry integration or an admission gate. src/microcosm_core/engine_room/demo.py admits this component as one staged bundle. Its CAPSULES tuple holds a CapsuleExercise for engine_room_metabolism_runtime covering the jewel targets metabolism_runtime and metabolism_reconciler. run_capsule imports the module and calls its evaluate_fixture_dir. audit_controller_coverage checks that each bundle has its expected surfaces on disk and that the covered targets match EXPECTED_JEWEL_TARGETS. The demo runs the bundles in sequence and reports whether they pass; it is not registry integration or an admission gate.

Validation Result record Path

PYTHONPATH=src python3 -m microcosm_core.engine_room.metabolism_runtime evaluate-fixtures \
  --input fixtures/first_wave/engine_room_metabolism_runtime/input \
  --json

PYTHONPATH=src ./repo-pytest tests/test_engine_room_metabolism_runtime.py -q
cd microcosm-substrate && PYTHONPATH=src ../repo-python scripts/build_doctrine_projection.py --check-paper-module-corpus

What the result records prove

The CLI passes only when every fixture case met its expected status, the pytest suite exercises each path, and the corpus check keeps the generated row in parity, showing reproducibility and nothing more. The CLI emits a record with organ_id: engine_room_metabolism_runtime; a pass means every fixture case met its expected status. The pytest suite exercises WAL and idempotency, expired-lease recovery, the contradicted-claim projection, each reconciliation rule, fixture replay, and the CLI. The corpus check keeps the generated JSON row in parity with the bundle registry. A pass shows the synthetic behaviour is reproducible and nothing more.