Engine Room Command-Run Singleflight

When two callers fire the same command at once, this component runs it exactly once and hands the second caller the first caller's captured output. When several agents or background tasks fire the same command at the same moment, the plain outcome is several identical subprocesses doing the same work at once. That wastes the machine, and where the command writes something it can corrupt shared state by writing twice. This component answers one question: when two callers ask for the same command at the same time, can the system run it exactly once and hand the second caller the first caller's captured output?

Is public refactor

It is a public refactor keeping the executable core: content-addressed keys, an fcntl leader lock, captured output that followers replay, and completed-run reuse on request. It is a public refactor of an internal command-run helper. It keeps the executable core: content-addressed command keys, an fcntl file lock that elects one leader per key, captured stdout and stderr that followers replay, and completed-run reuse when the caller asks for it. It does not import or expose the internal live command-run state tree, and it does not stand in for a scheduler.

Whole file is

The whole file is src/microcosm_core/engine_room/command_run_singleflight.py. The whole file is src/microcosm_core/engine_room/command_run_singleflight.py. Every function named below is defined there.

Purpose

The key spans argv, cwd, environment, and scope

Rather than compare names or argv alone, the component keys on argv, cwd, an environment slice, and scoped worktree state, so two runs collapse only while their code is identical. A loose deduplicator compares command names or memoizes by argv alone. That is wrong here, because the same argv can mean different work: edit a file the command reads and the correct answer changes. So the interesting decision is how this component decides two requests are the same. It does not compare names. It builds a key over the argv, the working directory, a small slice of the environment, and the scoped worktree state. Two runs collapse into one only while the code they would see is identical. Change a file in scope and the key changes, which forces a fresh run instead of serving a stale result.

How it works

One entry function, two role functions

The pipeline is short: one entry function takes a per-key lock, decides a role, and returns a result record, while the heavy work lives in two role functions. The pipeline is short. One entry function takes a per-key lock, decides a role, and returns a result record. The heavy work lives in two role functions.

build_command_key hashes into a dirty_fingerprint

build_command_key hashes argv, cwd, and a dirty_fingerprint over scoped Git or file state, keeping only a PYTHONPATH and PYTEST_ADDOPTS environment slice and no absolute paths. build_command_key consumes the argv, the resolved cwd, a resource class, the scoped paths, and an environment mapping. It hashes the argv, labels and hashes the cwd, and computes a dirty_fingerprint over the scope. Inside a Git repository that fingerprint carries HEAD, the porcelain status hash, the working diff hash, the staged diff hash, and per-file content hashes for the scoped paths. Outside Git it falls back to the scoped file-content fingerprint alone. The environment slice is narrow: PYTHONPATH and PYTEST_ADDOPTS. The key is a dictionary of hashes and labels, not absolute paths, so the records stay public-safe. run_command_singleflight then reduces it to a 24-character key_hash.

run_command_singleflight elects a role under the lock

run_command_singleflight rejects empty argv, then under an fcntl.LOCK_EX per-key lock reads the active record and becomes a follower, a reuse, or the leader. run_command_singleflight rejects empty argv with a ValueError before it builds anything. It resolves the cwd and state root, builds the key, and opens the per-key lock file under fcntl.LOCK_EX. Holding the lock, it reads the active record for this key. If an active run is still running and its process is alive or inside the pending window, this caller becomes a follower. If an active run has completed and the caller passed reuse_completed=True, it becomes a reuse. Otherwise it writes fresh active and run metadata, releases the lock, and becomes the leader.

_run_leader runs the subprocess once

_run_leader starts the subprocess once, captures stdout and stderr to per-run files, marks the run completed with its exit code, and returns a leader RunReceipt. _run_leader starts the subprocess once with subprocess.Popen, records the active run with its pid, then blocks on communicate to capture stdout and stderr. It writes those to per-run output files, marks the run completed with its exit code, updates the latest-by-key record, and appends leader lifecycle events. It returns a RunReceipt with role set to leader.

_wait_for_active replays without rerunning

_wait_for_active polls until the run completes and replays the leader's output with the same run_id, and on a dead process or timeout returns stale_or_timeout with exit code 124. _wait_for_active is the follower path. It polls the active record until the status reads completed, then reads the leader's captured output files and returns a record that carries the same stdout, stderr, exit code, and run_id as the leader. It never launches its own subprocess. If the active run's process dies or the run does not finish inside the wait window, the follower returns status: stale_or_timeout with exit code 124 and still does not rerun.

flowchart TD A["argv, cwd, env, scope paths"] --> B["build_command_key hash argv + cwd + env + scoped state"] B --> C["run_command_singleflight take per-key fcntl lock"] C --> D{"active run for this key?"} D -- "running" --> E["_wait_for_active follower replays captured output"] D -- "completed and reuse_completed" --> F["replay completed result"] D -- "none or stale" --> G["_run_leader run subprocess once, capture output"]

Negative cases

Two positive and two boundary fixture cases

evaluate_fixture_dir checks four cases: single_leader and completed_reuse pass, while scope_mutation_changes_key and missing_command_rejected guard the key and reject empty argv. evaluate_fixture_dir runs four named cases and checks each observed result against the expected flag. Two are positive: single_leader confirms one caller wins the lock and runs the command, and completed_reuse runs a counter command, then reuses the completed result and checks the counter stayed at 1. Two are boundary cases. scope_mutation_changes_key writes a scoped file, builds a key, rewrites the file, builds the key again, and requires the two key_hash values to differ, so edited scoped content cannot be laundered into an old run. missing_command_rejected calls the entry function with empty argv and requires the ValueError, so an empty command is never accepted. A pass is case_count: 4 and passed_case_count: 4.

Prior Art Grounding

Lineage in Go's singleflight package

The pattern is duplicate-call suppression from Go's singleflight, adapted to local subprocess runs with a content-addressed key, scoped state, and captured output replay, not a distributed lock. The pattern is duplicate-call suppression, the same idea as Go's singleflight package, `golang.org/x/sync/singleflight`, which defines a namespace of work where duplicate calls for one key share a single in-flight execution. This component borrows the leader and follower shape and the shared result, then adapts it to local subprocess runs with a content-addressed command key, scoped worktree state, captured output replay, and an explicit line between collapsing active duplicates and caching completed results. It is singleflight for command execution, not a distributed lock service.

Validation Result record Path

Exercise the four fixture cases:

PYTHONPATH=src python3 -m microcosm_core.engine_room.command_run_singleflight evaluate-fixtures \
  --input fixtures/first_wave/engine_room_command_run_singleflight/input \
  --json

Run the focused tests and the corpus parity check:

PYTHONPATH=src ./repo-pytest \
  tests/test_engine_room_command_run_singleflight.py \
  -q --basetemp /tmp/microcosm-command-run-singleflight
cd microcosm-substrate && PYTHONPATH=src python3 scripts/build_doctrine_projection.py --check-paper-module-corpus

The two-process race the tests add

The focused tests add a real two-process race where roles resolve to leader and follower, both share one run_id and counter=1, and the side effect increments exactly once. The focused tests add the real two-process race: two callers start the same command, their roles resolve to leader and follower, both records share one run_id, both replay counter=1, and the side-effect counter increments exactly once. A pass means the public fixture behavior and the bundle-backed JSON projection stay reproducible. These validation result records do not prove scheduler behavior, daemon behavior, or distributed locking.