Agent Completion Faithfulness Audit

Completion language needs observable evidence

The finish-line sentence an agent writes is admitted only claim by claim, once observable git and pytest evidence backs each part of it. agent_closeout_faithfulness_audit checks the sentence an agent writes when it finishes a task: "I committed the change, closed the ledger item, and the test passed." It runs the supplied public fixture through real git and pytest subprocesses and admits each claim only when an evidence object it can observe actually backs it.

A fixture repository is copied into a throwaway directory, committed with real git, and its HEAD is read back. A declared test span is run, and the exit code alone decides whether the span passed.

Four metadata-only records carry the result

It emits only metadata-only records: a result file, a board, a validation record, and an exported bundle validation record. It emits metadata-only result records: a result file, a board, a validation record, and an exported bundle validation record.

Purpose

A done report is only text

A finished-work report is just text: the commit, the ledger row, and the "passed" may or may not be real, and an audit that trusts the wording certifies nothing. When an agent reports that work is done, the report is text. The commit may or may not exist. The ledger row may or may not be there. "The test passed" can mean the test ran, or it can mean nothing was checked. A loose audit that trusts the wording certifies all of those as done and learns nothing.

Ran and passed are separate facts

It asks one question per fixture: is each completion claim backed by an evidence object that exists, and is "passed" backed by an explicit exit-zero check rather than by the phrasing of the claim? The component answers one question over a fixed fixture: is each completion claim backed by an evidence object that genuinely exists, and is a "passed" claim backed by an explicit exit-zero check rather than by the phrasing of the claim? The distinction it defends is narrow. "The span ran" and "the span passed" are separate facts. A completion sentence that conflates them is the exact failure mode here.

How it works

Two entrypoints forward to evaluate

run and run_agent_closeout_bundle are the two entrypoints, and both forward to evaluate, which carries the whole mechanism. The runtime is src/microcosm_core/organs/agent_closeout_faithfulness_audit.py. run and run_agent_closeout_bundle are the two entrypoints; both forward to evaluate, which carries the mechanism.

evaluate runs in this order:

1. Load closeout_claims.json and fixture_ledger.json. If public_fixture_repo is not a directory, emit CLOSEOUT_PUBLIC_FIXTURE_REPO_MISSING and stop.
2. _prepare_public_fixture_repo copies the fixture repo into a temporary directory and runs git init, two git config calls, git add ., and git commit, then git rev-parse HEAD. Six subprocesses in total. The observed HEAD is returned. Each subprocess row is a return code and two sha256 digests from _sha256_text, never raw output. _run_subprocess builds the five setup rows in that metadata-only shape.
3. _select_pytest_python picks the first interpreter that answers pytest --version with exit zero, so the span runs under an interpreter that actually has pytest installed.
4. Walk the claim rows. A commit claim verifies only when its commit_ref is HEAD or the observed head, else CLOSEOUT_FAKE_COMMIT_CLAIM. A task_ledger_cap claim verifies only when its cap_id is present in fixture_ledger.json, else CLOSEOUT_FAKE_CAP_CLAIM. A pytest_span claim runs python -m pytest <nodeid> -q inside the temporary repo: span_ran is true when the process exits 0 or 1, and passed is true only on exit 0. A span that did not run emits CLOSEOUT_FAKE_TEST_CLAIM. A claim whose expected_pass is true but whose pass_status_checked is false, or whose span did not exit zero, emits CLOSEOUT_PYTEST_PASS_STATUS_NOT_CHECKED.

A pass counts pass status separately

A pass records five counters, where verified means the evidence object exists or the span ran, and pass status is tallied separately only when the exit-zero check held. A pass records verified_claim_count, git_subprocess_count, pytest_span_ran_count, pytest_pass_status_checked_count, and head_verified_by_subprocess. Here verified means the evidence object exists or the span ran. Pass status is counted separately, only when the exit-zero check held. That keeps a referenced test distinct from a passed test in the record itself.

flowchart TD Claims["completion_claims.json commit, cap, span"] --> Eval["evaluate()"] Ledger["fixture_ledger.json cap rows"] --> Eval Repo["public_fixture_repo"] --> Prep["_prepare_public_fixture_repo git init..commit, rev-parse HEAD"] Prep --> Eval Eval --> Pytest["python -m pytest nodeid -q"] Pytest --> Eval Eval --> Pass["pass: verified counts, metadata-only record"] Eval --> Reject["reject: fake commit, cap, test, unchecked pass"]

Negative cases

One mutated row per rejection

evaluate_negative_case mutates one claim row at a time and confirms the matching rejection fires. The four cases named in EXPECTED_NEGATIVE_CASES: evaluate_negative_case mutates one claim row at a time and confirms the matching rejection fires.

• fake_commit_claim: rewrites the commit ref to all zeros, expects CLOSEOUT_FAKE_COMMIT_CLAIM.
• fake_cap_claim: points the cap at an id missing from the fixture ledger, expects CLOSEOUT_FAKE_CAP_CLAIM.
• fake_test_claim: names a test node that does not exist, expects CLOSEOUT_FAKE_TEST_CLAIM.
• unchecked_pass_claim: sets pass_status_checked to false, expects CLOSEOUT_PYTEST_PASS_STATUS_NOT_CHECKED.

Losing a case loses the proof

If any of these rejections stops appearing in the observed cases, the component no longer proves it refuses fabricated commit, cap, test-node, or unchecked-pass claims. If these stop appearing in the observed cases, the component no longer proves that it refuses fabricated commit, cap, test-node, or unchecked-pass claims.

Prior Art Grounding

Claim, artifact, and evidence stay separate

The posture is claim-then-evidence verification drawn from FEVER, TruthfulQA, and the artifact-evaluation tradition: commit, ledger, and span refs must line up with observed objects before completion language is admitted. The posture is claim-then-evidence verification rather than trust in fluent summary text. FEVER framed fact extraction and verification as a task over cited evidence. TruthfulQA showed that fluent answers can be untrue without an explicit truthfulness check. The wider artifact-evaluation tradition keeps a claim, its artifact, and its validation evidence as three separate things. This component applies that separation to agent completion: commit refs, ledger refs, and test spans must line up with observed objects before the completion language is admitted.

Validation Result record Path

./repo-pytest tests/test_agent_closeout_faithfulness_audit.py -q --basetemp=/tmp/microcosm_agent_closeout_faithfulness_audit_pytest
./repo-python scripts/build_doctrine_projection.py --check-paper-module-corpus

What a focused-test pass certifies

A focused-test pass means completion language was checked against fixture evidence across seven named witnesses and all four negative classes; the corpus check only proves the bundle and instance agree. A pass on the focused test means completion language was checked against fixture evidence: three verified claims, at least five git subprocess witnesses, one pytest witness, one span that ran, one checked pass status, head_verified_by_subprocess true, and observation of all four negative classes. The corpus check only proves that the source record and the generated instance agree.