Agent Benchmark Integrity Anti-Gaming Replay

Every claimed pass is recomputed

A claimed repair-benchmark "pass" is downgraded to a claim and recomputed from public references before this component will repeat it. Agent benchmark numbers are easy to state and hard to trust. This component takes a set of claimed repository-repair passes and refuses to repeat the word "pass" until each one can be replayed from public references that name its evaluator, its frozen configuration, and the evidence that the run was not gamed. It recomputes every verdict from scratch. A row that simply declares integrity_pass is treated as a claim to be checked, not as proof.

Public references in, metadata-only verdicts out

The inputs are public case ids, evaluator ids, configuration hashes, and evidence references; the output is a metadata-only record of which replays cleared the boundary and why. The component reads a fixture that models a repair benchmark with public case ids, evaluator ids, configuration hashes, and reference paths to file-access logs, contamination checks, trusted-reference scores, and output replays. It keeps issue bodies, oracle patches, hidden answers, and model-output data out of every result record. What it emits is a metadata-only record: which case replays cleared the boundary, which were quarantined, and why.

A consistency checker, not a leaderboard

It is a consistency checker over public references, not a benchmark runner, a leaderboard, or a score. It checks that a set of public references is internally consistent and traces back to a command that actually ran.

Purpose

What a bare pass-rate headline hides

A headline pass-rate hides the evaluator, configuration, leakage, and selection decisions that produced it; this gate makes the claim carry them or be quarantined. A single headline like "passes N percent of repair tasks" hides every decision that produced it: which evaluator ran, whether its configuration was frozen, whether the agent could see held-out answers, whether the test cases leaked into training, and whether one lucky attempt was promoted as the result. A loose benchmark claim carries none of that. This component exists to make the claim carry it, or be quarantined.

Declaring success is not a way through

Every reference the policy names must exist and pass, or the row recomputes to quarantine no matter what it declared. The gate is deliberately hostile to the easy path. The evaluator id must be on a locked list. The configuration hash must be one the policy declared in advance. File-access, contamination, trusted-reference, and output-replay references must exist and pass. The case id must be registered up front. If any of those is missing or contradicted, the row is recomputed as quarantine no matter what it declared.

A real command run backs every pass

Each integrity_pass must cite a digest-bound artifact from a completed, exit-zero run of this component's focused test, so no claim rests on prose alone. An integrity_pass must be backed by a sanitised real command run, not only by hand-written references. Each passing row cites a real_benchmark_trace_ref that has to resolve to a copied artifact carrying a completed, exit-zero run of this component's focused test, with sha256 digests bound to a recorded run id and an explicit list of omitted live material. This is what stops a benchmark claim from resting on prose.

How it works

One orchestrator, two entry points

_build_result loads the four fixture inputs and runs every validation stage in order; run and run_benchmark_integrity_bundle are thin wrappers over it. _build_result is the orchestrator. It loads the four fixture inputs (projection_protocol.json, locked_evaluator_policy.json, benchmark_cases.json, replay_observations.json), then runs each validation stage in order and returns one metadata-only result. The entry points run and run_benchmark_integrity_bundle wrap it. run validates the fixture with the twelve negative-case files included. run_benchmark_integrity_bundle validates the exported public bundle without them, and can reuse a prior result record when _freshness_basis shows the inputs, validator source, and evidence digests are unchanged.

Per-row recomputation is the centre of the mechanism

A row passes only when every required field is present, its evaluator and configuration hash are policy-locked, its real-run trace verifies, and no evidence check fails. The center of the mechanism is _validate_replay_row. It collects the required fields listed in REQUIRED_REPLAY_FIELDS, checks the evaluator id is locked and its configuration hash is policy-declared, checks the case id is in the roster, and confirms the real_benchmark_trace_ref resolves to a digest-verified real-run artifact that is also cited among the row's evidence references. For a positive row it opens the file-access, contamination, trusted-reference, and output-replay evidence and folds any failing check or contamination flag into a reason list. The rule at the end is one line: the computed verdict is integrity_pass only when no required field is missing and the reason list is empty, otherwise quarantine.

An independent second pass over the public trace

validate_public_trace recomputes every verdict a second time from independent trace spans and turns any disagreement into a blocking finding. It reads the trace built by build_public_benchmark_integrity_anti_gaming_trace, which recomputes each verdict from contamination, file-access, and locked-evaluator spans. If a span's recomputed verdict disagrees with the declared one, or its configuration hash is not policy-declared, or its evaluator is not locked, the disagreement becomes a finding. _build_result returns status=pass only when the non-public-state scan finds no blocking hit, every sub-validator passes, and no expected negative case is missing.

flowchart TD Inputs["projection, policy, cases, replay rows"] --> Source["validate_source_module_imports manifest digest + real-run floor"] Inputs --> Rows["_validate_replay_row recompute per row"] Source --> Rows Rows --> Verdict["integrity_pass or quarantine"] Verdict --> Trace["validate_public_trace independent recompute"] Trace --> Result["metadata-only result record"]

What the shipped fixture recomputes to

Three public cases recompute to two passes and one quarantine, and the public trace has to agree before the run can pass. On the shipped fixture the three cases repo_issue_public_001, repo_issue_public_002, and repo_issue_public_003 recompute to two integrity_pass rows and one quarantine.

Negative cases

Twelve gaming attempts the fixture must catch

Each named negative case maps to a required reason code, and a run fails if any expected case is not observed. EXPECTED_NEGATIVE_CASES names twelve gaming attempts the fixture must catch, each mapped to the reason code the component has to raise. They are an evaluator edit attempt, train/test leakage, a real-trace variant of train/test leakage, an oracle patch body leak, hidden-gold access, final-answer-only grading, a model-output data leak, a score overclaim, pass-k cherry-picking, an admitted misleading test, a private issue body leak, and an unregistered case replay. The negative fixtures are a required floor rather than decoration.

Prior Art Grounding

Goodhart, reward hacking, and leakage accounting

The design follows the reward-hacking and benchmark-leakage literature: a metric stops being evidence once it becomes a target, so the references must carry the proof. The component draws on the long-standing observation that an optimized metric loses its evidential force once it becomes a target, and on the AI-safety literature on reward hacking and specification gaming. Concrete Problems in AI Safety frames reward hacking as a practical accident risk, DeepMind's specification-gaming survey collects concrete examples, and Benchmarking Benchmark Leakage in Large Language Models motivates explicit leakage documentation. The engineering pattern is simple accounting: evaluator ids, configuration hashes, case rosters, and evidence references must be present and consistent before benchmark-style language is allowed.

Validation Result record Path

Fixture output stays outside the working tree

The fixture validator runs from the repository root and writes its result to a disposable path outside the working tree. Run the fixture validator from the repository root and write its result record outside the working tree:

Then run the exported-bundle validator:

cd microcosm-substrate && PYTHONPATH=src ../repo-python -m microcosm_core.organs.agent_benchmark_integrity_anti_gaming_replay run-benchmark-integrity-bundle --input examples/agent_benchmark_integrity_anti_gaming_replay/exported_benchmark_integrity_bundle --out /tmp/agent_benchmark_integrity_bundle_receipt --card > /tmp/agent_benchmark_integrity_bundle_card.json

The focused test and corpus check are:

cd microcosm-substrate && ../repo-pytest tests/test_agent_benchmark_integrity_anti_gaming_replay.py
./repo-python scripts/build_doctrine_projection.py --check-paper-module-corpus

What a pass does and does not mean

A pass means every verdict was recomputed from the public references and agreed with the independent trace; it does not mean a benchmark claims was produced. A pass means the fixture and bundle validators recomputed every verdict from public case ids, locked evaluator references, configuration hashes, evidence references, manifest digests, and the twelve negative cases, and agreed with the independent trace.