Component · spec & evidence

Agent Benchmark Integrity Anti Gaming Replay

Validates a synthetic benchmark-integrity record and flags the contamination cases it declares.

Contract

Does Checks a public benchmark-integrity example bundle that contains three copied source pattern provenance bodies under source_artifacts. The component verifies the source-module manifest digests, requires each replay row to cite those copied source artifacts, recomputes pass/quarantine verdicts from contamination, file-access, and locked-evaluator spans, and rejects common gaming attempts such as peeking at hidden answers, training on the test set, exposing the oracle patch, cherry-picking the best of many tries, or asserting a score. It still does not run real bug fixes or claim any benchmark claims.

Scope limit It authorizes only bounded public runtime validation over copied source-open pattern provenance bodies and metadata-only benchmark-integrity replay rows; it does not establish any benchmark or SWE-bench score, agent capability, external model service, live-repo mutation, private/oracle/hidden-gold body access, product progress, or launch-scope decision.

Run

microcosm agent-benchmark-integrity-anti-gaming-replay run-benchmark-integrity-bundle --input examples/agent_benchmark_integrity_anti_gaming_replay/exported_benchmark_integrity_bundle --out .microcosm/agent_benchmark_integrity_anti_gaming_replay

Evidence

ClassComputed projectionevidence 3/5Source-faithful refactor

Links to Research Replication Rubric Artifact Replay, Cold Evaluation Honesty Bundle

ai-safetyagent-evaluationred-teaming

Source

Source Design note · Source

Map Show on the map

Index Open compact index card

Paper module