Component · spec & evidence

Research Replication Rubric Artifact Replay

Audits whether a paper-replication claim carries the full evidence trail.

Contract

Does It checks whether a claim that an AI agent "replicated a research paper" comes with the paper trail real replication would leave behind. It re-runs nothing; instead it confirms the bundle names every required piece of evidence: a breakdown of the paper's contributions, a grading rubric, the list of allowed public inputs, a from-scratch repo scaffold, an experiment plan, the metric scripts, a roster of declared file-hashes for the outputs plus hashes that all stay inside that roster, a grader report, a capped compute/runtime budget, an ablation diff, a failure list, and a cold-rerun result record. It also catches eight ways a claim can cheat: reusing the original authors' code, leaking a hidden rubric, calling a run a "success" when only a write-up backs it, asserting a benchmark claims, leaking a private paper or dataset body, using unbounded compute, grading only the final answer, or pointing at a file-hash that was never declared. The work runs on two made-up sample papers (one machine-learning method, one computational-science study), and the generated result record shows which of the eight cheats each test case triggered, rather than taking "it was replicated" on trust.

Scope limit It validates the shape and presence of synthetic replay metadata and result record references only - it does not run any experiment, metric script, or rerun, excludes any claim that a paper was actually replicated, that a benchmark claims was achieved, or that the underlying science is correct, and it never calls providers, exposes private paper/data bodies, or authorizes public sharing or launch.

Run

PYTHONPATH=src python3 -m microcosm_core.organs.research_replication_rubric_artifact_replay run --input fixtures/first_wave/research_replication_rubric_artifact_replay/input --out receipts/first_wave/research_replication_rubric_artifact_replay

Evidence

ClassComputed projectionevidence 3/5Source-faithful refactor

Links to Materials Lab-Safety Refusal Replay, Toy-Transformer Attribution Replay, Prediction Oracle Reconciliation, Gridworld Counterfactual State Replay, Cold Evaluation Honesty Bundle

research-workflowsforecastingprovider operations

Source

Source Design note · Source

Map Show on the map

Index Open compact index card

Paper module