Research Replication Rubric Artifact Replay

Evidence bundle for a claimed paper replication

It turns the claim that an agent replicated a paper into an evidence bundle a cold runtime can check, reruns nothing scientific, and forces the packet's numbers to reproduce. research_replication_rubric_artifact_replay is a public Microcosm component that turns the sentence "an agent replicated a paper" into a bundle a cold runtime can check. It reruns nothing scientific. It reads a replay packet and confirms the packet names every piece of evidence a real replication would have left behind, then runs one small local metric so the numbers in the packet have to reproduce rather than be taken on trust.

It runs over two synthetic public papers, and a pass means the replay packet is accountable and digest-bound, not that a real paper was independently replicated. The component runs over two synthetic public papers, one machine-learning method and one computational-science study. A pass says the replay packet is structurally accountable, digest-bound, and negative-case tested. It does not say a real paper was independently replicated.

Purpose

Four things collapsed into one claim

Demos fold the paper, the artifact, the rubric, and the rerun evidence into a single word, so this component pulls "replicated" apart and asks for each object by name. Research-agent demos collapse four separate things into one claim: the paper, the runnable artifact, the grading rubric, and the evidence that an independent rerun happened. A loose check accepts any of them as proof of all four. This component refuses to treat "replicated" as one fact. It pulls the claim apart into the objects a real replication produces and asks for each one by name.

Fourteen fields, eight axes, four forbidden authorities

validate_replication_policy sets the floor by requiring fourteen replay fields and eight rubric axes and forcing four escape authorities false. validate_replication_policy sets the floor. It requires the fourteen fields in REQUIRED_REPLAY_FIELDS: paper_id, contribution_decomposition_ref, rubric_tree_ref, allowed_public_input_refs, scratch_repo_scaffold_ref, experiment_dag_ref, metric_script_refs, artifact_hash_refs, declared_artifact_hash_refs, grader_report_ref, cost_runtime_budget_ref, ablation_diff_ref, failure_taxonomy_ref, and cold_rerun_receipt_ref. It also requires the eight rubric axes in REQUIRED_RUBRIC_AXES: contribution decomposition, artifact replay, experiment DAG, metric script, grader alignment, budget boundary, failure taxonomy, and cold rerun. A policy that drops any field or axis is rejected. The policy must also declare four escape authorities false: hidden-rubric export, benchmark-performance claim, unbounded compute search, and final-answer-only grading. A packet cannot grant itself any of them.

How it works

The two entry points are run and run_replication_bundle. Both call _build_result and differ only in what they read. The two entry points are run and run_replication_bundle. Both call _build_result and differ only in what they read.

Fixture mode versus exported-bundle mode

run loads the fixture with its eight negative cases, while run_replication_bundle validates the public export without them and reuses an unchanged prior record. run is fixture mode. It reads the first-wave input directory with include_negative=True, so it loads the positive replay rows plus the eight negative-case files, and writes the result, board, validation, and sign-off records. run_replication_bundle is exported-bundle mode. It reads the public example bundle with include_negative=False, then writes exported_research_replication_bundle_validation_result.json. It can reuse a prior record when _freshness_basis shows the inputs are byte-identical, so a clean rerun does not rewrite an unchanged result.

_build_result runs the checks in order and folds them into one record. _build_result runs the checks in order and folds them into one record.

The metric replay that must reproduce

_execute_metric_script sums one named field over an allowed public input, and validate_execution_artifact_replay fails the row if the produced numbers do not match the declared hash. The executable core is _execute_metric_script. It reads a metric spec whose operation is sum_field, sums one named field over the rows of one allowed public input table, and returns the total plus a row count. It rejects a spec that is not a bare sum, and it rejects input rows that are missing or non-numeric. validate_execution_artifact_replay wraps that call: it reads execution_artifacts/execution_artifact_manifest.json, refuses a manifest that sets body_in_receipt, authorizes external model access, or names a private body, runs the metric, then compares the produced payload against the declared output artifact and its declared sha256 file. A replay row can name every correct reference and still fail here if the numbers do not reproduce, so the pass is not a field-presence check.

Verdicts from fields, not from fixture labels

validate_research_replays reads verdicts from row fields rather than declared labels, and validate_source_module_imports verifies copied-body digests while result records stay metadata-only. validate_research_replays derives its verdicts from the replay-row fields, not from any label the fixture declares. It records the keys it ignored, so a file that claims status: pass or names its own expected_error_codes cannot talk its way past the check. validate_source_module_imports reads the manifest at SOURCE_MODULE_MANIFEST_REF, requires source_import_class of copied_non_secret_macro_body, requires body_in_receipt to be false, and verifies each declared target digest. For an exact-copy row it also confirms the live source file exists and that the source and target bytes match. Result records carry counts, references, digests, and verdicts. They never carry a source body.

flowchart TD policy["Replication policy 14 fields + 8 axes + forbidden claims"] replays["Replay rows 2 synthetic papers + 8 negative cases"] artifacts["Execution manifest allowed input + metric spec + declared hash"] metric["Local metric replay sum_field over allowed input"] source["Source-module manifest 4 copied public bodies"] result["metadata-only result record counts, refs, digests, scope boundaries"] policy --> replays replays --> artifacts artifacts --> metric metric --> result source --> result

Negative cases

The fixture ships eight named negative files, one per key in EXPECTED_NEGATIVE_CASES. Each must trip its own error code: The fixture ships eight named negative files, one per key in EXPECTED_NEGATIVE_CASES. Each must trip its own error code:

• original-author code reuse
• hidden-rubric leakage
• report-only success
• benchmark-performance overclaim
• private paper or data body leakage
• unbounded compute search
• final-answer-only grading
• undeclared artifact-hash reference

Eight fakes plus mutation cases stay blocked

These name the eight ways a plausible fake tries to pass, and the focused tests add mutation cases that stay blocked even when the tampered bundle is made internally consistent. These are the eight ways a plausible fake tries to pass. A packet that reuses the original authors' code, calls a run a success on the strength of a write-up alone, asserts a benchmark number, or points at an output hash it never declared is rejected by field semantics, not by filename. The focused test suite adds mutation cases on top: digest mismatch, local body tamper, rehashed source swaps, wrong execution-artifact hashes, metric perturbation, input perturbation, output body tamper, and self-consistent input/output/hash rewrites. Each mutation stays blocked even when the tampered bundle is made internally consistent.

Prior Art Grounding

Borrowed from ACM artifact badging

The rubric-over-artifact shape follows ACM artifact badging, which separates available from functional from reproduced, made here into a runtime gate over a declared bundle. This is the rubric-over-artifact shape from systems and machine-learning artifact evaluation, where the ACM Artifact Review and Badging process separates "available" from "functional" from "reproduced". Microcosm borrows that separation and makes it a runtime gate over a declared evidence bundle. The result is fixture-bound replay evidence, not a reproducibility guarantee and not a peer-review verdict.

Validation Result record Path

Focused runtime validation:

./repo-pytest tests/test_research_replication_rubric_artifact_replay.py -q --basetemp=/tmp/microcosm_research_replication_rubric_artifact_replay_pytest

Paper-module corpus validation:

./repo-python scripts/build_doctrine_projection.py --check-paper-module-corpus

The runtime commands behind the records are:

What a pass over the records means

A pass certifies every check held, all eight negative cases were observed, and the records stayed metadata-only. A pass means the policy, replay rows, source-module digests, projection protocol, and local metric replay all held, all eight negative cases were observed, and the records stayed metadata-only.