Set 10 Cold Eval Honesty Bundle

One copied simulator, one bounded scorecard

It runs one real route-quality simulator, an exact copy of cold_eval.py, over a fixed public workspace and reports the scorecard without letting it become a claim about the live system. batch10_cold_eval_honesty_capsule takes one real route-quality simulator, runs it over a fixed public workspace, and reports the scorecard without letting that scorecard become a claim about the live system. The simulator is an exact copy of cold_eval.py, imported from the exported bundle rather than rewritten, so the numbers come from the same code the original produced.

Four assertions without a wider win claim

A green run asserts four things: the copied simulator ran, every task went to the idea-first arm, its extra route surface shows in the counts, and the forbidden claims stay forbidden. A green run says four things happened: the copied simulator executed, the recomputed scorecard has every task won by the idea-first arm, the extra route surface that arm reaches is visible in the counts, and the exercise manifest still forbids benchmark, navigation-truth, launch, and hosted-readiness claims. It does not say idea-first routing wins anywhere except this workspace.

Counts and statuses without source bodies

The result is a small metadata-only record carrying counts, engine statuses, and a scoring-policy name, but no source text or command output. The result is a small metadata-only record. It carries counts, engine statuses, and a scoring-policy name. It does not carry any source text or command output.

Purpose

A clean-sweep scorecard is easy to fake

A scorecard where one arm wins every task is easy to fake, so this component runs the real simulator and checks none of those shortcuts happened before treating the win as accounting. A scorecard that shows one arm winning every task is easy to fake. You can hand the winning arm the answer, inject the expected references into its route, or quietly let a non-public paths leak into the fixture and call the shape "measured". This component exists to run the real simulator and then check that none of those shortcuts happened, so the win is an accounting fact about a declared fixture and nothing more.

Expected references score coverage, never the route

The honesty turns on one property: expected references only ever score coverage and are never added to either arm's route, so idea-first wins by genuinely declaring more relevant files. The honest part turns on one property of the copied scorer. Each task lists the route references an answer should reach, but those expected references are only ever used to score coverage. They are never added to either arm's route. The flat arm is scored on what a reader reaches from README.md, docs/quickstart.md, and pyproject.toml. The idea-first arm is scored on what its entry packets declare. The idea-first arm wins because it genuinely declares more of the relevant files, not because the scorer leaked the target into the route.

How it works

Both entrypoints share one evaluator

Both entrypoints hand their fixture to run_crown_jewel_organ, where _evaluate turns each blocked engine into an explicit finding instead of a silent success. run and run_batch10_cold_eval_bundle both hand the fixture to run_crown_jewel_organ with _evaluate as the evaluator. _evaluate runs three engines in order and turns any non-pass into a structured finding, so a blocked engine surfaces as an error code rather than a silent success.

The copied simulator runs at a fixed time

_run_original_cold_eval requires the task file, runs the copied simulator at a fixed timestamp, and reads the scorecard and result record back from a temporary workspace. _run_original_cold_eval first requires evals/cold_agent_ab/tasks.json in the workspace. Without it the engine returns blocked with BATCH10_COLD_EVAL_TASKS_REQUIRED and never runs the simulator. With it, the engine copies the workspace, calls run_cold_eval at 2026-05-31T00:00:00Z so the run is deterministic, and reads runs/cold_agent_ab/seed_scorecard.json plus receipts/cold_agent_ab_seed.json back from the temporary copy. It emits the task count, the idea-first win count, the flat win count, the per-task winners, and the scoring-policy name. In the shipped fixture the idea-first win count is 3.

The scorecard audit rejects route and path leaks

_scorecard_shape_audit accepts only an all-idea-first result with visible route-count asymmetry and no private reference in the task fixture. _scorecard_shape_audit reads the same tasks file and walks its strings for any value containing a user-home prefix or the private source-repository root. It passes only when every winner is B.idea_first_packet, the idea-first route reference count is greater than the flat count, and no private reference was found. It records the two counts and the route_surface_asymmetry_visible flag so a reader can see the gap that produced the win.

The ceiling gate blocks promoted claims

_claim_ceiling_gate requires expected-reference injection to stay off and the exercise manifest to keep all four promoted claims explicitly forbidden. _claim_ceiling_gate loads batch10_cold_eval_exercise_manifest.json and passes only when at least one row was checked, expected_ref_injection_allowed is false, and the manifest's forbidden-claim list contains live_agent_benchmark_win, navigation_truth, release_approval, and hosted_public_readiness. This is where the run refuses to promote the shape.

The source manifest is checked before execution

Before any engine runs, the shared runner verifies the copied simulator's digest and required anchors so a tampered source copy cannot produce a passing scorecard. The source copy is checked before any of this. The shared runner validates the bundle's source_module_manifest.json against the copied file: the digest must match and the required anchors (SCORING_POLICY =, def _score_task(, def run_cold_eval(, expected_ref_injection_used, idea_first_packet_wins_fixture) must be present. A tampered digest fails with CROWN_JEWEL_SOURCE_DIGEST_MISMATCH before the engines run.

flowchart TD A["Fixture or exported bundle (workspace + exercise manifest)"] --> M["Source manifest check digest + required anchors"] M --> R["cold_eval_original_runner run copied run_cold_eval, read back scorecard"] R --> S["cold_eval_scorecard_shape_audit all-B winner + route asymmetry + no non-public refs"] S --> C["cold_eval_claim_ceiling_gate injection off, forbidden claims named"] C --> O["metadata-only result record and card"]

The compact card preserves the same result

result_card adds engine and copied-module counts to the full record, while bundle mode follows the same evaluator path with exported inputs. result_card projects a compact card from the full record and adds the engine count and the copied source-module count. run_batch10_cold_eval_bundle is the same path with the exported-bundle input mode.

Negative cases

Four perturbations must move the verdict

Each negative fixture mutates one input, reruns the same evaluator path, and must turn the result blocked with its named error code. The fixture ships four perturbations. Each one copies the input, applies a named mutation, re-runs the engines, and confirms the run turns blocked with the right code.

Missing tasks, route ties, injection, and non-public refs

The four mutations remove tasks, erase route asymmetry, enable expected-ref injection, or add a non-public paths, and each has a distinct required rejection. missing_tasks removes the tasks file, so _run_original_cold_eval returns BATCH10_COLD_EVAL_TASKS_REQUIRED. flat_route_can_win rewrites the route counts to be equal, so the shape audit loses its asymmetry and reports BATCH10_COLD_EVAL_NOT_ALWAYS_B_WIN. expected_ref_injection sets expected_ref_injection_allowed to true in the manifest, and the scope limit gate rejects it with BATCH10_COLD_EVAL_EXPECTED_REF_INJECTION_FORBIDDEN. private_fixture_ref injects a /Users/ probe into the tasks file, and the shape audit rejects it with BATCH10_COLD_EVAL_PRIVATE_REF_REJECTED. Each mutation runs through the same manifest and evaluator path as the positive case, so a perturbation that fails to move the outcome is itself a failure.

Prior Art Grounding

Evaluation reports need scenario and limit context

HELM and Model Cards motivate reporting what was measured, under which fixture assumptions, and which wider capability claims the score cannot support. This component follows evaluation-transparency and benchmark-hygiene practice: a scorecard should state what was measured, what fixture assumptions went in, and what the result can and cannot support. HELM frames evaluation as a scenario-bound surface rather than a single global capability claim. Model Cards for Model Reporting established pairing performance numbers with intended use and limitations. This component borrows that scorecard-plus-limits shape and narrows it to a deterministic route-quality fixture.

Validation Result record Path

Reader-verifiable commands, run from the microcosm-substrate/ public root:

A pass covers three engines and four rejections

The commands prove the copied simulator and all three engines ran under the declared fixture while every negative case blocked with its expected code. The fixture command writes the scorecard record and sign-off JSON. The bundle command validates the copied source, its manifest, the metadata-only cards, the injection policy, and the private-reference negative case. A pass means the copied simulator ran, the three engines passed, and the four negative cases each turned blocked with the expected code.