Belief-State Process Reward Replay

Every process reward must reconstruct from evidence

This component reads pre-recorded agent steps and asks of every step-level reward whether it reconstructs from lower-level public evidence or merely asserts its own correctness. This component checks a recorded set of agent steps and asks one question of every step-level reward: does it reconstruct from lower-level public evidence, or is it a label asserting its own correctness. It reads pre-recorded files. It never runs or watches a live agent.

Three toy tasks carry seven planted reward traps

The input models three partially observable toy tasks, and a reward is admitted only when its references recompute into one coherent chain with all seven planted traps observed. The input models three partially observable toy tasks: a terminal investigation, a mock purchase, and a small formal planner. For each step it carries a public belief summary, the verifier or observed feedback the reward cites, the process reward itself, the trajectory group the step belongs to, the outcome reward for that trajectory, and a cold-replay record. A reward is admitted only when all of those references recompute into one coherent chain and none of seven planted reward-hacking traps go unobserved.

metadata-only results explain each admitted reward

The result files record per step the belief summary, the reward, and the recompute and trap outcomes, so a reader sees why each reward was allowed without exposing hidden reasoning. The result files record, per step, the belief summary, the reward, and whether the recompute and trap checks passed. So a reader can see why each reward was allowed or blocked, without any hidden reasoning being exposed.

Purpose

Process-reward claims a linter cannot catch

Process-reward language is easy to assert and hard to verify, since a row can claim good reasoning while resting on a hidden label or gamed formatting a shape linter passes. Process-reward language is easy to assert and hard to verify. A row can claim a step earned a reward "for good reasoning" while the real basis is a hidden gold label, a neural-judge guess, or formatting that gamed the scorer. A shape linter does not catch this: every field can be present and well-typed and the claim can still be fake.

The end-to-end recompute failure mode

The guarded failure is a reward correct field by field that fails end-to-end recompute, with outcome rewards kept beside process rewards and every reference metadata-only to keep the check honest. The failure mode this component guards against is a reward that looks correct field by field but does not survive being recomputed end to end. A reward cannot point at a belief that points at a different episode, or cite feedback that belongs to another trajectory, and still count. Two design choices keep it honest. Outcome rewards are carried beside process rewards, so a final-answer score cannot be relabelled as step-level evidence. Every belief summary, feedback reference, and reward event is metadata-only, so the check proves the accounting structure without reading hidden reasoning.

How it works

One runtime exposes fixture and bundle modes

The same process-reward replay runtime supports a negative-bearing fixture mode and a manifest-backed exported-bundle mode. The runtime lives in belief_state_process_reward_replay.py and runs in two modes.

Two modes: fixture and exported bundle

In fixture mode run loads eight positive files and seven planted negatives before calling _build_result, while bundle mode's run_reward_bundle drops the negatives and makes the copied source-body floor mandatory. In fixture mode, run loads the eight positive input files named by INPUT_NAMES plus the seven planted negatives named by NEGATIVE_INPUT_NAMES, with the expected trap-to-code mapping in EXPECTED_NEGATIVE_CASES. It calls _build_result with include_negative=True, then _write_receipts. In bundle mode, run_reward_bundle validates an exported bundle with no negative inputs and makes the copied source-body floor mandatory by passing require_manifest.

_build_result orchestrates every check

_build_result orchestrates the per-file checks, semantic recompute, negative cases, secret scan, trace, and manifest, and passes only when all of them clear, blocking on any single failure. _build_result is the orchestrator. It loads the payloads, runs the per-file checks (validate_projection_protocol, validate_reward_policy, validate_task_episodes, validate_belief_states, validate_verifier_feedback, validate_reward_events, validate_trajectory_groups, validate_cold_replay), runs the cross-file check validate_semantic_recompute, runs validate_negative_cases, scans every input path for forbidden classes with scan_paths, builds the public trace, and checks the copied source-body manifest with _source_module_manifest_result. A pass requires: no expected negative case missing, zero blocking secret-scan hits, public trace status pass, no findings outside the expected traps, every per-file check pass, and in bundle mode the manifest check pass. Any single failure sets the status to blocked with the error codes attached.

It checks that the cited feedback belongs to the same episode, that the process reward references the same belief, episode, trajectory, feedback ref, and belief-discrepancy value, that the trajectory lists both that episode and that reward, that the trajectory's outcome reward is a real outcome event for the same episode, and that the cold replay exists and passed. Any inconsistency appends a precise reason code such as feedback_episode_mismatch, belief_discrepancy_mismatch, or trajectory_process_reward_missing. One blocked row blocks the whole result. This is the check a label-only fixture cannot fake.

Each planted trap must raise its code

validate_negative_cases confirms every planted trap triggers its expected error code, and a trap that fails to fire becomes a missing case that blocks the run. validate_negative_cases reads the planted traps and confirms each one triggers its expected error code, for example hidden_chain_of_thought_exported raising BELIEF_REWARD_HIDDEN_COT_EXPORT and neural_judge_only with no verifier reference raising BELIEF_REWARD_NEURAL_JUDGE_ONLY_LABEL. A trap that fails to trigger becomes a missing case and blocks the run.

Copied source-body floor for bundles

_source_module_manifest_result enforces the copied source-body floor, requiring each module to match its class, digest, and source authority while keeping result records metadata-only, raising a specific code on any breach. _source_module_manifest_result enforces the copied source-body floor for the exported bundle. Each declared module must classify as copied_non_secret_macro_body, use a body class, copy its body into source_modules/ while keeping result record fields metadata-only, exist as a target file, match its declared sha256 digest, and resolve its source reference to live source authority. Digest mismatch, missing target, wrong body class, result record-body leakage, count mismatch, and missing source authority each raise a specific code rather than passing quietly.

Result records and the metadata-only public card

_write_receipts writes the four result files while result_card projects a compact card that omits the heavy payload fields, and the trace emits metadata-only spans keeping body_in_receipt: false. _write_receipts writes the result, board, validation, and sign-off files. result_card projects a compact public card of counts, status, freshness digest, and reference handles. CARD_OMITTED_FULL_PAYLOAD_KEYS names the heavy fields kept off the card: findings, scans, trace bodies, row payloads, source imports, the scope limit, and the scope boundary text. The trace itself comes from build_public_belief_state_process_reward_trace, imported from agent_execution_trace.py, which emits metadata-only spans and keeps body_in_receipt: false.

Function to role:

flowchart TD run["run / run_reward_bundle load inputs"] perfile["Per-file checks projection, policy, episodes, beliefs, feedback, rewards, trajectories, cold replay"] recompute["validate_semantic_recompute rebuild one coherent chain"] negatives["validate_negative_cases 7 traps must trigger"] scan["scan_paths forbidden-class scan"] gate{"all checks pass, chain recomputes, every trap observed, no secret hit?"} pass["status: pass"] blocked["status: blocked with reason codes"] result records["_write_receipts + result_card body_in_receipt false"] run --> perfile perfile --> recompute recompute --> negatives negatives --> scan scan --> gate gate -->|yes| pass gate -->|no| blocked pass --> result records blocked --> result records

Negative cases

Seven falsification traps and forbidden keys

The seven planted traps each name an abuse a reward claim might smuggle in and must raise its code, while FORBIDDEN_KEYS blocks reasoning bodies and model-output data from any row. The seven planted traps are the falsification fixtures. Each names the abuse a process-reward claim might smuggle in, and each must raise its code for the run to pass: hidden chain-of-thought export, neural-judge-only labels, hidden gold labels, reward by formatting, verifier bypass, a benchmark-performance claim, and final-answer-only scoring. FORBIDDEN_KEYS blocks reasoning bodies, model-output data, gold-answer bodies, live-training run ids, and benchmark submission ids from appearing in any belief or reward row.

Prior Art Grounding

Three lineages: belief state, process reward, reward hacking

Three older ideas ground the component, partial-observability belief states, step-level process reward, and reward hacking, but it trains no model and only borrows their accounting form. Three older ideas ground this component. The belief-state lineage comes from work on acting under partial observability such as Kaelbling, Littman, and Cassandra's Planning and Acting in Partially Observable Stochastic Domains. The step-level reward lineage follows OpenAI's Let's Verify Step by Step, which separates process feedback from outcome-only supervision. The reward-hacking lineage comes from Concrete Problems in AI Safety. This component does not train a reward model or expose reasoning. It borrows the accounting form: public belief summaries, verifier-backed feedback, outcome rewards kept separate from process rewards, planted traps, and cold replay before a reward claim is admitted.

Validation Result record Path

Fixture output stays outside the working tree

The fixture validator runs from the repository root and writes its result to a disposable path outside the working tree. Run the fixture check from the repo root and write its result outside the working tree:

Then run the exported bundle check:

cd microcosm-substrate && PYTHONPATH=src ../repo-python -m microcosm_core.organs.belief_state_process_reward_replay run-reward-bundle --input examples/belief_state_process_reward_replay/exported_belief_state_process_reward_bundle --out /tmp/belief_state_process_reward_bundle_receipt --card > /tmp/belief_state_process_reward_bundle_card.json

What a passing regression run means

A pass on the focused regression test means all eight positive files cleared their checks, the chain recomputed, every planted trap fired, and no secret-class or body leak appeared. The focused regression test is tests/test_belief_state_process_reward_replay.py. A pass means all eight positive files cleared their checks, the chain recomputed, every planted trap triggered its code, and no secret-class or body leak was found.