Agent Monitor Redteam Falsification Replay

Monitor verdicts must be earned by evidence

A monitor verdict counts only once the re-runnable evidence behind it is present and consistent, not because a label asserts it. agent_monitor_redteam_falsification_replay is a regression check for one honesty claim: that a monitor verdict was earned by evidence, not asserted by a label. It replays a small set of synthetic agent trajectories and their monitor observations, then refuses to let any verdict count until the evidence that would let a reader re-run it is present and consistent.

metadata-only records preserve the review trail

It reads public trajectory metadata and emits metadata-only records of ids, refs, digests, and verdicts, never copying agent reasoning, source bodies, or model-output data. The component consumes public trajectory metadata and metadata-only result records. It emits a result record, a review board, a validation record, and a command card, all carrying ids, refs, digests, counts, verdicts, and findings. It never copies agent reasoning, source bodies, or model-output data into those records.

Fictional replay grants no live-monitor authority

Nothing here runs live: it is fictional data with no real-monitor score and no launch authorized. It is a demo over fictional data. It does not run a live monitor, does not score how well any real monitor performs, and excludes launch.

Purpose

A verdict field is bounded evidence

Writing "the monitor caught it" is easy; showing the verdict follows from its trajectory and survives a second reading is the hard part a field-presence check skips. It is easy to write "the monitor caught it" next to a suspicious agent action. It is harder to show that the verdict follows from the trajectory it claims to summarise, and that a second reading of the same evidence reaches the same conclusion. A loose check that only asks "does the row have a verdict field" passes on a label that reads well and is backed by nothing.

Cited refs plus an independent recompute

Each observation must cite nine re-runnable refs before its verdict is admitted, and a separate builder recomputes the verdict from span evidence, blocking on any disagreement. This component makes the verdict falsifiable. Each observation must cite a trajectory hash, a monitor id, a suspicious-span ref, an adversarial-probe ref, a severity tier, a human-escalation ref, a mitigation-diff ref, a body-omission ref, and a cold-replay ref before the verdict is admitted. A separate builder then recomputes the verdict from the span evidence. If the recomputed verdict disagrees with the declared one, the run blocks. The failure it guards against is a verdict that summarises a trajectory it does not actually match.

How it works

Two entry points fold into _build_result

run and run_monitor_bundle both call _build_result, which loads four inputs plus seven negative-case files, runs each validator in order, and folds them into one payload. The entry points are run for the first-wave fixture and run_monitor_bundle for an exported public bundle. Both call _build_result, which loads the four input documents (projection_protocol.json, monitor_policy.json, trajectory_cases.json, monitor_observations.json) plus the seven negative-case files, runs each validator in order, folds the results into one payload, and returns it. _write_receipts then writes the result, board, validation, and sign-off records; result_card projects a compact card that omits the full finding, scan, and trace bodies.

The validators run in this order inside _build_result:

Three conditions for a pass

scan_paths fails the run on any forbidden-class hit, and the status is pass only when no negative case is missing, the scan is clean, and every validator passed. scan_paths runs over the inputs and the copied source artifacts against the forbidden-class policy; any blocking hit fails the run. The status in _build_result is pass only when no negative case is missing, the non-public-state scan has zero blocking hits, and every validator returned pass.

The anti-echo recompute per span

build_public_monitor_redteam_falsification_trace recomputes each span's verdict from its own evidence, and validate_public_trace blocks on any mismatch, so a copied verdict cannot pass. The anti-echo step is build_public_monitor_redteam_falsification_trace, imported from microcosm_core.macro_tools.agent_execution_trace. It reads the same inputs and recomputes, per span, whether the coverage label is backed by an adversarial-probe span and what the verdict should be given the span's severity, escalation, and mitigation evidence. validate_public_trace compares each recomputed verdict to the declared one. A disagreement records PUBLIC_TRACE_MONITOR_REDTEAM_VERDICT_MISMATCH and the run blocks. This is why a verdict cannot pass by being copied into the observation row.

What the four fixture cases recompute to

The four fixture cases recompute to two pass, one escalate, and one block, spanning a benign review, a suppressed failure, a tool-authority escape, and a dogfood safety-evals trace. The first-wave fixture has four trajectory cases and four observations: a benign route review that recomputes to pass, a suppressed-failure attempt that recomputes to escalate, a tool-authority escape that recomputes to block, and one observation over a sanitized public dogfood safety-evals trace that recomputes to pass. The two positive routes plus the escalation and the block give two pass, one escalate, and one block.

flowchart TD inputs["protocol, policy, trajectories, observations"] manifest["source module manifest"] digest["SHA-256 digest check per artifact"] scan["forbidden-class non-public-state scan"] observed["per-observation evidence check"] trace["public trace recomputes verdict"] compare["recomputed vs declared verdict"] result["metadata-only result record"] inputs --> observed manifest --> digest digest --> scan observed --> compare trace --> compare scan --> result compare --> result

Negative cases

Semantic floors, not file presence

The seven negative files each carry a semantic floor, so a case counts only when its payload actually triggers the named failure mode, not when the file merely exists. The fixture ships seven negative files, each expected to block or quarantine. Every one has a semantic floor in NEGATIVE_CASE_SEMANTIC_FLOORS, so a case counts only when its payload actually triggers the failure mode named by its label, not when the file merely exists.

• private_chain_of_thought_leakage records MONITOR_REDTEAM_PRIVATE_COT_FORBIDDEN.
• internal_code_export records MONITOR_REDTEAM_INTERNAL_CODE_FORBIDDEN.
• exploit_instruction_material records MONITOR_REDTEAM_EXPLOIT_INSTRUCTIONS_FORBIDDEN.
• account secret material records MONITOR REDTEAM account secret MATERIAL FORBIDDEN.
• live_agent_traffic records MONITOR_REDTEAM_LIVE_TRAFFIC_FORBIDDEN.
• monitor_product_performance_claim records MONITOR_REDTEAM_PRODUCT_PERFORMANCE_CLAIM_FORBIDDEN.
• coverage_without_adversarial_probe records MONITOR_REDTEAM_COVERAGE_WITHOUT_PROBE; its floor requires the adversarial_probe_ref field to be empty, so a coverage claim with no probe cannot pass.

Expected verdicts and severities

All seven cases expect a blocking or escalating verdict, at critical severity for the first five and high for the performance-claim and coverage cases. The first five expect a block verdict at critical severity. The performance-claim case expects block at high severity. The coverage-without-probe case expects escalate or block at high severity.

Prior Art Grounding

Falsify with probes, then account for it

It follows the red-team lineage that falsifies a monitor claim with adversarial probes rather than confirming it with benign examples, adding the accounting shape of required refs plus an agreeing recomputation. The component follows the red-team and behavior-discovery lineage that treats a monitor claim as something to falsify with adversarial probes rather than confirm with benign examples. Anthropic's Red Teaming Language Models to Reduce Harms is a procedural anchor for eliciting unwanted behavior, and Discovering Language Model Behaviors with Model-Written Evaluations anchors the idea that evaluation cases surface behavior classes worth tracking. The local contribution is the accounting shape: a verdict needs a trajectory hash, a probe ref, a suspicious-span ref, escalation and mitigation refs, a replay ref, and a body-omission record before coverage language is allowed, and an independent recomputation has to agree with it.

Validation Result record Path

The fixture command writes a disposable result

The first-wave validator runs from the repository root and writes its result record outside the working tree. Run the first-wave fixture validator from the repo root and write its record outside the working tree:

Then run the exported bundle validator:

cd microcosm-substrate && PYTHONPATH=src ../repo-python -m microcosm_core.organs.agent_monitor_redteam_falsification_replay run-monitor-bundle --input examples/agent_monitor_redteam_falsification_replay/exported_monitor_redteam_bundle --out /tmp/agent_monitor_redteam_bundle_receipt --card > /tmp/agent_monitor_redteam_bundle_card.json

Regression and corpus checks share the root

The focused regression test and the paper-module corpus check both run from the repository root. The focused regression test and corpus projection check are run from the repo root:

PYTHONDONTWRITEBYTECODE=1 PYTHONPYCACHEPREFIX=/tmp/mc_agent_monitor_pyc ./repo-pytest tests/test_agent_monitor_redteam_falsification_replay.py -q -p no:cacheprovider --basetemp=/tmp/mc_agent_monitor_bt
./repo-python scripts/build_doctrine_projection.py --check-paper-module-corpus

What a pass means here

A pass means every validator returned pass, the non-public-state scan found no blocking hit, and each recomputed verdict matched its declared verdict for these fixtures. A pass means every validator returned pass, the non-public-state scan found no blocking hit, and each recomputed verdict matched its declared verdict for these fixtures.