Computer-Use Action Trace Replay

Component answers offline

The component answers, offline over a synthetic trace, whether a blocked action was later narrated as success and whether a claimed success has any state evidence behind it. A computer-use agent leaves behind a stream of screenshots, clicks, keystrokes, and "it worked" assertions. Reading that stream by hand does not tell you the two things that matter: whether an action that was blocked or sent for review got narrated later as a success, and whether a claimed success has any state evidence behind it. This component answers those two questions on a synthetic trace, offline, before any of the language reaches a reader.

Takes recorded local-agent

It takes a recorded local-agent run with its observations, actions, pre-action verdicts, and state changes, checking the trail is self-consistent against authority limits written down up front. It takes a recorded run of a local agent, the route it picked, the observations it logged, the actions it took, the authority verdict stamped before each action, and the state change each action produced, and checks that this recorded trail is well-formed and self-consistent. The record states up front where the agent's authority was supposed to stop, so the limits are written down and checkable rather than taken on faith.

Result is metadata-only

The result is a metadata-only record of counts, refs, hashes, and redaction posture over a recorded trail; it never watches a live agent. The result is a metadata-only record of counts, references, hashes, and redaction posture. It checks a recorded trail. It does not watch a live agent and it does not establish one stayed in bounds.

Purpose

The complete-looking trace that hides a lie

It guards against a complete-looking trace that hides a lie: a denied action reported as done, a success asserted with no oracle, a screenshot treated as proof. The failure this guards against is a trace that looks complete while hiding a lie. An action can be denied by policy and then reported as done. A success can be asserted with no diff, no oracle, and no before-and-after state. A screenshot can be treated as proof when the pixels prove nothing about what actually changed.

A typed join of verdict and transition

Every action must line up with a prior observation, a pre-action verdict, and a transition whose outcome matches that verdict, through a typed join rather than a screenshot replay. A loose reading of the log cannot catch these. You need every recorded action to line up, row by row, with a prior visible observation, a pre-action authority verdict, and a state-transition record whose outcome agrees with that verdict. If the verdict said allow, the transition has to show the action ran and an oracle confirmed the new state. If the verdict said block or review, the transition has to show the action did not run and the status reads blocked or review_required. The mechanism is a typed join across those tables, not a screenshot replay.

The screenshot is the one distrusted input

Inverting the usual trust, observations enter only as a screenshot_digest and visible_state_hash, with raw pixels and hidden state barred, so weight rests on verdict-transition agreement. The inversion is the point. Most action-trace tooling trusts the screenshot. This component treats the screenshot as the one thing it will not trust: observations enter only as a screenshot_digest and a visible_state_hash, and raw pixel bodies, hidden-state claims, and live-browser state are all required to be absent. The evidence that carries weight is the agreement between verdict and transition.

How it works

run_computer_use_action_trace_bundle and the check sequence

run_computer_use_action_trace_bundle resolves the input, decides fixture versus exported bundle, loads it through strict JSON, then runs a fixed sequence of checks before writing anything. The entry point is run_computer_use_action_trace_bundle. It resolves the input directory, decides whether the input is the full fixture with negative cases or the public exported bundle, loads the bundle through a strict JSON path, then runs a fixed sequence of checks in this order before it writes anything.

flowchart TD load["load bundle, strict JSON"] load --> proj["_validate_computer_use_projection_protocol"] proj --> pol["_validate_computer_use_interaction_policy"] pol --> ep["_validate_computer_use_episodes"] ep --> obs["_validate_computer_use_observations"] obs --> act["_validate_computer_use_actions"] act --> ver["_validate_computer_use_authority_verdicts"] ver --> tr["_validate_computer_use_state_transitions"] tr --> rec["_validate_computer_use_recovery_receipts"] rec --> cold["_validate_computer_use_cold_replay"] cold --> neg["_validate_computer_use_negative_cases fixture path only"] neg --> out["status pass, else blocked"]

Stages chain, so early breaks propagate

Each stage consumes the rows the earlier stage exported, so a break early in the chain propagates downstream. Each stage consumes the rows the earlier stage exported, so a break early in the chain propagates. The relevant functions:

Public trace spans and the pass rule

build_public_computer_use_trace emits bounded public spans, and status is pass only when findings are empty, no negative case is missing, the scan passes, and source-module digests verify. After the checks, build_public_computer_use_trace emits bounded public trace spans over refs, counts, and hashes. run_computer_use_action_trace_bundle collects the findings from every stage. It sets status to pass only when the positive findings are empty, no expected negative case is missing, the non-public-state scan passes, and the copied public source-module digests verify. Otherwise the status is blocked. The written record carries counts, refs, hashes, and the redaction posture, and omits raw screen bodies, account secrets, hidden screen state, model-output data, private source bodies, absolute local paths, and benchmark-score claims.

What the tracked fixture record reports

The tracked fixture record reports fixed counts: four episodes, eight actions and verdicts, eight transitions, one recovery, four cold-replay rows, one blocked action, and eight trace spans. The tracked fixture record reports four episodes, six observations, eight actions, eight authority verdicts, eight state-transition records, one recovery record, four cold-replay rows, one blocked action, and eight public trace spans.

Negative cases

Eight declared perturbations on the fixture path

_validate_computer_use_negative_cases feeds eight declared perturbations through the same posture and requires each to be rejected with its own code, failing the run if any is missing. The fixture path runs _validate_computer_use_negative_cases, which feeds eight declared perturbations through the same posture and confirms each one is rejected with a specific error code. The set is fixed in COMPUTER_USE_EXPECTED_NEGATIVE_CASES; a missing observation there fails the run.

The specific actions each row must reject

Rows are rejected for live-account or account secret actions, external mutation, unapproved purchase or send, destructive file actions, hidden-state claims, missing observations, and benchmark-score claims. Live account action is rejected when a row carries live_account_context or live_account_action. account secret entry is rejected when a row attempts it or carries a account secret value. External network mutation is rejected outright. A purchase or send is rejected when it is requested without a human_approval_ref. A destructive file action is rejected without a human_review_ref. A hidden screen-state claim is rejected when a row claims hidden state or omits its visible_state_hash. An action without a prior observation or affordance ref is rejected. A benchmark-score claim is rejected: a synthetic replay record cannot claim benchmark performance.

Prior Art Grounding

WebArena, Mind2Web, OSWorld, WebDriver accounting

The design borrows inspectable action-trace accounting from WebArena, Mind2Web, OSWorld, and WebDriver, requiring observations, verdicts, transitions, recovery, and cold replay to line up before an episode is credited. This component borrows the action-trace accounting pattern from web and desktop agent benchmarks that make action trajectories inspectable. WebArena and Mind2Web anchor realistic web-task evaluation, and OSWorld extends the concern to agents acting in real computer environments. Browser automation standards such as WebDriver are prior art for representing actions against visible browser state through a controlled protocol. The borrowed idea is the accounting: observations, affordances, actions, pre-action verdicts, transitions, recovery, and cold replay must line up before an episode is credited.

Validation Result record Path

Run from microcosm-substrate/:

PYTHONPATH=src ../repo-python -m microcosm_core.cli agent-route-observability-runtime \
  --input examples/agent_route_observability_runtime/exported_computer_use_action_trace_bundle \
  --out receipts/runtime_shell/demo_project/organs/agent_route_observability_runtime \
  validate-computer-use-bundle

What a clean bundle join means

A pass means the exported bundle joined cleanly, every action carrying an observation, verdict, and matching transition, with the cold replay reproducing them and the record staying metadata-only. A pass means the exported bundle joined cleanly: every action had an observation, a pre-action verdict, and a matching transition, the cold replay reproduced them, and the written record stayed metadata-only. The command writes the validation record under the named output directory. The public exported bundle has contains_negative_cases: false, so the eight-case floor is checked on the fixture path, and tests/test_agent_route_observability_runtime.py remains the negative-case authority.