Engine Room Navigation Fitness Benchmark

Costly cold-start failure

The costly cold-start failure is not a wrong answer but the wrong first command, so this component scores whether a navigation packet pointed at the stable ids a cold task actually needed. When an agent is dropped cold into a large repository, the costly failure is not a wrong answer. It is reaching for the wrong first command and landing on the wrong rows. This component makes that failure measurable. It reads cold-task fixtures and the route packets a navigation surface returned for them, then scores whether the packet pointed at the stable ids the task actually needed.

Is evaluator not

It is an evaluator, not a route runner: it scores pre-captured packets for recall, precision, forbidden-route hits, scent coverage, and latency, without calling the navigation kernel or generating routes. The component is an evaluator, not a route runner. It consumes fixture tasks and pre-captured packets, computes recall, precision, forbidden-route hits, scent coverage, and latency status, and emits one result record per task plus a suite-level summary. It does not call the private navigation kernel, does not validate embeddings, and does not generate routes of its own.

Runtime lives in

The runtime lives in src/microcosm_core/engine_room/navigation_fitness_benchmark.py. The runtime lives in src/microcosm_core/engine_room/navigation_fitness_benchmark.py. Public fixtures live in fixtures/first_wave/engine_room_navigation_fitness_benchmark/input.

Purpose

Why one score is the wrong shape

Most retrieval scores collapse correct and fast into one number, so a slow-but-correct route and a fast-but-wrong route look identical and land in the same repair queue when they need different fixes. Most retrieval scores collapse "correct" and "fast" into one number. That is the wrong shape for navigation triage. A packet can name every expected stable id and still be too slow. A packet can come back fast and still miss an id, carry a weak information scent, use a forbidden first route, time out, or error. If those collapse into a single score, a slow-but-correct route and a fast-but-wrong route look the same, and they land in the same repair queue when they need different fixes.

Sufficiency and latency stay apart

The evaluator keeps two axes apart: sufficiency asks whether the route pointed at the right things, latency asks whether it stayed under budget, and each is computed and reported independently. This evaluator keeps the two axes apart. Sufficiency answers "did the route point at the right things." Latency answers "did it stay under budget." The two are computed independently and reported side by side, so each failing route lands in the queue that matches its actual defect.

How it works

The input is a typed task

NavigationFitnessTask is a frozen dataclass fixing the task id, expected artifacts, forbidden routes, latency budget, and scent terms, and task_from_mapping degrades missing fields to visible public defaults. The unit of input is a typed task, not a free-form prompt. NavigationFitnessTask is a frozen dataclass that fixes the task id, family, prompt, route type, expected stable artifacts, forbidden first routes, latency budget in milliseconds, route role, and scent terms. task_from_mapping turns each raw fixture row into that record. Missing fields fall back to explicit public defaults (unknown_task, public_fixture, context_pack, a 1500 ms budget), so a thin fixture degrades to visible defaults rather than to hidden context.

Evaluate task is

evaluate_task is the core predicate. evaluate_task is the core predicate. It runs these steps against one task and one packet:

Computing recall, precision, and route checks

evaluate_task computes recall_at_packet and precision_at_packet, reads wall_ms, timed_out, and error off the packet, and checks each forbidden route against its first_contact_command. From those, evaluate_task computes recall_at_packet (found over expected) and precision_at_packet (found over selected), reads wall_ms, timed_out, and error off the packet, and checks each forbidden route against the packet's first_contact_command.

Sufficiency ladder versus latency status

Sufficiency is a first-failing-check ladder over timeout, error, missing id, weak scent, and forbidden route, while latency is a separate three-way status that never touches the sufficiency verdict. Sufficiency is a first-failing-check ladder. In order: a timeout gives failure kind route_timeout; an error gives route_error; any missing expected id gives missing_id; a failed scent check gives weak_scent; a forbidden first route gives forbidden_route. If none fire, sufficiency is pass. Latency is a separate three-way status: timeout if the packet timed out, pass if wall time is within budget, fail if it ran over. Latency never touches the sufficiency verdict.

Suite aggregation and two debt classes

evaluate_benchmark runs the predicate over every case and aggregates counts, percentiles, and exactly two debt classes, so a slow-but-correct route yields one latency debt and zero sufficiency debt. evaluate_benchmark lifts the per-task predicate to a suite. It runs evaluate_task over every case, then aggregates counts, p50_wall_ms, p95_wall_ms, per-route-type metrics from _route_type_metrics, and debt candidates from _debt_candidates. That last function emits exactly two debt classes. A sufficiency debt names the missing ids and forbidden-route hits and carries the failure kind as repair_class. A latency debt records wall time, budget, and latency status without marking the route semantically wrong. A single slow-but-correct route produces one latency debt and zero sufficiency debt.

Comparing result records against expectations

evaluate_case reports expectation_met only when observed status, every summary field, and every per-task field agree, and evaluate_fixture_dir passes only when every case matched. evaluate_case compares a suite result record against a fixture's stated expectations: an expected_status, an expected_summary of counts, and expected_task_statuses per task. It reports expectation_met only when the observed status, every summary field, and every per-task field agree. evaluate_fixture_dir walks a directory of case files, runs each through evaluate_case, and returns case_count, passed_case_count, and an overall status that passes only when every case matched. build_parser and main expose two subcommands, evaluate-benchmark and evaluate-fixtures, the latter printing the fixture-directory result as JSON without writing any durable output.

flowchart TD A["Fixture task expected ids, forbidden routes, latency budget, scent terms"] --> E["evaluate_task"] P["Captured route packet"] --> E E --> S["Sufficiency ladder timeout, error, missing id, weak scent, forbidden route"] E --> L["Latency status wall time vs budget"] S --> R["Per-task result record"] L --> R R --> B["evaluate_benchmark counts, p50/p95, debt candidates"] B --> C["evaluate_case observed vs expected"]

Negative cases

Four public fixture

The four public fixture cases pin the accounting rules. The four public fixture cases pin the accounting rules. Each names its own task id.

Two paraphrased cold tasks pass

heldout_paraphrase_pass runs two nonliteral cold tasks that must route to their expected stable ids, avoid forbidden first routes, satisfy scent terms, and stay under budget. heldout_paraphrase_pass runs two nonliteral cold tasks (heldout_theory_roof, heldout_agent_path_store). Both must route to their expected stable ids, avoid forbidden first routes, satisfy scent terms, and stay under budget.

A correct id with a forbidden route

adversarial_forbidden_route selects the right id but uses the forbidden --paper-module first command, so sufficiency fails with forbidden_route while latency stays pass. adversarial_forbidden_route runs adversarial_paper_module_query. The packet selects the right id (paper_modules:navigation_hologram_theory) but its first command uses the forbidden --paper-module route. Sufficiency fails with kind forbidden_route while latency stays pass, so a correct id does not rescue a banned first move.

A nearby row misses the expected id

missing_stable_id_negative selects a nearby row while the expected stable id is absent, so sufficiency fails with kind missing_id. missing_stable_id_negative runs missing_agent_diagnostics, where a nearby row is selected but the expected stable id is absent. Sufficiency fails with kind missing_id.

Slow but sufficient proves axis separation

latency_debt_negative selects the expected id at 830 ms against a 100 ms budget, so sufficiency passes, latency fails, and the suite reports one latency debt and zero sufficiency debt. latency_debt_negative runs slow_but_sufficient. The packet selects the expected id with a 100 ms budget and an 830 ms wall time. Sufficiency passes, latency fails, and the suite reports one latency debt and zero sufficiency debt. This is the case that proves the two axes stay separate.

Prior Art Grounding

Retrieval evaluation and information scent

The component applies information-retrieval evaluation and information-scent research to route packets: define expected targets, score recall and precision, penalize forbidden first moves, and keep latency apart from answer quality. The component applies information-retrieval evaluation and information-scent research to agent route packets rather than to document search: define the expected targets, score returned results for recall and precision, penalize forbidden first moves, and keep latency apart from answer quality. The retrieval-benchmark tradition of relevance judgments, precision, and recall is anchored by TREC. The information-scent framing follows Pirolli and Card's Information Foraging (Psychological Review, 1999). This is a fixture-scored evaluator, not a universal benchmark or a live-kernel proof.

Validation Result record Path

PYTHONPATH=src ./repo-pytest tests/test_engine_room_navigation_fitness_benchmark.py -q
cd microcosm-substrate && PYTHONPATH=src ../repo-python scripts/build_doctrine_projection.py --check-paper-module-corpus

You can also replay the fixtures directly:

cd microcosm-substrate && PYTHONPATH=src python3 -m microcosm_core.engine_room.navigation_fitness_benchmark evaluate-fixtures \
  --input fixtures/first_wave/engine_room_navigation_fitness_benchmark/input --json

What a pass means here

A pass means the public fixture behavior and the bundle projection remain reproducible; it admits no component, unblocks no Atlas owner lane, and excludes launch. A pass means the public fixture behavior and the bundle projection remain reproducible. It does not admit an accepted component, unblock the Atlas owner lane, or include launch operations.