Navigation Fitness Benchmark
Recomputes a curated route-packet navigation benchmark — recall, precision, forbidden-first-route, scent coverage, latency, and debt — over bounded public fixtures, accepting a case only when the recomputation matches its planted expectation.
The write-up
Scoring route packets against planted expectations
navigation_fitness_benchmark scores how well a navigation system found the right thing, and then checks its own scoring against what each case claimed would happen. navigation_fitness_benchmark scores how well a navigation system found the right thing, and then checks its own scoring against what each case claimed would happen. A case is a task paired with the route packet a router returned for it. The task says what was wanted (which exact items count as the answer), which first routes are off-limits, which cue words should appear, and how fast the answer should arrive. The packet says what the router actually selected and how long it took.
The component recomputes the verdict from the packet: recall and precision of the selected items against the expected ids, whether a banned first route was taken, whether the cue words were present, and whether the run stayed under its time budget. The component recomputes the verdict from the packet: recall and precision of the selected items against the expected ids, whether a banned first route was taken, whether the cue words were present, and whether the run stayed under its time budget. It emits a pass or fail per case plus the debt it found. A case is accepted only when that recomputation matches the expectation planted in the case. A packet that claims a pass it did not earn is rejected.
Purpose
A single navigation score is easy to inflate and easy to misread as either failure or marketing. A single navigation score is easy to inflate and easy to misread as either failure or marketing. If the harness trusts a packet's own claim that it passed, the number means nothing. This component does not trust the claim. It re-derives every metric from the packet and treats the case's stated expectation as a thing to be confirmed, not repeated. The unit of evidence is "the recomputation matched the expectation," not a headline number.
How it works
The mechanism lives in two files. engine_room/navigation_fitness_benchmark.py is the source-faithful public evaluator. The mechanism lives in two files. engine_room/navigation_fitness_benchmark.py is the source-faithful public evaluator. organs/navigation_fitness_benchmark.py wraps it as a bounded fixture runner and decides sign-off.
task_from_mapping reads a task row into a NavigationFitnessTask: expected artifact ids, forbidden first routes, a latency budget (default 1500 ms), and scent terms. task_from_mapping reads a task row into a NavigationFitnessTask: expected artifact ids, forbidden first routes, a latency budget (default 1500 ms), and scent terms.
evaluate_task is the scorer. It collects the packet's selected artifacts with _packet_artifacts, matches them against the expected set with _match_expected (an id ending in * matches by prefix), and computes recall as found over expected and precision as found over selected. evaluate_task is the scorer. It collects the packet's selected artifacts with _packet_artifacts, matches them against the expected set with _match_expected (an id ending in * matches by prefix), and computes recall as found over expected and precision as found over selected. It scans the packet's first_contact_command for any forbidden route, scores scent coverage with _scent_status, and sets a latency status by comparing wall time to the task budget. It then assigns one sufficiency status in a fixed priority order: route_timeout, route_error, missing_id, weak_scent, forbidden_route, otherwise pass. Latency is judged separately, so a slow run can still be sufficient.
evaluate_benchmark runs evaluate_task over every task in a case and aggregates: p50 and p95 wall time from _percentile, sufficiency and latency pass/fail counts, per route-type metrics, and debt candidates from _debt_candidates. evaluate_benchmark runs evaluate_task over every task in a case and aggregates: p50 and p95 wall time from _percentile, sufficiency and latency pass/fail counts, per route-type metrics, and debt candidates from _debt_candidates. Each failed sufficiency check becomes a sufficiency_debt row and each failed latency check a latency_debt row.
evaluate_case re-derives that benchmark and compares the recomputed result record against the case's planted expected_status, expected_summary, and expected_task_statuses. evaluate_case re-derives that benchmark and compares the recomputed result record against the case's planted expected_status, expected_summary, and expected_task_statuses. It returns expectation_met only when the recomputed status and every summary and per-task check agree with what the case declared.
The component wrapper closes the loop. _evaluate_case calls the real evaluate_case (it does not re-implement the maths or bake in answers) and derives observed_ok. The component wrapper closes the loop. _evaluate_case calls the real evaluate_case (it does not re-implement the maths or bake in answers) and derives observed_ok. For a positive case that means the recomputation both met the expectation and aligned with the declared expected_ok. For a negative case it means the recomputation did not meet the expectation and the specific recomputed failure marker for that planted defect is present. build_result runs the fixture set and returns pass only when the positives all hold, the negatives are all correctly rejected, and both expected negative case ids are present. result_card projects a metadata-only summary and run writes the result, board, validation, and sign-off records.
Diagram source & refs
flowchart TD Case["Task + route packet"] Task["task_from_mapping expected ids, forbidden routes, budget, scent terms"] Score["evaluate_task recall, precision, forbidden, scent, latency, sufficiency"] Bench["evaluate_benchmark p50/p95, counts, debt"] Match["evaluate_case recomputed == planted expectation?"] Accept["_evaluate_case observed_ok"] Case --> Task --> Score --> Bench --> Match --> AcceptNegative cases
Two planted negatives ship with the fixtures, and both carry a packet that declares expected_status: pass. Two planted negatives ship with the fixtures, and both carry a packet that declares expected_status: pass. missing_stable_id_rejected returns a packet that omits an expected id; the recomputation produces a missing_id failure, so the recomputed status is fail, the planted "it passed" expectation is not met, and the case is correctly rejected. forbidden_first_route_rejected returns a packet whose first-contact command uses a banned route; the recomputation produces a forbidden_route failure and rejects it the same way. A negative is only counted as handled when the exact marker for its defect appears, so a case that fails for the wrong reason does not silently pass as a rejection.
The latency_debt_pass positive shows the other side. Its one task runs 830 ms against a 100 ms budget, so it carries one latency debt candidate, yet its sufficiency still passes and the case is accepted. The latency_debt_pass positive shows the other side. Its one task runs 830 ms against a 100 ms budget, so it carries one latency debt candidate, yet its sufficiency still passes and the case is accepted. The component accepts honest debt; it does not require every packet to be debt-free.
Prior Art Grounding
The recall, precision, and prefix-matching here are ordinary information retrieval evaluation applied to router output. The rest is standard engineering practice for fixture-scoped regression evidence: state which public inputs can be checked, recompute the verdict rather than trusting the input's own claim, and hold a small set of positive and negative cases. The recall, precision, and prefix-matching here are ordinary information retrieval evaluation applied to router output. The rest is standard engineering practice for fixture-scoped regression evidence: state which public inputs can be checked, recompute the verdict rather than trusting the input's own claim, and hold a small set of positive and negative cases. The local lineage is the Plectis coverage contract, where a reader page publishes a scope limit and a rerun path instead of letting a generated structured source record stand in for the source.
Validation Result record Path
Run the component over its fixtures:
Positive and negative case recomputation
A pass means the two positives (clean_fanout_pass, latency_debt_pass) matched their planted expectations and the two negatives (missing_stable_id_rejected, forbidden_first_route_rejected) were rejected by recomputation. A pass means the two positives (clean_fanout_pass, latency_debt_pass) matched their planted expectations and the two negatives (missing_stable_id_rejected, forbidden_first_route_rejected) were rejected by recomputation. The paper-module coverage contract and corpus parity check guard this page itself:
PYTHONPATH=src ./repo-pytest tests/test_plectis_paper_module_coverage_contract.py -q --tb=short
PYTHONPATH=src ./repo-python scripts/build_doctrine_projection.py --check-paper-module-corpusScope boundary
Scope limit
A green run shows that curated public route-packet cases recomputed as expected: the positives were accepted, including one that carries an honest latency debt, and the negatives were rejected because the recomputation contradicted their planted "it passed" claim. A green run shows that curated public route-packet cases recomputed as expected: the positives were accepted, including one that carries an honest latency debt, and the negatives were rejected because the recomputation contradicted their planted "it passed" claim. That is the whole proof boundary. The component judges public fixtures only. It does not run the private kernel, does not capture packets from the live route runner, and does not test the underlying search or embeddings. It is not a universal navigation score and not a prover. It does not call any provider, does not export private state, and grants no launch, public sharing, or source-file changes.
Context & evidence
In short Navigation Fitness Benchmark surfaces the engine-room route-packet benchmark bundle. Each fixture pairs a navigation task (expected stable ids, forbidden first routes, latency budget, scent terms) with the route packet a router actually produced. The bundle recomputes recall/precision against the expected ids, checks the first route against the forbidden list, scores scent coverage, derives a latency verdict, and collects sufficiency/latency debt — then reports whether the recomputation matches the case's planted expectation. Two positive cases match their expectations and are accepted (one carrying an honestly-anticipated latency debt); two negative cases plant a route defect plus a deliberately-wrong "it passed" expectation, which the recomputation contradicts, so they are rejected with the real failure marker (missing_id, forbidden_route) firing. Scope: curated public fixtures only — not a live kernel run, not an embedding benchmark, not a universal navigation score, not launch-scope decision.
Scope limit Command-result record evidence over bounded public fixtures, not runtime-product completeness. A pass means the curated route-packet cases recomputed as expected (positives accepted, negatives rejected by recomputation with the expected markers); it does NOT establish live private-kernel navigation quality, embedding quality, universal benchmark authority, deployment posture, or launch/public sharing authorization. Result records are metadata-only command evidence, bounded evidence of system correctness.
Covers Navigation Fitness Benchmark
Source
Source Source module: src/microcosm_core/organs/navigation_fitness_benchmark.py · Source module: src/microcosm_core/engine_room/navigation_fitness_benchmark.py · Design note · Source registry