Finance Forecast Evaluation Spine

Component runs forecast-evaluation

This component runs forecast-evaluation statistics over synthetic, market-shaped fixtures, records both the numbers computed and the cases refused, and writes a metadata-only result record from the copied econometric code. finance_forecast_evaluation_spine runs forecast-evaluation statistics over synthetic, market-shaped fixtures and records both the numbers it computed and the cases it refused. It is a public component with runnable source. It reads a synthetic loss matrix and paired loss series, checks the request is admissible, runs the copied econometric code, and writes a metadata-only result record.

Value is evaluation

The value is the evaluation posture, not a claim about any asset. The value is the evaluation posture, not a claim about any asset.

Result record carries

The result record carries computed statistics when the fixture supports them and typed refusals when it does not, and both are treated as valid outcomes. The result record carries computed statistics when the fixture supports them and typed refusals when it does not. Both are treated as valid outcomes.

Purpose

Why model comparison is hard

A lower average loss does not establish better forecasting because losses are autocorrelated, samples are short, and a careless split lets a model peek past its forecast point. Comparing two forecasting models is harder than it looks. A lower average loss does not establish one model predicts better. Losses are autocorrelated, samples are short, and a careless time split lets a model peek past the point it was supposed to forecast from. A "loose" comparison, one that reports a mean and stops, produces a number that reads like a result but is not one.

The econometric machinery it carries

It carries the careful machinery economists use, Diebold-Mariano with a Bartlett HAC variance, the HLN correction, Hansen's SPA test, a model confidence set, and a stationary bootstrap, run on synthetic data only. This component carries the machinery economists use to answer the question carefully: the Diebold-Mariano loss-differential statistic with a Bartlett HAC long-run variance, the Harvey-Leybourne-Newbold small-sample correction, Hansen's test for superior predictive ability, a model confidence set, and a Politis-Romano stationary bootstrap. It runs them on synthetic data only, and it refuses when the input is inadmissible instead of returning a number that could be misread.

How it works

A thin runtime over SPEC

The component is a thin runtime over a CrownJewelSpec that fixes the four required inputs, names the three negative cases, binds the source manifest, and restricts reads to anchors in four copied modules. The component is a thin runtime over a CrownJewelSpec named SPEC. The spec fixes the four required inputs (family_loss_matrix.json, paired_loss_series.json, finance_boundary_policy.json, projection_protocol.json), names the three negative cases, binds the source manifest, and restricts source-open reads to required anchors in four copied modules: model_selection_stats.py, spa_statistics.py, loss_differentials.py, and family_loss_matrix.py.

Two entrypoints, one bundle difference

run and run_finance_forecast_bundle both delegate to run_crown_jewel_organ, differing only in that the bundle entrypoint sets input_mode to the exported-bundle mode. run and run_finance_forecast_bundle both delegate to run_crown_jewel_organ with evaluate and evaluate_negative_case. The bundle entrypoint sets input_mode to the exported-bundle mode; that is the only difference.

evaluate loads inputs, order matters

evaluate loads the loss matrix, paired loss series, and boundary policy, then calls _evaluate_payloads, where the order of operations matters. evaluate loads the loss matrix, the paired loss series, and the boundary policy, then calls _evaluate_payloads. That function is where the order of operations matters.

Guards run before any computation

_policy_findings and _lookahead_split_findings run first, blocking on any non-False advice flag or forecast-date leakage, so an inadmissible request never reaches a statistic. The guards run first. _policy_findings walks FALSE_BOUNDARY_FLAGS (investment_advice_authorized, trading_advice_authorized, live_market_data_authorized, track_record_claim_authorized, performance_claim_authorized, optimizer_mutation_authorized) and raises FINANCE_NO_ADVICE_OVERCLAIM for any flag that is not exactly False. _lookahead_split_findings parses the row dates with _parse_iso_date and raises FINANCE_LOOKAHEAD_SPLIT_FORBIDDEN when a row's subject_as_of reaches its event_start, or when an event window ends before it begins. If either guard produces a finding, or the source manifest did not pass, _evaluate_payloads returns a blocked record and no statistics run. An inadmissible request never reaches a computation.

Branching to standalone or subprocess stats

Only after the guards pass does _is_exported_finance_bundle branch to a declared, unexecuted contract for the exported bundle or a real child-interpreter subprocess for the first-wave fixture with scipy blocked. Only after the guards pass does the branch happen. _is_exported_finance_bundle decides the path. For the exported public bundle, _standalone_exported_statistics_contract returns a declared contract: the statistics are named and marked executed: false, with not_a_live_run and real_runtime_receipt: false, so the standalone bundle does not depend on a live source subprocess. For the first-wave fixture, _run_stats_subprocess (memoized by _run_stats_subprocess_cached) shells out to a child interpreter that imports the copied tools.finance modules. It runs reality_check_summary, spa_summary, and model_confidence_set_summary with min_sample 10, bootstrap_reps 40, and seed 1729, paired_loss_summary with min_paired 10, and stationary_bootstrap_indices with reps 5 and seed 1729. The subprocess deliberately blocks scipy from importing before calling harvey_leybourne_newbold_correction, which forces a typed refusal instead of a fabricated t-distribution p-value.

Checking the subprocess payload

Back in the parent, _evaluate_payloads checks each statistic field against its expected status, blocks on any mismatch, and keeps source bodies out with body_in_receipt: false and hashed stdout and stderr. Back in the parent, _evaluate_payloads reads the subprocess payload and checks each field: Reality Check and SPA must be computed_bootstrap, the model confidence set must report implemented, Diebold-Mariano must be computed_hac_normal_approximation, and the HLN entry must be a refusal with reason scipy_unavailable_for_t_distribution. Any mismatch becomes a finding and the record is blocked. Source bodies never enter the record; every emitted stats object keeps body_in_receipt: false, and the subprocess is summarized by stdout_sha256 and stderr_sha256 from _sha256_text.

Negative cases reuse the same path

evaluate_negative_case mutates one input to reproduce a specific fault, re-runs _evaluate_payloads, and confirms the expected stable error code appears. evaluate_negative_case reuses the same path. It loads the fixture, mutates one input to reproduce a specific fault (moving a row's subject_as_of onto its event_start for the leakage case, or setting investment_advice_authorized true for the overclaim case), re-runs _evaluate_payloads, and confirms the expected stable error code appears.

flowchart TD Inputs["Synthetic fixture family_loss_matrix, paired_loss_series, finance_boundary_policy"] Evaluate["evaluate loads inputs"] Guards["_policy_findings and _lookahead_split_findings"] Blocked["Blocked record no statistics run"] Branch{"Exported bundle?"} Standalone["_standalone_exported_statistics_contract declared, not recomputed"] Subprocess["_run_stats_subprocess DM/HAC, SPA, MCS, bootstrap, HLN refusal"] Record["metadata-only result record refs, digests, verdicts, scope boundaries"] Inputs --> Evaluate Evaluate --> Guards Guards -->|"finding or bad manifest"| Blocked Guards -->|"admissible"| Branch Branch -->|"yes"| Standalone Branch -->|"no"| Subprocess Standalone --> Record Subprocess --> Record Blocked --> Record

Negative cases

Fixture declares three

The fixture declares three negative cases, each bound to a stable code in EXPECTED_NEGATIVE_CASES. The fixture declares three negative cases, each bound to a stable code in EXPECTED_NEGATIVE_CASES.

Three faults, three stable codes

A moved forecast cutoff raises FINANCE_LOOKAHEAD_SPLIT_FORBIDDEN, a flipped advice flag raises FINANCE_NO_ADVICE_OVERCLAIM, and an unavailable scipy yields a typed HLN refusal instead of a guessed p-value. finance_leakage_lookahead_split moves a forecast cutoff onto the event it forecasts and must raise FINANCE_LOOKAHEAD_SPLIT_FORBIDDEN. finance_no_advice_overclaim flips a boundary flag to authorize advice and must raise FINANCE_NO_ADVICE_OVERCLAIM. finance_hln_dependency_refusal is the case where scipy is unavailable for the small-sample t-distribution; the correct behavior is a typed refusal carrying FINANCE_HLN_TYPED_REFUSAL_REQUIRED, not a guessed p-value.

Prior Art Grounding

Standard forecast-evaluation econometrics

The statistics are standard forecast-evaluation econometrics anchored by Diebold-Mariano, the HLN correction, Hansen's SPA test, and proper scoring rules, with a lookahead-split guard against temporal leakage. The statistics are standard forecast-evaluation econometrics, not a trading system. The core anchors are the Diebold-Mariano test for comparing predictive accuracy, the Harvey-Leybourne-Newbold small-sample correction (DOI reference), Hansen's test for superior predictive ability, and proper-scoring-rule work such as Gneiting and Raftery. The lookahead-split guard follows the financial machine-learning concern that temporal leakage makes backtests look stronger than they are.

Validation Result record Path

Run from repo

Run from the repo root, without mutating durable result or projection surfaces. Run from the repo root, without mutating durable result or projection surfaces:

PYTHONPATH=src ./repo-pytest tests/test_finance_forecast_evaluation_spine.py -q --basetemp=/tmp/microcosm_finance_forecast_evaluation_spine_pytest
./repo-python scripts/build_doctrine_projection.py --check-paper-module-corpus

To exercise component

To exercise the component directly and inspect a result record, run the CLI and read the emitted JSON. To exercise the component directly and inspect a result record, run the CLI and read the emitted JSON:

microcosm finance-forecast-evaluation-spine run --input fixtures/first_wave/finance_forecast_evaluation_spine/input --out receipts/first_wave/finance_forecast_evaluation_spine

What a pass means here

A green run computed the positive-fixture statistics, raised all three stable negative codes, kept the advice and live-data flags false, and admitted no source body. A pass means the positive fixture computed the statistics, the three negative cases raised their stable codes, the advice and live-data flags stayed false, and no source body entered the record.