M5 WP5.2: preregister the v2 nested protocol, candidate list and numerical gates - #30
Merged
Merged
Conversation
…rical gates Commit the complete v2 experimental protocol before any v2 experiment runs: outer leave-one-tournament-out scopes over WC2018/Euro2020/WC2022/Euro2024 in fixed order, inner match-grouped CV (match_id, k=5, sorted by match_date then match_id, index % 5, shuffle=false, no seed), mechanical simplicity-first tie behavior, fold-local preprocessing obligations, exactly four frozen candidate families with frozen hyperparameter search spaces, WP2.7 bootstrap mechanics extended with tournament-stratified pooled resampling (2000 replicates, seed 0, paired differences, 95% percentile intervals), calibration policy without isotonic, internal replacement rule, one-time sealed AFCON 2023 + Copa America 2024 qualification gate, stop conditions, and the binding evidence hierarchy that labels outer-LOTO results as internal selection evidence only. Artifacts: docs/modeling/wp5_2-v2-nested-protocol-contract.md, data/model/v2_protocol.json (machine-readable gates mirroring v2_evaluation_registry.json), backend/tests/test_wp5_2_protocol.py (allowed-key schema, threshold literals, registry consistency, executable target-free fold semantics on synthetic fixtures), ADR 0016. No model code changes; zero experiments executed under the protocol.
…ation fitting semantics Move the frozen target-free fold semantics into one production module, touchline.modeling.v2_folds (outer LOTO specs validated against the config's development pool with sealed-scope rejection, inner match-grouped assignment sorted by match_date then match_id with index % 5, disjoint non-degenerate partitions, loud failure on duplicate ids, missing dates, foreign or sealed scopes), and have the unit contract import and exercise that primitive directly; only materialized fold-manifest generation remains deferred to M7. Add real contract-prose consistency checks that pin the normative fragments of wp5_2-v2-nested-protocol-contract.md (scopes, inner rule, tie behavior, bundle evaluator, replacement, calibration, external gate, bootstrap, bins, slices, search spaces, evidence hierarchy) against the whitespace-normalized Markdown, so config, prose and tests cannot drift apart. Preregister the exact leakage-free calibration fitting procedure: per outer fold, intercept-only/Platt candidates are fitted only from inner-CV OOF raw predictions inside the outer training partition and applied to that fold's untouched outer-holdout predictions; the final refit sequence generates development OOF raw predictions, fits the selected calibrator there, refits the base model on all development rows and freezes the model-calibrator pair before sealed-set opening; define the external constant_prevalence reference as the full-development prevalence frozen before any sealed set is opened. No v2 experiment runs under any of this: the diff adds one production module imported by no serving or training path yet, config gates, prose and tests. Validation: 42 unit contracts green, mypy strict clean across 130 source files, ruff format and lint clean, docs link checker OK.
… frozen config Remove the caller-controlled split_count parameters from assign_inner_folds and inner_partition: k is now read from data/model/v2_protocol.json through inner_split_count, which rejects missing, non-integer or below-minimum values and fails unless the configured value equals the preregistered five, so caller code can never make k silently differ from the preregistered protocol. Inner-fold construction now rejects any match scope outside the v2 development pool - not only sealed sets - and accepts a keyword-only training_scopes narrowing that pins matches to one outer fold's training partition while itself refusing to widen beyond the frozen pool. Regression tests prove the override is gone from both signatures, that a tampered config k fails loudly with an ADR-required message, that a foreign unsealed scope (99, 9) is rejected on the inner path, and that another outer fold's held-out tournament cannot enter inner CV when the partition is declared. Semantics unchanged: (match_date, match_id) sort, index % 5, shuffle=false, no seed. No v2 experiment runs. Validation: 47 unit contracts green, mypy strict clean across 130 source files, ruff format and lint clean, docs link checker OK.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Commits the complete v2 experimental protocol before any v2 experiment runs — M5 work item 6, unblocked by WP5.1's sealing:
Boundaries
No v2 experiment executed anywhere in this diff; one production module added that no serving/training path imports yet; no changes to model code, migrations, frontend, v1 artifacts, or the deployed release.
Validation
Local: 47/47 tests green, mypy strict clean across 130 source files, ruff format+lint clean, docs link checker OK. CI is the broader acceptance gate for the full suite.