Skip to content

WO20u M1: WO20_SEED knob + the multi-seed confirmation scorer - #144

Open
ronaldtse wants to merge 3 commits into
mainfrom
wo20u-m1-m2
Open

ronaldtse wants to merge 3 commits into
mainfrom
wo20u-m1-m2

Conversation

@ronaldtse

Copy link
Copy Markdown
Contributor

Summary

  • train_wo20_tiny_hf.py gains the WO20_SEED knob (torch/numpy/python/DataLoader-worker seeding) so seeds 2/3 of the WO20 bridge-arm recipe run as explicit independent draws; default WO20_SEED=0 preserves the unseeded seed-1 behavior bit-for-bit. Seed recorded in the train log line and final_eval.json recipe.
  • New wo20_seed_confirm.py — the WO20u M1 confirmation scorer (WO20u: M1 bridge-arm multi-seed confirmation (pre-registered) + M2 int8 release candidates staged interscript-models#444): scores a seed's banked student predictions against G2a's released predictions (frontier-predictions-v1/run-007-r7-muon-6ep-final_preds.jsonl, sha f6f60619… verified in-job) under the identical sadeed evaluator (word-count mismatches skipped pairwise) and the identical count-weighted paired bootstrap (documents = resampling unit, 10k, seed 7). Writes wo20u-m1/<run>.confirm.json to the isx-training bucket; resume-safe via that marker.
  • Judgment pre-registered in interscript-models TODO.sota-2026/06 BEFORE the seed-2/3 GPU spend: the bridge lever confirms iff ≥2 of 3 seeds show separated-improvement vs G2a with non-overlapping CIs. The scorer's seed-1 calibration (must reproduce −0.4385pp CI [−0.6463, −0.2767]) gates any seeds-2/3 reading.

Test plan

  • Seed jobs launched and running with the knob (seed: WO20_SEED=2/3 in-job log lines)
  • Seed-1 calibration run reproduces the published seed-1 delta

The bridge arm's separated-improvement vs G2a (-0.4385pp, p=0) is a
single-seed result; the ledger's seed-variance caveat (2608.27729)
applies. Seeds 2 and 3 run the identical recipe (same scan bridges,
same 2.0 rung base, beta 0.1) as explicit independent draws:
WO20_SEED seeds torch/numpy/python/DataLoader-workers; the unseeded
seed-1 run stays as-is and is recorded as an uncontrolled draw.

wo20_seed_confirm.py scores any seed's banked student preds against
G2a's released preds (frontier-predictions-v1, sha-verified) with the
identical evaluator + count-weighted paired bootstrap; the seed-1
calibration (must reproduce -0.4385pp CI [-0.65, -0.28]) precedes any
seed-2/3 reading. Judgment pre-registered in interscript-models
TODO.sota-2026/06 before launch.
…er got it transitively via transformers, the CPU scorer did not)
…agraph arrays are already pp; the paired delta/CI/p were always correct - the seed-1 calibration matched the published verdict exactly)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant