Skip to content

WO20: r7 tiny-student scan + distill HF job (microkimi recipe) - #143

Merged
ronaldtse merged 5 commits into
mainfrom
wo20-ml-tiny
Nov 16, 2026
Merged

ronaldtse merged 5 commits into
mainfrom
wo20-ml-tiny

Conversation

@ronaldtse

Copy link
Copy Markdown
Contributor

Summary

  • train_wo20_tiny_hf.py — the WO20 campaign-A job (issue WO20: tiny-student distillation (Arabic r7) + static-int8 sweep on released artifacts interscript-models#441): kill-gated two-stage distillation of r7 into ByT5-small via the microkimi recipe (scan-first directional ridge-R² pairing with scrambled controls → frozen rank-256 bridges → the 2.0 rung recipe + β·MSE through the bridges), plus the full-protocol eval epilogue with paired bootstrap vs the in-job teacher.
  • Ran to completion across four resumable 4h HF-job windows (labels.sha-pinned 500-step checkpoints); verdict recorded in interscript-models#442: 4.0545 full-set DER-CE — GATE-FAIL vs teacher but SEPARATED-IMPROVEMENT over the published G2a frontier (−0.44pp, p=0) at half the epochs.
  • wo20_muon.py is a verbatim copy of the lineage optimizer for run reproducibility.

Test plan

  • Review the scan kill rule implementation against the pre-registration (TODO.sota-2026/06 on interscript-models)
  • Review the paired-bootstrap epilogue math (documents = resampling unit, count-weighted)

train_wo20_tiny_hf.py - two-stage, kill-gated. Stage scan: verify the r7
teacher release (teachers-arabic-v1, per-part sha256) and the canonical
labels (teacher_labels_r7.jsonl, e70ce991), capture teacher/student decoder
streams on shared teacher-forced positions, closed-form directional
ridge-R2 over every (student, teacher) layer pair with scrambled controls,
pre-registered kill rule (R2 >= 0.30 and >= 2x control), low-rank frozen
bridges on arm. Stage distill: the 2.0 rung verbatim (Muon 0.01 + AdamW 1e-4,
sequence KD alpha 0.5 T 2.0, BS 8, 3 epochs, labels <=384 B/side) plus the
single delta beta*MSE through the frozen bridges; step checkpoints every 500
with labels.sha resume; epilogue decodes student AND in-job teacher on the
full 1,200-paragraph SadeedDiac-25 windowed zero-skip protocol and emits the
paired bootstrap (documents = resampling unit) + final_eval.json.

wo20_muon.py: verbatim copy of the lineage optimizer (interscript-models
attic/gpu/muon.py) so the job /code mount carries the exact 2.0 routing.
…caps

The ara specs cap units at max_len 1450 B and labels at 2x max_len =
2900 B (accept_label) with collate truncation at those caps and gradient
checkpointing on - my 384-B default came from a different Pairs call
site and produced 112 usable pairs instead of the lineage's ~29k. Fix
the caps, the collate truncation, and disable the cache/checkpointing
for the eval decode.
@ronaldtse
ronaldtse merged commit d312c4d into main Nov 16, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant