Skip to content

refactor(zoo): name the transformer models' modules like their checkpoints, and warn on unread tensors - #101

Open
chmjkb wants to merge 10 commits into
mainfrom
@chmjkb/checkpoint-names-tier1
Open

chmjkb wants to merge 10 commits into
mainfrom
@chmjkb/checkpoint-names-tier1

Conversation

@chmjkb

@chmjkb chmjkb commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator

Description

Tier 1 of "modules own their checkpoint layout" (#70): the transformer models in the zoo name their modules exactly like their Hugging Face checkpoints, so their state-dict hooks no longer rename, drop or tie tensors.

  • loadStateDict({ strict }), PyTorch-style. A tensor in the file that the model doesn't read now gets a console.warn listing it and stays unbound; strict: true throws CHECKPOINT_MISMATCH as before. A missing tensor always throws. The progress total counts only bound tensors. No runner passes strict.
  • RMSNorm.scale → RMSNorm.weight, PyTorch's name. privacy-filter's hosted int8g640 file stores the old .scale names; its detector maps them (renameExportedNormScales).
  • Moonshine and Moonshine Streaming: fields follow model.encoder.* / model.decoder.* (self_attn, encoder_attn, input_layernorm, ...). The conv stem lives in the encoder as in the checkpoint; Streaming's decoder gets its own decoder.ts for pos_emb / proj.
  • MiniLM: embeddings.{word_embeddings,position_embeddings,LayerNorm}, encoder.layer.N.{attention.{self,output},intermediate,output}.
  • EmbeddingGemma 2: language_model.*, vision_tower.* + embed_vision.*, audio_tower.* + embed_audio.*.
  • Code style: a rule that a model's module tree matches its checkpoint, and that a hook only builds derived tensors, with a bad/good example.

Hook files after this, with zero move() calls left in all four:

model hook lines
MiniLM 102 → 79
Moonshine 141 → 84
Moonshine Streaming 93 → 71
EmbeddingGemma 2 334 → 256

What's left in them is fusion (q/k/v concatRows), scale folds, RoPE and conv permutes, and tensors computed on the CPU. Those move into core in tier 2.

Introduces a breaking change?

  • Yes
  • No

The public API (create<Task>(), models) is unchanged. Hosted files that carry tensors the model never reads now print one warning per load:

  • all-MiniLM-L6-v2 f16/f32: embeddings.position_ids, pooler.dense.weight, pooler.dense.bias
  • Moonshine Streaming tiny f16/f32: proj_out.weight

Re-exporting those files without the unused tensors removes the warning; that's left for the Hub upload step.

Type of change

  • Bug fix (change which fixes an issue)
  • New feature (change which adds functionality)
  • Documentation update (improves or adds clarity to existing documentation)
  • Performance (kernel tuning, graph building, anything related to performance)
  • Other (chores, tests, code style improvements etc.)

Testing instructions

  1. pnpm typecheck && pnpm lint && pnpm format:check && pnpm --filter runntime test:unit (1104 pass)
  2. pnpm --filter runntime exec vitest run --project gpu test/zoo test/core/nn test/core/kernels/matmul (427 pass, 3 expected fail that exist on the base too)

Checked outside this branch:

  • A model-level A/B (kept on a local branch, not part of this PR) loaded the real hosted f32, f16 and int8 g32 files on both sides and compared every output: max difference 0 on MiniLM and all three EmbeddingGemma 2 towers. Breaking one hook on purpose made it report 4.1.
  • Through the public runners, base vs this branch: identical MiniLM and EmbeddingGemma 2 embeddings, identical Moonshine tiny and Streaming tiny transcripts on 3 clips, identical privacy-filter spans on the hosted int8g640 file.

Related issues

Part of #70. Stacked on #99.

Checklist

  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have updated the documentation accordingly
  • My changes generate no new warnings

Additional notes

  • Field names copy the checkpoint, including snake_case and capitals (self_attn, LayerNorm); the code-style skill now says so.
  • Derived tensors (qkv, kv, qkv_norm, rel, q_scale, pos_x/pos_y) sit under the checkpoint-named parent.

@chmjkb
chmjkb added this pull request to stack #119 November 13, 2026 14:19
@chmjkb
chmjkb force-pushed the @chmjkb/checkpoint-names-tier1 branch 3 times, most recently from d6737d9 to 8d5d3a7 Compare November 16, 2026 10:11
Base automatically changed from @chmjkb/quant-embeddinggemma2-models to main November 16, 2026 10:14
@chmjkb
chmjkb force-pushed the @chmjkb/checkpoint-names-tier1 branch from 8d5d3a7 to 89c85f5 Compare November 16, 2026 10:15
/** Derives what the HF BertModel checkpoint stores differently from the
* model's parameters, as lazy tensors; nothing is fetched here. Every other
* tensor binds under its own name, since the modules are named like the
* checkpoint; the pooler and `position_ids` are left unread.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess we should still drop the tensors the model never reads (pooler, position_ids, proj_out). The hosted MiniLM and Moonshine Streaming files have them, so every load of a default model logs a warning now.

// Bind in state-dict (file) order, so each chunk is read once. Module order
// jumps around the payload and refetches evicted chunks. Keys a transform
// added sort last; they read nothing new.
const position = new Map([...sd.tensors.keys()].map((name, i) => [name, i]));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Model files download almost twice now: MiniLM f16 fetches 82.5 MB for a 45.4 MB file (45.5 MB on main), because hooks append the fused tensors at the end of the state dict and the loader goes back for chunks it already dropped. Maybe put a derived tensor where its first source sat in the file?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants