Skip to content

feat(core): add nn.FusedLinear and fuse the zoo's attention projections with it instead of hooks - #124

Draft
chmjkb wants to merge 24 commits into
@chmjkb/quantizer-kernel-agnosticfrom
@chmjkb/fused-linear
Draft

chmjkb wants to merge 24 commits into
@chmjkb/quantizer-kernel-agnosticfrom
@chmjkb/fused-linear

Conversation

@chmjkb

@chmjkb chmjkb commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Tier 2, PR A: the fuse. Adds nn.FusedLinear to core and replaces the hand-written fused Linears in the zoo with it. A model author lists the Linears; loading needs no hook for the fuse.

Read this first: the diff includes #101. The base branch (#123) is off main and does not have tier 1 (#101), which this PR needs (checkpoint-named module fields, strict). The first commit merges @chmjkb/checkpoint-names-tier1 into this branch. Review from feat(core): add nn.FusedLinear… on; the #101 part disappears from the diff once #101 lands.

API

q_proj = new nn.Linear(hidden, hidden);
k_proj = new nn.Linear(hidden, kvDim);
v_proj = new nn.Linear(hidden, kvDim);
qkv = new nn.FusedLinear([this.q_proj, this.k_proj, this.v_proj]);
// this.qkv.forward(x): one matmul, columns q | k | v
  • 1 to N parts, each with its own out features, all with the same in features.
  • The parts stay in the tree under their checkpoint names. namedParameters() lists q_proj.weight … and nothing for qkv. strict loading reports nothing for it.
  • The FusedLinear and its parts can be fields of any modules of the model (q_proj.linear in EmbeddingGemma 2's towers, or a sibling module).

What happens at load

parts stacked as runs on
all float floats: f16 when every part is F16, f32 otherwise float matmul
all quantized, same bits and group size ints and scale rows as stored (exact: groups run along in features) quantized matmul when a Linear of the whole size fits it, else dequantized floats with the warning
anything else (float + quantized, int8 + int4, two group sizes) dequantized floats float matmul, with step 2's warning naming the quantized parts

It never throws for a mix. After stacking, the weight goes through Linear's own load code, so the forward is the same dispatch as before (matmul or matmulQuantW, same attrs and inputs).

A fused layer that falls back to float is named in a warning, with its parts by their file tensors (sources):

  • parts stored alike, but the stacked size misses the quantized matmul: the existing warning, with …self.qkv (FusedLinear of …query.weight, …key.weight, …value.weight);
  • parts not stored alike: loadStateDict: fused layers loaded as float, their parts are not quantized alike: …self.qkv: …query.weight is float, …key.weight is int8 in groups of 32, …, listing every part.

Decisions and what each costs

  1. Two walks over the tree. namedParameters() is the checkpoint's view (the parts). A new internal storageParameters() is what gets bound, retyped by half() and freed by dispose(). They differ only at a FusedLinear: the parts are out, its stacked weight is in, under the FusedLinear's own path (…self_attn.qkv.weight, plus .bias and .weightScale). The key check uses the first, the bind loop the second.
    • Cost: code that binds or inspects parameters by walking namedParameters() misses the stacked weight. Two GPU tests did that and now walk storageParameters(). Progress callbacks still report …qkv.weight.
  2. The stacked weight lives in a hidden Linear inside the FusedLinear (an underscore field, which the module Proxy does not register). All of Linear's logic is reused: transpose, the quantized fit check, packing, weightScale, forward.
    • Cost: the underscore rule is an existing but implicit Proxy convention.
  3. A part used alone throws (Linear: this layer is a part of a FusedLinear, which holds its weight…). Its weight is never uploaded on its own. Making it work would mean either a second copy of the weight on the GPU or a full fused matmul plus a slice behind a call that looks cheap.
    • Cost: a model that needs one part alone in some path cannot fuse it.
  4. Bias rule: the stacked Linear has a bias when any part has one. A part without one adds zeros.
  5. Stacking runs after every transformStateDict hook, in a second pass (fuseStateDict) that loadStateDict makes over the tree. Field order does not matter: an edit a hook makes to a part is always in the stacked weight.
    • Cost: one more optional hook on Module.
  6. Bind order is the loader's job, for every tensor. Each lazy tensor names the file tensors it is read from (sources): a file tensor itself, a helper's output its inputs, a hand-written derivedTensor what its maker passes as from. loadStateDict binds by the first source's place in the file. A tensor that names no source binds where the file holds its own name, or last. So no hook, and not the FusedLinear, has to care where it puts a key, and the loader walks the file once.
    • Cost: sources is a new field on every lazy tensor, and a hand-written derivedTensor under a new key has to pass from to bind early. Without it, it binds last, as before.
  7. The FusedLinear and every part must be in the module tree, each in one field. A part outside it, a FusedLinear held in an array, an underscore field or a local, and a FusedLinear or a part held in two fields all fail the load with a message that names the paths.
    7a. A file tensor named like the stacked weight (…qkv.weight) is a tensor the model does not read: a warning, or an error under strict. The stacked weight still loads from the parts.
  8. A part may not have an activation, may be in one FusedLinear only, and may be listed once. All throw at construction, before any part is marked.
  9. The quantized fit is checked on the stacked size, as a hand-fused weight was. Two int4 parts of 4 outputs run quantized together (8 % 8 == 0) though neither would alone. Two int8 parts of 4 and 2 outputs load as float.

Assumptions

  • A bias is never quantized (the tool only quantizes tensors with 2 or more dims).

Fuse sites replaced

model site transform still in the hook (in place, under the checkpoint name)
MiniLM query/key/value, weight and bias headDim^-¼ on query and key
Moonshine encoder and decoder self-attention qkv, cross-attention kv 1/√headDim on cross q_proj
Moonshine Streaming encoder qkv (decoder shared with Moonshine) 1/√headDim on encoder q_proj and cross q_proj
EmbeddingGemma 2 text q_proj/k_proj/v_proj rope row order of q and k
EmbeddingGemma 2 vision {q,k,v}_proj.linear rope row order of q and k
EmbeddingGemma 2 audio {q,k,v}_proj.linear none for the weights
Higgs encoder HuBERT q_proj/k_proj/v_proj, weight and bias 1/√headDim on q_proj

EmbeddingGemma 2 audio: each projection is a clipped linear with its own bounds. They now bind under their own names. One matmul can only be clamped once, so the model clamps with q's bounds and the hook still refuses a checkpoint where k's or v's differ (CHECKPOINT_MISMATCH, at the read, as before). This check had no test; it has one now.

stateDictHooks.ts lines, before → after: MiniLM 79 → 68, Moonshine 84 → 49, Moonshine Streaming 71 → 65, EmbeddingGemma 2 272 → 271 (the fuse is gone, the per-part bounds added lines), privacy-filter 87 → 87. Higgs' attention hook: 6 → 5.

Left out, and why

  • privacy-filter q/k/v. Its model is not on checkpoint names yet (block.N.attn.qkv), and its hosted files, the default int8g640 included, already store the fused qkv under that name. With parts in the tree those files would need a split at load only to be stacked again. It needs the tier 1 rename and a decision on the exported files first.
  • qkv_norm in EmbeddingGemma 2 text and vision. It tiles two norm vectors over heads. It is not a Linear, so concatRows stays for it.
  • Gate/up. No model fuses them today: EmbeddingGemma's gate_proj carries a GELU in its epilogue, and privacy-filter's gate_up_proj is already one tensor in the checkpoint, which the hook splits.
  • No core helper ended up unused: concatRows is still used by FusedLinear, qkv_norm and privacy-filter.

Hook transforms that PRs B and C delete

  • B (attention scale as a kernel uniform): scaleWeight in MiniLM (query, key), Moonshine (scaleQuery on cross q_proj), Moonshine Streaming (encoder and cross q_proj), Higgs (q_proj).
  • C (rope({ layout })): ropeOrder / permuteRows on q_proj and k_proj in EmbeddingGemma 2 text and vision, and the same order inside fusedQkvNorm.

A/B

pnpm --filter runntime bench:ab zoo, tool from @chmjkb/zoo-ab-guard applied locally to both sides and not committed. A is this branch's base (the merge of #123 and #101), B is this branch. Hosted f32 and f16 files; int8g32 made with runntime quantize (MiniLM, Moonshine) or the existing EmbeddingGemma 2 int8 files; EmbeddingGemma 2 f32 widened from the f16 files.

model case A (µs) B (µs) Δ noise verdict max abs(A−B)
embeddinggemma2 text f32, 12 tokens in 32 rows 10591 10580 -0.1% ±3.0% same 0
embeddinggemma2 text f16, 12 tokens in 32 rows 9142 9199 +0.6% ±3.8% same 0
embeddinggemma2 text int8g32, 12 tokens in 32 rows 7133 7125 -0.1% ±3.0% same 0
embeddinggemma2 vision f32, 6×3 patches 7654 7730 +1.0% ±3.9% same 0
embeddinggemma2 vision f16, 6×3 patches 6064 6092 +0.5% ±3.0% same 0
embeddinggemma2 vision int8g32, 6×3 patches 4813 4850 +0.8% ±7.1% same 0
embeddinggemma2 audio f32, 1 s 13969 14015 +0.3% ±3.0% same 0
embeddinggemma2 audio f16, 1 s 11120 11157 +0.3% ±3.0% same 0
embeddinggemma2 audio int8g32, 1 s 10190 10196 +0.1% ±3.0% same 0
minilm base f32, 2 sentences in 32 rows 965 975 +1.0% ±3.0% same 0
minilm base f16, 2 sentences in 32 rows 898 906 +0.9% ±3.8% same 0
minilm base int8g32, 2 sentences in 32 rows 605 614 +1.4% ±5.5% same 0
moonshine-streaming tiny f32, 1 s, 12 tokens 2981 2978 -0.1% ±3.2% same 0
moonshine-streaming tiny f16, 1 s, 12 tokens 2786 2715 -2.5% ±3.1% same 0
moonshine-streaming tiny int8g32, 1 s, 12 tokens 2593 2600 +0.3% ±4.7% same 0
moonshine tiny f32, 1 s, 12 tokens 2781 2729 -1.9% ±4.1% same 0
moonshine tiny f16, 1 s, 12 tokens 2634 2643 +0.3% ±3.0% same 0
moonshine tiny int8g32, 1 s, 12 tokens 2259 2232 -1.2% ±3.0% same 0

18 of 18 cases: max diff 0, timing the same.

The suite has no Higgs case and there is no Higgs checkpoint on this machine. A local case ran the encoder's HuBERT branch on the unit tests' small random checkpoint (f32, int8, int4, 20 frames): max diff 0 on all three. It has not been run on the real Higgs file.

Bytes fetched on a first load

From the Hub, no cache, through the zoo runners (warmUp: false, counted by onBytes). Base is 9be43cc, the merge of #123 and #101 without this PR.

model file MB base MB this PR MB base chunks this PR chunks
MiniLM f16 43.3 78.7 43.3 13 8
Moonshine tiny f16 51.7 75.7 51.7 12 9
Moonshine Streaming tiny f16 84.0 104 72 15 11
EmbeddingGemma 2 text f16 517.0 817.9 517 105 67
EmbeddingGemma 2 all towers f16 1420.1 2512 1420.1 321 184

MiniLM, Moonshine and EmbeddingGemma 2 now fetch exactly their file size. Moonshine Streaming fetches 72 of 84 MB; I take that to be chunks that hold only tensors the model does not read, but did not check which. The base fetched more because its hook-made tensors bound after the rest of the file; the first version of this PR fetched more still.

On the final head:

  • Weight uploads at load, base against this PR, hashed per array: identical for MiniLM, Moonshine tiny and Moonshine Streaming tiny (f32, f16, int8, int4, each into an f32 and an f16 model) and EmbeddingGemma 2 text, vision and audio (f16, int8): 36 of 36. No EmbeddingGemma 2 int4 files on this machine.
  • The MiniLM zoo A/B (max diff 0) is from the head before the last two commits, which change the bind order and messages, not the weights.

Introduces a breaking change?

  • Yes
  • No

Zoo users see no change. runntime/core is not published.

Type of change

  • Bug fix (change which fixes an issue)
  • New feature (change which adds functionality)
  • Documentation update (improves or adds clarity to existing documentation)
  • Performance (kernel tuning, graph building, anything related to performance)
  • Other (chores, tests, code style improvements etc.)

Testing instructions

  1. pnpm typecheck && pnpm lint && pnpm format:check
  2. pnpm --filter runntime test:unit (1225 pass)
  3. pnpm --filter runntime test:gpu (1930 pass, 15 expected fail)
  4. pnpm --filter docs... run build

New tests, in test/core/nn/layers.test.ts and layers.gpu.test.ts:

  • 1, 2, 3 and 5 parts of different out sizes, with a bias, without, and mixed, checked against one Linear loaded from the hand-stacked rows, plus one case with literal values.
  • F16 parts into an f32 and a half() model; halfwords stay halfwords through the stack.
  • int8 and int4 parts stacked and loaded quantized, including parts that only fit the quantized matmul together.
  • Mixed parts (float + int8, int8 + int4, two group sizes, a stacked size that misses the kernel): dequantized floats, one warning naming the quantized parts.
  • The tree lists only the parts; strict load is silent; a missing part tensor is reported by its checkpoint name; an unread tensor still fails strict.
  • A part run alone throws; the parts stay unloaded.
  • The forward is the same op, attrs and inputs as a Linear of that size (float and int8).
  • half(), dispose(), load progress bytes, parts nested under wrappers, a hook that scaled a part first, and the constructor errors.
  • On the GPU, f32 and f16: the output matches a CPU loop per part, and matches one Linear over the stacked rows bit for bit.
  • Round 3: the loader binds by sources (a rename, a tensor built from two, one put back without sources, one from no file tensor); MiniLM and EmbeddingGemma 2 text each fetch every chunk of a chunked checkpoint once, in file order (both fail with the old order); a FusedLinear or a part in two fields; a file tensor named like the stacked weight, strict and not; the two float-fallback warnings.
  • Fix round: a FusedLinear declared before the module whose hook edits a part; parts in a sibling module; a FusedLinear outside the tree; a part's bias of the wrong length; the same part twice; parts left unmarked when the constructor throws; the bind order and the chunks fetched from a file read in chunks (each once, ascending).

Screenshots

Related issues

Part of #70. Stacked on #123, needs #101.

Checklist

  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have updated the documentation accordingly
  • My changes generate no new warnings

Additional notes

  • Docs: core/exporting-weights.mdx (a draft page) gets a short section on FusedLinear under "Loading it into your own model". The code-style skill's rule about hooks now says a fuse is a FusedLinear, not a hook.
  • The branch now also merges feat(tools): quantize any float weight in checkpoint layout, and load what no quantized kernel runs as float #123 at cce0265. The EmbeddingGemma 2 hooks keep the checkpoint names and take inTorchShape around the tensors they reshape.
  • The GPU suite is red on the commits from 1ec6179 through 957f080: two MiniLM GPU tests walked namedParameters() and were switched to storageParameters() only in 5e8e184. History stays as is; the head is green.
  • Kept as is: storageParameters() is internal, a part used alone throws, privacy-filter is not converted, and quantizedAsFloat stays on the state dict (with fusedPartsDiffer next to it).

https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK

… Linear branch

FusedLinear needs module fields that mirror checkpoint names and the strict
option of loadStateDict, which are on #101 and not on #123.

Claude-Session: https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK
…input as one stacked weight and one matmul

The parts stay the module fields the checkpoint names, and the FusedLinear
is one more field. At load it stacks the parts' rows: floats as floats, ints
with their scales when every part has the same bits and group size, the
dequantized floats for any other mix. namedParameters() lists the parts, so
the key check runs on checkpoint names; the new internal
storageParameters() lists what is bound, which differs only here.

Claude-Session: https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK
… instead of a hook

The hook still folds the score scale into query and key, now in place under their checkpoint names.

Claude-Session: https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK
…rojections with nn.FusedLinear

The hooks keep the 1/√headDim fold into q_proj, in place. The projection
width check moves to the core Linear hook.

Claude-Session: https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK
…d v with nn.FusedLinear

The hooks keep the rope row order of q and k, in place. The audio tower keeps
each projection's bounds under its own name and still refuses bounds that
differ, since one matmul is clamped with q's.

Claude-Session: https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK
…r instead of a hook

The hook keeps the score scale fold into q_proj, in place.

Claude-Session: https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK
…q, k and v are one fused weight

The leak test binds random weights itself and the quantized test lists the
weightScale names; both need the FusedLinear's stacked weight, not its parts.

Claude-Session: https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK
…near branch

The EmbeddingGemma 2 hooks keep the checkpoint names and take inTorchShape
around the reshaped tensors. The fused Linear's float fallback notes its
parts on the state dict, as Linear does.

Claude-Session: https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK
…where the file holds its parts

The stacked weight came from the FusedLinear's own transformStateDict hook.
It missed an edit by a hook that ran later, and its key sorted after every
other derived key, so the loader walked the file again for it.

Stacking moves to fuseStateDict, a second pass loadStateDict runs once all
transformStateDict hooks are done. It gets the path of every module, so the
parts can sit anywhere in the model. The stacked weight goes in the state
dict right before the first of its parts, and so does the bias.

Also:
- a FusedLinear held outside the module tree fails the load, named by a part
- a part's bias of the wrong length is reported under the part's name
- the constructor checks every part before it marks any, and names a part
  that is listed twice
- the float fallback warning names the fused layer with its parts' files

Claude-Session: https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK
…they bind in file order

The hooks took each tensor out and set it again, which moved it to the end
of the state dict. A FusedLinear binds at the place of its first part, so a
moved part pulled the stacked weight to the end with it.

Claude-Session: https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK
…port FusedLinear misuse at load

Every lazy tensor now names the file tensors it is read from: a file tensor
itself, a helper's output its inputs, a derivedTensor what its maker passes.
loadStateDict sorts by the first source's place in the file, so a hook that
takes a tensor out and puts a derived one at the end no longer makes the
loader fetch a chunk twice. A tensor that names no source binds where the
file holds its own name, or last. FusedLinear needs no placement of its own.

FusedLinear:
- one held in two fields, or a part held in two fields, fails the load with
  both paths
- parts that are not quantized alike get their own warning, which lists
  every part with how it is stored
- a file tensor named like the stacked weight is reported like any tensor
  the model does not read: a warning, an error under strict

Claude-Session: https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK
…niLM and EmbeddingGemma 2 text walk a file once

The MiniLM position table and EmbeddingGemma 2's qkv_norm, layer_scalar and
PLE projection bound after the rest of the file, which fetched the first
chunks again.

Claude-Session: https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant