Repository navigation
feat(core): add nn.FusedLinear and fuse the zoo's attention projections with it instead of hooks - #124
Draft
chmjkb wants to merge 24 commits into
Draft
feat(core): add nn.FusedLinear and fuse the zoo's attention projections with it instead of hooks#124chmjkb wants to merge 24 commits into
chmjkb wants to merge 24 commits into
Conversation
…vacy-filter's exported scale to it
… like their checkpoints
…weight the encoders read the dtype off
…skill, with an example
…he whole checkpoint
…BF16 checkpoint's bounds are not read as raw bits Claude-Session: https://claude.ai/code/session_019uvkwiFEsgcU5AJssd6Pjz
… Linear branch FusedLinear needs module fields that mirror checkpoint names and the strict option of loadStateDict, which are on #101 and not on #123. Claude-Session: https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK
…input as one stacked weight and one matmul The parts stay the module fields the checkpoint names, and the FusedLinear is one more field. At load it stacks the parts' rows: floats as floats, ints with their scales when every part has the same bits and group size, the dequantized floats for any other mix. namedParameters() lists the parts, so the key check runs on checkpoint names; the new internal storageParameters() lists what is bound, which differs only here. Claude-Session: https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK
… instead of a hook The hook still folds the score scale into query and key, now in place under their checkpoint names. Claude-Session: https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK
…rojections with nn.FusedLinear The hooks keep the 1/√headDim fold into q_proj, in place. The projection width check moves to the core Linear hook. Claude-Session: https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK
…d v with nn.FusedLinear The hooks keep the rope row order of q and k, in place. The audio tower keeps each projection's bounds under its own name and still refuses bounds that differ, since one matmul is clamped with q's. Claude-Session: https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK
…r instead of a hook The hook keeps the score scale fold into q_proj, in place. Claude-Session: https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK
…q, k and v are one fused weight The leak test binds random weights itself and the quantized test lists the weightScale names; both need the FusedLinear's stacked weight, not its parts. Claude-Session: https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK
…ights guide and the code-style skill Claude-Session: https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK
6 of 11 tasks
…near branch The EmbeddingGemma 2 hooks keep the checkpoint names and take inTorchShape around the reshaped tensors. The fused Linear's float fallback notes its parts on the state dict, as Linear does. Claude-Session: https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK
…where the file holds its parts The stacked weight came from the FusedLinear's own transformStateDict hook. It missed an edit by a hook that ran later, and its key sorted after every other derived key, so the loader walked the file again for it. Stacking moves to fuseStateDict, a second pass loadStateDict runs once all transformStateDict hooks are done. It gets the path of every module, so the parts can sit anywhere in the model. The stacked weight goes in the state dict right before the first of its parts, and so does the bias. Also: - a FusedLinear held outside the module tree fails the load, named by a part - a part's bias of the wrong length is reported under the part's name - the constructor checks every part before it marks any, and names a part that is listed twice - the float fallback warning names the fused layer with its parts' files Claude-Session: https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK
…they bind in file order The hooks took each tensor out and set it again, which moved it to the end of the state dict. A FusedLinear binds at the place of its first part, so a moved part pulled the stacked weight to the end with it. Claude-Session: https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK
…port FusedLinear misuse at load Every lazy tensor now names the file tensors it is read from: a file tensor itself, a helper's output its inputs, a derivedTensor what its maker passes. loadStateDict sorts by the first source's place in the file, so a hook that takes a tensor out and puts a derived one at the end no longer makes the loader fetch a chunk twice. A tensor that names no source binds where the file holds its own name, or last. FusedLinear needs no placement of its own. FusedLinear: - one held in two fields, or a part held in two fields, fails the load with both paths - parts that are not quantized alike get their own warning, which lists every part with how it is stored - a file tensor named like the stacked weight is reported like any tensor the model does not read: a warning, an error under strict Claude-Session: https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK
…niLM and EmbeddingGemma 2 text walk a file once The MiniLM position table and EmbeddingGemma 2's qkv_norm, layer_scalar and PLE projection bound after the rest of the file, which fetched the first chunks again. Claude-Session: https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Tier 2, PR A: the fuse. Adds
nn.FusedLinearto core and replaces the hand-written fused Linears in the zoo with it. A model author lists the Linears; loading needs no hook for the fuse.Read this first: the diff includes #101. The base branch (#123) is off
mainand does not have tier 1 (#101), which this PR needs (checkpoint-named module fields,strict). The first commit merges@chmjkb/checkpoint-names-tier1into this branch. Review fromfeat(core): add nn.FusedLinear…on; the #101 part disappears from the diff once #101 lands.API
namedParameters()listsq_proj.weight… and nothing forqkv.strictloading reports nothing for it.q_proj.linearin EmbeddingGemma 2's towers, or a sibling module).What happens at load
It never throws for a mix. After stacking, the weight goes through
Linear's own load code, so the forward is the same dispatch as before (matmulormatmulQuantW, same attrs and inputs).A fused layer that falls back to float is named in a warning, with its parts by their file tensors (
sources):…self.qkv (FusedLinear of …query.weight, …key.weight, …value.weight);loadStateDict: fused layers loaded as float, their parts are not quantized alike: …self.qkv: …query.weight is float, …key.weight is int8 in groups of 32, …, listing every part.Decisions and what each costs
namedParameters()is the checkpoint's view (the parts). A new internalstorageParameters()is what gets bound, retyped byhalf()and freed bydispose(). They differ only at a FusedLinear: the parts are out, its stacked weight is in, under the FusedLinear's own path (…self_attn.qkv.weight, plus.biasand.weightScale). The key check uses the first, the bind loop the second.namedParameters()misses the stacked weight. Two GPU tests did that and now walkstorageParameters(). Progress callbacks still report…qkv.weight.Linearinside the FusedLinear (an underscore field, which the module Proxy does not register). All ofLinear's logic is reused: transpose, the quantized fit check, packing,weightScale,forward.Linear: this layer is a part of a FusedLinear, which holds its weight…). Its weight is never uploaded on its own. Making it work would mean either a second copy of the weight on the GPU or a full fused matmul plus a slice behind a call that looks cheap.transformStateDicthook, in a second pass (fuseStateDict) thatloadStateDictmakes over the tree. Field order does not matter: an edit a hook makes to a part is always in the stacked weight.Module.sources): a file tensor itself, a helper's output its inputs, a hand-writtenderivedTensorwhat its maker passes asfrom.loadStateDictbinds by the first source's place in the file. A tensor that names no source binds where the file holds its own name, or last. So no hook, and not the FusedLinear, has to care where it puts a key, and the loader walks the file once.sourcesis a new field on every lazy tensor, and a hand-writtenderivedTensorunder a new key has to passfromto bind early. Without it, it binds last, as before.7a. A file tensor named like the stacked weight (
…qkv.weight) is a tensor the model does not read: a warning, or an error understrict. The stacked weight still loads from the parts.Assumptions
Fuse sites replaced
query/key/value, weight and biasheadDim^-¼on query and keyqkv, cross-attentionkv1/√headDimon crossq_projqkv(decoder shared with Moonshine)1/√headDimon encoderq_projand crossq_projq_proj/k_proj/v_proj{q,k,v}_proj.linear{q,k,v}_proj.linearq_proj/k_proj/v_proj, weight and bias1/√headDimonq_projEmbeddingGemma 2 audio: each projection is a clipped linear with its own bounds. They now bind under their own names. One matmul can only be clamped once, so the model clamps with q's bounds and the hook still refuses a checkpoint where k's or v's differ (
CHECKPOINT_MISMATCH, at the read, as before). This check had no test; it has one now.stateDictHooks.tslines, before → after: MiniLM 79 → 68, Moonshine 84 → 49, Moonshine Streaming 71 → 65, EmbeddingGemma 2 272 → 271 (the fuse is gone, the per-part bounds added lines), privacy-filter 87 → 87. Higgs' attention hook: 6 → 5.Left out, and why
block.N.attn.qkv), and its hosted files, the defaultint8g640included, already store the fusedqkvunder that name. With parts in the tree those files would need a split at load only to be stacked again. It needs the tier 1 rename and a decision on the exported files first.qkv_normin EmbeddingGemma 2 text and vision. It tiles two norm vectors over heads. It is not a Linear, soconcatRowsstays for it.gate_projcarries a GELU in its epilogue, and privacy-filter'sgate_up_projis already one tensor in the checkpoint, which the hook splits.concatRowsis still used byFusedLinear,qkv_normand privacy-filter.Hook transforms that PRs B and C delete
scaleWeightin MiniLM (query, key), Moonshine (scaleQueryon crossq_proj), Moonshine Streaming (encoder and crossq_proj), Higgs (q_proj).rope({ layout })):ropeOrder/permuteRowsonq_projandk_projin EmbeddingGemma 2 text and vision, and the same order insidefusedQkvNorm.A/B
pnpm --filter runntime bench:ab zoo, tool from@chmjkb/zoo-ab-guardapplied locally to both sides and not committed. A is this branch's base (the merge of #123 and #101), B is this branch. Hosted f32 and f16 files; int8g32 made withrunntime quantize(MiniLM, Moonshine) or the existing EmbeddingGemma 2 int8 files; EmbeddingGemma 2 f32 widened from the f16 files.18 of 18 cases: max diff 0, timing the same.
The suite has no Higgs case and there is no Higgs checkpoint on this machine. A local case ran the encoder's HuBERT branch on the unit tests' small random checkpoint (f32, int8, int4, 20 frames): max diff 0 on all three. It has not been run on the real Higgs file.
Bytes fetched on a first load
From the Hub, no cache, through the zoo runners (
warmUp: false, counted byonBytes). Base is9be43cc, the merge of #123 and #101 without this PR.MiniLM, Moonshine and EmbeddingGemma 2 now fetch exactly their file size. Moonshine Streaming fetches 72 of 84 MB; I take that to be chunks that hold only tensors the model does not read, but did not check which. The base fetched more because its hook-made tensors bound after the rest of the file; the first version of this PR fetched more still.
On the final head:
Introduces a breaking change?
Zoo users see no change.
runntime/coreis not published.Type of change
Testing instructions
pnpm typecheck && pnpm lint && pnpm format:checkpnpm --filter runntime test:unit(1225 pass)pnpm --filter runntime test:gpu(1930 pass, 15 expected fail)pnpm --filter docs... run buildNew tests, in
test/core/nn/layers.test.tsandlayers.gpu.test.ts:Linearloaded from the hand-stacked rows, plus one case with literal values.half()model; halfwords stay halfwords through the stack.Linearof that size (float and int8).half(),dispose(), load progress bytes, parts nested under wrappers, a hook that scaled a part first, and the constructor errors.Linearover the stacked rows bit for bit.Screenshots
Related issues
Part of #70. Stacked on #123, needs #101.
Checklist
Additional notes
core/exporting-weights.mdx(a draft page) gets a short section onFusedLinearunder "Loading it into your own model". The code-style skill's rule about hooks now says a fuse is aFusedLinear, not a hook.cce0265. The EmbeddingGemma 2 hooks keep the checkpoint names and takeinTorchShapearound the tensors they reshape.1ec6179through957f080: two MiniLM GPU tests walkednamedParameters()and were switched tostorageParameters()only in5e8e184. History stays as is; the head is green.storageParameters()is internal, a part used alone throws, privacy-filter is not converted, andquantizedAsFloatstays on the state dict (withfusedPartsDiffernext to it).https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK