Repository navigation
Conversation
…equantized floats, with one warning per load A quantized Linear weight off the quantized matmul's limits, and a quantized tensor bound to an embedding table, a CpuParameter, a norm or a plain parameter, used to throw CHECKPOINT_MISMATCH. They load as float now, in the model's dtype. QuantTensor.f32() gives the dequantized values, so a hook that reads a tensor as floats works on a quantized one. loadStateDict names every tensor read that way in one console.warn, Conv1d and ConvTranspose1d's float fallback included. A hook's own dequantize() is left out.
…ith no kernel or layer rules runntime quantize knew the quantized matmul: it took 2D tensors only, wanted in features a multiple of 8, out features of 4 or 8 and a group size of 4, and kept embeddings and lm_head float because only Linear loaded quantized. It knows the file format only now. A selected float tensor with 2 or more dims is stored as [first dim, everything else] in groups of --group-size along that row, so a Conv1d [out, in, k] is [out, in·k], the layout the conv loaders read. It stays as stored only when it is not float, has fewer than 2 dims, the group size does not divide the row, or an int4 row is odd. The header leaves out bits and groupSize: the reader takes them from each tensor. No default excludes: int8 embedding tables move an embedding's cosine by under 1e-4 on all-MiniLM-L6-v2 and EmbeddingGemma 2.
… rows, embedding tables too, float fallback at load
…ts layer has a quantized kernel The warning listed every quantized tensor read as floats, so a file with a quantized embedding table or Conv2d weight warned on every load, about tensors no quantized kernel could ever run. Linear, Conv1d and ConvTranspose1d now note a weight their quantized kernel cannot run on the state dict, and loadStateDict warns about those only, with every name. The onFloatRead callback on QuantTensor is gone. Claude-Session: https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK
…transposes or splits it A file stores a quantized tensor as rows [dim 0, everything else], so a hook that works past dim 0 cut through its scale groups: EmbeddingGemma 2's audio layer1 conv threw at load, its depthwise conv and vision position table and Higgs's weight norm refused the shape. inTorchShape gives such a hook the dequantized floats in torch's shape. The privacy-filter experts cannot be split from rows at all and would mix the experts' values, so the hook throws CHECKPOINT_MISMATCH on a quantized one. Claude-Session: https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK
… quality choice The default exclude went away with the kernel rules. It is back as *embed*, *pos_emb* and lm_head.*: not because the runtime cannot load a quantized table, it can, but because quantized embedding tables only rarely hold up. --exclude replaces it. Claude-Session: https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK
…onv models, say what loading as float costs - Embedding tables and lm_head stay float by default, as a quality choice. - A quantized tensor no quantized kernel runs is turned back into floats: a smaller download only, and 4 bytes per value for a table kept on the CPU. - MiniLM against the float model, 20 queries over 20 documents: int8 keeps cosine 0.9997 or higher, int4 falls as low as 0.93 and reorders the top 3 results for 12 queries, int8 for 2. Claude-Session: https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK
6 of 11 tasks
…ghts loaded as float A weight a hook stacked or scaled showed up as the helper expression, concatRows(scaleWeight(q.weight), ...). A QuantTensor now carries the names of the file tensors it is made of, and the warning lists those. Also: the privacy-filter comment gave the wrong reason for refusing the 2D expert bias, and a test built its file with a cast. Claude-Session: https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK
…ate the numbers' command and dataset - The tool skips *embed*, *pos_emb* and lm_head.* by name; --exclude none quantizes them too, and checking the model's outputs is the user's call. - --f16 matters for a default file: MiniLM int8 is 59 MB without it, 35 MB with it, against 45 MB for the F16 checkpoint. - Picking a setting: the page's own command on MiniLM, 20 queries over 20 documents. int8 0.9998 and 2 reordered top 3, int4 0.94 and 12. - Drop the sentence about a CPU table's bytes per value. Claude-Session: https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK
chmjkb
added a commit
that referenced
this pull request
Nov 17, 2026
…near branch The EmbeddingGemma 2 hooks keep the checkpoint names and take inTorchShape around the reshaped tensors. The fused Linear's float fallback notes its parts on the state dict, as Linear does. Claude-Session: https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The principle. Exporting weights should be as simple as MLX. Float weights are the PyTorch / Hugging Face checkpoint saved as safetensors. Quantized weights come from
runntime quantize, which writes ints and scales under the checkpoint's own names and knows nothing about kernels or layer types. The runtime fits each weight to its kernels at load, so a file never needs re-exporting because a kernel changed. A slowdown on minor layers is an acceptable price; the hot quantized paths must stay as they are.What changes.
tools/python:runntime quantizequantizes any float tensor with 2 or more dims that matches--include, stored as rows[dim0, prod(rest)]in groups of--group-sizealong the row. The kernel rules are gone; tensors named*embed*,*pos_emb*orlm_head.*are skipped by default, and quantizing them is opt-in.packages/runntime: a quantized tensor that no quantized kernel runs loads as its dequantized floats instead of throwingCHECKPOINT_MISMATCH. Oneconsole.warnper load lists the Linear, Conv1d and ConvTranspose1d weights whose quantized kernel could not run them. Zoo hooks that reshape, transpose or split an N-D tensor dequantize it first.core/exporting-weights.mdxand its file figure,tools/python/README.md.What the tool writes (format version 1, as
mainreads it):<name>_scale, F16)runntime.quant = {"version":1,"scheme":"symmetric"}[out, in]I8 [out, in], int4U8 [out, in/2][out, in/g][out, in, k]I8 [out, in·k], int4U8 [out, in·k/2][out, in·k/g][in, out, k]I8 [in, out·k], int4U8 [in, out·k/2][in, out·k/g]A selected tensor stays as stored only when the format can't hold it: not float, fewer than 2 dims, the group size doesn't divide the row, or (int4) the row is odd.
Decisions, why, and what each costs
1. Store every quantized tensor flattened to
[dim0, prod(rest)](chosen over keeping torch N-D shapes under a new format version).mainalready read, and it needs no format, reader or kernel change. Hosted files, Higgs included, keep loading.in·k, notin. An int4 row must be even. A hook that reshapes, transposes or splits an N-D tensor has to dequantize it first, withinTorchShape; the EmbeddingGemma 2 and Higgs hooks do. The privacy-filter experts cannot be recovered from rows, so that hook throwsCHECKPOINT_MISMATCHon a quantized expert tensor.2. Kernel rules removed from the tool (
in % 8,out % 4or% 8, group size% 4).3. Header without
bits/groupSize.4. Default excludes kept (
*embed*,*pos_emb*,lm_head.*); quantizing embeddings is opt-in with--exclude none. Decided by the user.embedding_projection.weightfloat, which is a plain Linear; the docs say so. The default-output file is larger than it could be (EmbeddingGemma 2 text: 414 MB with the tables float, 288 MB with them int8). Without--f16a default MiniLM int8 file from the F32 checkpoint is 59 MB, more than the 45 MB F16 checkpoint; with--f16it is 35 MB.5. Runtime float fallbacks.
QuantTensor.f32()returns the dequantized values instead of throwing, andreadFloatsfollows. The readers of stored bits (u16,halfWords,words) still throw.Linear: a weight off the quantized matmul's limits loads as float in the model's dtype, instead of throwing.Conv1d/ConvTranspose1d: same limits and same fallback as before.loadStateDict: a quantized tensor bound to any float parameter (Embedding on GPU or CPU, a norm, a plainParameter, a Conv2d weight) dequantizes on the CPU. A rawquantWslot still throws.6. The warning fires only where a quantized kernel exists and the file missed it. Decided by the user.
Linear,Conv1dandConvTranspose1dnote a weight their quantized kernel cannot run (shape or group size) on the state dict, andloadStateDictprints oneconsole.warnwith every name, not truncated:loadStateDict: quantized weights loaded as float, the quantized kernel of their layer cannot run their shape or group size: a, b.onFloatReadcallback from the first round is removed.QuantTensorcarries the file tensors it is made of (sources), so a weight a hook stacked or scaled listsq.weight, k.weight, not the helper expression.Assumptions
(o, c, tap)of a Conv1d is valuec·k + tapof rowo, which is what the conv loaders read. A unit test loads the tool's golden conv weight intoConv1d(16, 6, 4).createEmbedderby me: all-MiniLM-L6-v2 and the EmbeddingGemma 2 text tower. The other hook fixes are covered by unit tests on tensors stored as rows, not by loading a fresh export of the real model.Introduces a breaking change?
The public API is unchanged and existing files load as before.
runntime quantizenow quantizes conv weights by default, and its output needs a runtime with #97.Type of change
Testing instructions
pnpm typecheck && pnpm lint && pnpm format:checkpnpm --filter runntime test:unit(1165 pass)pnpm --filter runntime test:gpu(1910 pass, 15 expected fail)pnpm --filter docs... run buildtools/python:uv run pytest(51 pass),uv run ruff check,uv run ruff format --check,uv run pyright(0 errors)pnpm --filter runntime bench:ab kernels -b 8de7374 -k matmul,conv: 93 of 93 cases same, max abs diff 0. The sha is the base of this branch; a stale localmainref compares against the wrong commit.@chmjkb/zoo-ab-guardbranch, not committed here), against8de7374: MiniLM f32, f16 and int8 g32 all same, max abs diff 0. Its EmbeddingGemma 2 and Moonshine cases are written for the module layout of the checkpoint-names stack and do not import on this branch, so they did not run.New tests: Linear, Conv1d and ConvTranspose1d just inside and just outside each kernel limit with route comments, and a fallback Linear giving the same bits as a float Linear loaded from the dequantized weight; an Embedding on GPU and CPU, int4 and int8, f32 and f16, matching the dequantized floats without a warning; the warning (every name listed, 12 names untruncated, a hook-stacked weight, silent for a plain parameter, a fitting weight and a float file); each hook on a tensor stored as rows (EmbeddingGemma 2 position table, audio conv and depthwise conv, Higgs weight norm, privacy-filter experts refused); the tool's skip reasons, default excludes and their override, 3D and odd shapes, the header, the golden files read from both Python and TypeScript.
End to end, first round: the hosted all-MiniLM-L6-v2 f32 file, quantized with the tool and loaded through
createEmbedder({ modelPath }), against the float model. These files had the embedding tables quantized, which now takes--exclude none:--group-size 6Docs numbers, from the page's own command (
runntime quantize model.safetensors out.safetensors --bits 8 --f16, and--bits 4) on the hosted all-MiniLM-L6-v2 F32 file, throughcreateEmbedderat its default dtype. Dataset: 20 questions searched against 20 one-sentence answers, written for this run and not in the repo; "top 3" compares the ordered top 3 documents per query. Two runs gave the same numbers:--exclude noneQA's numbers on its own data differ (int8 0.99977; int4 0.947 and 9 of 20), which is why the page now names the command and the dataset size.
Screenshots
Related issues
Part of #70.
Checklist
Additional notes
Fixed in the second round, from review and QA:
layer1.conv.weightthrew at load from a fresh default export; its depthwise conv and vision position table, and Higgs's weight norm, refused a tensor stored as rows. They dequantize first now.CHECKPOINT_MISMATCHnaming the tensor.quantWrefusal inloadStateDict.test.ts, and the Linear fallback is checked bit for bit.Third round:
--exclude noneand the--f16size numbers are on the docs page and in the README.loadStateDict.test.ts.Still open:
draft: true, so the production build skips it; the edited figure was not checked in a browser.@chmjkb/fused-linear(feat(core): add nn.FusedLinear and fuse the zoo's attention projections with it instead of hooks #124) is stacked on this branch; history here is append-only.https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK