Skip to content

feat(tools): quantize any float weight in checkpoint layout, and load what no quantized kernel runs as float - #123

Draft
chmjkb wants to merge 9 commits into
mainfrom
@chmjkb/quantizer-kernel-agnostic
Draft

chmjkb wants to merge 9 commits into
mainfrom
@chmjkb/quantizer-kernel-agnostic

Conversation

@chmjkb

@chmjkb chmjkb commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Description

The principle. Exporting weights should be as simple as MLX. Float weights are the PyTorch / Hugging Face checkpoint saved as safetensors. Quantized weights come from runntime quantize, which writes ints and scales under the checkpoint's own names and knows nothing about kernels or layer types. The runtime fits each weight to its kernels at load, so a file never needs re-exporting because a kernel changed. A slowdown on minor layers is an acceptable price; the hot quantized paths must stay as they are.

What changes.

  • tools/python: runntime quantize quantizes any float tensor with 2 or more dims that matches --include, stored as rows [dim0, prod(rest)] in groups of --group-size along the row. The kernel rules are gone; tensors named *embed*, *pos_emb* or lm_head.* are skipped by default, and quantizing them is opt-in.
  • packages/runntime: a quantized tensor that no quantized kernel runs loads as its dequantized floats instead of throwing CHECKPOINT_MISMATCH. One console.warn per load lists the Linear, Conv1d and ConvTranspose1d weights whose quantized kernel could not run them. Zoo hooks that reshape, transpose or split an N-D tensor dequantize it first.
  • Docs: core/exporting-weights.mdx and its file figure, tools/python/README.md.

What the tool writes (format version 1, as main reads it):

ints scales (<name>_scale, F16)
header runntime.quant = {"version":1,"scheme":"symmetric"}
Linear [out, in] I8 [out, in], int4 U8 [out, in/2] [out, in/g]
Conv1d [out, in, k] I8 [out, in·k], int4 U8 [out, in·k/2] [out, in·k/g]
ConvTranspose1d [in, out, k] I8 [in, out·k], int4 U8 [in, out·k/2] [in, out·k/g]

A selected tensor stays as stored only when the format can't hold it: not float, fewer than 2 dims, the group size doesn't divide the row, or (int4) the row is odd.

Decisions, why, and what each costs

1. Store every quantized tensor flattened to [dim0, prod(rest)] (chosen over keeping torch N-D shapes under a new format version).

  • Why: it is one rule with no layer knowledge, it is exactly what the reader and the Conv1d / ConvTranspose1d loaders on main already read, and it needs no format, reader or kernel change. Hosted files, Higgs included, keep loading.
  • Cost: the file holds the flattened shape, not torch's. A conv's group size must divide in·k, not in. An int4 row must be even. A hook that reshapes, transposes or splits an N-D tensor has to dequantize it first, with inTorchShape; the EmbeddingGemma 2 and Higgs hooks do. The privacy-filter experts cannot be recovered from rows, so that hook throws CHECKPOINT_MISMATCH on a quantized expert tensor.

2. Kernel rules removed from the tool (in % 8, out % 4 or % 8, group size % 4).

  • Why: they describe today's quantized matmul, not the file format.
  • Cost: a layer the kernel can't take used to stay float in the file. Now it is quantized in the file and dequantized at load, so that layer uses float memory and takes a small load-time cost.

3. Header without bits / groupSize.

4. Default excludes kept (*embed*, *pos_emb*, lm_head.*); quantizing embeddings is opt-in with --exclude none. Decided by the user.

  • Why: a cautious default. The runtime can load a quantized lookup table now, so this is no longer a loader limit, and the int8 loss measured below is small. Whether a given model tolerates it is left to the user, and the docs say to compare outputs before shipping.
  • What was measured before the decision, worst cosine against the float model over 20 texts, int8, group size 32:
model tables float tables int8 drop
all-MiniLM-L6-v2 0.999736 0.999719 1.7e-5
EmbeddingGemma 2 text 0.999723 0.999688 3.5e-5
  • Int4 tables cost more: MiniLM's worst cosine goes from 0.932 to 0.925 on 8 texts. Not measured: Moonshine's tied embedding and output head, privacy-filter.
  • Cost: the skips are by name, so they also keep EmbeddingGemma 2's embedding_projection.weight float, which is a plain Linear; the docs say so. The default-output file is larger than it could be (EmbeddingGemma 2 text: 414 MB with the tables float, 288 MB with them int8). Without --f16 a default MiniLM int8 file from the F32 checkpoint is 59 MB, more than the 45 MB F16 checkpoint; with --f16 it is 35 MB.

5. Runtime float fallbacks.

  • QuantTensor.f32() returns the dequantized values instead of throwing, and readFloats follows. The readers of stored bits (u16, halfWords, words) still throw.
  • Linear: a weight off the quantized matmul's limits loads as float in the model's dtype, instead of throwing.
  • Conv1d / ConvTranspose1d: same limits and same fallback as before.
  • loadStateDict: a quantized tensor bound to any float parameter (Embedding on GPU or CPU, a norm, a plain Parameter, a Conv2d weight) dequantizes on the CPU. A raw quantW slot still throws.
  • Cost: float memory for those tensors. A CPU table that would have kept raw BF16 bits becomes f32.

6. The warning fires only where a quantized kernel exists and the file missed it. Decided by the user.

  • Linear, Conv1d and ConvTranspose1d note a weight their quantized kernel cannot run (shape or group size) on the state dict, and loadStateDict prints one console.warn with every name, not truncated: loadStateDict: quantized weights loaded as float, the quantized kernel of their layer cannot run their shape or group size: a, b.
  • A tensor no quantized kernel reads (an embedding table, a Conv2d weight, a norm) loads as float silently. The onFloatRead callback from the first round is removed.
  • Cost: a quantized Conv2d or embedding table gives no hint that it only saved download size; the docs say so instead.
  • The names are the file's: a QuantTensor carries the file tensors it is made of (sources), so a weight a hook stacked or scaled lists q.weight, k.weight, not the helper expression.

Assumptions

  • Flattening is row-major, so value (o, c, tap) of a Conv1d is value c·k + tap of row o, which is what the conv loaders read. A unit test loads the tool's golden conv weight into Conv1d(16, 6, 4).
  • The generic bind checks element count only, so a flattened tensor fills a parameter of another shape with the same size.
  • The quantized conv scale readers take any group size that divides the row.
  • Run end to end through createEmbedder by me: all-MiniLM-L6-v2 and the EmbeddingGemma 2 text tower. The other hook fixes are covered by unit tests on tensors stored as rows, not by loading a fresh export of the real model.

Introduces a breaking change?

  • Yes
  • No

The public API is unchanged and existing files load as before. runntime quantize now quantizes conv weights by default, and its output needs a runtime with #97.

Type of change

  • Bug fix (change which fixes an issue)
  • New feature (change which adds functionality)
  • Documentation update (improves or adds clarity to existing documentation)
  • Performance (kernel tuning, graph building, anything related to performance)
  • Other (chores, tests, code style improvements etc.)

Testing instructions

  1. pnpm typecheck && pnpm lint && pnpm format:check
  2. pnpm --filter runntime test:unit (1165 pass)
  3. pnpm --filter runntime test:gpu (1910 pass, 15 expected fail)
  4. pnpm --filter docs... run build
  5. In tools/python: uv run pytest (51 pass), uv run ruff check, uv run ruff format --check, uv run pyright (0 errors)
  6. pnpm --filter runntime bench:ab kernels -b 8de7374 -k matmul,conv: 93 of 93 cases same, max abs diff 0. The sha is the base of this branch; a stale local main ref compares against the wrong commit.
  7. Zoo A/B guard (the bench files of the local @chmjkb/zoo-ab-guard branch, not committed here), against 8de7374: MiniLM f32, f16 and int8 g32 all same, max abs diff 0. Its EmbeddingGemma 2 and Moonshine cases are written for the module layout of the checkpoint-names stack and do not import on this branch, so they did not run.

New tests: Linear, Conv1d and ConvTranspose1d just inside and just outside each kernel limit with route comments, and a fallback Linear giving the same bits as a float Linear loaded from the dequantized weight; an Embedding on GPU and CPU, int4 and int8, f32 and f16, matching the dequantized floats without a warning; the warning (every name listed, 12 names untruncated, a hook-stacked weight, silent for a plain parameter, a fitting weight and a float file); each hook on a tensor stored as rows (EmbeddingGemma 2 position table, audio conv and depthwise conv, Higgs weight norm, privacy-filter experts refused); the tool's skip reasons, default excludes and their override, 3D and odd shapes, the header, the golden files read from both Python and TypeScript.

End to end, first round: the hosted all-MiniLM-L6-v2 f32 file, quantized with the tool and loaded through createEmbedder({ modelPath }), against the float model. These files had the embedding tables quantized, which now takes --exclude none:

file worst cosine loaded as float
int8 0.999719 the 3 embedding tables
int8, --group-size 6 0.999887 every Linear and the tables
int4 0.925 the 3 embedding tables

Docs numbers, from the page's own command (runntime quantize model.safetensors out.safetensors --bits 8 --f16, and --bits 4) on the hosted all-MiniLM-L6-v2 F32 file, through createEmbedder at its default dtype. Dataset: 20 questions searched against 20 one-sentence answers, written for this run and not in the repo; "top 3" compares the ordered top 3 documents per query. Two runs gave the same numbers:

file worst cosine top 3 changed top 1 changed
int8 (default skips) 0.99982 2 of 20 0
int4 (default skips) 0.94171 12 of 20 0
int8, --exclude none 0.99981 2 of 20 0

QA's numbers on its own data differ (int8 0.99977; int4 0.947 and 9 of 20), which is why the page now names the command and the dataset size.

Screenshots

Related issues

Part of #70.

Checklist

  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have updated the documentation accordingly
  • My changes generate no new warnings

Additional notes

Fixed in the second round, from review and QA:

  • EmbeddingGemma 2 audio layer1.conv.weight threw at load from a fresh default export; its depthwise conv and vision position table, and Higgs's weight norm, refused a tensor stored as rows. They dequantize first now.
  • privacy-filter experts stored as rows would have been split wrongly without an error; the hook throws CHECKPOINT_MISMATCH naming the tensor.
  • Docs: an int4 warning for conv models (QA saw MobileNetV4, DepthART and YOLO26 break on int4; that finding is QA's, not reproduced here), what loading as float costs.
  • Tests: the exact message is asserted again for the quantW refusal in loadStateDict.test.ts, and the Linear fallback is checked bit for bit.

Third round:

  • Embeddings: the default skips stay, worded as an opt-out default instead of a quality claim; --exclude none and the --f16 size numbers are on the docs page and in the README.
  • Docs numbers re-run with the page's command, with the dataset stated; the CPU-table sentence is removed.
  • The warning lists the file's tensor names.
  • Nits: the privacy-filter comment about the 2D expert bias, and a cast in loadStateDict.test.ts.

Still open:

https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK

…equantized floats, with one warning per load

A quantized Linear weight off the quantized matmul's limits, and a quantized
tensor bound to an embedding table, a CpuParameter, a norm or a plain
parameter, used to throw CHECKPOINT_MISMATCH. They load as float now, in the
model's dtype.

QuantTensor.f32() gives the dequantized values, so a hook that reads a tensor
as floats works on a quantized one. loadStateDict names every tensor read that
way in one console.warn, Conv1d and ConvTranspose1d's float fallback included.
A hook's own dequantize() is left out.
…ith no kernel or layer rules

runntime quantize knew the quantized matmul: it took 2D tensors only, wanted
in features a multiple of 8, out features of 4 or 8 and a group size of 4, and
kept embeddings and lm_head float because only Linear loaded quantized.

It knows the file format only now. A selected float tensor with 2 or more dims
is stored as [first dim, everything else] in groups of --group-size along that
row, so a Conv1d [out, in, k] is [out, in·k], the layout the conv loaders read.
It stays as stored only when it is not float, has fewer than 2 dims, the group
size does not divide the row, or an int4 row is odd.

The header leaves out bits and groupSize: the reader takes them from each
tensor. No default excludes: int8 embedding tables move an embedding's cosine
by under 1e-4 on all-MiniLM-L6-v2 and EmbeddingGemma 2.
… rows, embedding tables too, float fallback at load
…ts layer has a quantized kernel

The warning listed every quantized tensor read as floats, so a file with a
quantized embedding table or Conv2d weight warned on every load, about tensors
no quantized kernel could ever run.

Linear, Conv1d and ConvTranspose1d now note a weight their quantized kernel
cannot run on the state dict, and loadStateDict warns about those only, with
every name. The onFloatRead callback on QuantTensor is gone.

Claude-Session: https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK
…transposes or splits it

A file stores a quantized tensor as rows [dim 0, everything else], so a hook
that works past dim 0 cut through its scale groups: EmbeddingGemma 2's audio
layer1 conv threw at load, its depthwise conv and vision position table and
Higgs's weight norm refused the shape.

inTorchShape gives such a hook the dequantized floats in torch's shape. The
privacy-filter experts cannot be split from rows at all and would mix the
experts' values, so the hook throws CHECKPOINT_MISMATCH on a quantized one.

Claude-Session: https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK
… quality choice

The default exclude went away with the kernel rules. It is back as
*embed*, *pos_emb* and lm_head.*: not because the runtime cannot load a
quantized table, it can, but because quantized embedding tables only rarely
hold up. --exclude replaces it.

Claude-Session: https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK
…onv models, say what loading as float costs

- Embedding tables and lm_head stay float by default, as a quality choice.
- A quantized tensor no quantized kernel runs is turned back into floats: a
  smaller download only, and 4 bytes per value for a table kept on the CPU.
- MiniLM against the float model, 20 queries over 20 documents: int8 keeps
  cosine 0.9997 or higher, int4 falls as low as 0.93 and reorders the top 3
  results for 12 queries, int8 for 2.

Claude-Session: https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK
…ghts loaded as float

A weight a hook stacked or scaled showed up as the helper expression,
concatRows(scaleWeight(q.weight), ...). A QuantTensor now carries the names of
the file tensors it is made of, and the warning lists those.

Also: the privacy-filter comment gave the wrong reason for refusing the 2D
expert bias, and a test built its file with a cast.

Claude-Session: https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK
…ate the numbers' command and dataset

- The tool skips *embed*, *pos_emb* and lm_head.* by name; --exclude none
  quantizes them too, and checking the model's outputs is the user's call.
- --f16 matters for a default file: MiniLM int8 is 59 MB without it, 35 MB
  with it, against 45 MB for the F16 checkpoint.
- Picking a setting: the page's own command on MiniLM, 20 queries over 20
  documents. int8 0.9998 and 2 reordered top 3, int4 0.94 and 12.
- Drop the sentence about a CPU table's bytes per value.

Claude-Session: https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK
chmjkb added a commit that referenced this pull request Nov 17, 2026
…near branch

The EmbeddingGemma 2 hooks keep the checkpoint names and take inTorchShape
around the reshaped tensors. The fused Linear's float fallback notes its
parts on the state dict, as Linear does.

Claude-Session: https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant