Repository navigation
feat(core): run Conv1d and ConvTranspose1d on quantized weights - #97
Merged
Merged
Conversation
xxpopielxx
force-pushed
the
feat/quant-conv
branch
from
November 5, 2026 07:35
f0fd850 to
5d15eab
Compare
…t version 2 A version 2 file sets bits by dtype and groupSize by scale shape per weight, so one file can mix them. Version 1 loads as before.
conv1d takes a quantW weight and runs on matmulQuantTiled, which gains the loader slot and a scale period so a group of input channels shares a scale in every tap. Linear, Conv1d and ConvTranspose1d share the loader.
xxpopielxx
force-pushed
the
feat/quant-conv
branch
from
November 5, 2026 10:22
5d15eab to
61187aa
Compare
This was referenced Nov 5, 2026
…ional bits and groupSize A file without bits or groupSize takes them from each tensor's dtype and scale shape. A file that sets them is checked against every tensor, as before.
5 of 11 tasks
chmjkb
requested changes
Nov 9, 2026
chmjkb
left a comment
Collaborator
There was a problem hiding this comment.
With this setup, the quantizer has to know about convs. It needs to know the kernel size to pick the group size. It also needs to swap the axes of ConvTranspose1d weights before quantizing.
The reason is that the kernel only reads scales in one fixed order. You can't regroup the scales after quantizing, so the file has to match the kernel.
Could we flip this around? The kernel could let each layer say where its scales are, like loadA does for the input. Then the quantizer wouldn't care about layers at all, and the files would stay in torch layout like the rest of runntime.
It might make the kernel a bit slower, so it needs a benchmark.
…s them matmulQuantTiled takes a scale loader next to loadA. Conv1d and ConvTranspose1d load quantized weights as torch's [out, in·kernel] and [in, out·kernel], one scale per group of values along each row, so a quantizer needs no layer knowledge.
chmjkb
approved these changes
Nov 12, 2026
6 of 11 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Quantized Conv1d and ConvTranspose1d, on top of #96.
aand a scale period, so conv1d runs on it with one scale per group of input channels in every tap.Introduces a breaking change?
Type of change
Testing instructions
pnpm --filter runntime test: new GPU cases for quantW conv1d and for Conv1d / ConvTranspose1d loaded from a quantized file against the dequantized float layer, plus unit tests for the reader, op and layer loaders.pnpm bench:ab kernels -b @chmjkb/quant-small-m -k matmul,conv: all 79 existing cases same, max abs diff 0. New quantW conv cases skip on the base.Screenshots
Related issues
#96
Checklist
Additional notes
Version 2 changes #96's file format, so it is its own commit. Int8 convs run at about float speed (compute-bound); they save weight memory.