Repository navigation
kolibri1 TT: the routed-expert slot dequant read its fp8 source at twice the element stride - #3443
Merged
Merged
Conversation
lu-zero
force-pushed
the
row/tt-kolibri-flips2
branch
from
October 11, 2026 03:52
519c466 to
35e4f59
Compare
… device token gate The B2b-ii device token gate compared the TT free-running greedy chain against the committed golden chain per step index and BROKE at the first divergence. Once the free-running chain splits from the golden, every later same-index comparison measures different histories — the exact flaw the decode-bench near-tie instrument (test_kolibri1_decode_bench.cpp, RunTeacherForced) already fixed. This mirrors that methodology in the B2b-ii gate: PASS 1 keeps the byte-comparison gate unchanged. PASS 2 runs the TT engine TEACHER-FORCED on the golden chain's tokens (decode inputs are generated_ids[d-1], never the engine's own argmax), so the engine's per-step logits share the golden prefix, and adjudicates every divergence under that common prefix: gap = row-top1 - row[golden], top-5 membership, 2.5-nat band. The committed goldens carry per-step TOKENS only (final_logits is one prefill-position top-5), so the distribution is the engine's own teacher-forced row — the documented fallback. A new CHECK (instr_hard_flips == 0) is the adjudicated verdict; it is RED today by design — the red quantifies the real defect. Walks that hit the streaming pool's no-slot refusal (kolibri1_tt_forward.cpp:424 — itself a finding: the pool cannot sustain a full 32-step decode on the golden prefix under VT_TT_HOST_FREE_DECODE=0) are recorded as aborted and the instrument keeps every prompt adjudicated so far. Measured on the P150 (2026-10-10, docs to follow in the row evidence): host-free ON — 182 free divergences, 182 teacher-forced flips, 0 near-ties, 182 HARD, worst gap 17.80 nats; host-free OFF — 33 free divergences, 49 teacher-forced flips, 2 near-ties, 47 HARD, worst gap 5.17 nats, 5 of 8 golden-prefix walks aborted in the pool refusal. Diagnosis only: no op is fixed and the existing gate behavior is kept. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]
tt-metal rms_norm bias (red micro-test kept; both repairs substrate-
refused; baseline routing verified restored)
The teacher-forced instrument (eb7cefb85) on row/tt-kolibri-residue
characterized the shared residue: OFF arm 47 HARD flips, worst gap 5.17
nats, on 'der Mond...', 'Translation to German...', 'x1 = 3...'. This
change localizes the FIRST diverging op and records why it is not yet
fixed.
Localization (VT_KOLIBRI1_TT_STAGE_DUMP emb/gam dumps added to both
arms, env-gated): the embedding gather and the input_layernorm gamma
stage BIT-EXACT on device (CPU 9.846392 vs TT 9.846392 at L0 dhn in the
eager arm; identical sums and first values). The divergence enters the
device rms_norm: in the host-free ON arm L0 dhn measures 9.510736
against the CPU 9.846392, a systematic ~3.4% scale-down with
bit-identical inputs and gamma.
The micro-test (test_kolibri1_tt_b2ii "SCRATCH dbg rmsnorm micro", RED,
kept) pins it: vt::RmsNorm TT vs CPU on random bf16 [1,2560] gives sum
ratio 0.9899 and max_abs 0.0625 (4 bf16 ULP) — the pinned
ttnn::rms_norm deviates ~1% SYSTEMATICALLY even on f32 tiles. Normed
3-4x per layer over 50 layers, that bias compounds into the multi-nat
logit shifts and the prompt-dependent hard flips (argmax survives only
where the golden distribution is peaked/repetitive).
Two repairs were tried and REFUSED by the substrate; both are recorded
in RmsNormKernel and reverted, because each regressed the gate:
1. Composed f32 chain (x^2 -> mean -> +eps -> rsqrt -> scale), exact
in the micro (ratio 0.9998, 1 bf16 ULP). Its [rows,1] row scale
cannot reach the model: plain multiply broadcasts padded-tile
garbage (gate collapsed, sporadic 1e37 logits), BcastOpDim::W is
numerically wrong (ratio 1.019), BcastOpDim::H is refused, and
repeat + same-shape multiply is refused by binary_ng ("Invalid
subtile broadcast type") in the model context.
2. Serving the eager arm's residual-free short-row norms from the
host f32 loop: the extra host round-trips of the attention/MoE
intermediates desync the device shadow — the gate collapsed from
26/33 to 0/8 with 1e37 garbage logits, under both the all-norms
and hidden-width-only narrowings.
The landed tree keeps baseline routing: the device token gate is
verified back at 26/33 with 47 HARD (post-revert device run), the TT
unit suite is 16/16, and the b2ii host cases are 13/13. The composed
arm stays behind VT_TT_RMSNORM_COMPOSED=1 for A/B.
Also recorded: the slot-pool no-slot refusal STILL fires on the fixed
base (4 of 8 teacher-forced walks abort, e.what()=="1") but the
diverging prompts never abort — the divergence and the pool refusal
are two defect classes; and a teacher-forced CPU-vs-TT walk scratch
("SCRATCH dbg TF prompt", VT_TF_DBG_PROMPT/VT_TF_DBG_STEPS) joins the
dump helpers for the next bisect.
The row issue Resolution carries the full dated evidence. The fix owes
either an upstream rms_norm whose f32 statistics match the oracle, a
broadcast primitive that survives the model geometry, or the host-free
shadow reconciliation this row already owes.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai/glm-5.3-flash [maki]
…strate The red micro's 0.9899 root cause lived below vllm.cpp: ttnn::rms_norm's default compute config hardwired fp32_acc=false, so the layernorm program factory ran Float16_b CBs with float32_reduction=false even for FLOAT32 input tiles — the sum of squares accumulated in bf16. The trial tt-metal tree /tmp/tt-metal-umdtrial now carries a reduction-precision patch (fp32_acc=true, approx_mode=false in rmsnorm_default_compute_config; the reduce scaler CB is Float32 under an f32 reduction) and /tmp/umdtrial-install was rebuilt from it; the record in the issue Resolution and docs/bench-evidence/kolibri1-tt-b2ii-20261009.md section 7 so every future gate on this stack uses the patched install. Evidence (same clean reset/cache recipe per leg): micro ratio 0.98992 -> 0.99972 vs the CPU row, max_abs 4 ULP -> 1 ULP; against the host-double sum rounded to bf16 the TT arm (0.99915) sits where the CPU oracle row does (0.99942) — bf16 output-store floor, no kernel bias left. B2b-ii gate: ON 26/33 with 7 hard (instrument 35 HARD, worst 3.19 nats), OFF 26/33 with 6 hard (instrument 105 HARD, worst 7.59 nats, deterministic across a clean reset). The argmax chain did not move materially: the remaining flips belong to the two open op-level drifts (kGdnDecode, kMatmulBTQuantGrouped), both re-confirmed failing in isolation; test_tenstorrent_backend shows no new isolated failures. The >=141 target stays open; this row's device work continues on those drifts. The micro now also prints the host-double-bf16 denominator and the per-element disagreement counts, which is what separated the quantization floor from the kernel bias. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]
element stride — fix the packed/bf16 offset split in the B2b-ii memo The memoized slot dequant advanced one byte_base over BOTH the packed fp8-e4m3 source (one byte per element) and the bf16 destination (two bytes per element). Gate read at offset 0 correctly; up dequanted down's slot region; down dequanted past the slot end. The NaN/Inf weights fed the routed ExpertMlp, whose outputs dominated every downstream stage: first-token logits were corrupted at layer 0, which is where the P150 gate's hard flips actually came from. Red-first via the teacher-forced instrument on the fixed rmsnorm substrate: downdequant ref-vs-memo mismatched 1,177,065 of 1,310,720 elements with NaN memo sums; the one-hot down_w probe returned inf/NaN from k=256 against a clean scale grid; slotcmp proved the staged slot bytes byte-exact (the readback pivot could not catch this — it verifies the slot against its own write image). After the fix the dequant mismatches 0/1,310,720 and the L0 moe stage matches the CPU arm exactly; the first diverging stage moves to L1 moe at ~3% relative (near-tie). The earlier attribution of the residue to kGdnDecode and kMatmulBTQuantGrouped is falsified at census level: the kolibri forward has no GDN op and its experts are fp8-block weights dequanted on host — neither op has a device arm in this model's dispatch. Recorded in the row issue with the re-derived numbers: gate flips move from 7 hard (ON, pre-fix) to 4 ON / 3 OFF recorded before a deterministic device-DRAM exhaustion now reached because the walks survive longer; the full 33-prompt tally stays blocked on that memoized-staging budget (open debt). The streaming slot-pool what()=="1" throw did not fire on the fixed substrate. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]
VT_TT_FORCE_HOST_PA/RESIDUAL/ROPE are the host-free bisect switches the host-free fix added, and VT_KOLIBRI1_TT_STAGE_DUMP is the stage-dump diagnostic the near-tie localization uses; all four are engineering switches, not deployment levers, so they take the allowlist classification the other TT diagnostic knobs hold. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]
lu-zero
force-pushed
the
row/tt-kolibri-flips2
branch
from
October 11, 2026 08:00
6e2851d to
1749ef6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The B2b-ii teacher-forced instrument (row/kolibri-tt-near-tie, merged via #3442) localized the kolibri1 P150 "prompt-dependent" hard flips to a real bug, not gate noise: the routed-expert slot dequant in kolibri1_tt_forward.cpp advanced its byte base by nk2, serving BOTH the fp8 source (1 B/element) and the bf16 destination (2 B/element) — up dequanted down's slot region, down dequanted past the slot end into NaN/Inf territory. Corruption entered at layer 0 and every downstream prompt difference was downstream amplification; prompts "survived" only where the golden distribution was peaked.
Evidence: a one-hot down_w device probe returned inf/NaN from k=256 with a clean scale grid; a reference dequant mismatched the memo dequant in 1,177,065 of 1,310,720 elements; slotcmp proved the slot BYTES were byte-exact (the readback pivot cannot catch a stride bug). Fix: separate packed_base (nk) and bf16_base (nk*2) offsets. After: dequant mismatch 0/1,310,720; layer-0 moe matches the CPU arm exactly; the first divergence moves to layer-1 moe at ~3% relative (near-tie class — the next localization target).
Verdicts: device gate hard flips 7 -> 4 (host-free ON) / 3 (OFF); teacher-forced worst gap 11.76 -> 10.05 nats; argmax now matches the CPU arm at 5/13 steps on the previously-worst prompt (was 0). The prior attribution to kGdnDecode/kMatmulBTQuantGrouped drift is FALSIFIED at op-census level: kolibri1 dispatches neither (census recorded in the issue).
Issue: ISSUE-LOCAL-01M4ER0E9HHM95YZYJB7T5FECN.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai/glm-5.3-flash [maki]