Extra patches for llama-cpp-rdna-boosts on AMD RDNA4 (gfx1201, Radeon AI PRO R9700), for the changes that are not part of rdna-boosts: mostly lossy prefill speed-ups, a faster decode / draft-verification matmul, a guard against replies that end inside the reasoning, an opt-in lenience for sampled drafts, faster wide draft verification, a smaller recurrent-state cache for draft verification, a smaller FFN compute buffer, and fixes of upstream llama.cpp code that we keep here. Most other bit-identical work goes to rdna-boosts itself. Patches only; MIT. Written with Claude (AI).
File names: <serial>-<upstream base>-<name>-v<version>.patch. The files in patches/ are for rdna-boosts
v15-6050b467c-r156 (upstream base 6050b467c); patches/r144/ has what r144 to r153 need besides them
(Applicable versions). Older sets: patches/r59/ for v12-097cf33d9-r51 to r59, patches/r50/ for
v16-097cf33d9-r50, patches/a55/ for v16-a55e952b8-rN.
| patch | what | switch | type |
|---|---|---|---|
0001-605-k1-v6 |
K-quant prefill matmuls via hipBLASLt | GGML_LF_KQ_BLAS(default on, fp8) |
speed/ lossy |
0002-605-f8-v5 |
fp8 prefill attention | GGML_LF_FA_FP8(default on) |
speed/ lossy |
0003-605-m4-v3 |
MXFP4 weights to fp8 for 0001 |
GGML_LF_MXFP4_DIRECT(default on) |
speed/ lossy |
0004-605-reasoning-guard-v2 |
no reply left inside the reasoning | LLAMA_REASONING_GUARD(default on) |
guard |
0005-605-backend-sampling-reserve-v1 |
no buffer reallocation per request | — (always on) |
fix |
0006-605-mmvk-v3 |
faster 1-8 token K-quant matmuls | GGML_CUDA_MMVK(default on) |
speed/ diff |
0007-605-spec-lenience-v1 |
keep more sampled-draft tokens | LLAMA_SPEC_LENIENCE(default off) |
speed/ lossy |
0008-605-glu-inplace-v1 |
an FFN's gate, up and GLU output not all live at once (150 MiB less compute buffer at -ub 2048) |
GGML_GLU_INPLACE(default on) |
memory |
0009-605-fa-band-split-v1 |
faster 9-16 row draft verification | GGML_CUDA_FA_BAND_SPLIT(default on) |
speed/ diff |
0012-605-dflash-adaptive-width-v1 |
DFlash2 verify width from the acceptance | LLAMA_DFLASH_ADAPTIVE(default on) |
speed |
0013-605-gdn-verify-anchor-v3 |
Qwen3.x draft verification keeps two recurrent-state planes, not one per draft token (about 1 GiB less VRAM); a moved cell keeps its rollback snapshots | LLAMA_RS_ANCHOR(default on); GGML_CUDA_GDN_NT(default on) |
speed, fix |
optional/0014-605-ngram-prompt-only-v2 |
n-gram drafts from the current prompt only, with a key check: for rewriting whole files | LLAMA_SPEC_NGRAM_PROMPT_ONLY(default on, with ngram-mod) |
speed/ trade-off |
0015-605-spec-align-v3 |
slots verifying together send drafts of one length | LLAMA_SPEC_ALIGN,LLAMA_SPEC_ALIGN_CUT(default on) |
speed |
0010, 0011 (Temporarily)
What each patch does, what it needs, the types, and the change log: PATCHES.md.
All are on by default except 0007; GGML_LF_ALL=0 turns off every switch left unset (SWITCHES.md).
Prefill speed-ups measured on Linux, R9700, Qwen3.8-27B UD-Q4_K_XL, DFlash2 draft, 8K / 32K prompt, on v16-a55e952b8-r8 (the
patch code is the same since); KLD measured again on r144 (2026-10-27 report):
| prefill | KLD vs official UD-Q5_K_M on r144 (zh / en) | |
|---|---|---|
| reference (no patch) | — | 0.0018 / 0.0006 (run-to-run floor) |
| K1 f16 | +14% / +15% | 0.0018 / 0.0005 (≈ floor) |
| K1 fp8 (the default) | +16% / +22% on top of f16 | 0.0026 / 0.0007 |
| F8 (v5) | +3% / +17% | 0.0018 / 0.0005 |
| M4 (MXFP4 models only) | +55% at 8K, +44% at 110K | +0.002 / +0.0006 over the MXFP4 weights themselves |
mmvk (0006) |
— (decode: 4.5-6% less time per step with drafts) | 0.0015 / 0.0003 (below the floor) |
| all defaults | 0.0025 / 0.0009 |
K1, F8 and M4 leave decode unchanged; 0006 changes decode only (for its KLD the decode path runs it at one token, GGML_CUDA_MMVK=4). K1 fp8 with F8 also raises the perplexity of the prompt's own logits (every position scored on the prefill path):
by 0.1-0.3% on new text in Chinese, English or Thai and on a German novel, 0.6-0.8% on texts the model knows by heart, and
2.4-3.6% on a text it half-remembers (Hong Lou Meng, our Chinese test text; this note used to read that as a Chinese
effect). Generation reads the prompt through the KV cache and sees far less (the KLD column; Thai at the level of Chinese
and English). For prompt-scoring workloads set GGML_LF_KQ_BLAS=f16. Details: 2026-10-12 report.
F8 v5 counts the row sum from the fp8 probabilities themselves: on a 170K-token reasoning conversation F8's own KLD drops by a fifth (0.000396 to 0.000307, floor 0.00021); with all defaults the change is below the floor, at the same prefill speed (report). On the table's 48K test v5 is within the floor of v4 (F8 alone 0.0019 / 0.0007 with v4; all defaults 0.0024 / 0.0008 with v4).
0009 changes only draft-verification batches of 9-16 rows (two slots verifying at once, or a long n-gram draft), which
neither the KLD run nor the standard benchmark has; each of those rows then gets the result a batch of 8 rows or fewer gives.
Two slots decoding at once at 50K each: 13-19% slower than one after the other without it, 33-38% faster with it.
0008: the graph allocator lets a GLU write into its gate's memory, so an FFN's gate, up and GLU output are no longer all
live at once: at -ub 2048 the compute buffer is 150 MiB smaller on Qwen3.8-27B and 70 MiB on Qwen3.6-35B-A3B. The output
is the same, and prefill speed did not change.
0013 v3 (LLAMA_RS_ANCHOR=0 turns the anchor off): draft verification on qwen35 / qwen35moe models (Qwen3.5 to 3.8) keeps, per slot, the
recurrent state before and after the last batch plus small per-token replay factors, instead of one full state per draft
token, and replays the accepted part when a draft is partly rejected. The output is the same. Qwen3.8-27B with DFlash2:
1.03 GiB less VRAM with 2 slots, 0.51 GiB with 1 (-c 131072); each decode step 0.3-0.4% faster, two slots decoding at
once 2-4% faster. Its fix, always on: with 3 or more slots, a slot that had just verified a draft could have its cell
moved by another slot's batch in the same decode, lose its rollback snapshots, and continue from another slot's state
(upstream llama.cpp has this too). With 1 or 2 slots nothing changes.
Every switch, its values, and the server parameters it needs: SWITCHES.md.
On Linux we also set ROCm's own HSA_USERPTR_FOR_PAGED_MEM=0. By default ROCm allocates pinned host memory
(hipHostMalloc, llama.cpp's ROCm_Host buffers) as ordinary pageable pages registered with the GPU; when the kernel swaps out
or moves one of them (memory pressure, compaction), the driver stops all GPU queues of the process until the page is
back (5-440 ms at a time, often several in a row). With 0 those buffers are allocated by the driver instead and cannot be
swapped or moved. Under the same memory pressure: 23 stalls (1.16 s in total over 19 requests) became none, with the
same output and speed (report). The cost: those
buffers (about 0.7 GB with every layer on the GPU) stay in RAM; with large weights left in host memory (partial offload
without mmap) they count against the driver's limit. Not needed on Windows.
Tested on rdna-boosts v15-6050b467c-r156: the patches in patches/ apply in number order on the plain release, build, and
give the History numbers. 0015 v3 needs r154 or later (it shares the release's own LLAMA_SPEC_ALIGN). For r144 to r153,
apply the same files except 0015 (its git apply --check fails there), then patches/r144/0015-605-spec-align-v1.patch;
on Windows use r147 or later (r143 to r146 do not compile there, rdna-boosts issue #231).
On another release, check with git apply --check first. A release on a new upstream base (not 6050b467c) needs a new
patch set.
Speed on the standard benchmark (Qwen3.8-27B, 32K-token prompt, R9700, Linux), tokens/s. original: plain rdna-boosts.
Each dated row: this repo's patches of that date on it, all on; the date links to the report (setup and KLD).
| date | build | Q5 prefill | Q5 decode | MXFP4 prefill | MXFP4 decode |
|---|---|---|---|---|---|
| original | v16-a55e952b8-r8 |
983 | 47.4 | 1151 | 54.7 |
| 2026‑10‑07 | + K1 v2, F8 v3.1, M4 v2 | 1958 | 47.5 | 2393 | 63.0 |
| 2026‑10‑12 | + K1 v3, F8 v3.2, M4 v2 | 1954 | 49.0 | 2388 | 62.9 |
| original | v16-097cf33d9-r50 |
1124 | 50.8 | 1344 | 60.8 |
| 2026‑10‑13 | + K1 v3, F8 v3.2, M4 v2 | 1959 | 50.6 | 2397 | 66.2 |
| original | v12-097cf33d9-r59 |
1125 | 51.1 | 1335 | 60.0 |
| 2026‑10‑15 | + K1 v3.2, F8 v3.3, M4 v2 | 2197 | 50.4 | 2365 | 66.2 |
| 2026‑10‑18 | + K1 v3.2, F8 v3.3, M4 v2, mmvk v2 | 2207 | 53.8 | 2389 | 67.2 |
| original | v15-6050b467c-r144 |
1154 | 50.7 | 1381 | 63.5 |
| 2026‑10‑27 | + K1 v4, F8 v4, M4 v3, mmvk v3 (the defaults) | 2293 | 56.1 | 2449 | 65.4 |
| original | v15-6050b467c-r156 |
1155 | 52.8 | 1384 | 57.9 |
| 2026‑10‑28 | + K1 v4, F8 v5, M4 v3, mmvk v3 (the defaults) | 2300 | 58.8 | 2467 | 61.0 |
Q5 = official UD-Q5_K_M; MXFP4 = FreedomAISVR MXFP4. From r144 to r156 the time per decode step did not change (MXFP4: 43.35 / 43.21 ms with the patches); the decode figures moved with the draft acceptance, as the generated text changed.
The patches are on by default. Speculative decoding, which gives much of the generation speed, needs a draft model and
these server parameters; this is what we run on an R9700 with Qwen3.8-27B (Linux; on Windows the same in PowerShell,
without HSA_USERPTR_FOR_PAGED_MEM and with --poll 0):
HSA_USERPTR_FOR_PAGED_MEM=0 ./llama-server -m Qwen3.8-27B-UD-Q4_K_XL.gguf -ngl 99 -c 131072 --kv-unified \
-ub 2048 -b 2048 -fa on -ctk q8_0 -ctv q8_0 \
-md Qwen3.8-27B-DFlash2-Q4_K_M.gguf -ngld 99 --spec-draft-n-max 5 \
--spec-type draft-dflash \
--spec-draft-sampling probabilistic-ub 2048 -b 2048 -fa on -ctk q8_0 -ctv q8_0: K1 and F8, the prefill speed-ups, need them (SWITCHES.md).-md ... -ngld 99 --spec-draft-n-max 5: the DFlash2 draft model, for Qwen3.8-27B only (z-lab/Qwen3.8-27B-DFlash2-GGUF, Q4_K_M). With any other model leave out the last three lines.--spec-type draft-dflash: DFlash2 alone. To rewrite whole files faster, applyoptional/0014and use--spec-type ngram-mod,draft-dflash --spec-ngram-mod-n-max 15 --spec-ngram-mod-n-min 1 --spec-ngram-mod-n-match 30instead: whole-file edits +33..+66 %, other requests about the same, one long English continuation -4 % (Qwen3.8-27B, one request at a time). The parameters are upstream's; rdna-boosts clamps the length to 15.--spec-draft-sampling probabilistic: llama.cpp's own; sampled drafts, verified exactly by rejection sampling (it matters at temperature > 0).0007(LLAMA_SPEC_LENIENCE, lossy, off by default) builds on it.HSA_USERPTR_FOR_PAGED_MEM=0: Linux only (Switches).--poll 0: Windows. The Windows build has no OpenMP, so llama.cpp's own CPU thread pool spins after the CPU part of every step: on a Ryzen 7 5700G with an R9700 the server used 4.2-4.4 cores while generating, 0.85-0.91 with--poll 0, at the same speed and with the same text. The Linux build uses OpenMP and does not need it.- The Windows package of the GitHub Actions workflow comes with
run-server-sample.ps1: these settings, with the two model paths to fill in.
To stew675 and the rdna-boosts contributors, and to the llama.cpp / ggml community.
Issues and pull requests may not get an answer here. For problems, try a frontier cloud model first.
本 repo 不一定會回應 issue 與 PR。遇到問題，建議先請雲端前沿模型協助處理。