🚀 [v0.3.48] Release Note: Stateful MTP Speculative Decoding Arrives

[JamePeng]

llama-cpp-python v0.3.48

Stateful MTP Speculative Decoding Arrives

I’m pleased to introduce the first complete stateful MTP speculative decoding implementation in llama-cpp-python.

This release adds built-in and external MTP model support, integrates speculative decoding directly into the high-level Llama generation lifecycle, and introduces the checkpointing, rollback, verification, performance statistics, and benchmarking tools needed for practical use.

Highlights

Built-in and external MTP support

Qwen3.8-27B MTP Performance Test | llama-cpp-python 0.3.47 vs 0.3.48
https://www.youtube.com/watch?v=tWkql6jAkak

MTP can now run in two modes:

• Built-in MTP using the target model’s internal NextN heads.
• External MTP using a separate draft GGUF model and context.

When no external draft model is supplied, Llama automatically enables the target model’s MTP tensors.

The implementation has currently been tested with built-in and external MTP models from the following families:

• Qwen3.5
• Qwen3.6
• Qwen3.8

For Qwen3.8 27B, I recommend starting with draft_n_max=2. The optimal value still depends on GPU hardware, backend, quantization, prompt structure, sampling configuration, and workload, so users should run the included benchmark before choosing production settings.

Stateful speculative engine lifecycle

This release introduces:

• SpecConfig
• SpeculativeType
• LlamaSpecEngine
• create_spec_engine()
• create_native_spec_engine()

The new lifecycle covers:

begin → process → draft → target verification → accept/rollback

It supports target verification batches, acceptance feedback, checkpoint reuse, recurrent-state rollback, deterministic cleanup, and state reset between requests.

How to use it ? see : https://github.com/JamePeng/llama-cpp-python#mtp-speculative-decoding

Faster large-vocabulary MTP

Special attention was given to models with very large vocabularies, including Qwen3.8’s 248,320-token vocabulary.

This release:

• uses backend sampling where available;
• reads compact candidate buffers instead of transferring complete vocabulary logits;
• lazily allocates full-vocabulary CPU fallback buffers;
• removes an unused duplicate candidate buffer;
• bulk-copies mixed token embeddings and NextN hidden-state rows;
• reduces repeated Python and ctypes overhead.

These changes significantly reduce memory traffic and Python-side work during MTP drafting.

Recurrent snapshots and checkpoint optimization

MTP draft contexts can now use recurrent snapshots for efficient rollback.

The implementation:

• reuses draft-time checkpoints during target verification;
• removes only rejected suffixes when native rollback is available;
• keeps fallback checkpoint tensors on-device;
• avoids unnecessary host transfers and accepted-prefix replay;
• checks native memory removal and rollback results;
• fails immediately instead of continuing with corrupted state.

Correct verification and timing boundaries

Speculative verification batches are now atomic.

The wrapper no longer dynamically splits [last_verified_token, draft...] when the backend reports insufficient KV space. Instead, it raises a clear error and recommends increasing n_batch/n_ctx or reducing the draft length. Ordinary prompt evaluation still retains adaptive batch fallback.

Performance accounting is also divided into explicit phases:

• target decode submission;
• target synchronization and verification wait;
• speculative hidden-state processing;
• draft generation;
• acceptance and rollback;
• checkpoint capture and restore;
• TTFT, active generation speed, and sustained generation speed.

This prevents target GPU synchronization time from being incorrectly reported as speculative process() overhead.

Updated n-gram decoding and benchmarks

N-gram K and K4V decoding now use the same stateful speculative lifecycle.

MTP and n-gram tuning parameters are separated:

• MTP uses draft_n_max.
• N-gram K/K4V use ngram_size_m.

The benchmark can independently scan n-gram configurations such as:

N = 6, 8, 10, 12
M = 8, 16, 32, 48

The Python K4V continuation limit now defaults to four entries per key, matching llama.cpp.

The benchmark supports ordinary decoding, n-gram K/K4V, built-in MTP, and external MTP, with portable command-line configuration and CSV results.

Important Multimodal Limitation

MTP speculative decoding in v0.3.48 is currently text-only.

It does not yet support multimodal prompts containing image, audio, or other MTMD embedding segments. In particular, users should not enable the new MTP path for vision or omni workloads and assume the media embeddings are synchronized into the MTP draft context.

This limitation applies to both built-in and external MTP modes.

The underlying llama.cpp infrastructure has started adding multimodal speculative hooks, and DFlash/DSpark already contain some embedding-batch handling, but the MTP implementation still requires additional work to preserve media positions, hidden-state boundaries, draft-context state, caching, and rollback correctly.

For now:

• use MTP only for text generation;
• continue using ordinary decoding for multimodal MTP models;
• do not pass negative MTMD pseudo-token IDs through the text-only speculative path.

Breaking Changes

• LlamaPromptLookupDecoding has been removed.
• New code should use SpecConfig and SpeculativeType.
• The old string-based n-gram mode argument is no longer supported.
• The legacy stateless draft_model path remains temporarily available for compatibility but is deprecated.

Additional Fixes

This release also includes:

• a Python 3.9 forward-reference import fix;
• safe MTP cleanup during interpreter shutdown;
• corrected MTMD ctypes pointer bindings;
• minimal MTMD binding import coverage;
• synchronized llama, MTMD, and GGML API bindings;
• an updated llama.cpp vendor revision;
• rewritten README, core Llama, and speculative decoding documentation.

What’s Next

In the next version, I will continue adapting the remaining speculative decoding implementations exposed by llama.cpp, including work around:

• Eagle3
• DFlash
• DSpark
• additional draft and self-speculative strategies
• multimodal speculative decoding
• cache and rollback efficiency
• reduced synchronization and data-transfer overhead

MTP support will also continue to receive compatibility, performance, and multimodal integration improvements based on real-world testing and community feedback.

Thank you to everyone willing to test built-in and external MTP models, compare benchmark results, and report model-specific behavior. Enjoy it :）

[View the complete changes](4854c7d...85913ab)

— JamePeng

[allanmeng]

今天做这个版本的预编译 wheel（llama-cpp-python 0.3.48，仓库：allanmeng/llama-cpp-python-sycl-windows），发现这样一个问题。这只是复现报告——不是报你 fork 的 bug，根因看起来在上游 ggml-org/llama.cpp 的 Python 包装层，我只是在你的 0.3.48 构建上复现了它。

现象

在 hybrid 视觉模型上（Qwen3.5-4B + mmproj，带 SWA 层），用大图（约 2336×1760，约 4015 个 vision tokens）：

• prefill 正常
• 首 decode token 崩溃：
  • failed to prepare attention ubatches
  • failed to find a memory slot for batch of size 1

根因（我追踪到的）

调用方传 ctx_checkpoints=0 会让 max_checkpoints<=0，从而强制 hybrid 模型走 llama.py 里的 Bypassing rollback fast-path。在大 prefill 下，这条路径没有给首个生成 token 留 KV / 解码槽 → 崩溃。

这个 fast-path（max_checkpoints<=0 分支）在 0.3.47 和 0.3.48 都存在——来自同步的 ggml-org/llama.cpp 子模块（0.3.48 是 bb4caa754）。它只是在 0.3.48 因为 handler 重写 + 子模块 bump 改变了命中它的参数组合，才稳定触发。

规避（已验证）

用默认 ctx_checkpoints=-1（内部映射为 16，启用 checkpoint 缓存 → 避开缺陷分支）。用 -1 时，同模型同大图在 n_ctx=8192 下干净跑通：约 82 t/s，hybrid checkpoint 的 save/restore 正常。

我的 ComfyUI 插件（comfyui-sg-llama-cpp）已在 1f0fc15 把默认值 0 → -1，我这边已解决。

wheel 本身没问题

纯 llama_cpp.Llama 在同模型同大图 n_ctx=8192 下正常（双重验证）。崩溃只在 ctx_checkpoints=0 这种调用姿势下出现。所以 SYCL 构建本身没毛病——只是把这个上游 fast-path 的行为同步给你，方便你下次同步 llama.cpp 时 tracking / 做 workaround。

复现环境

项目	版本 / 配置
OS	Windows 11
GPU	Intel Arc B580 (Battlemage)
Python	3.13.11
Intel oneAPI	2026.1 (SYCL, icx, INTEL target)
llama-cpp-python	0.3.48（你的 fork，release commit 7562297，tag v0.3.48-cu128-win-20260821）
llama.cpp 子模块	bb4caa754（llama.cpp 0.2.0）
测试模型	Qwen3.5-4B-Uncensored + mmproj-BF16（hybrid，带 SWA 层）
测试图像	2336×1760（约 4015 vision tokens）
参数	ctx_checkpoints=0，n_ctx=8192

[JamePeng]

你可以考虑把checkpoint_on_device设置为True，ctx_checkpoints数量设置为4-8左右，也不会影响gpu的拷贝吞吐。

[allanmeng]

根据 CTX 和 Device 我做了四个组对比

是不是我这种 VLM 图片反推场景对于显存压力太小了？ 感觉差异不大。

四组对比表

[JamePeng]

如果只是单次单轮反推的话，差别确实不大。如果保持模型加载，多轮单次的话，还是会有区别，尤其是词表越大的模型，缓存也越大，会消耗来回存储的带宽

[allanmeng]

嗯。感觉视频脚本的分镜头图片提示词的反推+扩写能用到。