🕵️Community Testing Request: MTP Speculative Decoding Feedback🤗

[JamePeng]

MTP speculative decoding community testing

Text-only MTP speculative decoding is now available in llama-cpp-python, with support for both:

• Built-in MTP/NextN heads included in the target GGUF
• External MTP draft GGUF models

The current implementation has primarily been tested with the Qwen3.5, Qwen3.6, and Qwen3.8 model families. Testing on additional hardware, backends, quantizations, and workloads is needed before the feature can be considered broadly validated.

Multimodal support is not available yet. Cache handling, rollback behavior, and decoding-efficiency optimizations are still in progress.

Feedback requested

Community testing would be especially helpful for:

• Built-in versus external MTP performance
• Different draft_n_max values
• CUDA, CPU, Vulkan, ROCm, SYCL, Metal, and other backends
• Different model sizes and quantization formats
• Large-vocabulary models
• Hybrid or recurrent models
• Long generations and repeated requests
• Acceptance rates, rollback frequency, and checkpoint overhead
• Output correctness or unexpected divergence
• Model loading, shutdown, memory usage, or runtime errors

For Qwen3.8 27B, draft_n_max=2 has performed well in current testing and is the recommended starting point. However, the optimal value can vary significantly with the GPU, backend, quantization, prompt, sampling configuration, and whether MTP is built in or external. Please benchmark multiple values when possible.

Suggested benchmark

The included benchmark can compare ordinary decoding, built-in MTP, and external MTP:

python -m examples.high_level_api.high_level_api_mtp_speculative -h

A broader speculative decoding benchmark is also available:

python -m examples.benchmark.benchmark_speculative -h

For reliable results, please use the same prompt and sampling settings for each mode, perform a warmup, and run multiple measured iterations.

Feedback template

Please include as much of the following information as possible:

llama-cpp-python version or commit:
llama.cpp vendor commit:
Operating system:
Python version:
CPU:
GPU:
Backend and build options:

Target model:
Target quantization:
MTP mode: built-in / external
External MTP model, if applicable:
MTP quantization:

draft_n_max:
draft_p_min:
n_ctx:
n_batch:
GPU layers:
Sampling settings:

Ordinary decoding speed:
MTP decoding speed:
Speedup:
TTFT:
Draft tokens proposed:
Draft tokens accepted:
Acceptance rate:
Mean accepted length:
Rollback count:
Checkpoint mode and timing:

Output match or first divergence:
Peak RAM/VRAM usage:
Warnings or errors:
Additional observations:

Logs containing the final Llama.generate: draft-mtp summary are particularly useful. Please remove private prompts, model paths, usernames, or other sensitive information before posting.

Positive results are welcome, but slower results, crashes, unsupported model reports, and unusual rollback behavior are equally valuable. They help identify where the implementation needs compatibility fixes or further optimization.

Thank you to everyone willing to test and share results.

— JamePeng

[craftingmod]

llama-cpp-python version or commit: c7693b48df57ec5f9a970dded2e599b956727421 (0.3.48-dev)
llama.cpp vendor commit: bb4caa7540188872173c44d161602d9271386413
Operating system: Windows 11 26H2 (26300.9168)
Python version: 3.13.12
CPU: Intel i9-13900K
GPU: NVIDIA RTX 3090 ti
Backend and build options: CUDA 13.2 (SM86)

Target model: Qwen3.8-27B-Q4_K_M.gguf (unsloth)
Target quantization: Q4_K_M
MTP mode: built-in
MTP quantization: Unknown (maybe Q4_0)

------
Result of `python -m examples.benchmark.benchmark_speculative --model "Qwen3.8-27B-Q4_K_M.gguf" --methods builtin-mtp --mtp-draft-tokens 2`

llama-cpp-python speculative decoding benchmark
  target model : Qwen3.8-27B-Q4_K_M.gguf
  methods      : ordinary, builtin-mtp
  cases        : 2 total
  MTP draft    : M=2
  scenarios    : crud, jsonl, table, report, essay, reasoning
  workload     : 2 x 256 tokens per scenario
  timing       : TTFT includes prompt eval; sustained excludes first token

[ordinary]
  model loaded in 5.849 s
  crud       run 1  | TTFT    58.15 ms | sustained    43.72 tok/s
  crud       run 2  | TTFT    60.34 ms | sustained    43.99 tok/s
  jsonl      run 1  | TTFT   140.64 ms | sustained    43.94 tok/s
  jsonl      run 2  | TTFT   135.65 ms | sustained    43.90 tok/s
  table      run 1  | TTFT    59.38 ms | sustained    43.90 tok/s
  table      run 2  | TTFT    62.86 ms | sustained    43.76 tok/s
  report     run 1  | TTFT    56.02 ms | sustained    44.65 tok/s
  report     run 2  | TTFT    55.52 ms | sustained    43.71 tok/s
  essay      run 1  | TTFT    58.53 ms | sustained    43.93 tok/s
  essay      run 2  | TTFT    58.91 ms | sustained    43.65 tok/s
  reasoning  run 1  | TTFT    57.99 ms | sustained    43.49 tok/s
  reasoning  run 2  | TTFT    55.21 ms | sustained    43.77 tok/s

[built-in MTP (M=2)]
  model loaded in 5.968 s
  crud       run 1  | TTFT   101.50 ms | sustained    71.42 tok/s | accept 78.5% | output exact
  crud       run 2  | TTFT   111.60 ms | sustained    72.20 tok/s | accept 78.5% | output exact
  jsonl      run 1  | TTFT   193.53 ms | sustained    79.30 tok/s | accept 92.2% | output diverged at token 48
  jsonl      run 2  | TTFT   178.44 ms | sustained    79.04 tok/s | accept 92.2% | output diverged at token 48
  table      run 1  | TTFT   113.34 ms | sustained    72.20 tok/s | accept 78.5% | output diverged at token 103
  table      run 2  | TTFT   110.19 ms | sustained    71.45 tok/s | accept 78.5% | output diverged at token 103
  report     run 1  | TTFT   106.39 ms | sustained    71.32 tok/s | accept 78.0% | output diverged at token 25
  report     run 2  | TTFT   106.38 ms | sustained    72.15 tok/s | accept 78.0% | output diverged at token 25
  essay      run 1  | TTFT   106.91 ms | sustained    66.62 tok/s | accept 69.6% | output diverged at token 22
  essay      run 2  | TTFT   100.75 ms | sustained    66.73 tok/s | accept 69.6% | output diverged at token 22
  reasoning  run 1  | TTFT   107.45 ms | sustained    66.06 tok/s | accept 67.9% | output exact
  reasoning  run 2  | TTFT   107.24 ms | sustained    66.16 tok/s | accept 67.9% | output exact

[summary]
  crud: CRUD boilerplate
    configuration              mean tok/s        min        max    speedup       output
    ordinary                        43.85      43.72      43.99     1.000x    reference
    builtin-mtp-m2                  71.81      71.42      72.20     1.637x        exact

  jsonl: Repetitive JSONL records
    configuration              mean tok/s        min        max    speedup       output
    ordinary                        43.92      43.90      43.94     1.000x    reference
    builtin-mtp-m2                  79.17      79.04      79.30     1.803x       div@48

  table: Markdown course table
    configuration              mean tok/s        min        max    speedup       output
    ordinary                        43.83      43.76      43.90     1.000x    reference
    builtin-mtp-m2                  71.83      71.45      72.20     1.639x      div@103

  report: Structured market report
    configuration              mean tok/s        min        max    speedup       output
    ordinary                        44.18      43.71      44.65     1.000x    reference
    builtin-mtp-m2                  71.73      71.32      72.15     1.624x       div@25

  essay: Low-repetition historical essay
    configuration              mean tok/s        min        max    speedup       output
    ordinary                        43.79      43.65      43.93     1.000x    reference
    builtin-mtp-m2                  66.68      66.62      66.73     1.523x       div@22

  reasoning: Quantitative-finance explanation
    configuration              mean tok/s        min        max    speedup       output
    ordinary                        43.63      43.49      43.77     1.000x    reference
    builtin-mtp-m2                  66.11      66.06      66.16     1.515x        exact

Output match or first divergence:
Peak RAM/VRAM usage: 16GB(RAM) / 16GB(VRAM)
Warnings or errors: none

My take 👀