[DOCS] Running large MoE models under VRAM pressure: autosplit_reserve sizing, offload requirements, and worker tuning

[sorasoras]

Environment: Windows 11 + exllamav3 1.4.9 (cu12) - 2x RTX 3080 20 GB - Qwen3.8-Flash-Next-exl3_3.05bpw_h5_ng5, 256K context, cache_mode: Q4, cpu_moe_offload_layers: 26, 2x 1536 MB reserve

Documentation gaps that cost real debugging time when running a large MoE model that does not fit in VRAM. Three items, all from the same session.

1. autosplit_reserve must cover the KV cache, recurrent states and activations

config_sample.yml describes it as "Reserve VRAM used for autosplit loading (default: 96 MB on GPU 0)", which reads like a loading-time working buffer. In practice the KV cache is created after the load, so the reserve has to cover it plus the recurrent states plus activation headroom. Too small a reserve surfaces as a confusing late failure:

RuntimeError: Insufficient VRAM in split for model and cache

Measured non-weight VRAM at 256K context / cache_mode: Q4 / max_batch_size: 4 / MTP on:

Component	GiB
KV cache (Q4, 12 full-attention layers)	1.65
Recurrent states (36 gated-delta layers, fp32, max_history = draft_num_tokens)	2.26
CUDA contexts + allocator pools + activation scratch (measured, not modelled)	~4.5

Note the recurrent-state term scales with draft_num_tokens (5 slots per sequence at draft_num_tokens: 4) -- a setting that looks unrelated to VRAM. Worth stating in the docs, and worth cross-referencing turboderp-org/exllamav3#405 (EXL3 reserve allocations ignored for device IDs > 0, open): if only device 0's entry takes effect, a symmetric array like [1536, 1536] leaves other devices under-reserved and makes this much harder to reason about.

2. MoE CPU offload: requirements and silent fallbacks

The config comment currently says only "Number of mixture-of-expert layers to offload to CPU inference (default: 0) / Only affects MoE models. Set a large value such as 999 to offload all layers". Missing:

• It is experimental (exllamav3 labels both modes that way).
• It requires layer-split, i.e. tensor_parallel: false.
• Eligibility is checked per layer and failures are silent: experts need .mul1 codebook markers, K = trellis.shape[-1] // 16 must be <= 8, and no mixed per-expert biases. Ineligible layers fall back to GPU with only a warning, so 999 does not guarantee 999 layers moved.
• All offloaded layers must share uniform expert dims and top-k.
• How to size it: each offloaded layer frees exactly that layer's routed-expert bytes from VRAM and costs the same in system RAM. On this checkpoint that is 909.4 MiB per layer, so the value should be derived from the VRAM shortfall.

3. Undocumented worker tuning env vars, with non-obvious effects

Env var	Default	Measured effect
EXL3_MOE_CPU_SLOTS	4 (max 8)	4 -> 8 = +22% decode (20.3 -> 24.7 T/s)
EXL3_MOE_CPU_THREADS	cpu_count/2 (16 here)	28 = -50% (9.2 vs 17-18 T/s); do not oversubscribe with SMT

Both are worth a mention in the config comments or docs, since the defaults are not obviously suboptimal and the direction of the thread count effect is counterintuitive.