FreeToken loads HF safetensors checkpoints directly. The checkpoints below are known-good — the prebuilt kernels are tuned for them; other checkpoints of the same architectures work too.
These families accept image input by default; pass --text-model-only to skip the vision encoder. The flags are described in the
CLI reference; each family reads them in its own units.
| Family | Image tokens | --image-min-tokens / --image-max-tokens |
--mm-processor-kwargs example |
|---|---|---|---|
DeepSeek-V4.1-Flash (ViT tower, streamed under --mm-encoder-weights host) |
one feature per 42x42 pixels, plus a learned row break per row and two delimiters; checkpoint cap 1024 total span tokens | maximum caps the entire span; minimum maps to a pixel-area floor before the cap; checkpoint minimum area is 544x544 | {"max_image_tokens": 512, "min_pixels": 295936} |
| Qwen3.6 (both variants, every listed weight format), Qwen3.8-Flash-Next, Qwen3-VL | one token per 32x32 pixels of the resized image, dynamic resolution | pixel areas in size.shortest_edge / longest_edge; checkpoint defaults 64 to 16384 tokens |
{"size": {"longest_edge": 1048576}} |
Gemma-4 26B-A4B, 31B (gemma4: ViT tower, streamed under --mm-encoder-weights host) |
one of the soft-token budgets 70 / 140 / 280 / 560 / 1120, every image scaled to its budget as far as the aspect ratio allows | the maximum picks the largest budget within it, below 70 is refused at start-up; the minimum has no effect | {"max_soft_tokens": 1120} |
Gemma-4 12B (gemma4_unified: linear patch embedder, resident under either placement) |
same budgets, one 48x48 super-patch per soft token | same as the tower releases | same |
GLM-5.3-Flash (glm5_next: ViT tower, streamed under --mm-encoder-weights host) |
one token per 28x28 pixels of the resized image, dynamic resolution on a canvas zero-padded to a 28-multiple | token counts, passed through as the processor's min_image_tokens / max_image_tokens; checkpoint defaults 16 to 8000 tokens |
{"max_image_tokens": 2048} |
Muse-Glimmer-30B (windowed ViT tower, streamed under --mm-encoder-weights host) |
one token per 28x28 pixels of the resized image, aspect ratio kept under a token cap; checkpoint default 4096 tokens | the maximum is the cap (max_image_tokens); the minimum has no effect |
{"max_image_tokens": 1024} |
MiniMax-M3 (minimax_m3: CLIP-style ViT tower, streamed under --mm-encoder-weights host) |
one token per 28x28 pixels of the resized image, dynamic resolution | pixel areas in size.shortest_edge / longest_edge; checkpoint defaults 4 to 576 tokens |
{"size": {"longest_edge": 1048576}} |
DeepSeek-V4-Flash-Vision (deepseek_v4: native-resolution ViT tower, streamed under --mm-encoder-weights host) |
one token per 3x3 grid of 14x14 patches, against a per-image cap; the block carries sentinel tokens and lead pads that align it to a 4-token compression stride, so its length depends on where it lands in the prompt | the maximum is a cap on the block (vision_max_n_token, checkpoint default 384) and lowers it, the minimum is read as the checkpoint's upscale floor and has no effect |
not read |
ft serve --moe-strategy {auto,fused,offload,cpu,hybrid} (--moe-backend is the deprecated old spelling):
- fused — experts resident on GPU (needs the VRAM). Serves every expert format except DeepSeek-V4's.
- offload — experts live in host RAM, an LRU cache of expert slots on GPU; misses stream over PCIe.
- cpu — misses are computed on the CPU instead of fetched.
- hybrid — per step, fetches some misses over PCIe and computes the rest on
CPU, overlapped. Run
ft bench bw --model <checkpoint>once per machine and model to calibrate the split. - auto — dense models always resolve to
fused; MoE models resolve tooffload. Unified-memory GPUs differ, see below.
The GPU and the CPU share one DRAM, so offload only copies experts between two
names for the same memory. On these GPUs auto resolves every MoE model to
fused: the default is the resident path only. DeepSeek-V4 experts
have no resident path and stay on offload. Offload-only flags
(--moe-cpu-layers, --moe-cache-size / -rate / -auto,
--moe-prefill-hit-d2d, --disable-moe-prefill-overlap) are ignored with a
warning; pass --moe-strategy offload to use them.
WSL caps CUDA pinned memory near half of RAM, so the expert banks of a large
model may not all pin. With --moe-cpu-layers auto, FreeToken loads the banks
first, then pins them layer by layer from the middle out until CUDA refuses one;
the remaining head and tail layers are OS-locked instead (resident, but with no
device address). After a refusal under WDDM (WSL and native Windows) it
unpins about 3 GiB more (FREETOKEN_PIN_PROBE_MARGIN_GB) so later CUDA
allocations still fit.
FREETOKEN_PIN_BUDGET_GB replaces the probe with a fixed budget: the locked
layers are then picked from a size estimate before the load, and a refused pin
stops the boot.
Under hybrid, a locked layer keeps GPU slots of its own pools. Each decode
step it fetches a share of its misses (--moe-locked-fetch, by default from the ft bench bw profile): CPU threads
copy the experts into a small pinned staging buffer
(--moe-locked-max-fetch rows per bank) and the GPU copies them into the
slots. The CPU computes the remaining misses, as on any hybrid layer. Under
offload, locked layers decode on the CPU.
The host link is ~450 GB/s per direction instead of PCIe's ~32-64 GB/s. The
offload expert gather reads host memory zero-copy and is latency-bound, so it
needs a wider grid to keep enough loads in flight. Set
FREETOKEN_H2D_BLOCKS_PER_BANK:
FREETOKEN_H2D_BLOCKS_PER_BANK=32 ft serve --model <model> --moe-strategy offload
On GH200 this raises the gather from ~220 to ~410 GB/s (PCIe-gather in
ft bench bw), matching the DMA ceiling. Run ft bench bw with the variable
set so the hybrid split is calibrated against the faster gather. Leave it
unset on PCIe GPUs: wider grids add no bandwidth there.
- A checkpoint that fails to load: from a source checkout, run
PYTHONPATH=python python scripts/checkpoint_diagnose/diagnose.py <checkpoint dir or HF repo id>. It reads the headers, not the weights, says whether FreeToken or the checkpoint is at fault, and for a local checkpoint gives the commands that fix a non-standard one; see its README. GGUF files are checked with scripts/gguf_check. ft checkpointconversion is optional — it pre-converts a checkpoint into FreeToken's fast-load format, andft serve --modelauto-detects the result.- An FTW converted by an older build with
--moe-backend fusedortritonkeeps its experts as dense weights and no longer loads; reconvert it withft checkpoint. - FTW files converted by builds before the quantization refactor may fail to load; see ftw-hotfix.md for the affected checkpoints and the repair tool.
- Muse-Glimmer keeps the attention gate as its own linear next to
qkv_proj; an FTW converted when the gate was fused intoqkvg_projno longer loads, reconvert it withft checkpoint. - NVFP4 global scales are kept in fp32. An NVFP4 FTW that stores them as fp16 (converted by an
older build) is refused at load; reconvert it with
ft checkpoint. - Qwen3.5 / Qwen3.6 / Qwen3.8-27B load their zero-centred norm weights as stored and add the 1 in
fp32 at runtime. An FTW of these models converted by an older build stores the norms with the 1
already added: it loads without an error but gives wrong output; reconvert it with
ft checkpoint. - An FTW converted before its family served images holds no vision encoder:
ft serverefuses it unless started with--text-model-only(or--mm-disable vision); reconvert it withft checkpoint, or add the encoder in place with scripts/ftw_hotfix.py. - DeepSeek-V4 checkpoints must keep the
inference/config.jsonsubdir — the authoritative model args are read from there. - DeepSeek-V4.1-Flash serves text and images from the HF checkpoint as shipped (fp8 block-32
dense weights, fp4 experts on the offload cache, 890 B/token global KV in packed
fp4/fp8 pools). Its two 98 GiB Engram tables stream from the checkpoint shards on
demand (keep them on a fast NVMe; the 6 GiB of table scales stay in host RAM), using
bounded pinned staging buffers; an FTW conversion copies the shards that hold them next
to the checkpoint.
SWA bounded replay (DeepSeek_V41_Tech_Report.pdf, shipped in the checkpoint, §3.2.2): a replayed
segment recomputes only the SWA KV of its last
n_wintokens and truncates each query's window to the segment. Encoder replay rebuilds the encoder SWA KV behind a prefix hit from the global KV alone (approximate; the report's fallback when the SWA KV of a hit has been evicted). Decoder replay runs the decoder layers on each prompt's lastn_wintokens only; their SWA KV is never prefix-cached and post-training simulated it. FreeToken keeps encoder SWA KV in the radix cache (no encoder replay);--swa-decoder-replay bounded(default) is the report's decoder replay, with a prefix hit stopping at leastn_wintokens before the prompt end so those tokens are prefilled;exactruns the decoder on every prompt token (reference numerics). Bounded output differs from exact by construction and does not depend on the prefill chunk size. Under expert offload the prefill time is bounded by streaming each layer's experts, so decoder replay saves decoder-layer compute, not prefill time. Runft bench bw --model <checkpoint>to measure local PCIe and CPU bandwidth before choosing a MoE strategy;--moe-strategy hybridsplits expert misses using those measurements. With--moe-cache-auto,--kv-reserve-tokenssets the token capacity reserved before the expert cache is allocated, subject to the attention pool's structural minimum; size it for the intended context and concurrency.--max-extend-lengthcontrols the prefill chunk size and its workspace. Image input uses the shared multimodal processor and encoder cache, including chunked prefill and prefix replay;--text-model-onlyskips the vision weights. DSpark speculative decoding is not served.reasoning_efforttakeslow,highormax; the 1-100 integer budget goes through the template kwargs with thinking on,"chat_template_kwargs": {"enable_thinking": true, "reasoning_effort": 37}, and the checkpoint's encoder validates it. Runtime window-cache controls use 128-token pages. - Qwen3.8-Flash-Next keeps a 47.7 GiB PLE n-gram table pinned in host RAM.