NInfer

한국어 안내: README.ko.md

This fork runs Qwen3.8-27B on two 16 GB GeForce RTX 5060 Ti cards without P2P, with two-GPU tensor parallelism (--tp 2 --devices 0,1). It combines:

Neroued/ninfer at d44ab584 (the engine);
this fork's two-GPU layer: split weights and collectives over a pinned-host mailbox, NVFP4/FP8 rank shards, Vision, the hybrid prefix cache, ngram copy drafting and DFlash2 at --tp 2;
Wallawalla47/ninfer-custom, merged whole, with its later fixes cherry-picked; its inherited notes below describe that fork's single-GPU setup, not this branch's TP2 recommendation;
ideas and the shard-compilation technique of ValerioDolci/ninfer-tp2.

Everything this fork adds was written by Claude Opus 5.5 in Claude Code; a human operator set the goals, approved production changes and ran the server. Usage, the recommended configuration, what the branch changes and every measurement: Two-GPU tensor parallelism on RTX 5060 Ti.

Measured on 2026-10-23 with the recommended configuration (QUASAR-QAT all-NVFP4 artifact, MTP with four drafts and n-gram copy drafting, fp8 KV, Vision, four concurrent requests): a 10K-token prompt at 7,007 tok/s (first token 1.46 s), code 128-144 tok/s and Korean 80-84 tok/s, 337 tok/s aggregate over four requests, 315,008 KV tokens; GSM8K 97/100, MMLU-Pro 121/140, HumanEval 78/82 and KMMLU 76/100. The same machine's comparison with Basalt, including power, temperatures and memory: Basalt vs Ninfer (Korean).

Merged: Wallawalla47/ninfer-custom notes

AI disclaimer: Everything added to this fork, including most of this README, was written with AI (mostly Claude Opus 5.5, Qwen3.8-27B running on NInfer, plus a few other AI systems I’ve been testing). It is likely to be neither complete nor entirely accurate. This is hobby development.

This is a personal fork of Neroued/ninfer. It follows upstream closely and adds changes on top: the history is upstream master (d44ab584), the Windows port, then one commit per fork change. The sections below explain what is different, grouped by topic, with credit given as best my AI agents can where a change came from someone else. The upstream README follows, copied unchanged, under the "Upstream README" heading. A huge thank you to Neroued for creating NInfer!

The short version. Compared with upstream, this fork:

builds and runs natively on Windows (and still on Linux)
uses a new prefix caching system, designed and implemented by Claude Opus 5.5, as the default. You set how much system RAM it may use with --host-cache-mib N; --prefix-cache-file PATH keeps the cache across restarts. On the agentic benchmark below it serves far more of each prompt from cache than upstream's system. Stop the server with Ctrl+C (twice) rather than by closing the window, because Windows does not always leave enough time after a window closes to save a large cache
keeps upstream's original prefix caching, with a raft of fixes, behind --use-original-prefix-caching (I worked on it before switching to a new design; I found the original too complex and fragile)
prefills INT8 and NVFP4 KV with faster prompt-attention kernels by default (a third to two thirds less prompt-attention time on long prompts); --use-original-int8-prefill-kernel and --use-original-nvfp4-prefill-kernel select upstream's kernels
adds ngram copy drafting (based on an implementation by remesis), which greatly speeds up copy-heavy workloads, with more than one concurrent request
overlaps each decode kernel's launch and weight loading with the kernel before it, and tunes the decode-width kernels, for faster speculative decode rounds with the same output
can offload the vision encoder to system RAM (--vision-offload on), based on the work of Valeriy Selitskiy (iamwavecut)
supports YaRN context extension up to 1M tokens (--rope-yarn-factor F, F from 1 to 4)
sizes the CUDA Graph memory allowance from measurement instead of an estimate that could reserve far more VRAM than ever used
lets --vram-headroom-mib N shrink the 1 GiB VRAM headroom left after --kv-capacity auto
adds a custom thinking budget message (--default-thinking-budget N with --thinking-budget-message "...")
accepts more tool-call formats and API options used by agent clients such as Claude Code, Qwen Code, Codex, Zed and GitHub Copilot, and fixes several tool-call and reasoning-output issues (mostly based on the work of others credited below); --tolerant-tool-calls recovers some broken tool calls
improves the console: optional colours (--log-colours on), a statistics panel at the bottom (--log-stats-panel off removes it) and a --help screen organised by category
contains various other fixes and improvements, including upstream pull requests merged before upstream did

I recommend using this with the NVIDIA NVFP4 artifact I’ve uploaded here, which runs a bit faster than the original artifact based on the Unsloth quant and takes up less VRAM: https://huggingface.co/wallawalla47/Qwen3.8-27B-NVIDIA-NVFP4-NInferV3

Quick start (Windows)

Prerequisites: Visual Studio 2026 (MSVC), the CUDA 13 toolkit, and FFmpeg + curl from vcpkg (x64-windows, at C:\vcpkg); adjust the paths at the top of build_native.bat for your machine.

build_native.bat configure
build_native.bat build

The server is build-windows\apps\Release\ninfer-serve.exe, with the FFmpeg, curl and zlib DLLs copied next to it. The launch I use on a single 32 GB RTX 5090 (stop any other resident model first):

ninfer-serve.exe qwen3_8_27b_nvfp4-nvidia.ninfer --host 127.0.0.1 --port 8080 --max-context 240000 --max-concurrency 2 --spec dflash2 --draft-tokens 7 --lm-head-draft --ngram-draft-tokens 15 --ngram-min-match 12 --kv-dtype int8 --preserve-thinking --host-cache-mib 52000 --pending-timeout-ms 900000 --prefill-chunk 4096 --kv-capacity auto --vram-headroom-mib 0 --log-colours on --ngram-archive-mib 2048 --ngram-session-mib 256 --ngram-native-sessions --request-log-jsonl log.json --default-thinking-budget 16384 --thinking-budget-message "Considering the limited time available to the user, I must stop thinking now. Time to act:" --tolerant-tool-calls

Add --prefix-cache-file PATH to keep the prefix cache across restarts. Stop the server by pressing Ctrl+C twice (the first press shows a prompt at the bottom of the console). The server then cancels running and queued requests, saves the cache and exits; pressing Ctrl+C once more exits without saving and deletes the unfinished file. ninfer-serve.exe --help lists every option by category.

Running on another PC needs an RTX 50-series GPU (the build targets sm_120a) and an NVIDIA driver of 580 or later (CUDA 13); no CUDA toolkit is needed. Copy the DLLs next to ninfer-serve.exe and install the Visual C++ redistributable if it is missing.

Quick start (Linux)

The fork builds and runs on 64-bit Linux too, including WSL2 (tested on Ubuntu 24.04 under WSL2 with CUDA 13.4 and GCC 13.3). Prerequisites: a CUDA 13 toolkit, CMake 3.28 or newer, a C++20 compiler, Ninja, pkg-config, and the FFmpeg and curl development packages. On Ubuntu 24.04:

sudo apt-get install -y build-essential cmake ninja-build pkg-config libavformat-dev libavcodec-dev libavutil-dev libswscale-dev libcurl4-openssl-dev
cmake --preset release
cmake --build build -j

The build also needs the CUDA NVTX headers (nvtx3/nvToolsExt.h). The full cuda-toolkit-13-4 package includes them; a toolkit installed component by component may not, and the build then stops at src/core/nvtx.h. Add cuda-nvtx-13-4 (the same minor version as the toolkit) in that case. The tp2-5060ti branch also builds natively on Ubuntu 26.04.1 with GCC 15.2 and CUDA 13.4.

The server is build/apps/ninfer-serve and takes the same options as on Windows, for example the launch line above with ./build/apps/ninfer-serve in place of ninfer-serve.exe.

Under WSL2 the GPU driver is the Windows NVIDIA driver (580 or later); do not install a Linux NVIDIA driver inside WSL. NVIDIA's wsl-ubuntu CUDA repository stops at CUDA 13.3, so for 13.4 add its ubuntu2404 repository and install only cuda-toolkit-13-4 (not cuda or cuda-drivers, which pull in a driver). Artifacts on a Windows drive load slowly through /mnt/, so copy the .ninfer file into the Linux filesystem first.

Performance: this fork vs upstream

Both benchmarks below compare this fork (the commits before this README; the measured build fe869b56 has the same source) with upstream + Windows port: upstream at the commit this fork is rebased on (d44ab584) plus only the Windows port (commit d0abd0cb on the branch upstream-Windows-Port). Everything ran on an RTX 5090 under Windows with the official Qwen3.8-27B NVFP4 artifact (qwen3_8_27b_nvfp4-official.ninfer), in September 2026.

Agentic coding workload

The closed-loop suite in bench/agentic_ab/ replays three coding-agent sessions plus eleven subagents: 130 requests with fan-outs, a concurrent subagent pair, compaction, retries, an abort and a solo wrap-up, with prompts of 25K-135K tokens and thinking on. Each arm's own answers are fed back as an agent client does, and the three main sessions take their turns in lock-step rounds, so every build meets the same order of session turns whatever its speed. Both builds completed every request on each of three workload seeds (42, 43, 44), which replay different observations.

Settings:

Fork arm: the launch flags of LaunchQwen3.8-27B-official-dflash2-ngram.bat:

--max-context 170000 --max-concurrency 2 --spec dflash2 --draft-tokens 7 --lm-head-draft
--ngram-draft-tokens 15 --ngram-min-match 12 --kv-dtype int8 --preserve-thinking
--host-cache-mib 52000 --pending-timeout-ms 900000 --prefill-chunk 4096 --kv-capacity auto
--vram-headroom-mib 0 --ngram-archive-mib 2048 --ngram-session-mib 256 --ngram-native-sessions
--default-thinking-budget 16384 --thinking-budget-message "Considering the limited time
available to the user, I must stop thinking now. Time to act:" --tolerant-tool-calls

Upstream arm: the same flags minus those upstream does not have, with the host RAM split the fork's original cache resolves from the same 52,000 MiB passed as explicit flags:

--max-context 170000 --max-concurrency 2 --spec dflash2 --draft-tokens 7 --lm-head-draft
--kv-dtype int8 --preserve-thinking --pending-timeout-ms 900000 --prefill-chunk 4096
--kv-capacity auto --default-thinking-budget 16384 --host-state-slots 123 --host-kv-mib 29020
--max-private-continuations 4 --max-long-anchors-per-continuation 27 --max-shared-prefixes 7

Context: 170,000 tokens, the largest context the upstream build starts with under these flags; at it the fork's device KV holds 227,648 tokens and upstream's 175,424. Sampling: temperature 1.0, top_p 0.95, top_k 20, max_tokens: 64000 on agent turns.

Each cell is the mean over the three seeds, with the lowest and highest seed in brackets; changes are computed per seed against that seed's upstream run.

Metric	Upstream + Windows port	Fork
Average time to first token (s)	14.0 (11.6-15.6)	2.7 (2.4-3.1), −80.5 %
Median time to first token (s)	8.72 (8.20-9.68)	0.63 (0.47-0.76), −92.7 %
90th-percentile time to first token (s)	36.6 (24.0-43.8)	6.7 (5.5-8.0), −80.5 %
Average TTFT, continuing-session turns (s)	13.97 (12.03-15.51)	2.57 (2.31-3.08), −81.6 %
Average TTFT, new long prompts (s)	10.5 (10.4-10.7)	8.3 (6.4-11.6), −21.6 %
Prompt tokens served from cache	69.2 % (66.7-73.5)	91.1 % (90.9-91.2)
Prompt tokens prefilled	1.83M (1.61-1.98)	0.51M (0.49-0.54), −71.7 %
Main-session turns that re-prefilled the whole prompt (of 75)	11.7 (9.0-15.0)	0
Subagent turns that re-prefilled the whole prompt (of 37)	23.7 (17.0-31.0)	0
Prefill tok/s, requests with no cache hit in any arm	6,085 (6,018-6,151)	8,389 (8,380-8,407), +37.9 %
Prefill tok/s, the same requests from 32K tokens	5,982 (5,913-6,051)	8,291 (8,281-8,308), +38.6 %
Output tok/s, one request decoding	197 (193-205)	214 (204-219), +8.8 %
Decode rounds/s, one request decoding (engine speed)	58.4 (56.1-59.7)	60.2 (59.2-61.8), +3.2 %
Tokens per round, one request decoding (acceptance)	3.38 (3.24-3.45)	3.56 (3.30-3.71)
Output tok/s, two requests decoding (combined)	360 (349-368)	355 (343-378), −1.3 %
Decode rounds/s, two requests decoding (engine speed)	57.3 (55.0-58.6)	54.7 (53.7-55.5), −4.3 %
Decode rounds that ran two requests	9.8 % (5.8-12.3)	35.4 % (30.6-38.4)
Output tok/s, all decoding at the run's own batching	214 (204-227)	270 (264-282), +26.3 %
Workload wall time (min)	20.7 (18.3-22.8)	12.0 (10.0-14.6), −41.7 %

Time to first token includes queueing: up to seven requests are in flight on two lanes. The average queue wait was 11.7 s upstream and 2.2 s on the fork, and without it the average time to first token was 2.31 s and 0.58 s.
Output tok/s counts decode tokens per second of the engine's own decode time. It splits into decode rounds/s (engine speed) and tokens per round (speculative acceptance, which moves with what the model happened to write). The fork's ngram drafting supplied 6-13 % of its output; upstream has none.
With two requests decoding, the fork's rounds also carry ngram drafting for both, which costs host time per round, and upstream decoded two requests together for only 15-70 s per seed. The concurrent decode benchmark below compares two-request decode without ngram drafting.

Concurrent decode

The decode-saturation suite of tools/bench/run_serve_concurrency.py starts a fresh server at each --max-concurrency C and decodes C requests at once, each up to 8,192 tokens, with DFlash2 K=7 and --lm-head-draft (no ngram drafting), stochastic sampling, --max-context 32768 --kv-capacity auto. The builds alternated point by point in two passes, C=1 to 8 and back.

C	Upstream tok/s	Fork tok/s	Fork time per decode round vs upstream (pass 1, pass 2)
1	198.3	196.6	−2.3 %, −2.4 %
2	365.8	380.0	−2.2 %, −2.3 %
3	516.1	533.9	−3.1 %, −3.1 %
4	649.6	677.4	−3.2 %, −3.2 %
5	748.6	789.0	−3.5 %, −3.5 %
6	868.7	879.7	−1.8 %, −1.7 %
7	965.7	1002.6	−1.4 %, −1.3 %
8	1076.0	1092.2	−1.3 %, −1.3 %

Tok/s is the mean of both passes. The fork's decode rounds are 1.3-3.5 % faster at every concurrency, the same in both passes; tok/s also moves with speculative acceptance on the sampled text, which is why C=1 is lower despite faster rounds (2.99 against 3.08 tokens per round).

Running the benchmarks

Commits: e2db558, 2838371, 9189a7e.

Build this fork, then the upstream control from the branch upstream-Windows-Port (details); stop any other server on the port first:

build_native.bat configure
build_native.bat build
git worktree add C:\ab\control\src upstream-Windows-Port
set AB_CONTROL_SRC=C:\ab\control\src
set AB_CONTROL_BUILD=C:\ab\control\build
bench\agentic_ab\build_control.bat configure
bench\agentic_ab\build_control.bat build

Agentic workload (about 1.7 hours for the two arms on three seeds). The runner reads the model path and launch flags from AB_LAUNCH_BAT, calibrates the largest context the control starts with, and writes report.md under profiles\bench\agentic_ab\. treatment is the fork and control upstream; an optional alt arm runs another build (AB_ALT_EXE) or the fork with other flags (AB_ALT_EXTRA_FLAGS, default --use-original-prefix-caching):

set AB_CONTROL_EXE=C:\ab\control\build\apps\Release\ninfer-serve.exe
set AB_LAUNCH_BAT=<a launch .bat with the fork flags above>
py -3.11 bench\agentic_ab\runner.py --arms treatment,control --seeds 42,43,44

Concurrent decode, one call per build (repeat --concurrency to sweep, or alternate single-point calls between the builds as above):

py -3.11 tools\bench\run_serve_concurrency.py --serve build-windows\apps\Release\ninfer-serve.exe ^
  --artifact q38=qwen3_8_27b_nvfp4-official.ninfer --mode dflash2_7 --suite decode-saturation ^
  --max-context 32768 --kv-capacity auto --concurrency 1 --concurrency 8 --output profiles\bench\cc-fork

Upstream uses the same command with the control checkout's own copy of the script and its ninfer-serve.exe, because its server writes an older request-log schema. On Windows that copy needs the fork's wait_for_final_throughput (commit 2838371), which reads the final statistics interval that Windows otherwise loses when the server is stopped.

What this fork changes

Each topic lists everything that affects it, whether written here or taken from elsewhere. "Upstream PR" means an open pull request on Neroued/ninfer that this fork merged before upstream did.

Picking individual changes. This fork's history is upstream master (d44ab584), then the Windows port, then one commit per fork change in dependency order, each a whole feature with its fixes folded in. Each item below links to its commit, and each commit message lists the earlier fork commits it builds on, so a change can be cherry-picked into another fork together with those prerequisites.

Hybrid prefix cache (the default)

Commit: a4665be.

Designed around how Qwen3.5-family models work: most of their layers are linear-attention (GDN) layers, whose recurrent state cannot be rebuilt from the KV cache, so resuming a prompt needs the KV of every earlier token plus a saved state at the exact token where the new prompt continues. The cache keeps the two apart and stores each as cheaply as it can. The design document explains the design and records every decision taken while building and running it, with the measurements behind it and the alternatives that were tried and reverted.

KV is cached per 64-token block, keyed by content (its tokens, any image in it, and the block before it), in a radix tree. A shared system prompt is stored once and costs its GPU pages once at any concurrency.
Saved model state is sparse. Snapshots are taken only at useful points: the end of the system prompt and tools, client cache breakpoints, the start of the assistant reply, the end of each answer, and a few points spread back through long history. Most cost no extra prefill work because they fall on prefill chunk boundaries.
Three tiers. Free VRAM after the model becomes GPU block cache (--kv-capacity defaults to auto); --host-cache-mib (default 8192, 0 = GPU only) is one pinned host RAM pool that blocks and snapshots share, split by how much prefill time each entry saves; and --prefix-cache-file PATH saves the host tier on shutdown and reloads it at startup (a smaller host tier keeps the most valuable entries). A file from a different model, KV format or ninfer-serve build is ignored and replaced.
Restores overlap the request's own work. Host RAM copies run on a separate stream in layer order and each layer waits only for its own data, so a long restored context costs little more than its new tokens.
Parallel requests with a new shared prefix prefill it once. Later requests wait for the first one's snapshot where the prompts diverge. Four requests with a new 13.9K-token system prompt: mean time to first token 1.48 s instead of 3.52 s.
Host eviction keeps what the next turns reuse, and a request waiting for a lane prefetches its host-only blocks: at the 52 GB production host tier, 9.2 % fewer prompt tokens prefilled and a 7.8 % shorter agentic workload.
Everything except --host-cache-mib is derived from --max-concurrency and --prefill-chunk; --device-snapshot-slots, --cache-taps-per-request, --cache-tap-ladder and --cache-tap-min-gap are optional overrides.
Ctrl+C stops cleanly and saves the cache: the first press asks for confirmation, the second answers running and queued requests with 503 and saves; one more press exits without saving. Commit: b14f8d6.

Original prefix cache: `--use-original-prefix-caching`

Upstream's checkpoint catalog (a saved state plus the KV at that exact point), kept with this fork's fixes. It is configured with --host-cache-mib or upstream's separate capacity flags, which require --use-original-prefix-caching. Details: resource scheduling and context cache.

GPU KV grows with the answer instead of being reserved up front: a request reserves its prompt plus a 4,096-token window and extends it as the answer grows, freeing just enough idle cache when the pool is full. Commit: 3614562.
Eviction takes the oldest entries first, only as many as needed, and moves them to host RAM before deleting them. Commit: b3c6442.
A checkpoint loses its value only when its own conversation has moved past it. Builds on upstream PR #300 by pkochubey (upstream issue #178). Commit: dbb1964.
One host RAM setting, --host-cache-mib, sizes the saved-state pool, the long anchors and host KV. Commit: a5c4039.
Long anchors are placed automatically at message boundaries, spaced further apart further back (--long-anchor-spacing). Commits: b83ac4b, 99e0552.
Aborted requests keep their prefilled prefix, so a retry carries on from there. Commit: 1dac1a2.
More time to find a cache plan (5 ms to 250 ms with the request's cost; upstream issue #229, approach suggested by Gene0Liu), and the shared-prefix list no longer fills up for good (upstream issue #251, approach suggested by albertov). Commits: 16fced4, 65b168e.
The default shared-prefix catalog is sized for a request's full candidate set (upstream PR #274 by giveen); cache planning cannot race with itself (by Gideon Zenz (gzenz)); no resource-underflow HTTP 500s when a release leaves shared pages exclusive. Commits: 34f7d18, 831326f, 3f43c97.
Real-model prefix-reuse scenarios and a smoke runner cover these fixes. Commit: ea722fa.

Prefill speed

Fast INT8 prompt attention (default for --kv-dtype int8): a FlashAttention-2 style kernel in which each warp keeps its query rows, scores and output in registers and accumulates P×V in FP16. The prefill chunk is rounded down to whole GPU waves (4096 runs as 3584). Per attention layer it takes 0.61-0.66× the time of upstream's INT8 prompt kernel on 3584-token chunks from an empty context to 128K. On the full ninfer-ppl-1m-v1 corpus it is closer to BF16 KV than the original INT8 kernel (4K windows: 4.8986 against 4.9027, BF16 KV 4.8948). --use-original-int8-prefill-kernel keeps upstream's kernel. Commit: 4c9a949.
Fast NVFP4 prompt attention (default for --kv-dtype nvfp4 over more than 2048 cached keys): QK runs on block-scaled FP4 Tensor Cores (8× the FP16 rate on RTX 5090) straight from the stored K codes, with Q as two NVFP4 terms (0.9 % RMS error); a single chunk whose rows alone would leave SMs idle splits its keys across CTAs. 0.34-0.67× the time of upstream's NVFP4 prompt kernel per attention layer; end to end, prefill is 3.5 % faster at a 16K-token prompt, 6.4 % at 32K and 14.4 % at 64K. Perplexity moves by +1.1e-3 nats per token (standard error 2.0e-3). --use-original-nvfp4-prefill-kernel keeps upstream's kernel. Commit: 8dcd89a.
Several requests can prefill at the same time, overlapping one request's prefill with others' prefill and decode. By David Oelfke in the gzenz/ninfer fork. Commit: 25e52f9.
Fused text q/k RMSNorm + RoPE at every width (14-22 % faster than three separate calls at the 3584-token chunk, one sincos per lane), and only for checkpoints with its built-in RoPE theta and epsilon. Commit: 2c8be5e.
Kernel tuning from upstream PRs: the text rmsnorm_rope route (#273, Michael Dementii), a Q6 34,816×5120 shape for the fused MLP gate_up (#284, bingchengcc), and, adapted from llmq (IST-DASLab, Erik Schultheis) by DuncanBetts, a fused NVFP4 RMSNorm + quantise for the attention input projection (#305) and a single-pass target log-probability kernel (#307). Commits: 3681fed, a4c112f, 2e68ce8, 53f51c5.

Decode speed

Overlapped decode kernels. Kernels in the decode CUDA Graph launch as programmatic dependents of the kernel before them (PDL): weight-streaming kernels load their first weight tiles while the previous kernel runs, and release the next kernel only after their own main loop. Output is unchanged token for token; greedy decode rounds were 2.2-2.5 % faster when this was introduced. Ideas tried and dropped are in RESEARCH_NOTES.md. Commit: 0c59ca6.
Decode kernels sized for verification widths: a third pipeline stage for the NVFP4 down projection up to 64 tokens (45.5-47.7 µs instead of 57.7-60.0 µs, and the A4 route from 8 tokens), and two 16-row tiles sharing each staged activation in the FP8 head and the Q8 DFlash2 drafter. Commits: 5db3795, 5db53c7.
Reciprocal NVFP4 activation quantizer on the Linear MMA route (upstream #327 by DuncanBetts): 2-5 % faster at 8-64 tokens; the other A4 routes keep the divisions, because opting them in changed the generated text. Commit: 58808ee.
Ngram copy drafting proposes the next tokens by copying matching text from earlier in the context, alongside MTP/DFlash/DFlash2. The single-request version is the original work of remesis in the remesis/ninfer fork (upstream issue #234); this fork extends it to --max-concurrency above 1 and builds each prompt's index while the prompt is prepared, off the engine worker. See ngram copy proposals. Commits: d2209f6, c5e390b, c54dacb.
NVFP4 KV groups pick the best of five scales (NVFP4 K and V, K8V4 V): each 16-value group maps its largest magnitude to 6, 4, 4.5, 5 or 5.5 and keeps the scale with the least squared error (Four Over Six, arXiv:2512.02010, generalized). RMS error of the 27B model's rotated K rows falls from 9.5 % to 8.5 %; perplexity moves within one standard error. Commit: a79c2cd.

Tool calls and reasoning output

More tool-call formats: the XML forms emitted by Claude Code and other agent tools (<function name="…">, <invoke>, <function_calls>, short <param> tags), also while streaming. Upstream PR #300 by pkochubey (upstream issue #276). Commit: 00ef353.
Repeated tool-call parameters keep the last value (upstream PR #299 by adubkov), and a quoted </parameter> stays in the value (upstream PR #318). Commits: c09e929, 69b1760.
Quoting </think> no longer ends the reasoning early: it only ends the reasoning when a line break or the end of the turn follows. Adapted from upstream PR #309 by Fedor Suchkov. Commit: 080af40.
--tolerant-tool-calls keeps a good call followed by junk, a final call cut off by the output limit (if a parameter is complete), repairs a missing > after the function name, and returns calls to undeclared tools. By David Oelfke in the gzenz/ninfer fork, ported onto this fork's parser. Commit: 9e28ab8.
A reasoning effort the chat template rejects renders as its nearest accepted one (the official Qwen3.8 template accepts only low, medium and xhigh), and --chat-template gains the froggeric v22.5 template. Commits: 9b7c58b, c65c819.

API and client compatibility

llama.cpp-style model details on /v1/models (upstream PR #162 by Hector Ramon Jimenez (hecrj)) and ignore_eos on chat completions (upstream PR #197 by Thireus). Commits: ad36334, f235041.
GitHub Copilot and other agent-host requests (custom tools, advisory tool_choice / strict / parallel_tool_calls, tool names up to 256 bytes, --usage-chunk-choice): the serving commits of upstream PR #316 by paq85 (Damian Sromek). Commit: 36fc03b.
Responses API options used by Codex and Zed Agent (reasoning.summary, include: ["reasoning.encrypted_content"]): upstream PR #295 by Macasacker, based on an earlier PR by Sha1rholder. Commit: d03fd36.
response_format json_object / json_schema is accepted (not enforced), a tool call cut off by the output or context limit is reported as cut off, and a request ending with an assistant message continues that reply (thinking off only): upstream PR #300 by pkochubey. Commits: d9f6c95, ad0600e, c2e2339.
Anthropic clients such as Qwen Code: forced, named and strict tool choices are accepted as advisory (upstream issue #223), a thinking budget at or above max_tokens is accepted, and API routes answer under a doubled /v1 prefix (a base URL ending in /v1). Commits: e56b408, f863ddc, e49a360.

Stability

Out-of-memory no longer stops the engine: only the affected requests fail and the engine carries on with the queue (by David Oelfke in the gzenz/ninfer fork), and recovery really leaves the engine empty. Commits: a8569f4, f8f23a2.
An aborted prefill can no longer write into another conversation's cached prefix: a lane's block-table copy still queued from an aborted request is waited for before the next request overwrites it, and activated block tables are published on the compute stream (upstream PR #320). Commits: 92d0cf7, 4291a7e.
Smaller fixes: a workspace scope opened before an arena reset no longer rolls the next phase's allocations back; a request an idle engine can never admit gets 503 instead of 500; token-count requests are bounded like generation requests; Windows servers detect clients that vanish without closing the connection; and a vision overlay suffix is encoded at the right position (fork PR #1 by Yunado). Commits: f67a284, edc9785, 7936838, f6af07f, 22e6ef1.

Models, conversion and vision

GGUF files as conversion sources: upstream PR #282 by giveen. Commit: a4f6aab.
NVIDIA ModelOpt NVFP4 and FP8 checkpoints as conversion sources, with the qwen3_8_27b_nvfp4_nvidia recipe storing the output head as FP8; Quasar NVFP4 conversion with DFlash2 heads and an indexed proposal head; third-party tokenizer settings rebuilt during conversion. Commits: 33afed8, de4623a, 5b73bba.
A qwen3_8_27b_q6 recipe and a grouped_mse scale-search method for groupwise quantisation; Q8 MTP and a general BF16 GEMM fallback for shapes without a dedicated kernel. Commits: a4c112f, afb274c, e172c02, 961ce1b.
--rope-yarn-factor F for YaRN context extension (F from 1 to 4, up to 1M tokens of context). Commit: 59de7e3.
--vision-offload on keeps the vision tower in pinned system RAM instead of VRAM and streams it to the GPU while an image is encoded, and --vision-max-merged N bounds the merged vision tokens per image or video. Based on the original work by Valeriy Selitskiy (iamwavecut), rewritten for this engine. Commit: 5f7350f.

Windows

Native build and run with MSVC and CUDA (see Quick start): static CUDA runtime, FFmpeg/curl from vcpkg, non-RDC NVFP4 kernels, TMA descriptors staged into device memory by a kernel (launches captured into a decode graph read a copy written once, so decode rounds replay no staging kernels), a built-in PNG decoder for the vision path, drive letters in converter recipe paths, and a build id in every product binary. The same port without any other fork change is the branch upstream-Windows-Port. Commits: 57d77e4, 5065bc5.
Linux still builds and runs (see Quick start (Linux)), checked under WSL2 Ubuntu 24.04 with CUDA 13.4; the FP64 attention oracle of the tests uses every host core. Commit: 18795ee.

Options and console

--log-colours on colours the console statistics, and a session statistics panel beneath the log shows session and last-ten averages of TTFT, cache hit rate, prefill and decode speed, the mean decode batch and drafter acceptance (--log-stats-panel off removes it). Engine messages and FFmpeg's log are ordinary log records that scroll above the panel. Commits: 9ff604a, 94f0ea6, 6fda154, 3ee3c1b.
Grouped --help by category on ninfer-serve and the ninfer CLI, with separate sections for the two prefix caching systems. Commit: 3c9b129.
--vram-headroom-mib N sets how much GPU memory --kv-capacity auto leaves spare (upstream always leaves 1 GiB). Commit: 1684538.
Measured CUDA Graph allowance: the KV sizing reserves 64 MiB plus 4 MiB per decode-graph executable, measured on an RTX 5090 across every speculative mode and concurrency; startup warns if the graphs ever use more. Commit: 61e082f.
--thinking-budget-message S sets the message inserted when a request reaches its --default-thinking-budget N. Commit: f3aaad7.

Kept in sync with upstream

This fork is rebased onto upstream master whenever upstream moves; the latest is d44ab584 (September 2026). Where upstream rewrote code this fork had changed, the rebase starts from upstream's version and carries the fork's change onto it only where an A/B test on the RTX 5090 shows the fork's version is faster. Once upstream adopts a change, it leaves this list.

Dropped in the rebase onto d44ab584, with the measurement that decided each:

K8V4 prompt and decode kernels: upstream's reorganized K8V4 attention was faster (decode by up to 43 %, prefill from 32K keys and at 1024-token chunks).
NVFP4 decode kernels with FP4 Tensor Core QK: not carried over. They were 0.64-0.92× the time of upstream's new NVFP4 decode at long contexts but up to 1.21× at 8K, and porting them means a new split-KV family in upstream's routes. The NVFP4 prompt kernel was carried over (above).
TMA-staged FP8 prefill GEMM (upstream PR #167 by Michael Dementii): faster than upstream's new TMA split-K schedules in operator timings (8-16 % on the input projections at 2048-4096 tokens), but prefill on the NVIDIA NVFP4/FP8 artifact was 1.5-3.5 % slower end to end with it at every call site from 2048 tokens, and 0.5-0.8 % slower on the output projections alone.
Narrow FP8 A8 decode-width linear_add tiles: upstream's tuned TMA schedules are as fast at 17-128 tokens for K=6144 (within 5 %) and faster for K=17408 (up to 24 %).
Split-KV attention for short prefill passes and verification widths, and its split-count fix at very long contexts: upstream's grouped split-KV routes now take single-row passes of up to 256 (BF16, INT8), 192 (NVFP4) or 80 (FP8, K8V4) new tokens, and its 300,001-key test passes.
Folding the attention output gate into the reduce (#268 and batched verification): worth about 1.5-2 µs per full-attention layer (about 0.2 % of a decode step), and upstream's per-format kernels would each need the fold.
Per-profile CUDA Graph topology classes: upstream made decode attention update-compatible across resource tiers.
Adopted by upstream: FP8 MMA on the MX datapath (#328 by DuncanBetts).

Model artifacts

Qwen3.8-27B-NVIDIA-NVFP4-NInferV3 (recommended): nvidia/Qwen3.8-27B-NVFP4, the Model Optimizer mixed NVFP4/FP8 checkpoint, converted with the qwen3_8_27b_nvfp4_nvidia recipe in tools/convert/official_recipes.py. The weights are imported bit-exact except the output head (NVFP4 → row-scale FP8), with the DFlash2 draft model and an indexed 131,072-row proposal head for --lm-head-draft.
Qwen3.8-27B-Quasar-NinferV3: QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4, the QAT-trained NVFP4 checkpoint, converted with tools/convert/quasar_nvfp4.py, with the same DFlash2 draft model and proposal head.

Both are single-file .ninfer artifacts for an RTX 5090 (sm_120a); each Hugging Face page has the creation outline and conversion report. The Quick start launch works for either.

Thanks

A big thank you to all the contributors to upstream NInfer and to the forks this one draws on — Neroued, Michael Dementii, Minnnn, DuncanBetts, Thireus, remesis, Valeriy Selitskiy (iamwavecut), Hector Ramon Jimenez (hecrj), giveen, bingchengcc, pkochubey, paq85, Macasacker, Sha1rholder, adubkov, Gideon Zenz (gzenz), David Oelfke, Fedor Suchkov, Yunado, and everyone else whose pull requests, reviews and commits made this fork possible — and a particular thank you to Neroued for creating NInfer, maintaining upstream so well, and for the work this branch builds on.

Upstream README (direct copy)

Everything below reproduces the upstream NInfer README at d44ab584, with an added TP2 callout and a link to the fork's ngram copy proposals guide. Its single-GPU requirements and limitations describe upstream, not this branch. Its original-cache serving examples require --use-original-prefix-caching when run with this fork; for current TP2 usage, follow the two-GPU guide.

This fork (tp2-5060ti) adds two-GPU tensor parallelism (--tp 2). It runs Qwen3.8-27B NVFP4 on two 16 GB RTX 5060 Ti cards without P2P. The branch was developed by Claude Opus 5.5 (medium reasoning effort) in Claude Code. See Two-GPU tensor parallelism on RTX 5060 Ti for usage and results. Everything below is the upstream README.

Selected checkpoints. Maximum single-GPU inference performance.

NInfer is a from-scratch C++/CUDA inference engine for Qwen3.5 Dense and MoE architectures on a single NVIDIA GeForce RTX 5090. It runs text, image, and video prompts through a local CLI or OpenAI-/Anthropic-compatible HTTP APIs. The runtime is deliberately specialized: one GPU, one resident model, and a startup-fixed capacity of one to eight active requests.

Five official artifacts are available. The quick-start commands use Qwen3.8-27B NVFP4.

Model	Weights	Artifact	Download and model card
Qwen3.6-27B	`groupwise-int`	`qwen3_6_27b.ninfer`	Qwen3.6-27B
Qwen3.6-27B	`nvfp4`	`qwen3_6_27b_nvfp4.ninfer`	Qwen3.6-27B NVFP4
Qwen3.8-27B	`groupwise-int`	`qwen3_8_27b.ninfer`	Qwen3.8-27B
Qwen3.8-27B	`nvfp4`	`qwen3_8_27b_nvfp4.ninfer`	Qwen3.8-27B NVFP4
Qwen3.6-35B-A3B	`groupwise-int`	`qwen3_6_35b_a3b.ninfer`	Qwen3.6-35B-A3B

Each v3 .ninfer artifact carries model configuration, encoded weights, logical bindings and frontend resources. Runtime execution uses those facts with the implemented model and Op capabilities. You can also convert your own weights, reuse an official recipe or choose another supported mixture of formats.

The current engine requires v3 artifacts. Existing official v2 downloads can be upgraded locally without downloading the weights again.

Quick start

NInfer requires 64-bit Linux, an NVIDIA GeForce RTX 5090, a CUDA toolkit supporting sm_120a, CMake 3.28 or newer, a C++20 host compiler, Ninja, pkg-config, FFmpeg development libraries (libavformat, libavcodec, libavutil, and libswscale), and libcurl >= 7.85. CUDA 13.1 is the validated development toolkit; CMake does not impose a CUDA version floor. The build rejects CUDA architectures other than sm_120a.

Build the product binaries:

git clone https://github.com/Neroued/ninfer.git
cd ninfer

cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

Tests and benchmarks are excluded from the default build. cmake --preset release configures the same product build; cmake --preset dev also enables tests and benchmarks and finds a Python 3 interpreter. Both presets use build/ and explicitly reset the build options. Machine-specific compiler and Python paths belong in the ignored CMakeUserPresets.json. See build organization and configuration for details.

There is no install target or packaged binary distribution; run NInfer from its source build tree. Python tools run independently of CMake; the standalone HBM probe has its own build command.

Download the artifact used by this example with the Hugging Face CLI:

hf download neroued/Qwen3.8-27B-nvfp4-NInfer \
  qwen3_8_27b_nvfp4.ninfer \
  --local-dir models

Start a long-running text/agent server with two active-request lanes and explicit Device/Host checkpoint capacity:

./build/apps/ninfer-serve models/qwen3_8_27b_nvfp4.ninfer \
  --max-context 240000 \
  --kv-capacity 240000 \
  --max-concurrency 2 \
  --kv-dtype fp8 \
  --device-state-slots 2 \
  --host-state-slots 8 \
  --host-kv-mib 8192 \
  --spec mtp --draft-tokens 3 \
  --lm-head-draft \
  --preserve-thinking

Each request has a 240,000-token logical ceiling. A shared 240,000-token Device KV pool serves admitted requests; two requests run concurrently when their combined reservations fit. The cache tiers provide two Device checkpoint slots, eight pinned Host State slots, and 8 GiB of pinned Host KV beyond the two active StateImages.

Send an OpenAI-style request:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3.8-27b",
    "messages": [{"role": "user", "content": "Reply with one short sentence."}],
    "max_tokens": 64
  }'

Run a one-shot CLI request with a 32,768-token allocation:

./build/apps/ninfer models/qwen3_8_27b_nvfp4.ninfer \
  --prompt "Explain prefill and decode, then give a concise conclusion." \
  --max-context 32768 \
  --max-new 8192 \
  --kv-dtype fp8 \
  --spec mtp --draft-tokens 3 \
  --lm-head-draft

Answer content is written to stdout. Human-readable startup/runtime diagnostics and the CLI-owned reasoning, timing, throughput, memory, and speculative-decoding report are written to stderr; reasoning and the result report remain unprefixed product output. On a terminal, weight materialization uses one transient progress line followed by a compact Engine-ready summary. Redirected stderr receives persistent readable progress without terminal control sequences. Use --log-level debug for complete startup detail. Option and local input errors remain direct command diagnostics. Use --messages FILE and --vision for structured image/video input; see the CLI guide and committed examples.

Resource-aware long-context reuse

A reusable prefix checkpoint contains KV and the complete continuation state for its exact prompt frontier. A Device-resident checkpoint resumes directly. Under pressure, the planner weighs Device retention, pinned Host State/KV, and eviction by immediate restore work and later reuse cost. Active requests retain their completion reservations.

See Resource scheduling and context cache for the algorithm and Serve TTFT benchmark for public-HTTP coverage of hot reuse, Host resume, eviction, shared prefixes, scheduling boundaries, and multimodal load.

Performance

Published measurements use an RTX 5090. The performance index links to per-model run records and the measurement rules. The tables below are excerpts from those detailed results. Qwen3.8 uses FP8 E4M3 row-256 KV; Qwen3.6 uses INT8 group-64 KV.

Concurrent MTP3 decode

Saturated decode used CUDA Graphs, MTP3, and one 8,192-token generation per active request. Throughput uses aggregate committed decode tokens from complete intervals whose actual decode batch equaled the configured concurrency. Acceptance covers the complete request wave; these rates are steady decode (tok/s).

Model profile	C=1 tok/s / accept	C=2 tok/s / accept	C=4 tok/s / accept	C=8 tok/s / accept
Qwen3.6-27B `groupwise-int`	185.8 / 68.2%	247.0 / 69.0%	309.5 / 68.4%	535.0 / 68.3%
Qwen3.6-27B `nvfp4`	202.4 / 69.3%	399.7 / 71.4%	699.7 / 69.3%	1,146.9 / 68.6%
Qwen3.6-35B-A3B `groupwise-int`	642.5 / 68.6%	907.2 / 66.3%	1,213.5 / 69.6%	1,380.7 / 68.0%
Qwen3.8-27B `groupwise-int`	136.5 / 44.4%	253.3 / 45.2%	398.1 / 46.1%	582.4 / 46.4%
Qwen3.8-27B `nvfp4`	147.7 / 46.2%	291.0 / 48.7%	522.2 / 45.8%	922.4 / 46.1%

Single-request serving

The serial serving corpus used CUDA Graphs, a 1,024-token prefill chunk, and five fixed seeds after warm-up. The table keeps one short-prefill, one extreme-prefill, and one structured-output MTP3 point for each published profile; the full context and scenario matrices are linked from each model below.

Model profile	7,680-token prefill	260,096-token prefill	Structured MTP3 decode
Qwen3.6-35B-A3B `groupwise-int`	17,705.4 tok/s	5,247.0 tok/s	779.6 tok/s
Qwen3.6-27B `groupwise-int`	3,218.1 tok/s	1,614.8 tok/s	193.0 tok/s
Qwen3.6-27B `nvfp4`	11,191.5 tok/s	2,510.6 tok/s	252.2 tok/s
Qwen3.8-27B `groupwise-int`	3,331.9 tok/s	2,139.4 tok/s	214.7 tok/s
Qwen3.8-27B `nvfp4`	12,819.1 tok/s	4,016.4 tok/s	231.7 tok/s

Evaluation

Capability scores were measured through NInfer's OpenAI-compatible serving route with thinking enabled, MTP3, and EvalScope 1.9.0 (0-shot, rule scoring, one sample per problem):

Model profile	AIME 2025	AIME 2026	GPQA-Diamond	ERQA	RealWorldQA
Qwen3.6-27B groupwise-int	86.67%	93.33%	86.87%	—	—
Qwen3.6-27B NVFP4	93.33%	93.33%	84.34%	—	—
Qwen3.6-35B-A3B groupwise-int	90.00%	90.00%	85.35%	—	—
Qwen3.8-27B groupwise-int	96.67%	96.67%	87.37%	66.25%	82.22%
Qwen3.8-27B NVFP4	96.67%	96.67%	90.40%	66.25%	83.53%

The Qwen3.6 rows used temperature 0.6 and presence penalty 1.0; the Qwen3.8 rows used temperature 1.0 and presence penalty 0.0. Multimodal evaluation used --vision and an 81,920-token context limit. Text evaluation used 262,144 tokens except Qwen3.8-27B NVFP4, which used 252,928 tokens to fit the RTX 5090 after weights. Each score is one sample per problem; model cards contain the correct/total counts and evaluation notes.

Startup notes

GPU residency is fixed at process startup. --spec selects speculative decoding residency, and --vision independently selects Vision residency. Qwen3.6-35B-A3B DFlash can be combined with Vision; it accelerates generated-text decode after multimodal prefill, not Vision encode itself.

Docker

Build the runtime image on a host with the NVIDIA Container Toolkit:

docker build --tag ninfer:local .

Mount the downloaded model and run the same example server profile:

docker run --rm \
  --gpus '"device=0"' \
  --publish 8080:8080 \
  --volume "$PWD/models:/models:ro" \
  ninfer:local \
  ninfer-serve /models/qwen3_8_27b_nvfp4.ninfer \
  --host 0.0.0.0 \
  --max-context 240000 \
  --kv-capacity 240000 \
  --max-concurrency 2 \
  --kv-dtype fp8 \
  --device-state-slots 2 \
  --host-state-slots 8 \
  --host-kv-mib 8192 \
  --spec mtp --draft-tokens 3 \
  --lm-head-draft \
  --preserve-thinking

Capabilities and limits

The official artifacts provide the following capabilities, with optional components enabled at startup:

text generation with thinking and non-thinking prompt modes;
image, multi-image, video, and mixed multimodal messages;
chunked prefill, exact-batch CUDA Graph decode, and startup-bounded batched decode;
MTP speculative decoding with draft windows from one to five;
BF16, INT8, FP8, NVFP4, and K8V4 KV storage;
offline causal-perplexity scoring;
private and shared exact-prefix reuse with Device/Host State and KV retention;
model-aware sampling defaults and explicit sampler overrides;
OpenAI Responses Core, OpenAI Chat Completions, and Anthropic Messages, including streaming, tools, local response state, token counting, and usage accounting.

The 35B-A3B target additionally supports DFlash with draft windows from one to fifteen for Text and image/video Vision prompts. Qwen3.8-27B artifacts with the DFlash2 companion weights support --spec dflash2 --draft-tokens 7 for the same Text/Vision Engine path, with draft counts 1..15 and either full or optimized proposal heads.

The product boundary remains intentionally small:

one RTX 5090 and one resident model per Engine;
a startup-fixed capacity of one to eight active requests with bounded FIFO ingress;
no request preemption, priority/QoS, active-request swapping, weight offload, multi-GPU, or distributed serving;
one shared startup-fixed KV pool across active requests and retained prefixes;
model architectures and format/shape combinations use explicitly implemented native paths;
parsed tool calls are returned to the client; NInfer does not execute tools;
the in-tree C++ headers are not distributed as an installed SDK.

--max-context is each sequence's logical limit. --kv-capacity sizes the shared Main Text KV pool used by active requests and retained prefixes; auto resolves the largest legal capacity at startup from the memory remaining after weights while keeping 1 GiB of sizing headroom. Explicit capacities remain fixed for the process lifetime.

Documentation

Run the relevant --help for the exact current option contract.

Support

NInfer is a personal project that I develop out of interest. If you find it useful and would like to support its continued development, you can support the project on Ko-fi.

Support is entirely voluntary. It is not a purchase or investment and does not come with financial returns, promised services or features, or a role in project decisions. The project's direction, priorities, technical choices, and release schedule remain independently determined by the maintainer.

License

NInfer is licensed under the Apache License 2.0.

The published artifacts are derived from Qwen/Qwen3.6-27B, Qwen/Qwen3.8-27B, and Qwen/Qwen3.6-35B-A3B. The Qwen3.6-27B NVFP4 artifact also uses the fixed packed weights from rdtand/Qwen3.6-27B-PrismaSCOUT-Blackwell-NVFP4-BF16-vllm. The Qwen3.8-27B NVFP4 artifact also uses the fixed mixed FP8/NVFP4 weights from unsloth/Qwen3.8-27B-NVFP4. These source repositories are distributed under Apache-2.0. Vendored dependencies retain their own license files under third_party/.

Name		Name	Last commit message	Last commit date
.codex		.codex
.github		.github
apps		apps
bench		bench
cmake		cmake
docs		docs
eval		eval
examples/cli		examples/cli
include/ninfer		include/ninfer
model-cards		model-cards
src		src
tests		tests
third_party		third_party
tools		tools
.clang-format		.clang-format
.clangd		.clangd
.dockerignore		.dockerignore
.gitattributes		.gitattributes
.gitignore		.gitignore
AGENTS.md		AGENTS.md
CMakeLists.txt		CMakeLists.txt
CMakePresets.json		CMakePresets.json
CONTRIBUTING.md		CONTRIBUTING.md
Dockerfile		Dockerfile
LICENSE		LICENSE
README.ko.md		README.ko.md
README.md		README.md
RESEARCH_NOTES.md		RESEARCH_NOTES.md
build_native.bat		build_native.bat
build_native_target.bat		build_native_target.bat

Latest commit

History

Folders and files

Repository files navigation

NInfer — two-GPU fork for 2x RTX 5060 Ti (tp2-5060ti)