[Peuqui]

Update 2026-10-28: rebased onto main c59b3c4. The two commits are unchanged apart from the regenerated docs/design/architecture/runtime_reference.md (git range-diff). _C rebuilt from this tree; the Test Plan files pass on this head without a failure: 277 passed on a Tesla V100, 270 passed and 7 skipped on a Quadro RTX 8000 (the test fakes that #1214 repaired are on main now). pre-commit run --from-ref origin/main --to-ref HEAD on this head: every hook passes except check-layering, which now reports vllm/config/speculative.py (model coupling 142 -> 144) next to vllm/model_executor/layers/fused_moe/runner/moe_runner.py (platform coupling 37 -> 39); main c59b3c4 reports the same two counts, so this PR adds none. pre-commit run mypy-3.10 --hook-stage manual: passed.

Purpose

ple_disk_gather_u8 (from #885) first calls mincore and MADV_WILLNEED for every missing page of a request and then copies all rows. Pages brought in by read-ahead have not been accessed yet, so under memory pressure they are among the first pages reclaim takes. In a long prefill the copy can then reach pages that were already reclaimed and fault them back in one synchronous page at a time.

We hit this with Qwen3.8-Flash-Next-180B-A4B NVFP4 on PP4 (188,956,217 of 320,001,536 PLE rows on the disk tier), served from a systemd service with MemoryHigh=16G on a host with 30 GiB of RAM, the checkpoint on an NVMe SSD behind USB. In one boot the time to first token of a 184,134-token prompt was 164.7 s. While it served the long prompts, a per-thread sampler of the PLE offload worker counted 586,325 of the worker's 586,354 major faults in its main thread, up to 4,040 per second (10 s window), and py-spy showed that thread in MappedRowGatherKernel.apply. In the A/B below the same prompt took 131.8 and 183.0 s in the two base boots.

The first commit copies the rows in address order and advises only the pages that lie within ple_disk_prefetch_window_kib ahead of the copy (new KernelConfig field, default 4096 KiB; 0 copies without advice). As before, only pages that cover requested rows are considered. Byte selection and output order are unchanged. The field is listed with the other PLE I/O fields that do not enter the compile hash. Both callers of prepare_mapped_row_gather pass it: the n-gram disk tier in ple_layer.py and the CPU-owned packed GGUF table in gguf_embedding.py.

The second commit repairs the ten tests in tests/kernels/ple/test_gguf_packed_embedding.py, which fail on main before they reach the code under test: their hand-built stand-ins lack the kernel_config that the packed GGUF embedding reads while it is constructed (82b296a) and the layer_name that the n-gram forward uses for its diagnostics (906f8ff). With the repair these tests run the packed GGUF reader through this gather again (222 gather calls with the 4 MiB window in that file on the Tesla V100).

Not a duplicate: gh pr list --repo 1CatAI/1Cat-vLLM --state all --search with "prefetch window", "MADV_WILLNEED", "ple_disk_gather", "madvise PLE", "PLE page cache", "PLE disk prefetch", "mincore", "ple_disk_rows", "PLE major faults" and "PLE disk stall", and the same searches over issues, find no PR or issue about the prefetch of this gather. #885 (merged) introduced the gather. No open PR changes csrc/ple_disk_rows.cpp, vllm/model_executor/kernels/ple/disk_rows.py, tests/kernels/test_ple_disk_rows.py or tests/kernels/ple/test_gguf_packed_embedding.py (git diff against each open PR head). git merge-tree against all 122 open PR heads adds conflicts only in the generated docs/design/architecture/runtime_reference.md, for #903, #1210, #1211, #1215, #1216 and #1217: the line anchor of KernelConfig.compute_hash, which tools/generate_architecture_reference.py --write regenerates.

AI assistance was used for this change. I reviewed every line and ran the tests and measurements below.

Test Plan

pytest tests/kernels/test_ple_disk_rows.py tests/kernels/ple tests/models/qwen4_exp/test_ple.py tests/config/test_ple_cascade_capabilities.py tests/config/test_runtime_default_ownership.py tests/tools/test_architecture_reference.py
pre-commit run --from-ref origin/main --to-ref HEAD
pre-commit run mypy-3.10 --hook-stage manual --from-ref origin/main --to-ref HEAD

The tests ran on main 6f3557c plus this PR (_C built from this tree, TORCH_CUDA_ARCH_LIST=7.0) and on main with its own build, once on a Tesla V100 and once on a Quadro RTX 8000.

Offline gather: the real FP8 PLE table of that checkpoint (128 shards, 320,001,536 rows of 160 B), 16 calls of 18,000 random row ids each (about the disk-tier share of a 2,048-token chunk), in a cgroup with memory.high = 3 GiB and 2.25 or 2.4 GiB of touched anonymous ballast, the table evicted from the page cache before each arm. The old operator (main's code) against this PR's operator with windows of 1, 4 and 16 MiB and with the whole request advised up front; SHA-256 over all output bytes.

Serving A/B on our fork (main 6f3557c plus our commits): fork commit cf50e1a (the two commits of this PR) against its parent 5841543, Flash-Next PP4 and TP2xPP2, torch.compile with CUDA graphs, MTP with 4 speculative tokens. Boots fix cold, base, fix, base, fix; the cold boot is not counted, and the PLE table is evicted from the page cache before every boot. The base arm runs the production tree, the fix arm the same entries with PYTHONPATH on the fork tree, its own compile cache and its own _C. Our fork also keeps a 0.5 GiB Python row cache in front of this gather and releases only the page tables of the rows it read; neither is on main. Per boot: four ~29k-token prompts, a decode benchmark (prose, code, edit), eight quality prompts at temperature 0, and prompts of 184,134 and 249,950 tokens. System-wide major faults and bytes read (/proc/vmstat) and file refaults of the service cgroup (memory.stat) are counted from the start to the end of each boot's measurements.

Path coverage

Path / provider	Format	Key conditions (shape, TP/EP, graph, capability)	Expected selection / fallback	Evidence (source / static / executed / measured)	Unverified scope
Native mapped row gather, plain and with row cache	FP8 n-gram PLE rows in safetensors	Linux, file-backed shards; window 0, one page, inside the request, whole request	rows copied in address order, pages advised up to the window ahead; window 0 copies without advice	executed (unit tests, offline gather) + measured (serving A/B, Flash-Next PP4 and TP2xPP2)	serving on main without our fork's Python row cache
Native mapped row gather for CPU-owned packed GGUF PLE	packed GGUF	prepare_mapped_row_gather from gguf_embedding.py	same operator with the configured window	executed (tests/kernels/ple/test_gguf_packed_embedding.py after the second commit)	serving not run
Python copy route	any	ple_disk_row_gather=false or native operator unavailable	unchanged	static

Acceleration and benchmark contract (required for performance changes)

• Default enabled or opt-in: enabled; ple_disk_prefetch_window_kib defaults to 4096 KiB.
• Required CLI options and environment switches: none; --kernel-config '{"ple_disk_prefetch_window_kib": 0}' copies without advice.
• KV cache dtype used for the benchmark: auto (FP16, the model dtype).
• Wheel SHA or source commit: our fork, fix arm cf50e1a, base arm 5841543; this PR is the same change on main (the csrc diff has the same patch-id as fork commit bf91f9b).
• PYTHONPATH, source overlays, or external native libraries used: fix arm with PYTHONPATH on the fork tree and its own native build; nothing else.
• User-entry route-hit, speed, and output-quality evidence: see Test Result.
• Existing kernel/backend registry and capability-rejection reasons: unchanged; the gather is still admitted by prepare_mapped_row_gather and recorded in ple_disk_row_readers.
• Per-engine configuration and deprecated-variable compatibility: KernelConfig field only, no environment variable.
• New/changed environment metadata and generated reference check: architecture reference regenerated (KernelConfig.compute_hash line anchor).
• Route snapshot command and intentional changed rows: none.
• SM70 registered variables / unregistered reads / model-parameter restrictions, before -> after: unchanged.

Test Result

• Tests on a Tesla V100: this PR 266 passed, 8 failed; main 213 passed, 18 failed. On a Quadro RTX 8000: this PR 259 passed, 8 failed, 7 skipped; main 210 passed, 14 failed, 7 skipped. The 43 additional tests are the new window cases in tests/kernels/test_ple_disk_rows.py, all passing; the GGUF tests repaired by the second commit pass as well (10 on the V100; on the RTX 8000 four of them skip, as on main). The remaining 8 failures are identical on main (test_host_result.py: the connector module has no envs; test_ple.py: Qwen4ExpNGramEmbedding fakes without layer_name or layer_multipliers); [Test] Repair stale fakes in PLE offload and SM70 decode-graph tests #1214 repairs them.
• pre-commit and mypy-3.10 pass.
• Offline gather, median time per call (less is better): old operator 482.4-489.2 ms, 4 MiB window 444.0-447.9 ms, 1 MiB 444.2-448.1 ms, 16 MiB 450.7-453.2 ms, whole request advised up front 478.7-484.8 ms. The SHA-256 of the output bytes is the same for every run and window. The old operator did not stall in this setup (193-278 major faults per 16 calls), so we could not reproduce the serving stall offline; there the bounded window saves about 8 % per call, while advising the whole request first, even in address order, is as slow as the old operator.
• Serving, time to first token in s (less is better), two base boots against two fix boots:

Entry	Prompt	Base	This change
PP4	~29k (3 per boot)	7.5-8.0	7.5-7.9
PP4	184,134	131.8 / 183.0	69.4 / 63.6
PP4	249,950	80.8 / 148.8	81.3 / 82.0
TP2xPP2	~29k (3 per boot)	11.7-11.8	11.7-11.8
TP2xPP2	184,134	108.5 / 127.1	78.6 / 98.6
TP2xPP2	249,950	113.5 / 137.9	125.5 / 136.8

• Counters per boot, base against fix (less is better): PP4 major faults 668,732 / 1,165,556 against 226,054 / 151,744, read from disk 10,147 / 12,427 MiB against 8,713 / 8,025 MiB, file refaults 831,151 / 1,419,179 against 459,143 / 290,915; TP2xPP2 major faults 241,783 / 375,864 against 119,797 / 324,274, file refaults 288,386 / 440,598 against 170,558 / 390,341. In the worst base boot the PLE worker had nothing swapped out during the measurements, so the re-reads are file pages, not swap.
• Decode is unchanged (ms per step, less is better; tok/s, more is better): edit task PP4 54 ms per step = 92.7-92.9 tok/s base against 92.6-92.8 tok/s, TP2xPP2 45 ms = 110.0-110.2 against 110.2 tok/s. The 400-token answers to the long prompts, base against this change: at 184,134 tokens PP4 62.1 / 61.5 ms = 51.4 / 51.9 tok/s against 61.9 / 62.7 ms = 49.2 / 48.5 tok/s, TP2xPP2 53.7 / 53.9 ms = 56.3 / 56.1 tok/s against 53.5 / 53.7 ms = 57.4 / 57.2 tok/s; at 249,950 tokens PP4 67.2 / 66.3 ms = 39.1 / 39.6 tok/s against 66.7 / 67.2 ms = 39.1 / 39.1 tok/s, TP2xPP2 58.5 / 58.4 ms = 47.7 / 47.8 tok/s against 58.1 / 58.7 ms = 47.7 / 47.2 tok/s. With MTP the tok/s also depend on how many draft tokens the answer text lets through, and these texts differ between the arms (see Outputs), for example 125 against 131 steps for the PP4 answer at 184,134 tokens.
• Outputs: the change reads the same bytes (unit tests, offline SHA-256). On PP4 the greedy texts are word-identical in all five boots of both arms. The long-context and quality answers are the same in every boot of an arm but differ between the arms (all six long-context answers; three of eight quality answers on PP4, five of eight on TP2xPP2), and on TP2xPP2 the greedy texts of the fix arm differ between its boots. This A/B still had the stale resident rows fixed in [Bugfix][Qwen4Exp] Decide the PLE cascade merge when the layer is built #1234 (the forward pass took the resident rows from the offload worker's raw buffer: zeros or rows of earlier chunks and requests), and the two arms ran different compile caches. With [Bugfix][Qwen4Exp] Decide the PLE cascade merge when the layer is built #1234 on top (our fork commit 6bbcd40), three boots per topology give byte-identical greedy texts and long-context answers. Greedy texts, long-context answers and quality answers were read by hand; both arms pass the same quality prompts.

🤖 Generated with Claude Code

https://claude.ai/code/session_01YaG8VjXn5eN3Aq7eZrA3dp