[yangzhuxinyzx]

Summary

This PR runs DeepSeek-V4.1-Flash-1017 on 4× V100-32G (54633) with expert offloading:

• GSQ-RCO IQ2_XXS GGUF (138 GB)
• Engram IQ4_NL (110.6 GB)
• DSpark5 draft

The hardware is TP4 behind a single x16 PCIe uplink at 12.2 GB/s, with 62 GB host RAM.

It adds the phase 2–3 work: prefill goals G1–G7, 512K context through lossless packed FP4 KV kept in host memory, and an OpenAI API launch with prefix cache, tool calls and thinking. Every new path has its own switch. The defaults are lossless.

The full handoff is benchmarks/deepseek_v41/HANDOFF.md. It covers launch, switches, results, dead ends and open items.

Changes

G1: activation memory

• MoE down output goes to a per-tile scratch plus an FP32 accumulator.
• expand_hc and large all-reduces run in place.
• hc_transition is windowed.
• all_gather goes through pynccl.
• empty_cache runs before big passes.
• Chunks go up to 16K.

G2: cold-expert staging ring

• Staging is issued ahead across tiles and layers.
• The host never blocks on a pinned row.

G3: routed down projection split by output rows (VLLM_V41_DOWN_ROWS)

• Removes the duplicated TP boundary blocks: −2.25 GiB per GPU and about 10% fewer PCIe bytes.
• Decode uses an IPC peer_gather kernel in v41k.

G4: long context

• Layers ≥ 1 keep their sliding window in 256-row rings.
• Packed KV (VLLM_V41_PACKED_KV):
  • Compressed latents use FP4 + E4M3 scales. They are stored once in a shared host memfd, plus a 32K GPU hot ring.
  • Index keys use FP4 + power-of-two scales and stay on the GPU.
  • Decode index scores run in a CUDA kernel, followed by a segment-parallel exact top-k.
• RoPE rotates on the fly past its table.
• The C2, Engram, guard and DSpark draft length limits now follow the target length.

G5: prefill compute

• 64 Engram reader threads.
• Adaptive indexer tile.
• Persistent-grid activation kernel.

G6: short prompts

• Prompts up to 1536 tokens stage only their routed experts.

G7: same-session prefix cache (VLLM_V41_SESSION_PREFIX)

• The scheduler accepts a hit only when the request continues the last executed sequence.
• The worker restore_prefix rewinds the Engram and C2 state.

Tooling (all under benchmarks/deepseek_v41/)

• v41k extension source plus build.py.
• serve-ds41-phase{2,3}.yaml.
• tools/run_serve{2,3}.sh and the validation tools.
• GPU micro-tests.

Configuration

• About 90 V4.1 switches are registered in vllm/envs.py. The generic modules read them through vllm.envs, so the layering ratchet does not grow.
• deepseek_v41_activations moved into the model package.
• Two boundary entries were added to tools/config_boundaries.py.

Launch

.venv/bin/python benchmarks/deepseek_v41/v41k/build.py            # once, inside ds41-compute
docker exec -d ds41-compute-20261024 bash benchmarks/deepseek_v41/tools/run_serve3.sh serve3
python3 benchmarks/deepseek_v41/tools/tcp_forward.py 8000 <container-ip> 18000   # 1cat-tunnel DFLASH2-API

Results (vs phase 1)

	Phase 1	This PR
TTFT short / 8K / 15K	5.6 / 27.7 / 44.0 s	0.8–1.1 / 5.8 / 11.1 s
TTFT 128K / 500K (packed)	n/a (16K max)	110 / 651 s, needle passes
Follow-up turn (prefix hit)	full prefill	0.9–1.7 s
Fresh-text decode	57.8 ms/round	53.5 ms (55.0 packed + 512K)
Steady VRAM	28.6 GiB	26.9 GiB

With packed KV, tokens at ≤16K are identical to the paged FP16 cache.

Test plan

• GPU micro-tests in benchmarks/deepseek_v41/tests/: kv_pack, packed attention, packed indexer, parallel top-k, window ring, G3 down, peer gather, G1 memory, RoPE tail. Each is bit-exact or equivalent.
• e2e 5/5 including the 16K check.
• Needle retrieval at 4K, 32K, 128K and 500K.
• API soak 80/80.
• Thinking, tool calls and multi-turn through the public API.
• Local hooks pass:
  • ruff check and format
  • SPDX
  • torch.cuda
  • forbidden imports
  • env registration and metadata
  • env reference
  • config inventory
  • layering
  • validate-config
  • lazy imports
• The API server was restarted on this exact tree and smoke-tested.
• Not run locally: clang-format, typos and mypy. The tools are not installed on 54633, and downloading them was avoided to save traffic.
• check-layering reports moe_runner.py platform coupling 37 → 39. This was already on main before this PR.

🤖 Generated with Claude Code