[Engine] STRATA_PREFILL_CPU_SHARE_BATCH=1 (opt-in): the prompt's CPU share with --batch, for a read while no slot decodes; short reads -21% to -31%

[Vop3p]

Problem and result

The prompt path's CPU share (STRATA_PREFILL_CPU_SHARE) is off whenever --batch is set: generate.cpp sets share_pool = o.batch <= 0 && !o.no_pool, so with "parallel": 2 the variable has no effect. The server runs with batch slots for concurrency, so a short read there always waits on the GPU alone: below the full-streaming chunk size (stream_all_min, here 2048) a chunk copies only its routed experts over PCIe, and on this machine that wait is about half of a 1-2K-token read.

With STRATA_PREFILL_CPU_SHARE_BATCH=1 the pool is handed to the prompt path as it is without batch slots, and before each read the serve loop checks the slots: a read while any slot is decoding takes no share (the slots' verify windows use the pool), a read with none does. The existing CPU-share rules then decide as before (which chunks, how many experts).

Result on 2x RX 6900 XT (below), 10 interleaved cold-start pairs: real-text reads under 2,048 tokens -28.8% (median), a 1,733-token one -20.9%, conversation continuations adding 455/781 tokens -30.7%, a 34K read -2.3% (its last chunk, 1,378 tokens); reads of 2,048 tokens and more unchanged. Without the variable the default path is byte-identical.

Related discussion

None found. I could not find a stated reason for batch <= 0 in the code or history; this keeps it as the default and only adds the opt-in.

Approach

• generate.cpp: share_pool = (o.batch <= 0 || share_batch) && !o.no_pool, with one line at load when the variable applies; a share_gate() before each read (in read_part's run and before the concurrent reader starts) calls Prefill::set_share_blocked(busy > 0) from the slots' active flags, and prints the decision for the first 4 reads.
• prefill.cpp: set_share_blocked() sets an atomic that the chunk's existing pool-hold test also checks (!g_share_blocked).
• The share still needs the CPU share itself enabled (STRATA_PREFILL_CPU_SHARE=auto or 1; on a layer split it is not on by default).

Validation

Environment: 2x RX 6900 XT (gfx1030, 16 GB, PCIe 4.0 x8 each), Ryzen 5 5600X (6 cores), 128 GB DDR4-3200, Ubuntu 26.04 (kernel 7.0.0), ROCm 10.0.0. Revision: main e88ee39 (v0.2.0-beta + 12 commits) and this branch on it. Built with setup.py's HIP options (-DSTRATA_PREFILL_MMQ=ON -DSTRATA_Q6K_EXPERTS=ON -DSTRATA_MMQ_KQUANTS=ON, gfx1030, the pinned ggml, Release). Model: Qwen3.8-Flash-Next GSQ-RCO IQ3_S, layer split over both cards.

Passed:

1. Default path byte-identical: deterministic config (parallel 1, no pipeline windows, STRATA_PREFILL_CPU_SHARE=0, STRATA_LOGPOS top-256 log-probabilities), 8 chats of real text, 4,067 scored positions: main and this branch without the variable cmp identical.

2. Speed: parallel 2 (--batch 2), --pipeline-windows 2 --batch-mtp, STRATA_PREFILL_CPU_SHARE=auto, STRATA_PREFILL_STREAM_MIN=2048, STRATA_HC_PERSIST=1, STRATA_HIP_PROMPT_F16=1, STRATA_SH_STREAM=1, a rocBLAS table via STRATA_ROCBLAS_TUNING. The same binary with and without the variable, 20 cold starts alternating A B A B; each arm sends the same 33 requests one after another (so no slot is decoding during a read): 27 single real-text prompts of 430-3,521 tokens (English prose, Chinese prose, C++), then a 34K-token conversation and 5 continuations of it adding 455-3,168 tokens. prompt_ms from /metrics. Median over the 10 pairs of each group's mean change:

   reads	median change	faster
   15 single prompts under 2,048 tokens (430-1,542)	-28.8% (pairs -28.6 to -29.0)	10/10
   1 single prompt of 1,733 tokens	-20.9%	10/10
   11 single prompts of 2,048+ tokens	-0.0% (pairs -0.2 to +0.1)	-
   34,146-token first read (4 chunks of 8,192 + one of 1,378)	-2.3%	10/10
   2 continuations adding 455 / 781 tokens	-30.7%	10/10
   3 continuations adding 2,136-3,168 tokens	+0.0%	-

   e.g. a 430-token Chinese prompt 1,722 -> 1,146 ms, a 455-token continuation 1,722 -> 1,168 ms. The =1 arms printed CPU share for this read: on (0 slot(s) decoding), the others no such line; 0 stalls.

3. ctest (STRATA_BUILD_TESTS=ON; the engine binary is the same one, sha256 unchanged): 137 tests, 131 passed. The 6 failures are the same 6 on main e88ee39 built the same way: ple_parity (needs a Q2_0 GGUF this machine does not have), hip_prefill_mmq_parity (SEGFAULT on main too), native_hadamard_test, tq_cpu_parity, tq_cpu_parity_avx2 ("this ggml has no trellis types"), isa_guard_check (tq_rows_avx512 not listed). The branch adds no test: the change is a scheduling condition; item 2 checks it end to end (the per-read confirmation line).

4. CUDA compile only (no NVIDIA GPU here): CUDA 13.2.2 (nvcc 13.2.86) with GCC 15.2, -DSTRATA_ENABLE_CUDA=ON -DSTRATA_Q6K_EXPERTS=ON -DSTRATA_MMQ_KQUANTS=ON -DCMAKE_CUDA_ARCHITECTURES=86: the engine builds; the one warning in generate.cpp (line 11576, an unused variable) is there on main too, none from the changed lines.

Failed: none observed.

Not run:

• Reads beside a decoding slot on this revision: by design they take no share; I measured that two concurrent requests are unchanged on 0.1.45 with this patch, not on 0.2.0.
• Output quality with the share on: the CPU computes some experts, so the bits differ from the GPU-only read and depend on timing, as with the CPU share without batch slots today; I did not measure KL for it on this revision.
• CUDA execution (compiled only, item 4), the SYCL and Metal builds (no toolkits here; SYCL builds its own copy of prefill.cpp, Metal compiles this one, and the new Prefill::set_share_blocked is defined in it, so no stub is needed - not verified by a Mac build), a single-GPU setup, Windows, other GPUs.

Limits

• Opt-in; it only lifts the batch <= 0 condition, the CPU-share rules themselves are unchanged.
• The gate is taken when a read starts: a slot that begins decoding during a long read does not take the pool back from it.
• The gain is for reads below the streaming chunk size on a machine where those wait on PCIe (here 2x x8, 45% of the experts in VRAM); a box with the experts resident would see little.

Developed with an AI coding assistant; every number above was measured on the machine described here.

🤖 Generated with Claude Code