feat(infra): add opt-in sample-level rollout refill

[dingzhiqiang]

Description

Release rollout capacity as individual episodes finish, so new prompts can start while earlier groups still contain stragglers. rollout.max_concurrent_samples opts into sample-level admission; leaving it unset preserves group-level admission. Groups still complete and pass the existing acceptance rules before being consumed, and the group-based staleness budget remains enforced.

For example, with group size 8 and 128 sample slots, eight completed episodes across unfinished groups can admit another eight-member group without waiting for one original group to finish. Reservations use attempt identities and idempotent progress notifications; uncertain remote submission does not release capacity or retry work as if it had stopped. The change covers local/remote v1 dispatch and v2 member completion, with English/Chinese documentation. Benchmark scripts and experiment reports are intentionally outside this PR; measurements are summarized below.

Related Issue

No public issue linked.

Type of Change

• ⚡ Performance improvement

Validation

• Full pre-commit run --all-files through the persistent cached container.
• pytest tests/test_sample_level_refill.py tests/test_deterministic_sampling.py -q — 47 passed (focused refill tests plus existing deterministic-sampling regressions). Only one new test file is included: 12 focused test functions for refill and safety invariants.
• Tests cover sample reservation limits, duplicate/stale progress, uncertain remote submission, rejection/cleanup, and batch delivery while original groups still have stragglers, including pause/resume.
• Real single-node GPU inference pilots below. Multi-node integration and actual weight-update training validation were not run for this change. Documentation text and generated CLI references were checked; a full docs site build was not run.

Performance evidence

Both pilots use eight NVIDIA L20X GPUs, frozen weights, a three-batch prompt window, and complete-group consumption. They are single paired runs, baseline first, not repeated estimates or training convergence results. Experiment contracts and limitations are provided below; no benchmark code or reports are added to the repository.

SWE-bench Verified: B16 × group8, 128 sample slots

Qwen3-4B-Instruct-2507; eight TP1 SGLang replicas. Ten batches consume 160 accepted groups / 1,280 episodes. Steady metrics cover the interval from the first batch ready to the tenth batch ready.

Metric	Group admission	Sample refill	Change
Ten batches ready	127.29 min	101.24 min	-20.5%
Steady window duration	117.05 min	90.68 min	-22.5%
Steady accepted groups/hour	73.82	95.28	+29.1%
Median batch interval	625.64 s	566.63 s	-9.4%
SM Active	35.67%	50.07%	+14.40 percentage points
NVML GPU utilization	68.99%	86.31%	+17.32 percentage points
Average active episodes	55.62	121.67	Same 128-slot maximum

First-batch latency was slightly worse (10.24 → 10.55 min). Completion-order selection changes the consumed prompt set (144/160 common prompts). Measurement-period rejected groups were 1/161 versus 3/163; including drain, 1/193 versus 9/195. Drain took 54.01 versus 53.49 min. Consumed episodes included 63 versus 73 harness failures, and reward-one rates were 13.13% versus 14.38%; these do not establish quality equivalence or improvement. No slow group was dropped to declare a measured batch ready.

Boba math: B16 × group8, 128 sample slots

Qwen3-4B thinking, maximum 16,384 new tokens, temperature 0.6, top-p 0.95, top-k 20, SGLang memory fraction 0.85. Six batches per mode; each ready batch pauses real inference for a fixed ten seconds to simulate a training interval, then resumes unfinished work. No training engine or weight synchronization is involved. Each mode consumes 96 groups and drains all 128 submitted groups / 1,024 samples.

Metric	Group admission	Sample refill	Change
Mean rollout interval, batches 2–6	105.20 s	95.13 s	-9.6%
Mean cycle including simulated training and switching	116.22 s	106.15 s	-8.7%
Steady accepted-group throughput	baseline	1.106× baseline	+10.6%
Complete-candidate time, including tail	baseline	0.958× baseline	-4.2%
Cycle-window SM Active	55.13%	55.32%	+0.19 percentage points
Mean output tokens, all candidates	10,729.01	10,573.39	-1.45%

Both runs rejected zero groups; no reward-based filtering was enabled. This configuration shows a time benefit with essentially unchanged SM Active. It is not a universal improvement: Boba B8/group8 at the same concurrency had 54.35 → 55.41 s steady rollout time (1.9% slower). Output-length differences and a single execution order limit causal precision.

GPU means are time-weighted across all eight GPUs. SWE measures continuous rolling consumption; Boba cycle metrics include simulated pauses after the first batch. DCGM was polled every second but source update timestamps were unavailable, so short-window hardware-counter changes should not be overinterpreted. The pilots do not prove actual colocated memory handoff, weight-version correctness, or downstream learning quality.

Controlled-length inference: isolating within-group stragglers

A paired synthetic-work pilot uses real Boba prompts and Qwen3-4B GPU decoding,
B=16, group size 8, 128 sample slots, a 48-prompt rolling window, and the same
10-second simulated training pause after each batch. Weights remain fixed. EOS
and stop conditions are disabled for this test; each completed episode must produce
exactly its assigned token count. This tests scheduling performance, not answer quality.

The within-group long-tail profile assigns seven 1,024-token episodes and one
16,384-token episode per prompt. The long member's index rotates by prompt ID.
Both modes complete 128 prompt groups / 1,024 episodes and exactly 3,014,656
output tokens, with zero rejected groups. The collector consumes six batches
(96 complete groups), then drains all remaining work. Audits verify identical
prompt manifests, unique episode identities, and every episode's actual length.

Metric	Group admission	Sample refill	Change
Mean rollout interval, batches 2–6	73.99 s	26.12 s	-64.7%
All 128 groups complete, including pauses and tail drain	733.13 s	453.93 s	-38.1%
Cycle-window SM Active, after first batch	44.92%	42.20%	-2.72 percentage points

The equal-length control assigns eight 4,096-token episodes per group. Both modes
produce exactly 4,194,304 output tokens: mean rollout time is 21.82 → 21.86 s,
and all-work completion is 315.95 → 316.22 s—essentially unchanged.

This supports the intended mechanism: sample completion frees admission capacity
while original groups retain their long members; later complete groups can become
available sooner. It does not show that SM Active must increase. Both modes
already have approximately 100% NVML GPU utilization during generation, and the
mix of GPU work and fixed pauses changes with the scheduling policy. Equal logical
output-token counts also do not imply identical recomputed prefill work after pauses.
These are single baseline-first paired runs, not repeated speedup estimates or an
end-to-end RL convergence result. Group-between and mixed-tail profiles are still
being evaluated and are not included in this claim.

Risks

Sample-level admission can accumulate more partially finished groups and change completion order. It preserves group consumption rules but may change which prompts enter a batch and increases the importance of staleness and host-memory monitoring. The default remains unchanged. Remote capacity is retained when termination is uncertain, favoring bounded concurrency over speculative retries.

Checklist

• I have read the Contributing Guide
• Pre-commit hooks pass (pre-commit run --all-files)
• Relevant tests pass; new tests added for new functionality
• Documentation updated (full site build not run)
• Branch is up to date with main
• Self-reviewed via /review-pr command
• This PR was created by a coding agent via /create-pr
• This PR is a breaking change