Finite replay cycles drop queued requests

[jay-tau]

Summary

When --replay-cycles completes, the producer sets the shared stop event before workers drain the queue. The benchmark exits successfully with incomplete results.

Reproduced locally on commit 7d3a14f with Python 3.14.6.

Reproduction

{
  printf '%s\n' 'Timestamp,Model,Request tokens,Response tokens,Total tokens,Log Type'
  for i in $(seq 0 11); do
    printf '%s,ChatGPT,100,20,120,Conversation log\n' "$i"
  done
} > /tmp/BurstGPT_repro.csv

mkdir -p /tmp/kv-cache-repro-cache

python3 kv-cache.py \
  --model tiny-1b \
  --use-burst-trace \
  --burst-trace-path /tmp/BurstGPT_repro.csv \
  --trace-speedup 0 \
  --replay-cycles 1 \
  --num-users 8 \
  --duration 60 \
  --max-requests 0 \
  --gpu-mem-gb 0 \
  --cpu-mem-gb 0 \
  --generation-mode none \
  --disable-multi-turn \
  --disable-prefix-caching \
  --prefill-only \
  --cache-dir /tmp/kv-cache-repro-cache \
  --seed 42

--trace-speedup 0 removes trace pacing; --max-requests 0 leaves duration as the hard cutoff.

Observed in this run:

Completed 1 replay cycle(s). Trace total_tokens sum: 1,440
Requests Completed: 1

Expected 12 completed requests. The process exited with status 0. The exact shortfall may vary because shutdown races with worker scheduling.

Cause

After enqueueing the final cycle, the producer sets stop_event. Workers then exit without draining queued requests.

Expected behavior

Finite replay completion should stop production and drain queued requests. Duration and request limits should retain hard-stop behavior.