A field report of two distinct bugs we hit bringing MiniMax‑M3 (MXFP8) up in a 2‑prefill / 2‑decode (2P2D) disaggregated configuration on AMD ROCm MI300, using vLLM with the MoRIIO KV‑transfer connector and mori for MoE all‑to‑all. For each bug: the symptom, how we debugged it, what we found (with a block diagram and a small math example), and the fix (or current status).
TL;DR
- Bug 1 (SOLVED): prefill's KV write to decode never fired because the "deferred write" expired after 60 s before decode's block addresses arrived. Fix: raise
defer_timeoutto 1800 s. KV now transfers 117/117 layers.- Bug 2 (OPEN): once KV flows, decode freezes after ~24 tokens on a GPU kernel that never returns (
cudaStreamSynchronize). Root cause is a data‑parallel collective divergence (same family as vLLM #43547). The known one‑line fix PR #44601 is insufficient for MM3 — proven end‑to‑end.
PREFILL tier (2 nodes) DECODE tier (2 nodes)
┌───────────────────────┐ ┌───────────────────────┐
│ vLLM prefill DP×8 │ KV over │ vLLM decode DP×8 │
request ─► computes KV cache │───RDMA────► │ generates tokens │──► tokens
│ (MoRIIO producer) │ (MoRIIO) │ (MoRIIO consumer) │
└───────────────────────┘ └───────────────────────┘
│ │
└──────── toy proxy (registers P/D, routes requests) ───────┘
Model: MiniMax‑M3, hybrid — 60 full‑attention layers + 57 linear/"lightning"
(Mamba‑style recurrent state) layers + MoE experts. = 117 KV layers.
Parallelism: data‑parallel 16 (8 per decode node, across 2 nodes) + expert‑parallel.
Backend: MultiprocExecutor (mp), enforce_eager (no CUDA graph).
Hardware: AMD MI300, ROCm, AITER kernels, mori all2all.
Two independent things must work: (1) the KV cache must physically arrive at decode, and (2) decode must then run its forward pass. We hit one bug in each.
Decode produced 0 tokens; GSM8K ≈ 0. No crash — decode just sat waiting for a
KV cache that never came, then aborted the requests (FINISHED_ABORTED).
In MoRIIO WRITE mode the prefill side is the active writer: it computes the KV cache and then RDMA‑writes it directly into decode's memory. But prefill can't write until it knows where — it needs decode's block addresses. So the write is deferred: queued on prefill until decode sends a "remote_blocks" notify that fills in the destination addresses.
PREFILL DECODE
─────── ──────
1. compute KV ✔
2. queue a DEFERRED write ⏳ (dest unknown yet)
│
│ ◄──── 3. "remote_blocks" notify (my block addrs)
4. addresses arrive → fire RDMA write ──────────► 5. KV lands → start decoding
6. send write_done ───────────────────────────► 6. write_done received ✔
The gate prefill waits on is a dictionary keyed by transfer id:
done_remote_allocate_req_dict[transfer_id]. It gets populated when decode's
notify (step 3) is received. Until then, the write stays queued.
We built an instrumentation overlay (moriio_overlay_dp16fix_instr) that bind‑mounts
patched moriio_connector.py + moriio_engine.py with MORIIODBG log lines across
the entire KV lifecycle:
decode: NOTIFY-BLOCKS (nblocks, tokens) ← did decode send its addresses?
prefill: MSG (dispatch type) ← did prefill receive a message?
prefill: RBLOCKS-RECV / EMPTY / TERMINAL-DROP / STORED ← what happened on receipt?
prefill: WRITE-DONE-SENT / WRITE-FAIL / WRITE-NOBLOCKS ← did the write fire?
The failing runs (defer_timeout=60) showed this exact signature:
prefill: "Deferred write task for request …-0-… expired after 60.0s
(remote blocks never arrived), marking done"
prefill: "Reaped N deferred sends with no finished_sending notification …
lost async KV completion notifications"
prefill: "Could not find tx-… in transfer_id_to_request_id lookup table"
decode : MORIIODBG XFER-TABLE-SWAP size=0 (destination table never populated)
decode : status=FINISHED_ABORTED (gave up waiting)
Then the key clue: a retry‑wave notify from decode arrived ~18 s after the 60 s
expiry — i.e. the address did eventually come, just too late, and prefill had
already marked the transfer terminal and dropped the late notify at
moriio_engine.py:784 (_is_transfer_terminal_locked → return).
defer_timeout defaulted to 60 s (set in MoRIIO: DEFAULT_DEFER_TIMEOUT in
moriio_common.py, consumed by the deferred‑write poller in moriio_engine.py).
For MM3 2P2D the decode side admits requests and sends its block addresses tens of
seconds late relative to prefill (prefill computes KV fast; decode's per‑request
notify lags, and retry waves land even later). So the sequence was:
✗ THE BUG (defer_timeout = 60s)
────────────────────────────────
t=0s prefill queues deferred write (waiting for decode addresses)
t=60s timeout → prefill marks transfer TERMINAL, discards the pending write
t=78s decode's remote_blocks notify FINALLY arrives
→ hits the "already terminal" guard → DROPPED
→ done_remote_allocate_req_dict never populated
→ write never fires → no write_done → decode aborts → 0 tokens
Think of it as a deadline mismatch:
Prefill's patience window : [ 0s ─────────── 60s ] (defer_timeout)
Decode's address arrival : ▲ 78s
└ 18s TOO LATE
The address arrives at 78 s, the window closes at 60 s. 78 > 60, so the
valid‑but‑late message is thrown away. There is nothing wrong with the data — it's
purely that the timeout (60) < arrival latency (78). Widen the window past the
worst‑case arrival and the message lands in time.
✔ THE FIX (defer_timeout = 1800s)
Prefill's patience window : [ 0s ───────────────────────────── 1800s ]
Decode's address arrival : ▲ 78s ← now comfortably inside → write fires
Set defer_timeout = 1800 (30 min) so the deferred write waits long enough for
decode's late notify to arrive and populate the destination table. It is a
kv_connector_extra_config key (env vars VLLM_MORIIO_* are deprecated/ignored):
# delivered via a knob in vllm_disagg.sh moriio_tuning_kv(); then at submit:
export MORIIO_DEFER_TIMEOUT=1800Zero correctness risk — it transfers all layers, it just waits longer.
prefill: MORIIODBG RBLOCKS-RECV → RBLOCKS-STORED (table populated — never happened before)
prefill: MORIIODBG WRITE-DONE-SENT writes=117/117 (ALL layers transferred:
60 full‑attn + 57 linear/Mamba)
prefill: 0 WRITE-FAIL, 0 WRITE-NOBLOCKS
decode : write_done received → decode STARTS generating
writes=117/117 matters: it proves the 57 Mamba/linear "state" layers also
transfer. Those carry the compressed prompt context for ~half the network — a
tempting "fix" of skipping them to dodge a problem would produce a run that
completes with silently wrong answers. Never skip them.
KV transfer is fully fixed. The run then immediately hit Bug 2.
With KV now flowing, decode generates ~23–24 tokens per request and then freezes
forever. Other ranks spam No available shared memory broadcast block found in 60 seconds for ~25 minutes, then the engine dies with
TimeoutError: RPC call to sample_tokens timed out.
py‑spy and gdb aren't in the container, so we captured the stack with
srun --overlap onto the decode node + docker exec + kill -ABRT <worker_pid>
(and PYTHONFAULTHANDLER). The stuck stack was unambiguous:
decode worker (async output thread):
torch/cuda/streams.py:254 synchronize() ← waits on a CUDA event
← gpu_model_runner.py get_output
← multiproc_executor async_output_busy_loop
decode worker (main thread):
shm_broadcast.py:198 dequeue (idle, waiting for the above to finish)
So a GPU kernel launched during the decode forward never completes;
cudaStreamSynchronize never returns; everything else jams behind it. We confirmed
the workers were busy‑spinning at 96–199 % CPU (not idle), and the NCCL flight
recorder produced no dump even 44 min past its 20 min timeout — i.e. it is not
an NCCL collective timing out, it's a kernel that hangs.
We isolated it with a 2×2 matrix:
│ intra‑node DP8 │ cross‑node DP16
────────────────────┼───────────────────────┼──────────────────────
DSV3 (MLA attn) │ — │ ✔ 0.948 (no hang)
MM3 1P1D (sparse) │ ✔ 0.91 (no hang) │ —
MM3 2P2D (sparse) │ — │ ⛔ HANG
Only the cell with both MM3's sparse/"lightning" attention and cross‑node DP16 hangs. So the trigger is the interaction of MM3's data‑dependent attention routing with the cross‑node data‑parallel collective.
In data‑parallel MoE, the 16 DP ranks run in lockstep. Each decode step, before
the experts run, all ranks execute a small all‑reduce to agree how many tokens
each holds (coordinate_batch_across_dp), then an all‑to‑all to route tokens to
experts. Both are collectives: they complete only when all 16 ranks call the
same one. If a single rank takes a different path and skips (or resizes) the
collective, the other 15 wait forever.
NORMAL STEP THE BUG
─────────── ───────
Rank0 ─┐ Rank0..14 ─► post all_reduce(N) ─► WAIT ⏳
Rank1 ─┤ all_reduce(N) │ (need all 16)
... ├──► sum ──► all2all ──► next Rank15 ─► different branch X never posts N
Rank15 ┘ ▲ │
└ done only when all 16 post all_reduce kernel spins forever
= cudaStreamSynchronize never returns
= shm_broadcast stall → RPC timeout
For MM3, the divergent branch is data‑dependent: the sparse/lightning‑attention router makes per‑rank decisions that depend on the actual token values, so most steps are fine and then one particular batch makes a single rank drop out → freeze. That is exactly why it runs for ~24 tokens and then locks.
All‑reduce is just "everyone adds their number, everyone gets the total" — and it's a barrier, so it can't return the total until every contribution is in.
Round 5, healthy (2 ranks shown):
Rank A contributes 3 ┐
├─ all_reduce SUM ─► total = 3 + 2 = 5 ─► both continue ✔
Rank B contributes 2 ┘
Round 5, the bug:
Rank A contributes 3 ─► computing 3 + ??? … waiting for B's number
Rank B — skipped round 5 (its router took a different path this step) —
│
total = 3 + ??? ─────────┘ B's number never arrives
▼
Rank A's GPU sits on "3 + ???" FOREVER ← the hang
3 + ??? can never resolve because the missing operand is a message from B that B
decided not to send. The kernel doesn't error — it spins (96 % SM, 0 % memory
bandwidth), and synchronize() never returns.
vLLM issue #43547 describes this exact class for MoE+DP, and PR #44601 fixes
one cause of divergence: a rank with zero tokens skipping its forward step.
It broadens the "run a dummy step anyway" rule from external_launcher backends to
all data_parallel_size > 1.
We delivered it via bind‑mount and verified it was genuinely applied (the mount echoed on all 4 ranks — a positive control we had missed on an earlier run). Result:
Run 210826: PR#44601 mounted ×4 ✔ → decode STILL froze at ~24 tokens ⛔
Conclusion: PR #44601 is insufficient for MM3 2P2D. The divergence is not merely the zero‑token case — it's the deeper, data‑dependent sparse‑attention × cross‑node all2all interaction, i.e. a kernel‑level deadlock, not a scheduling miss.
A deep search of the vLLM / ROCm‑mori issue trackers found no exact public match, but the failure class is well documented (umbrella RFC #48720; sibling MiniMax reports #46097, #41725; DP‑branch sibling #44185 / PR #44954). Ranked levers:
- ROCm DP flags (documented MI300 requirement):
VLLM_ALL2ALL_BACKEND=allgather_reducescatter+--disable-nccl-for-dp-synchronization--distributed-executor-backend mp. Moves DP sync off RCCL (which misbehaves cross‑node) — directly matches "cross‑node hangs / single‑node fine".
- all2all backend swap
mori_low_latency → mori_high_throughput(InterNodeV1LL, new, tuned for the small‑batch decode regime → InterNodeV1, older/proven). Issue #21306: naive all2all works where the optimized kernels hang. HIP_LAUNCH_BLOCKING=1— cheap diagnostic (mori #342): if the hang vanishes, it's an async dispatch/combine quiescence bug and (2) should help.- Verify PR #25906 ("Fix DP/EP hang", merged v0.11.0) is present — it fixed a
DP
BatchDescriptorusing unpadded per‑ranknum_tokens, a data‑dependent cross‑rank deadlock very close to ours.
⚠️ Correctness caution: mori #475 / #507 report that InterNodeV1 (thehigh_throughputtarget of fix 2) has cross‑node combine correctness bugs on some NICs → silently wrong outputs. After any backend swap, re‑verify the GSM8K score, don't just check that the hang cleared.
- Positive‑control discipline: an earlier run "had the fix mounted" — but the ×4 mount was a different overlay. Always prove the fix actually loaded (a marker in the logs) before concluding "the fix didn't work." Mount ≠ executed.
- Symptom ≠ cause: the loud
gloo Connection closed by peer [X]error named a victim rank that died 1 s after the real culprit (the master's RPC timeout). Order events by timestamp before blaming the named party. - Never skip the Mamba/linear layers to dodge a transfer problem — they carry recurrent prompt state; skipping them yields silently wrong answers.
- vLLM #43547 (MoE + DP zero‑token dispatch deadlock) · PR #44601 (partial fix)
- RFC #48720 (TP/DP step‑consensus guard — frames #44601 as one instance)
- #46097, #41725 (MiniMax‑M2.7 collective‑desync hang) · #44185 / PR #44954 (DP branch divergence)
- #21306 (all2all backend hang; naive works) · PR #25906 (Fix DP/EP hang)
- ROCm/mori #342 (async quiescence), #475/#507 (InterNodeV1 combine correctness)
- AMD ROCm vLLM optimization guide (DP all2all backend + disable‑nccl‑for‑dp guidance)