Skip to content

fix(ple): add rocm disk wait-sync support - #776

Open
LZ-QWQ wants to merge 1 commit into
FlashML-org:mainfrom
LZ-QWQ:fix/rocm-ple-wait-sync-review-20261020
Open

LZ-QWQ wants to merge 1 commit into
FlashML-org:mainfrom
LZ-QWQ:fix/rocm-ple-wait-sync-review-20261020

Conversation

@LZ-QWQ

@LZ-QWQ LZ-QWQ commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Related: #697

ROCm disk PLE currently uses gate fallback. This PR adds HIP wait-sync and prevents deadlocks when submitting large graphs with wait-sync.

It adds HIP signal memory, stream memops and capture probing (auto probes; wait requires support). Use HIP signal memory for Engram's wait-sync flag.

#697 uses early cuGraphUpload to avoid the first-launch deadlock on CUDA. In our ROCm 10.1.0 large-graph checks, hipGraphUpload followed by stream synchronization did not prevent the deadlock. This may be related to the current HIP upload implementation: ihipGraphUpload returns after handle validation and stream resolution, while hipGraphUpload still marks preparatory work as TODO. To avoid this deadlock, this PR starts a HIP-only CPU worker before replay to fill PLE rows and signal the flag independently of graph submission.

CUDA's existing allocation, execution and synchronization logic is unchanged.

Validation

The baseline uses main's unchanged production code, with the new 16,384-kernel test included in the test suite.

Test files:

  • tests/kernels/test_row_store.py
  • tests/models/qwen4_exp/test_ple_disk.py
  • tests/models/deepseek_v41/test_engram.py
GPU Runtime Main + test PR
RTX4090 CUDA 13.0 18 passed 18 passed
W7900D ROCm 10.1.0 15 passed, 3 skipped 18 passed, 0 skipped
R9700_x8 ROCm 10.1.0 15 passed, 3 skipped 18 passed, 0 skipped

The new regression and two existing Engram worker cases now run on ROCm.

Performance reference

Qwen/Qwen3.8-Flash-Next-FP8, 256 input / 128 output tokens, single GPU/request, MoE offload cache 1024, graph batch 1, ROCm 10.1.0. Compare main/gate, PR/gate and PR/wait under identical settings.

GPU Metric Main/gate PR/gate PR/wait Wait vs PR gate
R9700_x8 Decode tok/s 17.190 17.172 19.365 +12.77%
R9700_x8 Request (s) 10.497 10.465 9.562 -8.63%
W7900D Decode tok/s 12.676 12.665 12.780 +0.91%
W7900D Request (s) 14.529 14.542 14.445 -0.67%

Each configuration was tested in two independent runs, with three measurements after warmup per run. The table reports the average of the two run medians.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant