[hadihonarvar]

Overview

On Radeon RDNA3 (RX 7900 XTX, gfx1100), the rocm/vllm:rocm7.14.1_rdna_ubuntu24.04_py3.14_pytorch_2.11_vllm_0.23.0 image answers garbage for Qwen3 models from the first or second request onward. The cause is one non_blocking host-to-device copy from pageable memory in async_copy_to_gpu. This PR stages that copy through pinned memory, which is what upstream main has done since vllm-project#56888. That change landed two weeks after this branch was cut.

Claims

• Fixes wrong output for Qwen3 (1.7B and 8B checked) on gfx1100 with this release's default engine (Model Runner V2).
• The change is in one function, with no API change, and covers all five callers of async_copy_to_gpu (model_runner.py ×3, pp_utils.py, structured_outputs.py).
• No effect on models that were already correct (Qwen2.5 1.5B and 7B: identical output before and after).

Validation

Repro. Run on the published image above, unmodified, with one RX 7900 XTX:

# repro.py: same prompt six times at temperature 0, so every answer should be identical
from vllm import LLM, SamplingParams

if __name__ == "__main__":
    llm = LLM("Qwen/Qwen3-1.7B", max_model_len=2048, gpu_memory_utilization=0.6)
    messages = [{"role": "user", "content": "What is the capital of France? Answer in one word, then name its river."}]
    params = SamplingParams(temperature=0, max_tokens=24)
    for i in range(6):
        out = llm.chat(messages, params, chat_template_kwargs={"enable_thinking": False}, use_tqdm=False)
        print(i, repr(out[0].outputs[0].text), flush=True)

Run (published image, default engine)	Before	After this PR
Qwen3-1.7B, run 1	0/6 correct: ': 1.0.0.0\n: 1.0.0.1…', then '1111111111…' ×5	6/6 'Paris, Seine.'
Qwen3-1.7B, run 2	0/6 correct (same output)	6/6 'Paris, Seine.'
Qwen3-8B (gpu_memory_utilization=0.88), 2 runs	first answer right at best, then ' of of of …'	12/12 'Paris. Seine.'
Qwen2.5-7B-Instruct	6/6 'Paris\nSeine'	6/6 'Paris\nSeine' (unchanged)

CI coverage. No CI job runs gfx1100, so this PR adds no test. All validation above was run by hand on an RX 7900 XTX (ROCm 7.14.1, torch 2.11.0+rocm7.14.1, HIP 7.14.60850). ruff check and ruff format pass on the changed file.

Details

How the copy was found (each step on the same card and image)

1. Not the model, card or kernels. Qwen2.5 1.5B/7B are correct on the same card; only Qwen3 fails. Turning off prefix caching, CUDA graphs (--enforce-eager), the custom RMSNorm/RoPE ops, switching ROCM_ATTN↔TRITON_ATTN, fp16↔bf16, and hipBLASLt↔rocBLAS changed nothing.
2. A host/device ordering problem. AMD_SERIALIZE_KERNEL=3, HIP_LAUNCH_BLOCKING=1 and AMD_SERIALIZE_COPY=3 each make it 4/4 correct. Serializing only the copies is enough.
3. Host-to-device copies. Forcing every non_blocking copy to blocking fixes it; forcing only device-to-host copies does not; forcing only host-to-device copies does.
4. One call site. Forcing each host-to-device site to blocking in turn: only model_runner.py:879, async_copy_to_gpu(query_start_loc_np, out=self.input_buffers.query_start_loc), fixes it. query_start_loc_np is a fresh pageable np.empty.
5. What arrives. A wrapper that snapshots out right after each copy, and again after torch.cuda.synchronize(), shows 5 to 14 copies per request at that site where elements 1..15 of the 257 hold stale zeros instead of 29 (one 29-token request). They are wrong in both snapshots, so the data never lands; it is not merely late. The host source is unchanged afterwards. Every other copy is correct.

Root cause, as far as it is isolated. async_copy_to_gpu issues out.copy_(x, non_blocking=True) straight from pageable memory. The comment it replaces says pinning was removed on purpose. On this ROCm stack, that copy intermittently fails to deliver its first bytes while the model runner overlaps steps. I could not reproduce the loss in a standalone PyTorch loop (pageable or pinned sources, default or side stream, 16 B to 256 KiB, 0/64-byte alignment, a concurrent device-to-host copy on another stream, 1/3/8 copies back to back: all 0 failures). So the exact HIP-side mechanism is not isolated here and may be worth a separate look by the HIP team. Within vLLM the failing operation is pinned down to this one copy.

The fix. If the source is pageable, copy it into a pinned tensor (torch.empty(..., pin_memory=True).copy_(x)), then issue the non_blocking copy. The staging tensor comes from PyTorch's caching host allocator, which keeps it alive until the copy completes. Upstream replaced this function with async_tensor_h2d in vllm-project#56888, and that does exactly this. The removed comment worried about "sporadic stalls … due to CUDA driver contention" from explicit pin_memory(). This path does not call cudaHostAlloc per copy: it reuses blocks from the caching allocator, which is what main ships. The added host work is one memcpy of at most max_num_reqs + 1 int32 per step.

Why a backport rather than a rebuild. The RDNA images for v0.23.0 come from this branch, and vLLM v0.30.0+ already contains the fix. Either merging this, or publishing an RDNA image built from a release that contains vllm-project#56888, resolves it. Tracking issue: #1404.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

Contact: Hadi Honarvar, hadi@opod.io or hadi.work.ca@gmail.com (https://opod.io)

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors.

You ask your reviewers to trigger select CI tests on top of fastcheck CI.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

🚀