Repository navigation
[https://nvbugs/6501404][fix] Unwaive test_row_linear_norm_fusion[2-hidden:16-seqlen:2] - #16911
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. ℹ️ Recent review info
Walkthrough
ChangesSymmetric all-reduce output handling
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested reviewers: Pre-merge checks:
 |
| Check name | Status | Explanation |
|---|---|---|
| Linked Issues check |  |
Check skipped because no linked issues were found for this pull request. |
| Out of Scope Changes check | |
Check skipped because no linked issues were found for this pull request. |
| Title check | |
The title uses the required NVBugs and fix format and accurately identifies the waiver removal. It does not mention the primary NCCL output-allocation fix, but it describes a real part of the changese… |
| Description check | |
The description is detailed and explains the issue, solution, affected behavior, test coverage, test plan, links, and review status. It does not reproduce the PR checklist, but the core required techn… |
Full details: Title check
-
Explanation
The title uses the required NVBugs and fix format and accurately identifies the waiver removal. It does not mention the primary NCCL output-allocation fix, but it describes a real part of the changeset.
Full details: Description check
-
Explanation
The description is detailed and explains the issue, solution, affected behavior, test coverage, test plan, links, and review status. It does not reproduce the PR checklist, but the core required technical and testing information is present.
✨ Finishing Touches: 💡 1
-
⚔️ Resolve merge conflicts: 💡
-
✅ Conflict resolution request accepted.
❌ Error resolving conflicts.- Resolve merge conflict in branch
repair-bot-bug6501404
- Resolve merge conflict in branch
🧪 Generate unit tests (beta)
-
- Create PR with unit tests
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Comment @coderabbitai help to get the list of available commands.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
cpp/tensorrt_llm/thop/allreduceOp.cpp (1)
564-569: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueMake the window allocation result
const.
windowOutputandwindowBuffer1are never reassigned.Proposed fix
- auto [windowOutput, windowBuffer1] = createNCCLWindowTensor(rawComm, input.sizes(), input.scalar_type()); + auto const [windowOutput, windowBuffer1] + = createNCCLWindowTensor(rawComm, input.sizes(), input.scalar_type());As per coding guidelines, “declare unmodified variables
const.”🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@cpp/tensorrt_llm/thop/allreduceOp.cpp` around lines 564 - 569, Declare the structured-binding variables windowOutput and windowBuffer1 as const in the createNCCLWindowTensor result within the surrounding allreduce operation, preserving the existing validity check and outputTensor assignment.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@cpp/tensorrt_llm/thop/allreduceOp.cpp`:
- Around line 564-569: Declare the structured-binding variables windowOutput and
windowBuffer1 as const in the createNCCLWindowTensor result within the
surrounding allreduce operation, preserving the existing validity check and
outputTensor assignment.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 99f0fbf3-1e74-44af-92b9-b0dbc4a1acd8
📒 Files selected for processing (1)
cpp/tensorrt_llm/thop/allreduceOp.cpp
f74413d to
2593d10
Compare
brnguyen2
left a comment
brnguyen2
left a comment
There was a problem hiding this comment.
Making the output allocation follow the same gate as the input is the right consistency fix regardless of the bug — with a 64-byte tensor and a ~290 KB threshold at 2 ranks, the old code skipped registration for the input and then ran a full collective allocateAndRegisterBuffer for the output, which is both inconsistent and wasteful.
What I don't follow is the causal story. The failure in nvbugs/6501404 was on 2×H100, where NVLink is present, so minRegistrationThreshold is never SIZE_MAX on that path and the mechanism the new comment describes never fires there. allocateAndRegisterBuffer is also written specifically so every rank reaches the min-allreduce even when ncclMemAlloc fails asymmetrically. So this change plausibly removes a collective from the hot path, but it isn't shown to remove the one that hung — and the linked bug records that the failure could not be reproduced.
So I'd land the gating change on its own merits and keep the waiver until there's evidence the hang is actually gone. A concrete way to get that evidence without holding up this PR: open a separate draft PR that (a) removes the waiver, (b) adds instrumentation around the symmetric allreduce path (log per-rank windowBuffer0.isValid(), bufferSizeBytes, minRegistrationThreshold, and entry/exit of allocateAndRegisterBuffer), and (c) runs only the failing stage with the test list trimmed to test_row_linear_norm_fusion (and neighbors if needed) so repeated runs are cheap on capacity and turnaround. Re-run that until the hang reproduces; the instrumentation then tells you which rank diverged and why. Once you have a pre-fix hang and a post-fix pass on the same stage, dropping the waiver here is easy to justify.
|
/bot run |
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
|
/bot kill |
|
/bot run |
|
PR_Github #64957 [ run ] triggered by Bot. Commit: |
|
PR_Github #64957 [ run ] completed with state
|
|
/bot run |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
a7161f4 to
4749fee
Compare
4749fee to
18b7cab
Compare
|
Bug engineer ownership @kris1025, as the engineer assigned to the linked bug, you are responsible for driving this bot PR to resolution. Please coordinate review, address feedback and required code changes, get CI passing, and verify the original failure before driving it to merge. If the PR is obsolete or superseded, close it with a reason. Keep this PR updated with blockers and the next step. |
42aa8ad to
baeb776
Compare
|
I am not able to reproduce the bug. I would suggest to remove the test waiver and do nothing else. |
…ration threshold runNCCLAllReduceSymmetric allocated its window-backed output buffer with an unconditional createNCCLWindowTensor, bypassing the minRegistrationThreshold gate that the input path a few lines above already applies. That threshold is set to SIZE_MAX when neither NVLink nor MNNVL is supported, so on such topologies the collective ncclAllReduce plus cudaStreamSynchronize inside allocateAndRegisterBuffer never completes and every rank hangs, which the CI stage reports as "Test terminated unexpectedly". Apply the same gate to the output allocation: request a window buffer only when the input already obtained one or the message is at least as large as the threshold. Gating on the threshold rather than on mIsNVLINKSupported / mIsMNNVLSupported keeps the symmetric-memory fast path enabled wherever it works and still honors TLLM_NCCL_MIN_REGISTRATION. The pre-existing invalid-buffer fallback to torch::empty_like covers the skipped case, so no new error path is introduced. Signed-off-by: handongl <handongl@nvidia.com>
test_row_linear_norm_fusion needs 2 GPUs and has been flaky under nvbugs/6501404. Mark it @pytest.mark.post_merge so it is dropped from the blocking pre_merge 2-GPU glob (unittest/_torch/multi_gpu -m "not post_merge") on l0_dgx_h100, and add a matching 2-GPU post_merge condition block that collects it via -m "post_merge" so coverage is retained in the non-blocking post_merge stage. Signed-off-by: linquanh <linquanh@nvidia.com>
Signed-off-by: linquanh <linquanh@nvidia.com>
Signed-off-by: linquanh <linquanh@nvidia.com>
Signed-off-by: handongl <handongl@nvidia.com> [nvbugs/6501404][chore] Sort ray_orchestrator waive entry The file-contents-sorter pre-commit hook requires waives.txt to be byte-sorted. The newly added ray_orchestrator waive was placed after the sampler entry; reorder it before to satisfy the hook. Signed-off-by: linquanh <linquanh@nvidia.com>
…RMS_NORM release Rebasing onto main brings in the RMS_NORM release of the internal output window in runNCCLAllReduceSymmetric (NVIDIA#20232), which reads windowBuffer1 after the fused fallback. The threshold-gated output allocation declared windowBuffer1 inside the gate, so that read was out of scope. Declare windowBuffer1 at function scope. It stays invalid unless the gate allocates a valid window, which is exactly when outputTensor is window-backed and must be released, so the release keeps its meaning. Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com>
The failure does not reproduce, so this PR now only removes the
test_row_linear_norm_fusion[2-hidden:16-seqlen:2] waiver. Revert the
other changes it carried:
- the output-window gating in allreduceOp.cpp ("Gate NCCL symmetric
output window on the registration threshold", "Gate output window on
uniform threshold" and "Keep the output window handle in scope for the
RMS_NORM release");
- the @pytest.mark.post_merge marker on test_row_linear_norm_fusion and
the 2-GPU H100 post_merge block in l0_dgx_h100.yml ("Move
test_row_linear_norm_fusion to post_merge");
- the DGX_H100-2_GPUs-PyTorch-Others-Post-Merge-1 stage in
L0_Test.groovy ("Add H100 post-merge stage").
allreduceOp.cpp, test_linear.py, l0_dgx_h100.yml and L0_Test.groovy now
match the base branch, so test_row_linear_norm_fusion runs again in the
existing pre-merge `unittest/_torch/multi_gpu -m "not post_merge"` entry.
Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com>
baeb776 to
7fc43aa
Compare
|
/bot run |
|
PR_Github #81162 [ run ] triggered by Bot. Commit: |
|
PR_Github #81162 [ run ] completed with state
|
|
/bot run |
|
update the PR title. |
|
PR_Github #81325 [ run ] triggered by Bot. Commit: |
|
PR_Github #81325 [ run ] completed with state |
kris1025
changed the title
The PR is much simpler now, only a single test is unwaived.
|
🤖 Coding Agent task started to fix merge conflicts. |
|
Add Carrot credits or activate Agent usage billing to use Autofix |
Summary
This PR updates the NCCL symmetric allreduce path so the output tensor no longer unconditionally requests a window-backed allocation.
Previously,
runNCCLAllReduceSymmetric()applied the registration threshold when deciding whether to copy the input into a new window, but always calledcreateNCCLWindowTensor()for the output. Small messages could skip that input allocation while still entering the output window allocation/registration path. Window allocation is a collective, so taking that path on some ranks and not others is the failure mode this change avoids.The new behavior requests a window-backed output tensor only when:
bufferSizeBytes >= minRegistrationThresholdIf no valid output window is obtained, the existing
torch::empty_like(...)fallback is used. The threshold still honorsTLLM_NCCL_MIN_REGISTRATION.This is not the same predicate as the input path. The input can already be window-backed for a sub-threshold message:
searchBuffer()returns a validwindowBuffer0when the caller reused a previously registered tensor, and that pointer is used assendbufregardless of size. The output gate does not includewindowBuffer0.isValid(). In that casencclAllReducesees a registered send buffer and a plain recv buffer. All ranks still take the same branches when SPMD inputs match, so this is a send/recv mix on the symmetric kernel (possible degrade / fallback), not an unpairedcreateNCCLWindowTensorhang.Test coverage
test_row_linear_norm_fusionaspost_merge.unittest/_torch/multi_gpu -m "post_merge" TIMEOUT (90)unittest/_torch/multi_gpu/test_linear.py::test_row_linear_norm_fusion[2-hidden:16-seqlen:2]Test plan
test_row_linear_norm_fusion[2-hidden:16-seqlen:2].createNCCLWindowTensorfor the output.Links
Dev Engineer Review
runNCCLAllReduceSymmetricto request the output window only whenbufferSizeBytes >= minRegistrationThreshold. This is intentionally size-only and does not mirror the input'swindowBuffer0.isValid()reuse path.torch::empty_likefallback when symmetric-memory allocation is unavailable.outputTensor.data_ptr()directly toncclAllReduce.TLLM_NCCL_MIN_REGISTRATIONbehavior.unittest/_torch/multi_gpu/test_linear.py::test_row_linear_norm_fusion[2-hidden:16-seqlen:2].QA Engineer Review
tests/integration/test_lists/waives.txt.