[None][chore] Clean up GPU architecture handling by tongyuantongyu · Pull Request #20137 · NVIDIA/TensorRT-LLM

tongyuantongyu · 2026-10-12T06:32:40Z

Dev Engineer Review

Removes -real suffixes from build defaults, documentation, CI, Docker, Slurm, and runtime guidance.
Adds C++ and Python APIs for built and supported CUDA architectures.
Adds runtime architecture validation during executor and visual-generation worker setup.
Removes SM121-to-SM120 aliasing and standardizes SM120-family checks through isSM120Family and is_sm_120f.
Updates attention, GEMM, MoE, Marlin, MLA, quantization, speculative-decoding, and visual-generation dispatch.
Expands ARM64 defaults to SM103, SM107, SM120, and SM121. Verify build size, build time, and kernel availability.
Some MoE compile-time paths still distinguish capability 120 from the wider SM120 family. Verify intended SM121 behavior.
Review-finding severity counts are unavailable.
Supplied CI results include failures and an aborted run. No successful full-validation result is available.

QA Engineer Review

Adds tests/unittest/others/test_cuda_architecture_check.py for architecture metadata, exact matching, built-versus-supported checks, and device validation.
Updates tests for architecture parsing and SM120-family behavior across attention, MoE, Marlin, MNNVL, visual generation, model loading, and model-specific paths.
Adds the architecture-check test to tests/integration/test_lists/test-db/l0_a10.yml.
No changed test is shown in a manual-QA list.
Coverage verdict: needs follow-up. The tests cover the new APIs and family predicates, but SM121 behavior and failed CI paths require further validation.

Per-File QA Perspective

Source and build files

cpp/cmake/modules/cuda_configuration.cmake: Verify suffix normalization, SM121 support, default selection, and rejection of architectures without compiled kernels.
cpp/include/tensorrt_llm/common/cudaArchitectures.h: Verify the public architecture-query and validation API.
cpp/tensorrt_llm/common/cudaArchitectures.cpp: Verify built-versus-supported classification and all device error paths.
cpp/tensorrt_llm/common/CMakeLists.txt: Verify compile definitions contain the intended architecture lists.
cpp/include/tensorrt_llm/common/cudaUtils.h and tensorrt_llm/_utils.py: Verify SM121 reporting and consistent SM120-family predicates.
Changed attention, GEMM, MoE, Marlin, MLA, quantization, speculative-decoding, and visual-generation files: Verify family dispatch, SM121 behavior, changed FP8/NVFP4 restrictions, and unsupported-device errors.
cpp/tensorrt_llm/nanobind/bindings.cpp: Verify Python exposure of architecture metadata and validation methods.
tensorrt_llm/executor/base_worker.py and tensorrt_llm/_torch/visual_gen/executor.py: Verify validation occurs after device selection and before model or distributed initialization.
cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/CMakeLists.txt: Verify architecture 100, 103, 107, and 100f filtering and cubin registration.
scripts/build_wheel.py: Verify semicolon-separated architecture parsing, suffix warnings, and Volta rejection.
Build, Docker, Jenkins, Slurm, example, and documentation files: Verify unsuffixed architecture values match executed commands and platform defaults.

Test files

tests/unittest/others/test_cuda_architecture_check.py: Covers architecture metadata, exact matching, build support, and device validation. It is listed in the A10 CI test list.
tests/unittest/scripts/test_build_wheel_copy_back.py: Covers unsuffixed architecture parsing and suffix compatibility cases.
Changed attention, MoE, Marlin, MNNVL, visual-generation, model-loading, and model-specific tests: Verify updated SM120-family skips, dispatch guards, and mocks.
tests/integration/test_lists/test-db/l0_a10.yml: Verify the architecture-check test runs in the intended A10 job.

Description

Cherry pick [None][feat] Enable Rubin architecture, fail early on unsupported architectures #19950
- Enable Rubin architecture properly
- Explicitly check that TensorRT-LLM is built with the architecture it runs on
Remove the special aliasing of SM121 -> SM120
- -a "121" is now required to support SM121. -a "120;121" still builds only the sm120f version of the kernel.
- SM12x detection is now migrated to is_sm_120f instead of manually test sm in [120, 121], to align with is_sm_100f.
Remove the -real suffix when defining architectures during build
- cuda_configuration.cmake normalizes that, and user should specify the exact architectures they want to support. Adding -real now generate warning, so users don't misunderstand the meaning.
- Cleanup the doc.

Test Coverage

Build tweaks. Passing build and current tests are enough.

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

tongyuantongyu · 2026-10-12T07:34:43Z

/bot run

tensorrt-cicd · 2026-10-12T07:40:49Z

PR_Github #77227 [ run ] triggered by Bot. Commit: a8d6315 Link to invocation

tensorrt-cicd · 2026-10-12T14:32:20Z

PR_Github #77227 [ run ] completed with state SUCCESS. Commit: a8d6315
/LLM/main/L0_MergeRequest_PR pipeline #63665 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

tongyuantongyu · 2026-10-14T04:28:08Z

/bot run

tensorrt-cicd · 2026-10-14T04:35:37Z

PR_Github #77677 [ run ] triggered by Bot. Commit: 0000366 Link to invocation

tensorrt-cicd · 2026-10-14T04:56:07Z

PR_Github #77677 [ run ] completed with state FAILURE. Commit: 0000366
/LLM/main/L0_MergeRequest_PR pipeline #64036 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

tongyuantongyu · 2026-10-14T07:26:08Z

/bot run

tongyuantongyu · 2026-10-14T08:05:17Z

/bot run

tensorrt-cicd · 2026-10-14T08:12:03Z

PR_Github #77741 [ run ] triggered by Bot. Commit: 095d567 Link to invocation

tensorrt-cicd · 2026-10-14T16:58:10Z

PR_Github #77741 [ run ] completed with state SUCCESS. Commit: 095d567
/LLM/main/L0_MergeRequest_PR pipeline #64087 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

tensorrt-cicd · 2026-10-27T11:31:38Z

PR_Github #80878 [ run ] triggered by Bot. Commit: 88d1a70 Link to invocation

tensorrt-cicd · 2026-10-27T15:17:31Z

PR_Github #80878 [ run ] completed with state SUCCESS. Commit: 88d1a70
/LLM/main/L0_MergeRequest_PR pipeline #66605 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

tongyuantongyu · 2026-10-28T02:42:14Z

/bot run --disable-fail-fast

# Conflicts: # tensorrt_llm/_torch/modules/linear.py

tongyuantongyu · 2026-10-28T03:24:46Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-28T03:32:13Z

PR_Github #81057 [ run ] triggered by Bot. Commit: 5e6d38d Link to invocation

tensorrt-cicd · 2026-10-28T11:35:51Z

PR_Github #81057 [ run ] completed with state FAILURE. Commit: 5e6d38d
/LLM/main/L0_MergeRequest_PR pipeline #66771 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

tongyuantongyu · 2026-10-29T02:09:06Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-29T02:34:04Z

PR_Github #81305 [ run ] triggered by Bot. Commit: 5e6d38d Link to invocation

tensorrt-cicd · 2026-10-29T04:28:12Z

PR_Github #81305 [ run ] completed with state SUCCESS. Commit: 5e6d38d
/LLM/main/L0_MergeRequest_PR pipeline #66997 completed with status: 'SUCCESS'

CI Report

Link to invocation

# Conflicts: # tensorrt_llm/_torch/modules/linear.py

tongyuantongyu · 2026-10-29T06:05:38Z

/bot skip --comment "trivial merge"

tensorrt-cicd · 2026-10-29T06:13:07Z

PR_Github #81364 [ skip ] triggered by Bot. Commit: 272ced9 Link to invocation

tensorrt-cicd · 2026-10-29T07:23:38Z

PR_Github #81364 [ skip ] completed with state SUCCESS. Commit: 272ced9
Skipping testing for commit 272ced9

Link to invocation

PR NVIDIA#20137 gives the dense FP8 block-scale Linear its SM121 path. Signed-off-by: Dorian Donaj Magasic <dorian.magasic@turintech.ai>

Resolve linear.py against NVIDIA#20833, so the tuned call uses the new weight argument, and adapt cublasScaledMM.cpp to NVIDIA#20137, which removed the queryRealSmArch argument of getSMVersion(). Signed-off-by: Dorian Donaj Magasic <dorian.magasic@turintech.ai>

….3: Restore Mistral Large 3 multimodal inference Forward-port of release/1.3 PR NVIDIA#19788 (release/1.3 commit 48c796c) to main. Adapted to main: adopts main's MultimodalItemMetadata contract from _build_mm_item_metadata and exposes it through get_mm_item_metadata, and the new tests call get_mm_item_metadata and compare against the tuples that MultimodalItemMetadata now stores. Also: the HF tokenizer load in MistralHFInputProcessor follows the trust_remote_code argument like the processor load next to it, and build_fmha_head104_tests requests dynamic NVRTC linking and passes plain architecture numbers (NVIDIA#20137) like the other configure-only gtest fixtures, so the target does not depend on a configure cache left by an earlier fixture. Signed-off-by: Pranav Shrestha <254760092+pranav-nvidia@users.noreply.github.com> Co-authored-by: Pranav Shrestha <254760092+pranav-nvidia@users.noreply.github.com> Signed-off-by: Simeng Liu <simengl@nvidia.com>

….3: Restore Mistral Large 3 multimodal inference Forward-port of release/1.3 PR NVIDIA#19788 (release/1.3 commit 48c796c) to main. Adapted to main: adopts main's MultimodalItemMetadata contract from _build_mm_item_metadata and exposes it through get_mm_item_metadata, and the new tests call get_mm_item_metadata and compare against the tuples that MultimodalItemMetadata now stores. Also: the HF tokenizer load in MistralHFInputProcessor follows the trust_remote_code argument like the processor load next to it, and build_fmha_head104_tests requests dynamic NVRTC linking and passes plain architecture numbers (NVIDIA#20137) like the other configure-only gtest fixtures, so the target does not depend on a configure cache left by an earlier fixture. It also regenerates the FMHA sources, so a stale generation stamp cannot hide the head-104 generator changes, and fmhaHead104Test is registered with add_tllm_gtest (OWNER, LABELS, FULL_STACK) as main asks for new C++ tests (NVIDIA#21022). Signed-off-by: Pranav Shrestha <254760092+pranav-nvidia@users.noreply.github.com> Co-authored-by: Pranav Shrestha <254760092+pranav-nvidia@users.noreply.github.com> Signed-off-by: Simeng Liu <simengl@nvidia.com>

github-actions Bot assigned tongyuantongyu Oct 12, 2026

github-actions Bot added the VisualGen label Oct 12, 2026

tongyuantongyu force-pushed the ytong/unify_sm branch from 66c9d11 to a8d6315 Compare October 12, 2026 06:39

tongyuantongyu force-pushed the ytong/unify_sm branch from a8d6315 to 0000366 Compare October 14, 2026 04:27

tongyuantongyu force-pushed the ytong/unify_sm branch from 0000366 to d9516ca Compare October 14, 2026 07:26

tongyuantongyu force-pushed the ytong/unify_sm branch from d9516ca to 095d567 Compare October 14, 2026 08:05

tongyuantongyu force-pushed the ytong/unify_sm branch from 095d567 to d77a6f6 Compare October 15, 2026 09:38

tongyuantongyu marked this pull request as ready for review October 15, 2026 09:42

tongyuantongyu requested review from a team as code owners October 15, 2026 09:42

Merge remote-tracking branch 'public/main' into ytong/unify_sm

5e6d38d

# Conflicts: # tensorrt_llm/_torch/modules/linear.py

erictsai-nv approved these changes Oct 28, 2026

View reviewed changes

litaotju approved these changes Oct 28, 2026

View reviewed changes

chenfeiz0326 mentioned this pull request Oct 29, 2026

[None][chore] Mass merge release/1.3 branch back to main (batch 2) #20843

Open

1 task done

Merge remote-tracking branch 'public/main' into ytong/unify_sm

272ced9

# Conflicts: # tensorrt_llm/_torch/modules/linear.py

tongyuantongyu enabled auto-merge (squash) October 29, 2026 06:04

tongyuantongyu merged commit 87ce6f1 into NVIDIA:main Oct 29, 2026
8 checks passed

tongyuantongyu deleted the ytong/unify_sm branch October 29, 2026 09:46

Dorijan10 mentioned this pull request Oct 29, 2026

[None][feat] Support non-gated FP8 block-scale MoE on SM121 #19806

Open

This was referenced Oct 30, 2026

[None][feat] Add NVFP4_PLUS (E2M1 + UE5M3 per 32) CuteDSL MoE on Rubin #21242

Open

[TRTLLM-17328][chore] Remove quantized FlashInfer TRTLLM-Gen MoE leaves and FlashInfer two-sided alltoall #21013

Open

chzblych mentioned this pull request Nov 1, 2026

[None][infra] Align the arm64 default CUDA_ARCHS in docker/Makefile with the aarch64 wheel build #20480

Closed

1 task done

Conversation

tongyuantongyu commented Oct 12, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dev Engineer Review

QA Engineer Review

Per-File QA Perspective

Source and build files

Test files

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

tongyuantongyu commented Oct 12, 2026

Uh oh!

tensorrt-cicd commented Oct 12, 2026

Uh oh!

tensorrt-cicd commented Oct 12, 2026

Uh oh!

tongyuantongyu commented Oct 14, 2026

Uh oh!

tensorrt-cicd commented Oct 14, 2026

Uh oh!

tensorrt-cicd commented Oct 14, 2026

Uh oh!

tongyuantongyu commented Oct 14, 2026

Uh oh!

tongyuantongyu commented Oct 14, 2026

Uh oh!

tensorrt-cicd commented Oct 14, 2026

Uh oh!

tensorrt-cicd commented Oct 14, 2026

Uh oh!

tensorrt-cicd commented Oct 27, 2026

Uh oh!

tensorrt-cicd commented Oct 27, 2026

Uh oh!

tongyuantongyu commented Oct 28, 2026

Uh oh!

tongyuantongyu commented Oct 28, 2026

Uh oh!

tensorrt-cicd commented Oct 28, 2026

Uh oh!

tensorrt-cicd commented Oct 28, 2026

Uh oh!

tongyuantongyu commented Oct 29, 2026

Uh oh!

tensorrt-cicd commented Oct 29, 2026

Uh oh!

tensorrt-cicd commented Oct 29, 2026

Uh oh!

tongyuantongyu commented Oct 29, 2026

Uh oh!

tensorrt-cicd commented Oct 29, 2026

Uh oh!

tensorrt-cicd commented Oct 29, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

tongyuantongyu commented Oct 12, 2026 •

edited by coderabbitai Bot

Loading