Repository navigation
[None][perf] enable Qwen3.*-VL encoder CUDA graphs via shared setup - #20213
yechank-nvidia wants to merge 8 commits into
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. ℹ️ Recent review info
WalkthroughThis change adds shared encoder attention-metadata preparation, automatic graph-shape configuration, and adapter-driven graph replay. Engine setup configures local encoders. Qwen3-VL, GLM5, and RADIO connect vision encoders to the shared graph lifecycle. ChangesMultimodal encoder graph setup
Priority: Estimated code review effort: 4 (Complex) | ~45 minutes Change: Feature Sequence Diagram(s)sequenceDiagram
participant PyTorchModelEngine
participant setup_mm_encoder_cuda_graphs
participant MultimodalEncoderMixin
participant MultimodalEncoderGraphRunner
PyTorchModelEngine->>setup_mm_encoder_cuda_graphs: Configure local encoder graphs
setup_mm_encoder_cuda_graphs->>MultimodalEncoderMixin: Select configuration and enable capture
MultimodalEncoderMixin->>MultimodalEncoderGraphRunner: Build runner and capture graphs
Suggested reviewers: Merge Risk:
 |
| Check name | Status | Explanation | Resolution |
|---|---|---|---|
| Docstring Coverage |  |
Docstring coverage is 26.06% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 142 functions across 17 files. | Write docstrings for the functions missing them to satisfy the coverage threshold. |
-
Check name Status Explanation Linked Issues check Check skipped because no linked issues were found for this pull request. Out of Scope Changes check Check skipped because no linked issues were found for this pull request. Title check The title clearly identifies the main change: enabling Qwen3-VL encoder CUDA graphs through shared setup. The [None][perf]prefix does not prevent understanding the change.Description check The description clearly explains the problem, solution, configuration, interfaces, validation, test results, benchmarks, limitations, and pending reviews. It does not use the required `## Test Coverag… Full details: Description check
-
Explanation
The description clearly explains the problem, solution, configuration, interfaces, validation, test results, benchmarks, limitations, and pending reviews. It does not use the required
## Test Coverageheading or include the## PR Checklist, but it provides the required test and validation details in dedicated sections, so the description is mostly complete.
-
✨ Finishing Touches: 💡 1
-
🛠️ Fix failing CI checks: 💡
-
- Commit to this branch
- Create a new PR
🧪 Generate unit tests (beta)
-
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Comment @coderabbitai help to get the list of available commands.
yechank-nvidia
changed the title
yechank-nvidia
changed the title
|
/bot run --disable-fail-fast |
|
PR_Github #78277 [ run ] triggered by Bot. Commit: |
|
Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements. |
|
PR_Github #78277 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #78787 [ run ] triggered by Bot. Commit: |
|
PR_Github #78787 [ run ] completed with state
|
yechank-nvidia
added
the
api-compatible
|
/bot run --disable-fail-fast |
|
PR_Github #78913 [ run ] triggered by Bot. Commit: |
|
PR_Github #78913 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #79064 [ run ] triggered by Bot. Commit: |
f3a97d8 to
56a285d
Compare
|
/bot run --disable-fail-fast |
|
PR_Github #79806 [ run ] triggered by Bot. Commit: |
|
PR_Github #79806 [ run ] completed with state
|
56a285d to
e8bbb88
Compare
|
/bot run --disable-fail-fast |
|
PR_Github #80053 [ run ] triggered by Bot. Commit: |
|
PR_Github #80053 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #80150 [ run ] triggered by Bot. Commit: |
|
PR_Github #80150 [ run ] completed with state Resource maintenance in LLM/main/L0_Test-x86_64-Multi-GPU #3320: B200 multi-GPU stages are paused to relieve the CI queue backlog (109 running / 459 pending on 2026-10-22). Owner: @sharabeshj-nvda. The CI team decides when to lift it. |
e8bbb88 to
48125b9
Compare
|
/bot run --disable-fail-fast |
|
PR_Github #80328 [ run ] triggered by Bot. Commit: |
|
PR_Github #80328 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #80377 [ run ] triggered by Bot. Commit: |
|
PR_Github #80377 [ run ] completed with state
|
Capture configured encoders through MultimodalEncoderMixin after weights and metadata are ready, before KV-cache profiling. Reuse MultimodalEncoderGraphRunner while keeping tensor specs, metadata providers, preprocessing, and mergers in each model. Add Qwen3-VL block graphs with eager deepstack mergers and share fixed-buffer packed attention metadata refresh with Qwen2-VL and GLM. Cover capture ordering, duplicate startup, padded replay, and TRTLLM/FlashInfer backends. Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
Select each eager merger by its checkpoint index while preserving the layer order of captured deepstack states. Cover reordered indexes on TRTLLM and FlashInfer, and mark graph logic regressions for CPU CI collection. Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
Expose encoder graph settings through the mixin property and remove the redundant metadata token-budget fallback. Keep the originally configured encoder budget through scheduler expansion and LLM token clamping, and access declared attention metadata fields directly. Cover explicit and default encoder budgets with item scheduling enabled and disabled in the real engine constructor. Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
Reuse the existing graph teardown in MultimodalEncoderMixin so Qwen, RADIO and GLM honor the engine NCCL window ownership policy. Cover repeated teardown and capture in the existing lifecycle test. Regenerate serving schemas for automatic capture settings and validate their YAML bounds. Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
48125b9 to
d551779
Compare
|
/bot run --disable-fail-fast |
|
PR_Github #80408 [ run ] triggered by Bot. Commit: |
Description
multimodal_config.encoder_cuda_graphexists, but Qwen3-VL-based vision encoders do not capture or replay encoder graphs. RADIO and GLM also initialize their graphs through separate weight-loading hooks. Add Qwen3-VL encoder graph dispatch and give supported encoders a shared lifecycle after weights and attention metadata are ready, before KV-cache sizing.Reuse
MultimodalEncoderMixinandMultimodalEncoderGraphRunner: the mixin captures each configured encoder once, while models retain their block functions, tensor specs, metadata providers, preprocessing, and mergers. The mixin also shares graph teardown across Qwen3-VL, RADIO and GLM, preserving the engine's NCCL window ownership policy. Share packed no-KV attention metadata refresh across Qwen2-VL, Qwen3-VL, and GLM, preserving captured buffer addresses. Qwen3-VL captures raw deepstack states and applies eager mergers in the same order as eager execution, including reordered deepstack indexes. Capture the largest bucket first to initialize shared attention workspace before smaller graphs.Adds optional prototype
max_batch_sizeandmax_num_tokensfields for automatic encoder graph shapes; no dependencies are added.Configuration
Set a batch-size limit; each encoder adapter generates its capture shapes automatically. The default token cap comes from
encoder_max_num_tokens, or the LLMmax_num_tokenswhen unset. Capture uses the configured budget before any scheduling-only increase for a large atomic item. Each metadata provider sizes its graph-owned buffers from the adapter-generated capture key, including layout-specific padding. Graph capture does not enlarge eager metadata buffers.The same schema works for audio adapters: for a model with both graph-capable towers, add
audio: {max_batch_size: 4}. An optional per-towermax_num_tokenslimits capture capacity. Production audio towers still need their own graph integration and model validation; the common interface is tested with batch/time inputs, masks, and downsampled outputs.Batch size counts encoder contexts, such as images, video frame groups, or audio chunks, rather than concurrent requests or scheduled atomic items. All context counts up to the cap are captured within the token budget. Existing explicit
bucketsremain available as an alternative for advanced tuning. Graphs stay disabled by default, and out-of-range shapes use eager execution.The new config fields require telemetry/privacy CODEOWNER review. The LLM args manifest was regenerated and checked; it is unchanged. Serving YAML schemas include the new fields and preserve main's current defaults; native-version validation limits are listed below.
Encoder Interfaces
Each graph-capable encoder declares its configuration key through
encoder_cuda_graph_modalityand selects only its own entry withconfigure_encoder_cuda_graph. The engine validates requested keys against all local towers together before capture, allowing vision and audio configurations to coexist when both towers implement graph support.EncoderGraphIOAdapterdefines automatic capture shapes, captured input allocation, layout compatibility, padding lengths, input/mask refresh, and output collection. The defaultPackedEncoderGraphIOAdapterreuses token-axis tensor specs for the existing vision encoders. Custom adapters can use batch/time dimensions and downsampled outputs.EncoderMetadataProvideris generic over the encoder's metadata type; graph-read buffer addresses remain checked across replays. Implemented model adapters cover vision encoders; batched masked/downsampled layouts are exercised by a synthetic test adapter.Validation
abb94ab8fcat head48125b921b, preserving main's mRoPE setup and NCCL capture/cleanup ownership. 191 CPU tests passed, 13 skipped across multimodal config validation, encoder graph runner/engine logic, and serving schemas. Changed-file pre-commit checks passed. The LLM args manifest was regenerated and is unchanged.trtllm::allreduce_pg_by_nameoperator; full native/GPU/serving validation awaits new-head CI. One whole-schema freshness check was excluded because that wheel'sMambaStateConfig/BlockReuseConfigdefinitions differ from current main. Encoder graph schema fields match regeneration; checked-in snapshots retain main's current native config definitions. Earlier runtime/performance results below remain pinned to their measured revisions.c183c2f80f: 244 passed, 6 skipped, 1 deselected. Covers automatic and explicit configurations, Qwen3-VL with TRTLLM/FlashInfer and reordered deepstack indexes, RADIO, GLM, metadata refresh, and shared startup/replay. Boundary tests cover a bucket equal to the encoder token budget, both partially and fully occupied, with graph-owned metadata independent of eager token/context capacities. The masked batch/time adapter replays downsampled outputs for batch sizes 1, 2, and 3. Modality tests cover separate tower configs and validation before any capture.c183c2f80f: 59 passed. Regenerated the LLM args manifest; no manifest changes. The new prototype fields still require telemetry/privacy CODEOWNER review.c183c2f80f: all 7 cases per model match eager bit-for-bit, with eager fallback disabled. Each uses onlymax_batch_size=8plus the engine's 2048-token budget and automatically captures 19 shapes. Cases cover 196/256/400 tokens, image batches 1/2/3/8, mixed resolutions, and a three-context video. Qwen3-VL includes deepstack outputs at layers 5, 11, and 17.c183c2f80fwithmax_batch_size=8and no explicit buckets: two image resolutions generate responses successfully. The configured encoder token budget is 2048; capture remains bounded by that configured budget when scheduling raises its effective limit for a large atomic item.cpu_only; Qwen3-VL and shared runner GPU replay are registered inl0_h100.yml, and the existing GLM suite is registered inl0_b200.yml. Multi-GPU validation remains pending. Serving performance measurements below are fromd2f06b8with explicit buckets.Earlier follow-up
b0c446885erecorded 41 CPU tests, 164 encoder/model tests (6 skipped, 1 deselected), and four targeted budget regressions passing. The warmup test fixture usesMultimodalConfig()defaults to provide the realencoder_cuda_graphattribute.Encoder Microbenchmark
Measured at
d2f06b8, H200 NVL, BF16, one context, 256 encoder tokens. Fullencode_batchedincludes preprocessing and mergers on synthetic pixel tensors, with fixed attention capacity 4096. Five warmups per mode, followed by the median of five rounds of 20 forwards.These measurements compare CUDA graphs enabled versus eager encoder execution. They do not measure the refactor alone or serving throughput.
Online Serving Benchmark (AIPerf)
Qwen3.8-27B, BF16, one H200 NVL, PR head
d2f06b8, matching main L0_PostMerge 3017 binaries, AIPerf 0.8.0. Streaming OpenAI chat requests use one distinct 224×224 PNG each (processor grid 16×16 / 256 vision tokens), 206 total prompt tokens and exactly 128 output tokens. Thinking is disabled. KV block reuse and encoder reuse are disabled.Server settings:
max_batch_size=128,max_num_tokens=4096,max_seq_len=1024,encoder_max_batch_size=8, chunked prefill and overlap enabled, decoder CUDA graph sizes[1,2,4,8,16,32,64,128]with padding in both conditions. Only encoder graphs are toggled. The ON buckets cover(256*n, n)forn=1..8; replay diagnostics are enabled. The configured encoder token budget is 4096; startup raises its effective budget to the model's 65536-token maximum atomic item. Encoder item count remains capped at 8.Each condition/concurrency runs 128 warmup requests followed by 1024 measured requests, three times. The table reports medians of the per-run metrics. All 12,288 measured requests succeeded with 128 output tokens. All 6,912 ON images, including warmup, used graph replay with zero runtime bucket misses; the max-size startup profiling dummy uses eager fallback.
Encoder Batch Scaling
Separate full
encode_batchedmicrobenchmark atd2f06b8, 256 vision tokens per image, BF16, H200 NVL, synthetic pixels and real weights. Fixed attention capacity 32769; capture buckets cover batches 1, 8, 32, 64 and 128. Five warmups per mode, median of five rounds of 20 forwards. Both models match eager bit-for-bit at every batch, including all three Qwen3-VL deepstack outputs.With this capture configuration, large encoder batches of 64/128 are slower than eager. The online results above use encoder microbatches capped at 8 even at client concurrency 128. These results establish the benefit for that workload/configuration; they do not claim a throughput gain at every encoder batch size.