yechank-nvidia · Oct 26, 2026 · Oct 15, 2026 · Oct 15, 2026 · Oct 16, 2026 · Oct 16, 2026
@@ -93,6 +93,12 @@ HuggingFace Model → LLM API → PyTorch Executor
 - **Disaggregated serving**: separates prefill (context) and decode (generation) across GPUs
   - KV cache exchange via NIXL (default), UCX, or MPI
 
+### Multimodal request data
+
+Read the [multimodal developer guide](docs/source/developer-guide/multimodal.md) before changing multimodal paths.
+Reuse `MultimodalParams`; request Params own cache references.
+Encoder and LLM forward Params are execution views.
+
 ## Key Files
 
 | File | Role |

@@ -0,0 +1,208 @@
+<!--
+SPDX-License-Identifier: Apache-2.0
+Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+-->
+
+# Multimodal Request Data
+
+Multimodal preprocessing and execution share `MultimodalParams`, defined in
+[inputs/multimodal.py](source:tensorrt_llm/inputs/multimodal.py).
+
+| Type or member | Responsibility |
+|---|---|
+| `MultimodalParams.multimodal_data` | Processed model inputs, embeddings or shared-tensor handles, and model-specific auxiliary data. |
+| `MultimodalParams.item_metadata` | Reads the single `MultimodalItemMetadata` record in `multimodal_data`. |
+| `MultimodalItemMetadata` | Prompt-ordered item references, optional encoder costs and output row counts, and input partitions. |
+| `MultimodalItemPartition` | Describes one field's item axis and optional packed lengths, inside `item_metadata.input_partitions`. |
+| `MultimodalInput` | Hashes, prompt positions, and exact runs converted into existing C++ request fields. |
+| `MultimodalRuntimeData` | Derived information for the current LLM forward or prefill chunk. |
+| `MultimodalEncoderRequestState` | Request-owned readiness and cache references; reads lengths from the admitted item metadata. |
+
+## Process Boundaries and Ownership
+
+```text
+Main / input processor
+  -> prompt token IDs + MultimodalParams (processed data and item metadata)
+Worker handoff
+  -> restore tensors; convert MultimodalInput to C++ request fields
+Executor / LlmRequest.py_multimodal_params
+  -> admission creates encoder_state where item scheduling is supported
+  -> encoder selection creates Params views with selected data and metadata
+  -> LLM forward creates Params views with embeddings and chunk runtime
+  -> executor cleanup releases request-owned cache references
+```
+
+Only the request Params own `encoder_state` cache references. Execution views
+leave that state unset, and request Params leave `multimodal_runtime` unset.
+The model owns the `TensorLRUCache`; the executor owns request cleanup.
+Text-only requests do not need a Params aggregate. Live cache references and
+CUDA events stay in the executor rather than the cross-process payload.
+
+The worker clears the processor-side `multimodal_input` after converting it to
+C++ fields. Persistent encoder caching lazily reconstructs one Python
+`MultimodalInput` per request. Selected inputs copy this wrapper before slicing
+its hashes, and prefill/termination cleanup clears it.
+
+Inline encoding and prefetch write encoder outputs back to the request.
+Item-scheduled LLM chunks keep their assembled embeddings local to the forward.
+Execution paths retain their CUDA event synchronization responsibilities.
+
+## Item Metadata and Selection
+
+`item_refs` identifies processed items by modality and modality-local index, in
+prompt order. A processed item is not necessarily one original media object or
+one tensor. Prompt token counts, physical encoder token costs, and output
+embedding rows are distinct quantities.
+
+Input preparation publishes the common record directly. The frontend's
+`mm_item_order` is consumed during preprocessing; execution reads only
+`item_metadata.item_refs`. Output row counts live in
+`item_metadata.output_embedding_lengths`. Admission, full-request and selected-item
+encoding, cache keys, and encoder output export all read the same record.
+The encoder-result boundary exports lengths as lists.
+
+Item refs, costs, and output rows are immutable tuples; the constructor also
+accepts lists. Encoder state references this record without copying its lengths.
+Producers fill input partitions before admission and validate contracts at
+producer/admission boundaries, avoiding repeated validation per forward.
+
+Partitions describe how to select processed fields: shared fields have no item
+axis, batched fields have an item axis, and packed fields also provide host-side
+per-item lengths. Contiguous tensor selections can share storage; reordered
+selections may copy. Treat shared input storage as read-only. Model hooks and
+`EncoderGroup` still own compatible batching, padding, masks, encoder arguments,
+and output fusion.
+
+## Processor Hooks
+
+Processors implement
+`get_mm_item_metadata(prompt_token_ids, multimodal_data, mm_item_order=None)` and
+return `MultimodalItemMetadata`. Generic preparation supplies prompt order and
+known output rows; model processors add encoder costs and input partitions.
+Partial model records retain known prompt rows. Conflicting order or rows fail
+at input preparation. Gemma4 and Nemotron add partitions; Qwen and Mistral also
+provide physical encoder costs.
+
+Unknown costs or row counts remain `None`. Common metadata does not enable item
+scheduling: models can supply order and output rows with
+`encoder_token_lengths=None` and keep full-request encoding. Model capability
+and configuration decide whether items are scheduled separately.
+
+Custom processors must use this common hook and type. The former
+`get_mm_encoder_item_metadata` hook, `MultimodalEncoderItemMetadata` tuple, and
+Params order/length fallbacks have been removed.
+
+## Onboarding a Model
+
+Follow [Adding a New Model](../models/adding-new-model.md) for model definition,
+weight loading, and registration. This section covers the additional multimodal
+processor and execution contracts.
+
+Start with full-request encoding and add item scheduling only when the model
+can encode its items independently. Use these contracts rather than inventing
+encoder costs or inferring item axes from matching tensor sizes.
+
+| Capability | Required information |
+|---|---|
+| Common request metadata | `item_refs` in prompt order. Unknown encoder costs and output rows stay `None`. |
+| Grouped output assembly | Exact `output_embedding_lengths`; `EncoderGroup` needs them to split and reorder rows. |
+| Item scheduling | Exact positive `encoder_token_lengths`, output row counts, and an item-selection implementation. |
+| Persistent item caching | Output row counts and stable content/processor hashes. Partial hits also require correct item selection. |
+
+Startup profiling calls `get_dummy_mm_data` directly, bypassing the normal
+processor wrapper. For grouped encoders, its returned data must include the
+common `MultimodalItemMetadata` record with item references and exact output
+row counts.
+
+For example, a processor for a full-request audio encoder can return:
+
+```python
+return MultimodalItemMetadata(
+    item_refs=[("audio", 0)],
+    encoder_token_lengths=None,
+    output_embedding_lengths=[num_output_rows],
+)
+```
+
+This supports output assembly without declaring schedulable encoder costs.
+If output row counts are unknown too, the model must resolve them before using
+grouped assembly or item caching. A model-specific full-request encoder can
+compute those counts from its result. Metadata alone never enables scheduling.
+
+To opt in, the model sets `supports_mm_encoder_item_scheduling = True` and
+implements `encode_multimodal_inputs` and item selection. The engine also
+checks `disable_mm_encoder`, the scheduling policy, and supported combinations
+in `mm_item_scheduling_enabled()` and
+`validate_mm_encoder_scheduling_compatibility()`. Under item scheduling, missing
+costs or output rows fail admission; they do not silently switch one request to
+full-request encoding. Keep the capability disabled until the model can supply
+the complete scheduling contract.
+
+### Selecting Unusual Payloads
+
+Declare `input_partitions` for ordinary batched, packed, and shared fields.
+For example, two audio items with two and one clips use:
+
+```python
+input_partitions = {
+    "audio": {
+        "input_audio_features": MultimodalItemPartition(item_lengths=(2, 1)),
+        "feature_attention_mask": MultimodalItemPartition(item_lengths=(2, 1)),
+        "audio_num_clips": MultimodalItemPartition(),
+    },
+}
+```
+
+If a payload has multiple item axes, nested variable-length structures, or
+selection that must rebuild sibling fields, override
+`build_multimodal_encoder_input(param, local_item_indices, modality)`. Return a
+new execution `MultimodalParams` view and leave the request's payload and cache
+references intact. The mixin then slices the common item metadata and hashes
+using prompt-global indices. Payload indices and prompt indices can differ:
+for `[('video', 0), ('image', 0)]`, the image has local index `0` and prompt
+index `1`.
+
+Returning `None` from `get_multimodal_item_partitions` uses the existing
+packed/stacked compatibility selector. New models should declare partitions or
+override selection explicitly; the compatibility selector cannot describe an
+arbitrary model layout. An empty partition dict means every field is shared,
+so it is appropriate only when selecting an item requires no payload changes.
+
+### Item Boundaries and Model Hooks
+
+An item is the smallest independently encodable unit, not necessarily one
+uploaded media object. Several crops, frames, or audio clips can belong to one
+item. A common item reference names one modality. Keep coupled audio/video
+data in one model-owned payload under that item's modality, or keep full-request
+encoding if independent item execution is not possible. Separate audio and
+video references imply that either may be selected independently.
+
+Use `after_full_multimodal_embeddings` for request-level preparation after
+cache lookup or encoding. Its tensors are not always full-request tensors: a
+mixed batch containing item-scheduled chunk outputs already gathers active
+rows. Use `after_active_multimodal_embeddings` for row-dependent transforms,
+such as splitting Qwen3-VL deepstack features, immediately before fusion.
+
+Model-specific auxiliary values belong in `multimodal_data`. Declare their
+selection rules alongside the primary fields, or rebuild them in the custom
+selection hook. Add a field to `MultimodalItemMetadata` only when shared
+preprocessing/execution consumers need it, and update validation, `select()`,
+producer merging, and process-handoff tests together. Keep lengths and
+partitions as host metadata; CUDA-to-host copies can synchronize even when
+written as `.cpu().tolist()`.
+
+## Encoder/Prefill Handoff
+
+For encoder/prefill disaggregation (E/P), model processor hooks return
+`(prompt_token_ids, MultimodalParams)`. Use `MultimodalParams.from_embeddings`
+to attach embeddings or shared-tensor handles. See
+[inputs/registry.py](source:tensorrt_llm/inputs/registry.py) for the hook contract.
+
+`DisaggregatedParams` remains the transport envelope used by external
+orchestrators such as Dynamo. The prefill processor still rebuilds model-specific
+prompt layout; its local Params result does not replace the wire protocol.
+Custom E/P hooks must follow the tuple return contract. Review Python hook
+compatibility separately from transport fields when changing this interface.
+
+These contracts do not change the supported model/modality matrix or impose a
+common batching policy on every model.
@@ -43,6 +43,13 @@ During each iteration of its background loop, the `PyExecutor` performs the foll
 - Output Handling: Updates the partial outputs for ongoing requests and finalizes the results for any requests that have reached completion, returning them to the user.
 
 
+## Multimodal Requests
+
+Multimodal input preparation and execution share `MultimodalParams`. The executor
+owns request cache references; encoder and LLM forwards use separate execution views.
+See the [multimodal developer guide](multimodal.md) for the data contracts,
+process boundaries, and encoder/prefill handoff.
+
 ## Runtime Optimizations
 
 TensorRT LLM enhances inference throughput and reduces latency by integrating a suite of runtime optimizations, including CUDA Graph, [Overlap Scheduler](../features/overlap-scheduler.md), [Speculative decoding](../features/speculative-decoding.md), etc.

@@ -1,3 +1,8 @@
+<!--
+SPDX-License-Identifier: Apache-2.0
+Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+-->
+
 # Multimodal Support in TensorRT LLM
 
 TensorRT LLM supports a variety of multimodal models, enabling efficient inference with inputs beyond just text.
@@ -26,6 +31,11 @@ Further optimizations are under development and will be updated as they become a
 
 Please refer to the latest multimodal [support matrix](../models/supported-models.md#multimodal-feature-support-matrix-pytorch-backend).
 
+## Adding a Model
+
+See the [multimodal onboarding guide](../developer-guide/multimodal.md#onboarding-a-model)
+for processor metadata, payload selection, model hooks, and encoder/prefill handoff.
+
 ## Optional dependencies
 
 Video input decoding uses OpenCV (`cv2`) and is not installed by default. Install it only for models with video inputs:

@@ -101,6 +101,7 @@ Welcome to TensorRT LLM's Documentation!
    :caption: Developer Guide
 
    developer-guide/overview.md
+   developer-guide/multimodal.md
    developer-guide/perf-analysis.md
    developer-guide/perf-benchmarking.md
    developer-guide/moe-perf-guide.md

@@ -1,3 +1,8 @@
+<!--
+SPDX-License-Identifier: Apache-2.0
+Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+-->
+
 # Adding a New Model
 
 ## Table of Contents
@@ -15,6 +20,10 @@
 
 This guide provides a step-by-step process for adding a new model in PyTorch Backend.
 
+For multimodal models, also follow the
+[multimodal onboarding guide](../developer-guide/multimodal.md#onboarding-a-model)
+for processor metadata, payload selection, encoder scheduling, and E/P handoff.
+
 ## Prerequisites
 
 Before you begin, ensure you have the following:

@@ -13,6 +13,21 @@ All published functionality in the Release Notes has been fully tested and verif
 ### API Changes
 
 - <span style="color: red">**[BREAKING CHANGE] Removed obsolete C++ executor orchestration helpers.**</span> The internal `RequestWithId` type, its `serializeReqWithIds` / `deserializeReqWithIds` functions, all production `insertRequestInOrder` overloads, and the unused `requestUtils.h` and `orchestratorUtils.h` helpers have been removed. These helpers belonged to the retired C++ executor orchestration path; downstream code linking their exported symbols must stop using them and rebuild. The current `Request` / `Response` interfaces and serialization remain available. Executor factory and worker constructors no longer accept the unused `engine` and `batched_logits_processor` arguments; callers should omit them and continue passing model configuration through `llm_args` and `hf_model_dir`. Unused decoder, request-cancellation, executor, legacy-model, orchestration, and speculative-decoding MPI tag names have also been removed; values 1–23, 127–129, and 1023–1025 are retired. The default, KV-cache event, and context-transfer tags retain their existing values. Internal `LogprobParams` and `LogProbsResult` tuples now contain only prompt-logprob fields; generation logprobs remain supplied by the sampler. The unused `TLLM_EXECUTOR_PERIODICAL_RESP_IN_AWAIT` environment variable has been removed.
+- <span style="color: red">**[BREAKING CHANGE] Unified multimodal processor metadata.**</span>
+  Multimodal processor extensions now use `get_mm_item_metadata()` and
+  `MultimodalItemMetadata`, replacing `get_mm_encoder_item_metadata()` and
+  `MultimodalEncoderItemMetadata`. The removed `DisaggPrefillMultimodalInputs`
+  result is replaced by `(prompt_token_ids, MultimodalParams)` from
+  `build_disagg_prefill_multimodal_inputs()`, which also accepts the keyword-only
+  `mm_hashes` argument. Custom processors must migrate these hooks; processors
+  defining the removed metadata hook now fail at wrapper creation.
+  Also removed: `MultimodalParams(mm_item_order=...)`, the
+  `multimodal_embedding_lengths` and `layout_metadata` keys in `multimodal_data`,
+  `registry.get_multimodal_encoder_item_metadata`, and `LlmRequest.py_mm_item_order`.
+  Read item order and output lengths through `MultimodalParams.item_metadata`.
+  Grouped encoders must also include item metadata with exact output row counts
+  in `get_dummy_mm_data()` results used for startup profiling. See the
+  [multimodal developer guide](developer-guide/multimodal.md).
 
 ### Fixed Issues
 

@@ -1,3 +1,8 @@
+<!--
+SPDX-License-Identifier: Apache-2.0
+Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+-->
+
 # Adding a New Model in PyTorch Backend
 
 ## Table of Contents
@@ -15,6 +20,10 @@
 
 This guide provides a step-by-step process for adding a new model in PyTorch Backend.
 
+For multimodal models, also follow the
+[multimodal onboarding guide](../developer-guide/multimodal.md#onboarding-a-model)
+for processor metadata, payload selection, encoder scheduling, and E/P handoff.
+
 ## Prerequisites
 
 Before you begin, ensure you have the following:

@@ -30,6 +30,11 @@ The single-step flow of PyExecutor involves:
 - Running the decoder using the model forward outputs for the scheduled requests.
 - Adding output tokens for each request and handling finished requests.
 
+## Multimodal Request Data
+
+See the [multimodal developer guide](../developer-guide/multimodal.md) for
+shared data contracts, process boundaries, and request ownership.
+
 ## Model Engine
 
 The core component of `PyExecutor` is the `ModelEngine`, responsible for executing the model's forward pass efficiently on the GPU.