From 78334b7250bead9ff2c886c53124673cb04290a8 Mon Sep 17 00:00:00 2001 From: Sergiy Fefilatyev Date: Mon, 14 Sep 2026 16:34:15 -0700 Subject: [PATCH 1/9] Update inference benchmarks documentation for Cosmos3 Refactor inference_benchmarks.md into an index that separates Generator and Reasoner results. Generator coverage now points at model-specific pages so audiovisual and action modalities can be reported without collapsing this page, while the Reasoner serving sections stay inline. --- inference_benchmarks.md | 317 ++-------------------------------------- 1 file changed, 16 insertions(+), 301 deletions(-) diff --git a/inference_benchmarks.md b/inference_benchmarks.md index a12e1bc7..3496c9ba 100644 --- a/inference_benchmarks.md +++ b/inference_benchmarks.md @@ -1,22 +1,21 @@ # Inference Benchmarks -These tables collect inference benchmarks for Cosmos3. **Generator** sections measure visual-generation and world-model latency across PyTorch, vLLM-Omni, Diffusers, and NIM, including image and video generation, forward and inverse dynamics, and policy generation. **Reasoner** sections measure VLM serving and token-generation performance for text, image, and video inputs through vLLM and Hugging Face Transformers. - -Generator results are published incrementally from internal benchmark runs. **Empty cells mean that combination has not been measured yet** — not that it is unsupported. See the notes under each table for workload details and data-source definitions. - -## Table of Contents - -- [Cosmos3-Edge Generator](#cosmos3-edge-generator) - - [vLLM-Omni](#vllm-omni) - - [PyTorch](#pytorch) -- [Cosmos3-Nano Generator](#cosmos3-nano-generator) - - [Text-to-Video (t2v)](#text-to-video-t2v) - - [Image-to-Video (i2v)](#image-to-video-i2v) - - [Text-to-Image (t2i)](#text-to-image-t2i) -- [Cosmos3-Super Generator](#cosmos3-super-generator) - - [Text-to-Video (t2v)](#text-to-video-t2v-1) - - [Image-to-Video (i2v)](#image-to-video-i2v-1) - - [Text-to-Image (t2i)](#text-to-image-t2i-1) +These reports collect inference benchmarks for Cosmos3. **Generator** pages measure image, video, audiovisual, and action-generation latency across PyTorch, vLLM-Omni, Diffusers, and NIM where results are available. **Reasoner** sections below measure VLM serving and token-generation performance for text, image, and video inputs through vLLM and Hugging Face Transformers. + +Results are published incrementally from internal benchmark runs. **Empty cells mean that combination has not been measured yet** — not that it is unsupported. See each model page or section for workload details and data-source definitions. + +## Generator benchmarks + +Generator results are organized by model so future checkpoints can be added without expanding this page into another wall of tables. Use the section links for modality-oriented navigation. + +| Model | Primary vision generation | Additional audiovisual generation | Action generation | +|---|---|---|---| +| Cosmos3-Edge | [i2v](inference_benchmarks/cosmos3-edge-generator.md) | Not reported | Not reported | +| Cosmos3-Nano | [t2v, i2v, t2i](inference_benchmarks/cosmos3-nano-generator.md#primary-vision-generation) | [v2v, t2av, v2av, i2av](inference_benchmarks/cosmos3-nano-generator.md#additional-audiovisual-generation) | [Forward dynamics, inverse dynamics, policy](inference_benchmarks/cosmos3-nano-generator.md#action-generation) | +| Cosmos3-Super | [t2v, i2v, t2i](inference_benchmarks/cosmos3-super-generator.md#primary-vision-generation) | [v2v, t2av, v2av, i2av](inference_benchmarks/cosmos3-super-generator.md#additional-audiovisual-generation) | [Forward dynamics, inverse dynamics, policy](inference_benchmarks/cosmos3-super-generator.md#action-generation) | + +## Reasoner benchmarks + - [Cosmos3-Edge Reasoner](#cosmos3-edge-reasoner) - [RTX PRO 4500 Blackwell Server Edition](#rtx-pro-4500-blackwell-server-edition) - [RTX PRO 6000 Blackwell Server Edition](#rtx-pro-6000-blackwell-server-edition) @@ -39,290 +38,6 @@ Generator results are published incrementally from internal benchmark runs. **Em - [B200](#b200-1) - [B300](#b300-1) -## Cosmos3-Edge Generator - -These tables report **Cosmos3-Edge** Generator latency in seconds for **image-to-video (i2v)**. Measurements use one GPU or one integrated computing platform. Lower latency is better, and empty cells indicate that a run has not been completed. - -Unless otherwise noted, visual-generation benchmarks use **480p resolution**. Video benchmarks generate **121 frames**. vLLM-Omni values report end-to-end latency, while PyTorch values report average generation latency. - -### vLLM-Omni - -| GPU or Platform | Image-to-Video | -|---|---:| -| B200 SXM 192 GB | 7.09 | -| B300 | 8.74 | -| H200 SXM 141 GB | 13.13 | -| H200 NVL | 14.15 | -| H100 SXM 80 GB | 13.33 | -| H100 NVL 96 GB | 17.33 | -| H20 SXM 96 GB | 53.94 | -| RTX PRO 6000 Blackwell Server Edition | | -| DGX Station | 7.13 | -| DGX Spark | 89.41 | -| Jetson AGX Thor T5000, 128 GB, MAXN | | -| Jetson T3000, 32 GB, 1100 MHz | | -| Jetson T2000, 16 GB, 702 MHz, THOR_NANO | | - -### PyTorch - -| GPU or Platform | Image-to-Video | -|---|---:| -| B200 SXM 192 GB | 7.45 | -| B300 | 6.84 | -| H200 SXM 141 GB | 12.31 | -| H200 NVL | 14.07 | -| H100 SXM 80 GB | 12.68 | -| H100 NVL 96 GB | 16.42 | -| H20 SXM 96 GB | 52.83 | -| RTX PRO 6000 Blackwell Server Edition | 21.92 | -| DGX Station | 6.31 | -| DGX Spark | 103.36 | -| Jetson AGX Thor T5000, 128 GB, MAXN | | -| Jetson T3000, 32 GB, 1100 MHz | | - -Notes: -1. All measurements use one GPU or one integrated computing platform. -2. Values are average end-to-end or generation latency in seconds; lower is better. -3. Unless otherwise specified, visual-generation measurements use **480p resolution**. -4. Video measurements (i2v) generate **121 output frames**. -5. PyTorch values report average generation latency rather than diffusion-only latency. - -## Cosmos3-Nano Generator - -These tables report **Cosmos3-Nano** generator latency in seconds for **image-to-video (i2v)**, **text-to-image (t2i)**, and **text-to-video (t2v)** - the primary vision-generation modes of the omni-model. Benchmarks use BF16 precision, batch size 1, and identical prompts, seeds, and sampler settings across engines where noted below. Video workloads follow the standard Cosmos3 generation profile (189 frames at 24 FPS unless a resolution tier limits frame count). - -Four integration paths are compared. **PyTorch** reports average generation (sampling) time from OSS reference inference with CUDA Graphs enabled where supported. **vLLM-Omni** reports total pipeline time at **720p** on supported GPUs. **Diffusers** reports end-to-end generation time through the Hugging Face `Cosmos3OmniPipeline` without custom CUDA graphs at **256p/1**, **480p/1**, and **720p/1** (320×192, 832×480, and 1280×720). **NIM** reports end-to-end request latency using NIM latency profiles with FP8 precision, including request processing, video generation, output encoding, and returning the response. Empty cells indicate a run has not been completed for that GPU, engine, resolution, or tensor-parallel width - tables are filled in as benchmark campaigns finish. - -### Text-to-Video (t2v) - -| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | -|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| **RTX PRO 6000 Blackwell** | PyTorch | 13.95 | | 4.90 | 180.81 | | | 786.37 | 225.45 | 127.57 | -| | vLLM-Omni | 10.65 | 5.06 | 3.78 | 105.61 | 35.93 | 23.76 | 369.67 | 114.30 | 68.66 | -| | Diffusers | 11.20 | | | 112.00 | | | 392.00 | | | -| | NIM | 7.93 | 4.69 | 4.00 | 82.69 | 33.57 | 24.49 | 318.69 | 107.83 | 68.50 | -| **H20** | PyTorch | 30.57 | | | 257.51 | | | 931.39 | 268.88 | 157.71 | -| | vLLM-Omni | 28.58 | 10.20 | 7.70 | 256.97 | 77.42 | 47.53 | 929.81 | 260.75 | 148.46 | -| | Diffusers | 30.20 | | | 258.00 | | | 926.00 | | | -| | NIM | 17.81 | 7.84 | 6.14 | 192.07 | 62.80 | 41.67 | 771.37 | 223.24 | 132.68 | -| **H100 NVL** | PyTorch | 10.03 | 4.27 | 3.95 | 84.12 | 29.18 | 21.46 | 297.27 | 94.15 | 61.63 | -| | vLLM-Omni | 9.25 | 3.68 | 3.15 | 80.75 | 27.48 | 18.77 | 311.13 | 88.25(*) | 54.01(*) | -| | Diffusers | 11.00 | | | 90.00 | | | 324.20 | | | -| | NIM | 7.09 | 3.77 | 3.65 | 68.01 | 25.67 | 20.51 | 267.73 | 86.18 | 57.38 | -| **H200 NVL** | PyTorch | 8.17 | | | 69.79 | | | 244.39 | 77.35 | 45.70 | -| | vLLM-Omni | 7.44 | 3.27 | 2.33 | 64.58 | 21.31 | 12.92 | 240.05 | 69.63 | 39.17 | -| | Diffusers | 9.00 | | | 74.00 | | | 276.20 | | | -| | NIM | 5.87 | 3.38 | 3.00 | 57.10 | 21.74 | 15.04 | 229.63 | 71.32 | 43.34 | -| **H100 80GB HBM3** | PyTorch | 7.61 | 3.50 | 3.17 | 59.83 | 21.23 | 14.37 | 207.78 | 66.94 | 41.81 | -| | vLLM-Omni | 6.97 | 3.45 | 3.49 | 58.17 | 19.95 | 13.46 | 202.29 | 62.82 | 37.80 | -| | Diffusers | 9.00 | | | 68.00 | | | 240.00 | | | -| | NIM | 5.72 | 3.36 | 3.12 | 51.73 | 20.26 | 14.82 | 199.46 | 65.32 | 41.66 | -| **H200 141GB HBM3** | PyTorch | 7.53 | 3.34 | 3.19 | 60.18 | 20.84 | 13.97 | 214.28 | 67.48 | 41.26 | -| | vLLM-Omni | 6.79 | 3.25 | 3.42 | 58.14 | 19.77 | 12.97 | 208.36 | 63.27 | 37.49 | -| | Diffusers | 9.00 | | | 67.00 | | | 239.60 | | | -| | NIM | 5.72 | 3.26 | 3.10 | 51.90 | 20.09 | 14.61 | 200.63 | 64.57 | 40.70 | -| **B200** | PyTorch | 4.56 | 2.78 | 2.79 | 33.20 | 13.20 | 9.69 | 114.85 | 39.75 | 26.27 | -| | vLLM-Omni | 4.03 | 2.43 | 3.49 | 32.04 | 12.63 | 10.09 | 107.84 | 35.29 | 22.87 | -| | Diffusers | 7.00 | | | 36.80 | | | 117.00 | | | -| | NIM | 3.72 | 2.74 | 2.89 | 26.68 | 12.66 | 10.10 | 93.33 | 35.04 | 24.64 | -| **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | 4.46 | 4.11 | 5.44 | 32.18 | 13.83 | 11.57 | 102.10 | 35.68 | 24.33 | -| | Diffusers | 39.40 | | | 63.40 | | | 139.40 | | | -| | NIM | 4.51 | 3.54 | 4.11 | 27.49 | 14.51 | 11.86 | 90.39 | 37.28 | 26.19 | - -### Image-to-Video (i2v) - -| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | -|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| **RTX PRO 6000 Blackwell** | PyTorch | | | | 182.14 | | | 788.80 | 226.25 | 127.79 | -| | vLLM-Omni | 11.04 | 5.48 | 4.24 | 107.77 | 38.05 | 25.95 | 375.01 | 119.27 | 73.57 | -| | Diffusers | 12.00 | | | 112.00 | | | 397.00 | | | -| | NIM | 8.23 | 5.28 | 4.64 | 84.58 | 35.74 | 26.63 | 326.96 | 112.27 | 73.31 | -| **H20** | PyTorch | 31.36 | | | 257.10 | | | 933.07 | 268.99 | 158.10 | -| | vLLM-Omni | 29.50 | 11.26 | 8.64 | 261.56 | 81.93 | 52.06 | 940.16 | 271.37 | 158.76 | -| | Diffusers | 31.00 | | | 258.00 | | | 925.00 | | | -| | NIM | 18.67 | 9.04 | 7.38 | 195.10 | 67.64 | 46.30 | 774.88 | 233.92 | 143.15 | -| **H100 NVL** | PyTorch | 10.19 | 4.31 | 3.99 | 84.50 | 28.69 | 21.52 | 298.57 | 95.76 | 60.58 | -| | vLLM-Omni | 9.62 | 4.11 | 3.63 | 82.61 | 29.35 | 20.73 | 286.33 | 92.23(*) | 58.02(*) | -| | Diffusers | 11.00 | | | 91.00 | | | 325.20 | | | -| | NIM | 7.39 | 4.36 | 4.28 | 69.39 | 27.67 | 22.43 | 272.29 | 90.26 | 61.55 | -| **H200 NVL** | PyTorch | 8.27 | | | 69.99 | | | 246.62 | 77.69 | 45.99 | -| | vLLM-Omni | 7.83 | 3.69 | 2.78 | 66.39 | 22.93 | 14.58 | 243.52 | 73.26 | 42.86 | -| | Diffusers | 9.00 | | | 74.00 | | | 275.20 | | | -| | NIM | 6.25 | 3.97 | 3.60 | 58.64 | 23.33 | 16.75 | 232.47 | 75.13 | 47.22 | -| **H100 80GB HBM3** | PyTorch | 7.64 | 3.47 | 3.21 | 59.95 | 21.40 | 14.43 | 207.87 | 67.52 | 41.66 | -| | vLLM-Omni | 7.37 | 3.81 | 3.97 | 59.77 | 21.68 | 15.12 | 205.97 | 66.52 | 41.51 | -| | Diffusers | 9.00 | | | 68.00 | | | 239.80 | | | -| | NIM | 6.08 | 3.89 | 3.71 | 53.02 | 22.20 | 16.61 | 202.59 | 69.02 | 45.16 | -| **H200 141GB HBM3** | PyTorch | 7.65 | 3.37 | 3.17 | 60.51 | 21.01 | 14.07 | 214.80 | 67.14 | 41.00 | -| | vLLM-Omni | 7.28 | 3.63 | 3.83 | 59.64 | 21.35 | 14.67 | 209.65 | 66.65 | 40.77 | -| | Diffusers | 9.00 | | | 67.20 | | | 240.00 | | | -| | NIM | 6.04 | 3.80 | 3.65 | 53.25 | 21.89 | 16.23 | 203.66 | 68.30 | 44.29 | -| **B200** | PyTorch | 4.60 | 2.77 | 2.81 | | 13.07 | 9.66 | 113.90 | 40.01 | 26.58 | -| | vLLM-Omni | 4.33 | 2.77 | 3.84 | 33.09 | 13.79 | 11.39 | 110.19 | 37.76 | 25.68 | -| | Diffusers | | | | | | | 116.00 | | | -| | NIM | 4.05 | 3.43 | 3.60 | 27.72 | 14.02 | 11.43 | 95.57 | 37.76 | 27.29 | -| **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | 5.61 | 4.67 | 5.90 | 33.45 | 15.06 | 13.13 | 104.75 | 38.27 | 26.87 | -| | Diffusers | 28.60 | | | 65.60 | | | 139.60 | | | -| | NIM | 4.50 | 4.96 | 5.03 | 28.90 | 16.06 | 13.25 | 92.59 | 39.49 | 29.00 | - -### Text-to-Image (t2i) - -| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | -|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| **RTX PRO 6000 Blackwell** | PyTorch | 2.99 | | | 4.51 | | | 7.12 | 3.18 | 2.70 | -| | vLLM-Omni | 1.59 | 1.54 | 1.59 | 2.87 | 1.55 | 1.81 | 4.99 | 2.32 | 1.96 | -| | Diffusers | 2.00 | | | 4.00 | | | 5.00 | | | -| **H20** | PyTorch | 3.06 | | | 6.51 | | | 12.31 | 4.28 | 3.06 | -| | vLLM-Omni | 1.73 | 2.46 | 3.22 | 4.92 | 2.57 | 7.24 | 10.73 | 4.24 | 3.59 | -| | Diffusers | 3.00 | | | 6.00 | | | 10.00 | | | -| **H100 NVL** | PyTorch | 2.77 | 2.45 | 2.57 | 2.83 | 2.56 | 2.51 | 4.21 | 2.57 | 2.64 | -| | vLLM-Omni | 1.55 | 1.75 | 1.91 | 1.92 | 1.81 | 10.82 | 3.44 | 1.83 | 1.90 | -| | Diffusers | 3.00 | | | 3.00 | | | 4.00 | | | -| **H200 NVL** | PyTorch | | | | 2.85 | | | 3.58 | 2.62 | 2.64 | -| | vLLM-Omni | 1.53 | 2.01 | 1.96 | 1.58 | 1.91 | 17.71 | 2.81 | 1.94 | 1.94 | -| | Diffusers | 3.00 | | | 3.00 | | | 4.00 | | | -| **H100 80GB HBM3** | PyTorch | 3.01 | 2.66 | 2.56 | 3.01 | 2.59 | 2.75 | 3.45 | 2.73 | 2.77 | -| | vLLM-Omni | 1.61 | 2.45 | 3.18 | 1.53 | 2.35 | 7.02 | 2.61 | 2.45 | 3.03 | -| | Diffusers | 3.00 | | | 3.00 | | | 4.00 | | | -| **H200 141GB HBM3** | PyTorch | 2.96 | 2.59 | 2.70 | 3.04 | 2.78 | 2.77 | 3.28 | 2.84 | 2.77 | -| | vLLM-Omni | 1.57 | 2.38 | 3.16 | 1.52 | 2.37 | 7.05 | 2.60 | 2.33 | 3.20 | -| | Diffusers | 3.00 | | | 3.00 | | | 4.00 | | | -| **B200** | PyTorch | | 2.39 | 2.59 | 2.75 | 2.43 | 2.56 | 2.87 | 2.58 | 2.62 | -| | vLLM-Omni | 1.49 | 2.21 | 3.27 | 1.20 | 2.05 | 7.58 | 1.77 | 2.20 | 3.41 | -| | Diffusers | | | | | | | 3.00 | | | -| **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | 1.97 | 4.52 | 5.82 | 1.81 | 4.16 | 71.19 | 2.34 | 4.09 | 5.62 | -| | Diffusers | 36.20 | | | | | | 41.00 | | | - -Notes: -1. All times measured on identical workloads (same seed, sampler settings, prompt). -2. 4×/8× GPU configurations use tensor parallelism. -3. vLLM-Omni numbers are for the upcoming public release in the vLLM-Omni repo; subject to change before GA. Values marked with (*) are pre-release vLLM-Omni measurements on H100 NVL and may change before GA. -4. Diffusers numbers use the HuggingFace `diffusers` integration without custom CUDA graphs; reported at 256p/1, 480p/1, and 720p/1 (single-GPU only). -5. PyTorch numbers report average generation (sampling) time from OSS inference benchmarking. -6. At 256p, multi-GPU configurations on B300 may underperform single-GPU due to small-workload TP overhead; single-GPU is recommended at this resolution. -7. NIM numbers use latency profiles with FP8 precision and report end-to-end `Request Latency s`, including request processing, video generation, output encoding, and returning the response. - -## Cosmos3-Super Generator - -These tables report **Cosmos3-Super** generator latency in seconds for **image-to-video (i2v)**, **text-to-image (t2i)**, and **text-to-video (t2v)**. The 32B checkpoint targets higher-quality world generation; expect longer runtimes than Nano at the same resolution. Benchmarks use BF16 precision, batch size 1, and matched prompts, seeds, and sampler settings. Video workloads follow the standard Cosmos3 profile (189 frames at 24 FPS where applicable). - -As with Nano, four engines are tracked: **PyTorch** (OSS generation/sampling time), **vLLM-Omni** (total pipeline time at 720p on supported GPUs), **Diffusers** (Hugging Face `Cosmos3OmniPipeline` end-to-end time at **256p/1**, **480p/1**, and **720p/1** — 320×192, 832×480, and 1280×720), and **NIM** (end-to-end request latency using latency profiles with FP8 precision, including request processing, video generation, output encoding, and returning the response). Super coverage is narrower than Nano in early releases - for example, vLLM-Omni and Diffusers runs exist primarily on B200 and select H200 configurations. **Empty cells are pending measurements**, not unsupported configurations. - -### Text-to-Video (t2v) - -| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | -|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| **RTX PRO 6000 Blackwell** | PyTorch | | | | | | | | 789.03 | 427.16 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| | NIM | | 12.65 | 13.99 | | 104.25 | 99.05 | | 350.74 | 286.02 | -| **H20** | PyTorch | | | | | | | | | 492.41 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| | NIM | | 20.07 | 12.95 | | 192.45 | 110.71 | | 734.37 | 395.56 | -| **H100 NVL** | PyTorch | | | 16.83 | | 101.27 | 64.14 | | 330.04 | 186.19 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| | NIM | | 8.77 | 12.73 | | 73.37 | 66.07 | | 267.64 | 197.32 | -| **H200 NVL** | PyTorch | | | | | | | | 258.34 | 139.37 | -| | vLLM-Omni | 27.54 | | 5.06 | 252.33 | | 36.66 | 911.49 | 245.51 | 123.85 | -| | Diffusers | 33.00 | | | 286.80 | | | 1036.00 | | | -| | NIM | 17.13 | 6.79 | 4.43 | 200.00 | 58.55 | 32.87 | 811.41 | 223.00 | 117.98 | -| **H100 80GB HBM3** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| | NIM | | 6.98 | 5.89 | | 55.10 | 35.52 | | 198.13 | 114.92 | -| **H200 141GB HBM3** | PyTorch | | 14.82 | 11.82 | | 70.27 | 41.78 | | 224.43 | 123.49 | -| | vLLM-Omni | 25.61 | | 5.87 | 219.11 | | 35.26 | 769.63 | 212.30 | 111.94 | -| | Diffusers | 31.00 | | | 251.60 | | | 886.20 | | | -| | NIM | 15.95 | 6.14 | 4.28 | 174.71 | 52.94 | 30.94 | 695.89 | 194.34 | 106.16 | -| **B200** | PyTorch | | 5.59 | 4.09 | 114.38 | 35.73 | 21.39 | 407.50 | 118.38 | 65.93 | -| | vLLM-Omni | 13.84 | | 4.76 | 114.08 | | 22.09 | 390.28 | 113.31 | 62.11 | -| | Diffusers | | | | 127.20 | | | 414.40 | | | -| | NIM | 9.09 | 4.26 | 3.38 | 82.39 | 27.83 | 17.74 | 314.68 | 92.25 | 53.43 | -| **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | 14.57 | | 6.68 | 109.03 | | 22.67 | 366.66 | 108.58 | 60.73 | -| | Diffusers | 54.20 | | | 155.40 | | | 424.80 | | | -| | NIM | 9.67 | 5.23 | 5.19 | 79.73 | 28.97 | 18.39 | 292.35 | 92.31 | 54.07 | - -### Image-to-Video (i2v) - -| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | -|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| **RTX PRO 6000 Blackwell** | PyTorch | | | | | | | | 795.14 | 427.96 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| | NIM | | 13.17 | 14.48 | | 106.50 | 100.23 | | 356.60 | 289.93 | -| **H20** | PyTorch | | | | | | | | 931.74 | | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| | NIM | | 21.16 | 14.06 | | 196.86 | 114.49 | | 745.12 | 405.68 | -| **H100 NVL** | PyTorch | | 20.85 | 16.96 | | 99.56 | | | 331.40 | 186.47 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| | NIM | | 9.32 | 13.30 | | 75.31 | 67.34 | | 271.46 | 201.39 | -| **H200 NVL** | PyTorch | | | | | | | | 265.33 | 138.31 | -| | vLLM-Omni | 27.90 | | 5.52 | 254.29 | | 38.51 | 915.05 | 248.89 | 127.32 | -| | Diffusers | 33.00 | | | 287.20 | | | 1034.60 | | | -| | NIM | 17.51 | 7.43 | 5.04 | 201.45 | 60.15 | 34.48 | 817.35 | 226.38 | 121.35 | -| **H100 80GB HBM3** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| | NIM | | 7.50 | 6.49 | | 56.33 | 36.81 | | 200.97 | 118.77 | -| **H200 141GB HBM3** | PyTorch | | 14.87 | 11.80 | | 70.45 | 42.10 | | 224.36 | 123.57 | -| | vLLM-Omni | 25.47 | | 6.32 | 220.70 | | 36.90 | 766.33 | 215.03 | 117.52 | -| | Diffusers | 31.00 | | | 249.20 | | | 879.20 | | | -| | NIM | 16.39 | 6.74 | 4.90 | 175.95 | 54.46 | 32.51 | 699.13 | 197.96 | 109.55 | -| **B200** | PyTorch | 14.71 | 5.63 | 4.12 | | 35.70 | 21.25 | 397.31 | 117.98 | 65.91 | -| | vLLM-Omni | 14.13 | | 5.31 | 115.17 | | 23.26 | 393.02 | 115.69 | 64.82 | -| | Diffusers | 19.20 | | | | | | 414.80 | | | -| | NIM | 9.36 | 4.83 | 4.12 | 83.19 | 29.09 | 19.14 | 316.76 | 94.65 | 55.92 | -| **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | 14.14 | | 7.19 | 111.42 | | 23.91 | 368.73 | 111.41 | 63.25 | -| | Diffusers | 54.20 | | | 151.80 | | | 425.00 | | | -| | NIM | 9.73 | 5.58 | 5.94 | 80.51 | 30.17 | 20.62 | 294.11 | 93.77 | 56.76 | - -### Text-to-Image (t2i) - -| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | -|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| **RTX PRO 6000 Blackwell** | PyTorch | | | | | | | | 92.61 | 93.11 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H20** | PyTorch | | | | | | | | | 20.92 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H100 NVL** | PyTorch | | 19.73 | 19.86 | | | | | 20.68 | 19.87 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 NVL** | PyTorch | | | | | | | | 32.64 | 33.16 | -| | vLLM-Omni | 2.73 | | 3.20 | 6.28 | | 17.71 | 11.02 | 4.22 | 3.26 | -| | Diffusers | 5.00 | | | 8.00 | | | 12.00 | | | -| **H100 80GB HBM3** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 141GB HBM3** | PyTorch | | 13.62 | 13.48 | | 13.33 | 13.53 | | 13.78 | 13.50 | -| | vLLM-Omni | 2.83 | | 4.50 | 5.70 | | 7.16 | 10.24 | 4.23 | 4.43 | -| | Diffusers | 5.00 | | | 8.00 | | | 11.00 | | | -| **B200** | PyTorch | | 4.10 | 4.27 | 4.78 | 4.13 | 4.48 | 7.25 | 4.28 | 4.65 | -| | vLLM-Omni | 2.32 | | 4.58 | 3.29 | | 9.10 | 6.02 | 3.09 | 4.43 | -| | Diffusers | | | | | | | 8.00 | | | -| **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | 5.05 | | 7.62 | 3.79 | | 72.65 | 7.08 | 5.79 | 7.24 | -| | Diffusers | 38.80 | | | 39.40 | | | 40.40 | | | - -Notes: -1. All times measured on identical workloads (same seed, sampler settings, prompt). -2. 4×/8× GPU configurations use tensor parallelism. -3. vLLM-Omni numbers are for the upcoming public release in the vLLM-Omni repo; subject to change before GA. Current vLLM-Omni coverage is B200 at 720p. -4. Diffusers numbers use the HuggingFace `diffusers` integration without custom CUDA graphs; reported at 256p/1, 480p/1, and 720p/1 (single-GPU only). -5. At 256p, multi-GPU configurations on B300 may underperform single-GPU due to small-workload TP overhead; single-GPU is recommended at this resolution. -6. PyTorch numbers report average generation (sampling) time from OSS inference benchmarking. -7. NIM numbers use latency profiles with FP8 precision and report end-to-end `Request Latency s`, including request processing, video generation, output encoding, and returning the response. - ## Cosmos3-Edge Reasoner These tables report **Cosmos3-Edge** reasoner serving performance through **vLLM**. Unlike the Generator benchmarks, Reasoner workloads produce autoregressively generated text and measure time to first token (TTFT), end-to-end request latency, request throughput, and output-token throughput. Lower is better for latency metrics; higher is better for throughput. From 8e394855405f0dff3dda519ba657eb0047e280a1 Mon Sep 17 00:00:00 2001 From: Sergiy Fefilatyev Date: Mon, 14 Sep 2026 16:36:00 -0700 Subject: [PATCH 2/9] Add per-model Cosmos3 Generator inference benchmark pages. Split Edge, Nano, and Super generator results into dedicated pages so audiovisual and action modalities can be reported without collapsing the landing page. --- .../cosmos3-edge-generator.md | 49 ++ .../cosmos3-nano-generator.md | 503 ++++++++++++++++++ .../cosmos3-super-generator.md | 503 ++++++++++++++++++ 3 files changed, 1055 insertions(+) create mode 100644 inference_benchmarks/cosmos3-edge-generator.md create mode 100644 inference_benchmarks/cosmos3-nano-generator.md create mode 100644 inference_benchmarks/cosmos3-super-generator.md diff --git a/inference_benchmarks/cosmos3-edge-generator.md b/inference_benchmarks/cosmos3-edge-generator.md new file mode 100644 index 00000000..f0e40cd0 --- /dev/null +++ b/inference_benchmarks/cosmos3-edge-generator.md @@ -0,0 +1,49 @@ +# Cosmos3-Edge Generator Inference Benchmarks + +[Back to the inference benchmark index](../inference_benchmarks.md) + +These tables report **Cosmos3-Edge** Generator latency in seconds for **image-to-video (i2v)**. Measurements use one GPU or one integrated computing platform. Lower latency is better, and empty cells indicate that a run has not been completed. + +Unless otherwise noted, visual-generation benchmarks use **480p resolution**. Video benchmarks generate **121 frames**. vLLM-Omni values report end-to-end latency, while PyTorch values report average generation latency. + +### vLLM-Omni + +| GPU or Platform | Image-to-Video | +|---|---:| +| B200 SXM 192 GB | 7.09 | +| B300 | 8.74 | +| H200 SXM 141 GB | 13.13 | +| H200 NVL | 14.15 | +| H100 SXM 80 GB | 13.33 | +| H100 NVL 96 GB | 17.33 | +| H20 SXM 96 GB | 53.94 | +| RTX PRO 6000 Blackwell Server Edition | | +| DGX Station | 7.13 | +| DGX Spark | 89.41 | +| Jetson AGX Thor T5000, 128 GB, MAXN | | +| Jetson T3000, 32 GB, 1100 MHz | | +| Jetson T2000, 16 GB, 702 MHz, THOR_NANO | | + +### PyTorch + +| GPU or Platform | Image-to-Video | +|---|---:| +| B200 SXM 192 GB | 7.45 | +| B300 | 6.84 | +| H200 SXM 141 GB | 12.31 | +| H200 NVL | 14.07 | +| H100 SXM 80 GB | 12.68 | +| H100 NVL 96 GB | 16.42 | +| H20 SXM 96 GB | 52.83 | +| RTX PRO 6000 Blackwell Server Edition | 21.92 | +| DGX Station | 6.31 | +| DGX Spark | 103.36 | +| Jetson AGX Thor T5000, 128 GB, MAXN | | +| Jetson T3000, 32 GB, 1100 MHz | | + +Notes: +1. All measurements use one GPU or one integrated computing platform. +2. Values are average end-to-end or generation latency in seconds; lower is better. +3. Unless otherwise specified, visual-generation measurements use **480p resolution**. +4. Video measurements (i2v) generate **121 output frames**. +5. PyTorch values report average generation latency rather than diffusion-only latency. diff --git a/inference_benchmarks/cosmos3-nano-generator.md b/inference_benchmarks/cosmos3-nano-generator.md new file mode 100644 index 00000000..4d77a606 --- /dev/null +++ b/inference_benchmarks/cosmos3-nano-generator.md @@ -0,0 +1,503 @@ +# Cosmos3-Nano Generator Inference Benchmarks + +[Back to the inference benchmark index](../inference_benchmarks.md) + +These tables report **Cosmos3-Nano Generator** latency in seconds. Lower is better. Empty cells mean that a run has not been completed for that GPU, engine, resolution, or GPU count; they do not indicate that a combination is unsupported. + +## Table of Contents + +- [Benchmark methodology](#benchmark-methodology) +- [Workload definitions](#workload-definitions) +- [Primary vision generation](#primary-vision-generation) + - [Text-to-Video (t2v)](#text-to-video-t2v) + - [Image-to-Video (i2v)](#image-to-video-i2v) + - [Text-to-Image (t2i)](#text-to-image-t2i) +- [Additional audiovisual generation](#additional-audiovisual-generation) + - [Video-to-Video (v2v)](#video-to-video-v2v) + - [Text-to-Audio-and-Video (t2av)](#text-to-audio-and-video-t2av) + - [Video-to-Audio-and-Video (v2av)](#video-to-audio-and-video-v2av) + - [Image-to-Audio-and-Video (i2av)](#image-to-audio-and-video-i2av) +- [Action generation](#action-generation) + - [Forward Dynamics — AV](#forward-dynamics--autonomous-vehicle-av) + - [Forward Dynamics — Camera](#forward-dynamics--camera) + - [Forward Dynamics — Robot](#forward-dynamics--robot) + - [Inverse Dynamics — AV](#inverse-dynamics--autonomous-vehicle-av) + - [Inverse Dynamics — Robot](#inverse-dynamics--robot) + - [Policy — AV](#policy--autonomous-vehicle-av) + - [Policy — Robot](#policy--robot) + +## Benchmark methodology + +The primary t2v, i2v, and t2i tables preserve the previously published benchmark campaigns across PyTorch, vLLM-Omni, Diffusers, and NIM. Those tables use BF16 precision, batch size 1, and matched prompts, seeds, and sampler settings where documented. Video workloads follow the standard Cosmos3 generation profile of 189 frames at 24 FPS unless a resolution tier limits frame count. + +The additional audiovisual and action tables come from PBR `#308197`, **Cosmos3-Generator OSS Inference Benchmarking 32B and 8B (189 frames)**. PyTorch values are average generation (sampling) latency from the native OSS path, using **CUDA Graphs disabled** and the **latency** automatic-sharding preset. Columns combine the 256p, 480p, and 720p tiers with 1, 4, or 8 GPUs. Values are rounded to two decimal places. The workbook does not include B300, so B300 is retained as an empty reservation. + +The PBR establishes the reported timing matrix but does not expose every prompt and action payload in this repository. The linked public recipes explain modality behavior and provide representative payloads; their example-specific frame counts and action chunk sizes should not be treated as the exact internal benchmark inputs. + +## Workload definitions + +| Workload | Input | Output | +|---|---|---| +| Video-to-video (`v2v`) | Text prompt and source video | Generated video | +| Text-to-audio-and-video (`t2av`) | Text prompt | Synchronized video and sound | +| Video-to-audio-and-video (`v2av`) | Text prompt and source video | Generated video with synchronized sound | +| Image-to-audio-and-video (`i2av`) | Text prompt and source image | Generated video with synchronized sound | +| Forward dynamics | Initial visual observation and an action trajectory | Future-observation rollout video | +| Inverse dynamics | Observed video | Recovered action trajectory; some serving integrations also return video | +| Policy | Initial visual observation, instruction, and optional state | Predicted action trajectory and, for general Generator paths, a rollout video | + +The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound-producing modes `t2vs`, `v2vs`, and `i2vs`. See the [audiovisual cookbook](../cookbooks/cosmos3/generator/audiovisual/README.md) for generation inputs and the [action cookbook](../cookbooks/cosmos3/generator/action/README.md) for action representations and output contracts. vLLM-Omni request shapes are maintained in the [Nano recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-Nano.md). Public recipes cover more embodiments than this PBR; the action tables below intentionally use only its measured domain rows. + +## Primary vision generation + +### Text-to-Video (t2v) + +| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | +|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 13.95 | 6.46 | 4.90 | 180.81 | 57.23 | 35.71 | 786.37 | 225.45 | 127.57 | +| | vLLM-Omni | 10.65 | 5.06 | 3.78 | 105.61 | 35.93 | 23.76 | 369.67 | 114.30 | 68.66 | +| | Diffusers | 11.20 | | | 112.00 | | | 392.00 | | | +| | NIM | 7.93 | 4.69 | 4.00 | 82.69 | 33.57 | 24.49 | 318.69 | 107.83 | 68.50 | +| **H20** | PyTorch | 30.57 | 11.25 | 7.87 | 257.51 | 80.18 | 50.49 | 931.39 | 268.88 | 157.71 | +| | vLLM-Omni | 28.58 | 10.20 | 7.70 | 256.97 | 77.42 | 47.53 | 929.81 | 260.75 | 148.46 | +| | Diffusers | 30.20 | | | 258.00 | | | 926.00 | | | +| | NIM | 17.81 | 7.84 | 6.14 | 192.07 | 62.80 | 41.67 | 771.37 | 223.24 | 132.68 | +| **H100 NVL** | PyTorch | 10.03 | 4.27 | 3.95 | 84.12 | 29.18 | 21.46 | 297.27 | 94.15 | 61.63 | +| | vLLM-Omni | 9.25 | 3.68 | 3.15 | 80.75 | 27.48 | 18.77 | 311.13 | 88.25(*) | 54.01(*) | +| | Diffusers | 11.00 | | | 90.00 | | | 324.20 | | | +| | NIM | 7.09 | 3.77 | 3.65 | 68.01 | 25.67 | 20.51 | 267.73 | 86.18 | 57.38 | +| **H200 NVL** | PyTorch | 8.17 | 3.75 | 3.14 | 69.79 | 24.23 | 15.10 | 244.39 | 77.35 | 45.70 | +| | vLLM-Omni | 7.44 | 3.27 | 2.33 | 64.58 | 21.31 | 12.92 | 240.05 | 69.63 | 39.17 | +| | Diffusers | 9.00 | | | 74.00 | | | 276.20 | | | +| | NIM | 5.87 | 3.38 | 3.00 | 57.10 | 21.74 | 15.04 | 229.63 | 71.32 | 43.34 | +| **H100 80GB HBM3** | PyTorch | 7.61 | 3.50 | 3.17 | 59.83 | 21.23 | 14.37 | 207.78 | 66.94 | 41.81 | +| | vLLM-Omni | 6.97 | 3.45 | 3.49 | 58.17 | 19.95 | 13.46 | 202.29 | 62.82 | 37.80 | +| | Diffusers | 9.00 | | | 68.00 | | | 240.00 | | | +| | NIM | 5.72 | 3.36 | 3.12 | 51.73 | 20.26 | 14.82 | 199.46 | 65.32 | 41.66 | +| **H200 141GB HBM3** | PyTorch | 7.53 | 3.34 | 3.19 | 60.18 | 20.84 | 13.97 | 214.28 | 67.48 | 41.26 | +| | vLLM-Omni | 6.79 | 3.25 | 3.42 | 58.14 | 19.77 | 12.97 | 208.36 | 63.27 | 37.49 | +| | Diffusers | 9.00 | | | 67.00 | | | 239.60 | | | +| | NIM | 5.72 | 3.26 | 3.10 | 51.90 | 20.09 | 14.61 | 200.63 | 64.57 | 40.70 | +| **B200** | PyTorch | 4.56 | 2.78 | 2.79 | 33.20 | 13.20 | 9.69 | 114.85 | 39.75 | 26.27 | +| | vLLM-Omni | 4.03 | 2.43 | 3.49 | 32.04 | 12.63 | 10.09 | 107.84 | 35.29 | 22.87 | +| | Diffusers | 7.00 | | | 36.80 | | | 117.00 | | | +| | NIM | 3.72 | 2.74 | 2.89 | 26.68 | 12.66 | 10.10 | 93.33 | 35.04 | 24.64 | +| **B300** | PyTorch | | | | | | | | | | +| | vLLM-Omni | 4.46 | 4.11 | 5.44 | 32.18 | 13.83 | 11.57 | 102.10 | 35.68 | 24.33 | +| | Diffusers | 39.40 | | | 63.40 | | | 139.40 | | | +| | NIM | 4.51 | 3.54 | 4.11 | 27.49 | 14.51 | 11.86 | 90.39 | 37.28 | 26.19 | + +### Image-to-Video (i2v) + +| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | +|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 14.08 | 6.51 | 4.92 | 182.14 | 57.43 | 35.74 | 788.80 | 226.25 | 127.79 | +| | vLLM-Omni | 11.04 | 5.48 | 4.24 | 107.77 | 38.05 | 25.95 | 375.01 | 119.27 | 73.57 | +| | Diffusers | 12.00 | | | 112.00 | | | 397.00 | | | +| | NIM | 8.23 | 5.28 | 4.64 | 84.58 | 35.74 | 26.63 | 326.96 | 112.27 | 73.31 | +| **H20** | PyTorch | 31.36 | 11.39 | 7.97 | 257.10 | 80.53 | 50.63 | 933.07 | 268.99 | 158.10 | +| | vLLM-Omni | 29.50 | 11.26 | 8.64 | 261.56 | 81.93 | 52.06 | 940.16 | 271.37 | 158.76 | +| | Diffusers | 31.00 | | | 258.00 | | | 925.00 | | | +| | NIM | 18.67 | 9.04 | 7.38 | 195.10 | 67.64 | 46.30 | 774.88 | 233.92 | 143.15 | +| **H100 NVL** | PyTorch | 10.19 | 4.31 | 3.99 | 84.50 | 28.69 | 21.52 | 298.57 | 95.76 | 60.58 | +| | vLLM-Omni | 9.62 | 4.11 | 3.63 | 82.61 | 29.35 | 20.73 | 286.33 | 92.23(*) | 58.02(*) | +| | Diffusers | 11.00 | | | 91.00 | | | 325.20 | | | +| | NIM | 7.39 | 4.36 | 4.28 | 69.39 | 27.67 | 22.43 | 272.29 | 90.26 | 61.55 | +| **H200 NVL** | PyTorch | 8.27 | 3.76 | 3.09 | 69.99 | 24.22 | 15.12 | 246.62 | 77.69 | 45.99 | +| | vLLM-Omni | 7.83 | 3.69 | 2.78 | 66.39 | 22.93 | 14.58 | 243.52 | 73.26 | 42.86 | +| | Diffusers | 9.00 | | | 74.00 | | | 275.20 | | | +| | NIM | 6.25 | 3.97 | 3.60 | 58.64 | 23.33 | 16.75 | 232.47 | 75.13 | 47.22 | +| **H100 80GB HBM3** | PyTorch | 7.64 | 3.47 | 3.21 | 59.95 | 21.40 | 14.43 | 207.87 | 67.52 | 41.66 | +| | vLLM-Omni | 7.37 | 3.81 | 3.97 | 59.77 | 21.68 | 15.12 | 205.97 | 66.52 | 41.51 | +| | Diffusers | 9.00 | | | 68.00 | | | 239.80 | | | +| | NIM | 6.08 | 3.89 | 3.71 | 53.02 | 22.20 | 16.61 | 202.59 | 69.02 | 45.16 | +| **H200 141GB HBM3** | PyTorch | 7.65 | 3.37 | 3.17 | 60.51 | 21.01 | 14.07 | 214.80 | 67.14 | 41.00 | +| | vLLM-Omni | 7.28 | 3.63 | 3.83 | 59.64 | 21.35 | 14.67 | 209.65 | 66.65 | 40.77 | +| | Diffusers | 9.00 | | | 67.20 | | | 240.00 | | | +| | NIM | 6.04 | 3.80 | 3.65 | 53.25 | 21.89 | 16.23 | 203.66 | 68.30 | 44.29 | +| **B200** | PyTorch | 4.60 | 2.77 | 2.81 | 33.08 | 13.07 | 9.66 | 113.90 | 40.01 | 26.58 | +| | vLLM-Omni | 4.33 | 2.77 | 3.84 | 33.09 | 13.79 | 11.39 | 110.19 | 37.76 | 25.68 | +| | Diffusers | | | | | | | 116.00 | | | +| | NIM | 4.05 | 3.43 | 3.60 | 27.72 | 14.02 | 11.43 | 95.57 | 37.76 | 27.29 | +| **B300** | PyTorch | | | | | | | | | | +| | vLLM-Omni | 5.61 | 4.67 | 5.90 | 33.45 | 15.06 | 13.13 | 104.75 | 38.27 | 26.87 | +| | Diffusers | 28.60 | | | 65.60 | | | 139.60 | | | +| | NIM | 4.50 | 4.96 | 5.03 | 28.90 | 16.06 | 13.25 | 92.59 | 39.49 | 29.00 | + +### Text-to-Image (t2i) + +| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | +|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 2.99 | 2.36 | 2.38 | 4.51 | 2.46 | 2.48 | 7.12 | 3.18 | 2.70 | +| | vLLM-Omni | 1.59 | 1.54 | 1.59 | 2.87 | 1.55 | 1.81 | 4.99 | 2.32 | 1.96 | +| | Diffusers | 2.00 | | | 4.00 | | | 5.00 | | | +| **H20** | PyTorch | 3.06 | 2.65 | 2.74 | 6.51 | 2.84 | 2.89 | 12.31 | 4.28 | 3.06 | +| | vLLM-Omni | 1.73 | 2.46 | 3.22 | 4.92 | 2.57 | 7.24 | 10.73 | 4.24 | 3.59 | +| | Diffusers | 3.00 | | | 6.00 | | | 10.00 | | | +| **H100 NVL** | PyTorch | 2.77 | 2.45 | 2.57 | 2.83 | 2.56 | 2.51 | 4.21 | 2.57 | 2.64 | +| | vLLM-Omni | 1.55 | 1.75 | 1.91 | 1.92 | 1.81 | 10.82 | 3.44 | 1.83 | 1.90 | +| | Diffusers | 3.00 | | | 3.00 | | | 4.00 | | | +| **H200 NVL** | PyTorch | 2.75 | 2.42 | 2.50 | 2.85 | 2.52 | 2.62 | 3.58 | 2.62 | 2.64 | +| | vLLM-Omni | 1.53 | 2.01 | 1.96 | 1.58 | 1.91 | 17.71 | 2.81 | 1.94 | 1.94 | +| | Diffusers | 3.00 | | | 3.00 | | | 4.00 | | | +| **H100 80GB HBM3** | PyTorch | 3.01 | 2.66 | 2.56 | 3.01 | 2.59 | 2.75 | 3.45 | 2.73 | 2.77 | +| | vLLM-Omni | 1.61 | 2.45 | 3.18 | 1.53 | 2.35 | 7.02 | 2.61 | 2.45 | 3.03 | +| | Diffusers | 3.00 | | | 3.00 | | | 4.00 | | | +| **H200 141GB HBM3** | PyTorch | 2.96 | 2.59 | 2.70 | 3.04 | 2.78 | 2.77 | 3.28 | 2.84 | 2.77 | +| | vLLM-Omni | 1.57 | 2.38 | 3.16 | 1.52 | 2.37 | 7.05 | 2.60 | 2.33 | 3.20 | +| | Diffusers | 3.00 | | | 3.00 | | | 4.00 | | | +| **B200** | PyTorch | 2.68 | 2.39 | 2.59 | 2.75 | 2.43 | 2.56 | 2.87 | 2.58 | 2.62 | +| | vLLM-Omni | 1.49 | 2.21 | 3.27 | 1.20 | 2.05 | 7.58 | 1.77 | 2.20 | 3.41 | +| | Diffusers | | | | | | | 3.00 | | | +| **B300** | PyTorch | | | | | | | | | | +| | vLLM-Omni | 1.97 | 4.52 | 5.82 | 1.81 | 4.16 | 71.19 | 2.34 | 4.09 | 5.62 | +| | Diffusers | 36.20 | | | | | | 41.00 | | | + +Notes: +1. All times measured on identical workloads (same seed, sampler settings, prompt). +2. 4×/8× GPU configurations use tensor parallelism. +3. vLLM-Omni numbers are for the upcoming public release in the vLLM-Omni repo; subject to change before GA. Values marked with (*) are pre-release vLLM-Omni measurements on H100 NVL and may change before GA. +4. Diffusers numbers use the HuggingFace `diffusers` integration without custom CUDA graphs; reported at 256p/1, 480p/1, and 720p/1 (single-GPU only). +5. PyTorch numbers report average generation (sampling) time from OSS inference benchmarking. +6. At 256p, multi-GPU configurations on B300 may underperform single-GPU due to small-workload TP overhead; single-GPU is recommended at this resolution. +7. NIM numbers use latency profiles with FP8 precision and report end-to-end `Request Latency s`, including request processing, video generation, output encoding, and returning the response. + +## Additional audiovisual generation + +### Video-to-Video (v2v) + +A text prompt and source video condition a generated continuation or transformation. + +| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | +|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 13.73 | 6.49 | 4.88 | 180.54 | 57.44 | 35.65 | 785.41 | 224.94 | 127.19 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H20** | PyTorch | 30.44 | 11.35 | 7.94 | 257.79 | 80.52 | 50.56 | 930.96 | 268.82 | 157.81 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 NVL** | PyTorch | 10.18 | 4.29 | 3.95 | 81.73 | 28.86 | 21.40 | 297.83 | 94.37 | 61.31 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 NVL** | PyTorch | 8.10 | 3.73 | 3.12 | 69.70 | 24.17 | 15.14 | 250.33 | 76.59 | 45.94 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 80GB HBM3** | PyTorch | 7.50 | 3.47 | 3.16 | 59.75 | 21.10 | 14.29 | 207.36 | 67.05 | 41.54 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 141GB HBM3** | PyTorch | 7.48 | 3.35 | 3.11 | 60.23 | 20.99 | 13.94 | 214.03 | 66.90 | 41.06 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B200** | PyTorch | 4.52 | 2.74 | 2.78 | 32.53 | 13.06 | 9.68 | 114.16 | 39.67 | 26.33 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B300** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | + +### Text-to-Audio-and-Video (t2av) + +A text prompt produces synchronized video and sound. + +| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | +|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 14.13 | 6.47 | 4.89 | 180.98 | 57.21 | 35.77 | 794.74 | 225.73 | 127.53 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H20** | PyTorch | 30.82 | 11.21 | 7.87 | 256.41 | 80.22 | 50.51 | 922.35 | 268.73 | 158.03 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 NVL** | PyTorch | 9.99 | 4.27 | 3.95 | 82.32 | 28.50 | 21.46 | 297.88 | 94.45 | 60.25 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 NVL** | PyTorch | 8.20 | 3.75 | 3.08 | 69.88 | 24.30 | 15.17 | 250.46 | 78.00 | 45.75 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 80GB HBM3** | PyTorch | 7.60 | 3.43 | 3.18 | 59.09 | 21.18 | 14.32 | 207.87 | 66.74 | 41.71 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 141GB HBM3** | PyTorch | 7.55 | 3.33 | 3.13 | 60.17 | 20.86 | 13.90 | 211.59 | 67.02 | 41.24 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B200** | PyTorch | 4.63 | 2.81 | 2.81 | 32.60 | 13.17 | 9.64 | 113.89 | 39.59 | 26.33 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B300** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | + +### Video-to-Audio-and-Video (v2av) + +A text prompt and source video produce transformed video with synchronized sound. + +| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | +|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 13.73 | 6.45 | 4.88 | 180.72 | 57.30 | 35.73 | 786.11 | 224.99 | 127.45 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H20** | PyTorch | 30.42 | 11.35 | 7.94 | 257.83 | 80.48 | 50.55 | 921.82 | 268.81 | 158.07 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 NVL** | PyTorch | 9.91 | 4.35 | 3.95 | 81.96 | 29.02 | 21.53 | 297.51 | 94.64 | 60.36 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 NVL** | PyTorch | 8.10 | 3.73 | 3.09 | 69.59 | 24.39 | 15.08 | 246.51 | 76.46 | 46.27 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 80GB HBM3** | PyTorch | 7.49 | 3.46 | 3.17 | 59.86 | 21.25 | 14.21 | 207.55 | 67.24 | 41.63 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 141GB HBM3** | PyTorch | 7.49 | 3.35 | 3.10 | 60.15 | 21.00 | 13.93 | 214.44 | 67.13 | 41.05 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B200** | PyTorch | 4.51 | 2.76 | 2.82 | 32.62 | 13.05 | 9.61 | 113.72 | 39.28 | 26.18 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B300** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | + +### Image-to-Audio-and-Video (i2av) + +A text prompt and source image produce video with synchronized sound. + +| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | +|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 14.08 | 6.48 | 4.91 | 181.64 | 57.25 | 35.69 | 787.81 | 226.71 | 127.85 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H20** | PyTorch | 31.34 | 11.36 | 7.97 | 259.06 | 80.55 | 50.63 | 924.39 | 269.21 | 157.97 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 NVL** | PyTorch | 10.45 | 4.31 | 3.97 | 84.37 | 28.74 | 21.46 | 298.61 | 94.02 | 60.57 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 NVL** | PyTorch | 8.29 | 3.76 | 3.08 | 71.20 | 24.34 | 15.05 | 248.35 | 76.82 | 46.40 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 80GB HBM3** | PyTorch | 7.73 | 3.46 | 3.11 | 59.13 | 21.24 | 14.30 | 207.93 | 67.50 | 41.75 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 141GB HBM3** | PyTorch | 7.66 | 3.35 | 3.14 | 60.55 | 20.98 | 13.99 | 214.82 | 67.35 | 41.21 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B200** | PyTorch | 4.62 | 2.78 | 2.80 | 32.82 | 13.09 | 9.69 | 113.85 | 39.76 | 26.56 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B300** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | + +## Action generation + +Forward dynamics is reported separately for AV, camera, and robot inputs. Inverse dynamics and policy are reported for AV and robot because those are the only domain rows in this PBR; no camera row is inferred. For each domain, PyTorch is populated from the PBR while vLLM-Omni and Diffusers rows are reserved for future measurements. + +### Forward Dynamics — Autonomous Vehicle (AV) + +| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | +|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 16.87 | 12.13 | 8.96 | 16.86 | 12.18 | 8.95 | 16.85 | 12.14 | 8.94 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H20** | PyTorch | 31.47 | 19.97 | 13.77 | 31.16 | 19.91 | 13.78 | 31.45 | 19.95 | 13.78 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 NVL** | PyTorch | 10.32 | 7.36 | 6.46 | 10.66 | 7.35 | 6.31 | 10.64 | 7.43 | 6.32 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 NVL** | PyTorch | 8.30 | 6.18 | 4.41 | 8.31 | 6.16 | 4.39 | 8.33 | 6.14 | 4.39 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 80GB HBM3** | PyTorch | 7.80 | 5.69 | 4.30 | 7.87 | 5.68 | 4.34 | 7.79 | 5.68 | 4.30 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 141GB HBM3** | PyTorch | 7.77 | 5.51 | 4.16 | 7.76 | 5.56 | 4.18 | 7.81 | 5.54 | 4.16 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B200** | PyTorch | 4.75 | 3.88 | 3.38 | 4.75 | 3.85 | 3.37 | 4.73 | 3.87 | 3.36 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B300** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | + +### Forward Dynamics — Camera + +| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | +|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 16.82 | 12.01 | 8.80 | 16.82 | 12.01 | 8.80 | 16.81 | 12.01 | 8.81 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H20** | PyTorch | 31.40 | 19.68 | 13.52 | 31.10 | 19.69 | 13.50 | 31.40 | 19.75 | 13.52 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 NVL** | PyTorch | 10.61 | 7.35 | 6.19 | 10.32 | 7.27 | 6.31 | 10.57 | 7.35 | 6.25 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 NVL** | PyTorch | 8.24 | 6.08 | 4.32 | 8.26 | 6.09 | 4.32 | 8.26 | 6.09 | 4.33 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 80GB HBM3** | PyTorch | 7.78 | 5.64 | 4.24 | 7.83 | 5.61 | 4.21 | 7.75 | 5.60 | 4.21 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 141GB HBM3** | PyTorch | 7.73 | 5.44 | 4.08 | 7.75 | 5.44 | 4.09 | 7.75 | 5.49 | 4.08 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B200** | PyTorch | 4.70 | 3.79 | 3.23 | 4.74 | 3.78 | 3.27 | 4.71 | 3.78 | 3.23 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B300** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | + +### Forward Dynamics — Robot + +| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | +|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 3.74 | 3.15 | 2.62 | 3.73 | 3.14 | 2.63 | 3.72 | 3.13 | 2.59 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H20** | PyTorch | 7.00 | 4.78 | 3.59 | 7.01 | 4.79 | 3.59 | 7.01 | 4.78 | 3.59 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 NVL** | PyTorch | 2.49 | 2.15 | 2.18 | 2.50 | 2.16 | 2.12 | 2.55 | 2.11 | 2.15 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 NVL** | PyTorch | 2.08 | 2.14 | 2.11 | 2.09 | 2.03 | 2.08 | 2.09 | 2.03 | 2.09 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 80GB HBM3** | PyTorch | 2.00 | 2.14 | 2.15 | 2.01 | 2.14 | 2.17 | 1.98 | 2.15 | 2.25 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 141GB HBM3** | PyTorch | 1.93 | 2.14 | 2.30 | 1.93 | 2.14 | 2.15 | 1.93 | 2.11 | 2.21 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B200** | PyTorch | 1.28 | 1.89 | 1.99 | 1.30 | 1.91 | 2.00 | 1.31 | 1.91 | 1.94 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B300** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | + +### Inverse Dynamics — Autonomous Vehicle (AV) + +| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | +|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 16.74 | 12.05 | 8.79 | 16.75 | 12.02 | 8.78 | 16.75 | 12.02 | 8.78 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H20** | PyTorch | 31.22 | 19.66 | 13.52 | 30.99 | 19.68 | 13.52 | 31.22 | 19.70 | 13.53 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 NVL** | PyTorch | 10.47 | 7.13 | 6.16 | 10.45 | 7.16 | 6.26 | 10.44 | 7.10 | 6.14 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 NVL** | PyTorch | 8.12 | 5.96 | 4.19 | 8.13 | 5.97 | 4.20 | 8.10 | 5.98 | 4.20 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 80GB HBM3** | PyTorch | 7.62 | 5.54 | 4.15 | 7.63 | 5.52 | 4.16 | 7.68 | 5.53 | 4.14 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 141GB HBM3** | PyTorch | 7.51 | 5.36 | 4.00 | 7.64 | 5.34 | 3.99 | 7.51 | 5.37 | 3.99 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B200** | PyTorch | 4.60 | 3.74 | 3.20 | 4.63 | 3.73 | 3.19 | 4.58 | 3.76 | 3.23 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B300** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | + +### Inverse Dynamics — Robot + +| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | +|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 3.70 | 3.12 | 2.61 | 3.70 | 3.11 | 2.60 | 3.70 | 3.11 | 2.57 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H20** | PyTorch | 6.94 | 4.72 | 3.53 | 6.94 | 4.72 | 3.52 | 6.89 | 4.71 | 3.52 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 NVL** | PyTorch | 2.53 | 2.12 | 2.09 | 2.47 | 2.07 | 2.09 | 2.54 | 2.11 | 2.09 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 NVL** | PyTorch | 2.05 | 1.99 | 2.09 | 2.05 | 2.01 | 2.07 | 2.05 | 2.05 | 2.05 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 80GB HBM3** | PyTorch | 1.96 | 2.09 | 2.16 | 1.96 | 2.07 | 2.18 | 1.95 | 2.17 | 2.13 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 141GB HBM3** | PyTorch | 1.90 | 2.06 | 2.10 | 1.90 | 2.07 | 2.13 | 1.91 | 2.12 | 2.09 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B200** | PyTorch | 1.28 | 1.91 | 1.91 | 1.27 | 1.87 | 1.97 | 1.28 | 1.86 | 1.91 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B300** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | + +### Policy — Autonomous Vehicle (AV) + +| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | +|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 16.88 | 12.13 | 8.91 | 16.89 | 12.16 | 8.93 | 16.91 | 12.13 | 8.93 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H20** | PyTorch | 31.51 | 19.90 | 13.77 | 31.51 | 19.94 | 13.79 | 31.50 | 19.97 | 13.78 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 NVL** | PyTorch | 10.66 | 7.34 | 6.27 | 10.65 | 7.41 | 6.33 | 10.35 | 7.33 | 6.27 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 NVL** | PyTorch | 8.31 | 6.17 | 4.38 | 8.32 | 6.16 | 4.39 | 8.31 | 6.14 | 4.38 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 80GB HBM3** | PyTorch | 7.82 | 5.68 | 4.31 | 7.82 | 5.67 | 4.33 | 7.80 | 5.67 | 4.32 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 141GB HBM3** | PyTorch | 7.76 | 5.52 | 4.15 | 7.69 | 5.58 | 4.16 | 7.66 | 5.51 | 4.16 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B200** | PyTorch | 4.75 | 3.89 | 3.34 | 4.77 | 3.88 | 3.34 | 4.74 | 3.87 | 3.28 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B300** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | + +### Policy — Robot + +| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | +|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 3.74 | 3.14 | 2.60 | 3.74 | 3.15 | 2.62 | 3.74 | 3.14 | 2.59 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H20** | PyTorch | 6.95 | 4.79 | 3.60 | 7.00 | 4.78 | 3.59 | 7.01 | 4.78 | 3.60 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 NVL** | PyTorch | 2.50 | 2.11 | 2.12 | 2.51 | 2.11 | 2.13 | 2.55 | 2.17 | 2.16 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 NVL** | PyTorch | 2.11 | 2.03 | 2.10 | 2.09 | 2.15 | 2.10 | 2.09 | 2.02 | 2.11 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 80GB HBM3** | PyTorch | 1.99 | 2.12 | 2.20 | 1.99 | 2.16 | 2.19 | 1.99 | 2.11 | 2.23 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 141GB HBM3** | PyTorch | 1.93 | 2.10 | 2.21 | 1.93 | 2.10 | 2.16 | 1.94 | 2.11 | 2.10 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B200** | PyTorch | 1.30 | 1.91 | 1.95 | 1.30 | 1.92 | 1.91 | 1.30 | 1.90 | 2.00 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B300** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | + +Additional-modality notes: +1. All reported values are average PyTorch generation (sampling) latency in seconds; lower is better. +2. PyTorch values use `CUDA_GRAPH=No` and the `latency` automatic-sharding preset. +3. The `/1`, `/4`, and `/8` suffixes denote the number of GPUs used by the benchmark run. +4. vLLM-Omni and Diffusers rows are intentionally empty reservations for future benchmark campaigns. +5. Empty cells indicate unmeasured combinations, not unsupported combinations. diff --git a/inference_benchmarks/cosmos3-super-generator.md b/inference_benchmarks/cosmos3-super-generator.md new file mode 100644 index 00000000..f7164232 --- /dev/null +++ b/inference_benchmarks/cosmos3-super-generator.md @@ -0,0 +1,503 @@ +# Cosmos3-Super Generator Inference Benchmarks + +[Back to the inference benchmark index](../inference_benchmarks.md) + +These tables report **Cosmos3-Super Generator** latency in seconds. Lower is better. Empty cells mean that a run has not been completed for that GPU, engine, resolution, or GPU count; they do not indicate that a combination is unsupported. + +## Table of Contents + +- [Benchmark methodology](#benchmark-methodology) +- [Workload definitions](#workload-definitions) +- [Primary vision generation](#primary-vision-generation) + - [Text-to-Video (t2v)](#text-to-video-t2v) + - [Image-to-Video (i2v)](#image-to-video-i2v) + - [Text-to-Image (t2i)](#text-to-image-t2i) +- [Additional audiovisual generation](#additional-audiovisual-generation) + - [Video-to-Video (v2v)](#video-to-video-v2v) + - [Text-to-Audio-and-Video (t2av)](#text-to-audio-and-video-t2av) + - [Video-to-Audio-and-Video (v2av)](#video-to-audio-and-video-v2av) + - [Image-to-Audio-and-Video (i2av)](#image-to-audio-and-video-i2av) +- [Action generation](#action-generation) + - [Forward Dynamics — AV](#forward-dynamics--autonomous-vehicle-av) + - [Forward Dynamics — Camera](#forward-dynamics--camera) + - [Forward Dynamics — Robot](#forward-dynamics--robot) + - [Inverse Dynamics — AV](#inverse-dynamics--autonomous-vehicle-av) + - [Inverse Dynamics — Robot](#inverse-dynamics--robot) + - [Policy — AV](#policy--autonomous-vehicle-av) + - [Policy — Robot](#policy--robot) + +## Benchmark methodology + +The primary t2v, i2v, and t2i tables preserve the previously published benchmark campaigns across PyTorch, vLLM-Omni, Diffusers, and NIM. Those tables use BF16 precision, batch size 1, and matched prompts, seeds, and sampler settings where documented. Video workloads follow the standard Cosmos3 generation profile of 189 frames at 24 FPS unless a resolution tier limits frame count. + +The additional audiovisual and action tables come from PBR `#308197`, **Cosmos3-Generator OSS Inference Benchmarking 32B and 8B (189 frames)**. PyTorch values are average generation (sampling) latency from the native OSS path, using **CUDA Graphs disabled** and the **latency** automatic-sharding preset. Columns combine the 256p, 480p, and 720p tiers with 1, 4, or 8 GPUs. Values are rounded to two decimal places. The workbook does not include B300, so B300 is retained as an empty reservation. + +The PBR establishes the reported timing matrix but does not expose every prompt and action payload in this repository. The linked public recipes explain modality behavior and provide representative payloads; their example-specific frame counts and action chunk sizes should not be treated as the exact internal benchmark inputs. + +## Workload definitions + +| Workload | Input | Output | +|---|---|---| +| Video-to-video (`v2v`) | Text prompt and source video | Generated video | +| Text-to-audio-and-video (`t2av`) | Text prompt | Synchronized video and sound | +| Video-to-audio-and-video (`v2av`) | Text prompt and source video | Generated video with synchronized sound | +| Image-to-audio-and-video (`i2av`) | Text prompt and source image | Generated video with synchronized sound | +| Forward dynamics | Initial visual observation and an action trajectory | Future-observation rollout video | +| Inverse dynamics | Observed video | Recovered action trajectory; some serving integrations also return video | +| Policy | Initial visual observation, instruction, and optional state | Predicted action trajectory and, for general Generator paths, a rollout video | + +The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound-producing modes `t2vs`, `v2vs`, and `i2vs`. See the [audiovisual cookbook](../cookbooks/cosmos3/generator/audiovisual/README.md) for generation inputs and the [action cookbook](../cookbooks/cosmos3/generator/action/README.md) for action representations and output contracts. vLLM-Omni request shapes are maintained in the [Super recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-Super.md). Public recipes cover more embodiments than this PBR; the action tables below intentionally use only its measured domain rows. + +## Primary vision generation + +### Text-to-Video (t2v) + +| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | +|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | | 65.10 | 66.04 | | 201.16 | 118.90 | | 789.03 | 427.16 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| | NIM | | 12.65 | 13.99 | | 104.25 | 99.05 | | 350.74 | 286.02 | +| **H20** | PyTorch | | 40.74 | 27.72 | | 276.24 | 152.46 | | 930.45 | 492.41 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| | NIM | | 20.07 | 12.95 | | 192.45 | 110.71 | | 734.37 | 395.56 | +| **H100 NVL** | PyTorch | | 20.73 | 16.83 | | 101.27 | 64.14 | | 330.04 | 186.19 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| | NIM | | 8.77 | 12.73 | | 73.37 | 66.07 | | 267.64 | 197.32 | +| **H200 NVL** | PyTorch | | 24.90 | 24.78 | | 82.19 | 47.69 | | 267.66 | 142.35 | +| | vLLM-Omni | 27.54 | | 5.06 | 252.33 | | 36.66 | 911.49 | 245.51 | 123.85 | +| | Diffusers | 33.00 | | | 286.80 | | | 1036.00 | | | +| | NIM | 17.13 | 6.79 | 4.43 | 200.00 | 58.55 | 32.87 | 811.41 | 223.00 | 117.98 | +| **H100 80GB HBM3** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| | NIM | | 6.98 | 5.89 | | 55.10 | 35.52 | | 198.13 | 114.92 | +| **H200 141GB HBM3** | PyTorch | | 14.82 | 11.82 | | 70.27 | 41.78 | | 224.43 | 123.49 | +| | vLLM-Omni | 25.61 | | 5.87 | 219.11 | | 35.26 | 769.63 | 212.30 | 111.94 | +| | Diffusers | 31.00 | | | 251.60 | | | 886.20 | | | +| | NIM | 15.95 | 6.14 | 4.28 | 174.71 | 52.94 | 30.94 | 695.89 | 194.34 | 106.16 | +| **B200** | PyTorch | 14.66 | 5.59 | 4.09 | 114.38 | 35.73 | 21.39 | 407.50 | 118.38 | 65.93 | +| | vLLM-Omni | 13.84 | | 4.76 | 114.08 | | 22.09 | 390.28 | 113.31 | 62.11 | +| | Diffusers | | | | 127.20 | | | 414.40 | | | +| | NIM | 9.09 | 4.26 | 3.38 | 82.39 | 27.83 | 17.74 | 314.68 | 92.25 | 53.43 | +| **B300** | PyTorch | | | | | | | | | | +| | vLLM-Omni | 14.57 | | 6.68 | 109.03 | | 22.67 | 366.66 | 108.58 | 60.73 | +| | Diffusers | 54.20 | | | 155.40 | | | 424.80 | | | +| | NIM | 9.67 | 5.23 | 5.19 | 79.73 | 28.97 | 18.39 | 292.35 | 92.31 | 54.07 | + +### Image-to-Video (i2v) + +| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | +|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | | 65.42 | 66.54 | | 202.26 | 118.95 | | 795.14 | 427.96 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| | NIM | | 13.17 | 14.48 | | 106.50 | 100.23 | | 356.60 | 289.93 | +| **H20** | PyTorch | | 41.37 | 28.22 | | 278.23 | 153.19 | | 931.74 | 491.93 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| | NIM | | 21.16 | 14.06 | | 196.86 | 114.49 | | 745.12 | 405.68 | +| **H100 NVL** | PyTorch | | 20.85 | 16.96 | | 99.56 | 64.17 | | 331.40 | 186.47 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| | NIM | | 9.32 | 13.30 | | 75.31 | 67.34 | | 271.46 | 201.39 | +| **H200 NVL** | PyTorch | | 24.81 | 24.70 | | 82.51 | 47.60 | | 269.85 | 141.62 | +| | vLLM-Omni | 27.90 | | 5.52 | 254.29 | | 38.51 | 915.05 | 248.89 | 127.32 | +| | Diffusers | 33.00 | | | 287.20 | | | 1034.60 | | | +| | NIM | 17.51 | 7.43 | 5.04 | 201.45 | 60.15 | 34.48 | 817.35 | 226.38 | 121.35 | +| **H100 80GB HBM3** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| | NIM | | 7.50 | 6.49 | | 56.33 | 36.81 | | 200.97 | 118.77 | +| **H200 141GB HBM3** | PyTorch | | 14.87 | 11.80 | | 70.45 | 42.10 | | 224.36 | 123.57 | +| | vLLM-Omni | 25.47 | | 6.32 | 220.70 | | 36.90 | 766.33 | 215.03 | 117.52 | +| | Diffusers | 31.00 | | | 249.20 | | | 879.20 | | | +| | NIM | 16.39 | 6.74 | 4.90 | 175.95 | 54.46 | 32.51 | 699.13 | 197.96 | 109.55 | +| **B200** | PyTorch | 14.71 | 5.63 | 4.12 | 112.40 | 35.70 | 21.25 | 397.31 | 117.98 | 65.91 | +| | vLLM-Omni | 14.13 | | 5.31 | 115.17 | | 23.26 | 393.02 | 115.69 | 64.82 | +| | Diffusers | 19.20 | | | | | | 414.80 | | | +| | NIM | 9.36 | 4.83 | 4.12 | 83.19 | 29.09 | 19.14 | 316.76 | 94.65 | 55.92 | +| **B300** | PyTorch | | | | | | | | | | +| | vLLM-Omni | 14.14 | | 7.19 | 111.42 | | 23.91 | 368.73 | 111.41 | 63.25 | +| | Diffusers | 54.20 | | | 151.80 | | | 425.00 | | | +| | NIM | 9.73 | 5.58 | 5.94 | 80.51 | 30.17 | 20.62 | 294.11 | 93.77 | 56.76 | + +### Text-to-Image (t2i) + +| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | +|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | | 93.20 | 93.18 | | 92.22 | 93.39 | | 92.89 | 93.47 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H20** | PyTorch | | 14.58 | 14.18 | | 19.93 | 16.46 | | 26.26 | 20.92 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 NVL** | PyTorch | | 19.73 | 19.86 | | 19.78 | 19.80 | | 20.68 | 19.87 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 NVL** | PyTorch | | 32.51 | 32.86 | | 32.53 | 33.05 | | 32.64 | 33.16 | +| | vLLM-Omni | 2.73 | | 3.20 | 6.28 | | 17.71 | 11.02 | 4.22 | 3.26 | +| | Diffusers | 5.00 | | | 8.00 | | | 12.00 | | | +| **H100 80GB HBM3** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 141GB HBM3** | PyTorch | | 13.62 | 13.48 | | 13.33 | 13.53 | | 13.78 | 13.50 | +| | vLLM-Omni | 2.83 | | 4.50 | 5.70 | | 7.16 | 10.24 | 4.23 | 4.43 | +| | Diffusers | 5.00 | | | 8.00 | | | 11.00 | | | +| **B200** | PyTorch | 4.51 | 4.10 | 4.27 | 4.78 | 4.13 | 4.48 | 7.25 | 4.28 | 4.65 | +| | vLLM-Omni | 2.32 | | 4.58 | 3.29 | | 9.10 | 6.02 | 3.09 | 4.43 | +| | Diffusers | | | | | | | 8.00 | | | +| **B300** | PyTorch | | | | | | | | | | +| | vLLM-Omni | 5.05 | | 7.62 | 3.79 | | 72.65 | 7.08 | 5.79 | 7.24 | +| | Diffusers | 38.80 | | | 39.40 | | | 40.40 | | | + +Notes: +1. All times measured on identical workloads (same seed, sampler settings, prompt). +2. 4×/8× GPU configurations use tensor parallelism. +3. vLLM-Omni numbers are for the upcoming public release in the vLLM-Omni repo; subject to change before GA. Current vLLM-Omni coverage is B200 at 720p. +4. Diffusers numbers use the HuggingFace `diffusers` integration without custom CUDA graphs; reported at 256p/1, 480p/1, and 720p/1 (single-GPU only). +5. At 256p, multi-GPU configurations on B300 may underperform single-GPU due to small-workload TP overhead; single-GPU is recommended at this resolution. +6. PyTorch numbers report average generation (sampling) time from OSS inference benchmarking. +7. NIM numbers use latency profiles with FP8 precision and report end-to-end `Request Latency s`, including request processing, video generation, output encoding, and returning the response. + +## Additional audiovisual generation + +### Video-to-Video (v2v) + +A text prompt and source video condition a generated continuation or transformation. + +| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | +|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | | 64.80 | 66.31 | | 201.59 | 118.92 | | 787.37 | 427.19 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H20** | PyTorch | | 40.93 | 27.98 | | 277.39 | 153.02 | | 931.12 | 491.75 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 NVL** | PyTorch | | 20.75 | 16.79 | | 99.57 | 63.84 | | 328.83 | 183.64 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 NVL** | PyTorch | | 24.87 | 24.74 | | 82.22 | 47.60 | | 268.16 | 142.23 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 80GB HBM3** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 141GB HBM3** | PyTorch | | 14.66 | 11.76 | | 70.52 | 41.85 | | 223.86 | 123.09 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B200** | PyTorch | 14.30 | 5.59 | 4.07 | 111.49 | 35.68 | 21.43 | 395.97 | 117.46 | 65.62 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B300** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | + +### Text-to-Audio-and-Video (t2av) + +A text prompt produces synchronized video and sound. + +| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | +|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | | 65.38 | 65.79 | | 202.34 | 118.87 | | 788.88 | 429.78 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H20** | PyTorch | | 40.69 | 27.73 | | 277.33 | 152.53 | | 930.63 | 492.38 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 NVL** | PyTorch | | 20.78 | 16.86 | | 99.28 | 64.17 | | 329.26 | 183.81 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 NVL** | PyTorch | | 24.88 | 24.79 | | 82.49 | 47.67 | | 267.86 | 142.06 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 80GB HBM3** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 141GB HBM3** | PyTorch | | 15.05 | 11.74 | | 70.09 | 41.49 | | 223.96 | 123.35 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B200** | PyTorch | 14.52 | 5.64 | 4.07 | 112.99 | 35.93 | 21.57 | 395.20 | 118.67 | 65.93 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B300** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | + +### Video-to-Audio-and-Video (v2av) + +A text prompt and source video produce transformed video with synchronized sound. + +| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | +|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | | 64.67 | 66.10 | | 202.31 | 118.88 | | 785.77 | 428.64 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H20** | PyTorch | | 40.89 | 28.00 | | 276.26 | 152.78 | | 930.81 | 492.63 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 NVL** | PyTorch | | 20.65 | 16.79 | | 99.90 | 64.69 | | 323.95 | 183.62 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 NVL** | PyTorch | | 24.88 | 24.73 | | 82.36 | 47.67 | | 268.78 | 143.06 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 80GB HBM3** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 141GB HBM3** | PyTorch | | 14.85 | 11.74 | | 70.42 | 41.59 | | 223.23 | 123.06 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B200** | PyTorch | 14.28 | 5.60 | 4.07 | 111.84 | 35.64 | 21.48 | 406.66 | 117.16 | 65.30 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B300** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | + +### Image-to-Audio-and-Video (i2av) + +A text prompt and source image produce video with synchronized sound. + +| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | +|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | | 65.54 | 66.27 | | 202.65 | 119.05 | | 790.09 | 429.26 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H20** | PyTorch | | 41.33 | 28.11 | | 277.13 | 153.33 | | 931.18 | 493.51 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 NVL** | PyTorch | | 20.83 | 16.85 | | 99.62 | 64.19 | | 329.80 | 186.66 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 NVL** | PyTorch | | 24.81 | 24.69 | | 82.65 | 47.65 | | 269.74 | 142.18 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 80GB HBM3** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 141GB HBM3** | PyTorch | | 15.09 | 11.82 | | 69.80 | 41.95 | | 224.65 | 123.42 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B200** | PyTorch | 14.84 | 5.64 | 4.12 | 113.32 | 35.84 | 21.70 | 407.30 | 117.95 | 65.20 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B300** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | + +## Action generation + +Forward dynamics is reported separately for AV, camera, and robot inputs. Inverse dynamics and policy are reported for AV and robot because those are the only domain rows in this PBR; no camera row is inferred. For each domain, PyTorch is populated from the PBR while vLLM-Omni and Diffusers rows are reserved for future measurements. + +### Forward Dynamics — Autonomous Vehicle (AV) + +| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | +|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | | 79.50 | 59.39 | | 61.21 | 59.31 | | 61.15 | 59.35 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H20** | PyTorch | | 67.73 | 41.51 | | 67.83 | 41.47 | | 67.85 | 41.42 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 NVL** | PyTorch | | 28.53 | 21.00 | | 28.69 | 21.00 | | 28.56 | 21.04 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 NVL** | PyTorch | | 24.40 | 22.66 | | 24.38 | 22.67 | | 24.46 | 22.65 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 80GB HBM3** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 141GB HBM3** | PyTorch | | 19.88 | 14.41 | | 19.87 | 14.41 | | 19.88 | 14.41 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B200** | PyTorch | 13.59 | 9.29 | 6.33 | 13.55 | 9.29 | 6.37 | 13.86 | 9.32 | 6.34 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B300** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | + +### Forward Dynamics — Camera + +| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | +|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | | 60.75 | 59.62 | | 60.69 | 59.39 | | 60.61 | 59.55 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H20** | PyTorch | | 67.86 | 41.13 | | 67.89 | 41.10 | | 67.88 | 41.11 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 NVL** | PyTorch | | 28.73 | 21.05 | | 28.76 | 20.98 | | 28.65 | 21.00 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 NVL** | PyTorch | | 24.33 | 22.55 | | 24.35 | 22.54 | | 24.37 | 22.54 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 80GB HBM3** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 141GB HBM3** | PyTorch | | 19.85 | 14.30 | | 19.92 | 14.30 | | 20.29 | 14.29 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B200** | PyTorch | 13.73 | 9.32 | 6.27 | 13.72 | 9.27 | 6.27 | 13.58 | 9.28 | 6.27 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B300** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | + +### Forward Dynamics — Robot + +| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | +|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | | 75.52 | 56.35 | | 55.04 | 56.38 | | 54.88 | 56.01 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H20** | PyTorch | | 20.93 | 15.60 | | 20.91 | 15.61 | | 20.78 | 15.59 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 NVL** | PyTorch | | 13.61 | 12.90 | | 13.68 | 12.90 | | 13.57 | 12.90 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 NVL** | PyTorch | | 20.25 | 20.16 | | 20.25 | 20.15 | | 20.25 | 20.15 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 80GB HBM3** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 141GB HBM3** | PyTorch | | 9.34 | 8.74 | | 9.34 | 8.74 | | 9.33 | 8.70 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B200** | PyTorch | 3.50 | 2.91 | 2.98 | 3.49 | 3.06 | 3.00 | 3.47 | 2.96 | 3.00 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B300** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | + +### Inverse Dynamics — Autonomous Vehicle (AV) + +| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | +|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | | 60.92 | 59.20 | | 60.87 | 59.14 | | 60.95 | 59.07 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H20** | PyTorch | | 67.37 | 41.17 | | 67.09 | 40.96 | | 67.37 | 41.06 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 NVL** | PyTorch | | 28.42 | 20.76 | | 28.39 | 21.05 | | 28.41 | 20.69 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 NVL** | PyTorch | | 24.16 | 22.40 | | 24.17 | 22.40 | | 24.18 | 22.42 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 80GB HBM3** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 141GB HBM3** | PyTorch | | 20.05 | 14.30 | | 19.68 | 14.32 | | 19.91 | 14.16 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B200** | PyTorch | 13.38 | 9.08 | 6.11 | 13.37 | 9.13 | 6.15 | 13.36 | 9.08 | 6.11 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B300** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | + +### Inverse Dynamics — Robot + +| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | +|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | | 55.15 | 56.28 | | 55.12 | 56.33 | | 54.83 | 56.00 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H20** | PyTorch | | 20.71 | 15.51 | | 20.83 | 15.52 | | 20.84 | 15.51 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 NVL** | PyTorch | | 13.51 | 12.51 | | 13.54 | 12.51 | | 13.50 | 12.51 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 NVL** | PyTorch | | 20.21 | 20.11 | | 20.18 | 20.10 | | 20.19 | 20.12 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 80GB HBM3** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 141GB HBM3** | PyTorch | | 9.30 | 8.68 | | 9.28 | 8.68 | | 9.29 | 8.65 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B200** | PyTorch | 3.42 | 2.89 | 2.96 | 3.49 | 3.02 | 2.92 | 3.41 | 2.90 | 3.03 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B300** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | + +### Policy — Autonomous Vehicle (AV) + +| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | +|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | | 61.03 | 59.66 | | 61.19 | 59.51 | | 61.04 | 59.49 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H20** | PyTorch | | 67.86 | 41.54 | | 67.56 | 41.51 | | 67.62 | 41.49 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 NVL** | PyTorch | | 28.61 | 21.02 | | 28.77 | 21.00 | | 28.54 | 21.31 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 NVL** | PyTorch | | 24.40 | 22.64 | | 24.46 | 22.63 | | 24.47 | 22.64 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 80GB HBM3** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 141GB HBM3** | PyTorch | | 20.31 | 14.54 | | 19.87 | 14.41 | | 20.14 | 14.52 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B200** | PyTorch | 13.71 | 9.33 | 6.36 | 13.73 | 9.31 | 6.34 | 13.60 | 9.32 | 6.34 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B300** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | + +### Policy — Robot + +| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | +|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | | 55.13 | 56.31 | | 55.47 | 56.29 | | 54.92 | 55.95 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H20** | PyTorch | | 20.90 | 15.60 | | 20.88 | 15.60 | | 20.92 | 15.62 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 NVL** | PyTorch | | 13.57 | 12.56 | | 13.58 | 12.55 | | 13.60 | 12.54 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 NVL** | PyTorch | | 20.25 | 20.16 | | 20.26 | 20.17 | | 20.26 | 20.16 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 80GB HBM3** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 141GB HBM3** | PyTorch | | 9.33 | 8.72 | | 9.35 | 8.75 | | 9.34 | 8.70 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B200** | PyTorch | 3.48 | 2.95 | 3.07 | 3.49 | 3.02 | 2.98 | 3.48 | 2.92 | 3.05 | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B300** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | + +Additional-modality notes: +1. All reported values are average PyTorch generation (sampling) latency in seconds; lower is better. +2. PyTorch values use `CUDA_GRAPH=No` and the `latency` automatic-sharding preset. +3. The `/1`, `/4`, and `/8` suffixes denote the number of GPUs used by the benchmark run. +4. vLLM-Omni and Diffusers rows are intentionally empty reservations for future benchmark campaigns. +5. Empty cells indicate unmeasured combinations, not unsupported combinations. From 0becc437962ce9543c4c53f3a78245d5e32ba82c Mon Sep 17 00:00:00 2001 From: Sergiy Fefilatyev Date: Mon, 14 Sep 2026 16:52:48 -0700 Subject: [PATCH 3/9] Add vLLM-Omni audiovisual and action latencies for Nano and Super. Fill t2av/i2av from PBR 308195 and AV action plus Policy-DROID from PBR 308481 so the new modality tables are no longer empty reservations. --- .../cosmos3-nano-generator.md | 114 ++++++++++++------ .../cosmos3-super-generator.md | 114 ++++++++++++------ 2 files changed, 150 insertions(+), 78 deletions(-) diff --git a/inference_benchmarks/cosmos3-nano-generator.md b/inference_benchmarks/cosmos3-nano-generator.md index 4d77a606..45079a47 100644 --- a/inference_benchmarks/cosmos3-nano-generator.md +++ b/inference_benchmarks/cosmos3-nano-generator.md @@ -25,14 +25,15 @@ These tables report **Cosmos3-Nano Generator** latency in seconds. Lower is bett - [Inverse Dynamics — Robot](#inverse-dynamics--robot) - [Policy — AV](#policy--autonomous-vehicle-av) - [Policy — Robot](#policy--robot) + - [Policy — DROID](#policy--droid) ## Benchmark methodology The primary t2v, i2v, and t2i tables preserve the previously published benchmark campaigns across PyTorch, vLLM-Omni, Diffusers, and NIM. Those tables use BF16 precision, batch size 1, and matched prompts, seeds, and sampler settings where documented. Video workloads follow the standard Cosmos3 generation profile of 189 frames at 24 FPS unless a resolution tier limits frame count. -The additional audiovisual and action tables come from PBR `#308197`, **Cosmos3-Generator OSS Inference Benchmarking 32B and 8B (189 frames)**. PyTorch values are average generation (sampling) latency from the native OSS path, using **CUDA Graphs disabled** and the **latency** automatic-sharding preset. Columns combine the 256p, 480p, and 720p tiers with 1, 4, or 8 GPUs. Values are rounded to two decimal places. The workbook does not include B300, so B300 is retained as an empty reservation. +The additional audiovisual and action tables come from three internal benchmark reports. PyTorch values are from PBR `#308197`, **Cosmos3-Generator OSS Inference Benchmarking 32B and 8B (189 frames)**: average generation (sampling) latency from the native OSS path, using **CUDA Graphs disabled** and the **latency** automatic-sharding preset. vLLM-Omni values for text-to-audio-and-video (`t2av`/`t2vs`) and image-to-audio-and-video (`i2av`/`i2vs`) are from PBR `#308195`. vLLM-Omni action values are from PBR `#308481`, **Cosmos3-Generator vLLM-Omni Inference Benchmarking (action)**, measured with the `vllm/vllm-omni:cosmos3` image and the official action cookbook samples; `DIFFUSION_ATTENTION_BACKEND=TORCH_SDPA` was not set. Forward-dynamics cells are the mean of `av_forward`, `av_left`, and `av_right`; inverse-dynamics cells are the mean of `av_inverse_0` and `av_inverse_1`. That action sweep reports 1/2/4 GPUs only: 8-GPU Ulysses runs failed a sequence-length divisibility check, so `/8` action cells stay empty. The 2-GPU columns are omitted to keep the published table layout. Policy-DROID is a separate checkpoint and was measured only at 480p on one GPU. Super was not measured on H20, H100 NVL, or H100 80GB HBM3. Values are rounded to two decimal places. -The PBR establishes the reported timing matrix but does not expose every prompt and action payload in this repository. The linked public recipes explain modality behavior and provide representative payloads; their example-specific frame counts and action chunk sizes should not be treated as the exact internal benchmark inputs. +These reports establish the reported timing matrix but do not expose every prompt and action payload in this repository. The linked public recipes explain modality behavior and provide representative payloads; their example-specific frame counts and action chunk sizes should not be treated as the exact internal benchmark inputs. ## Workload definitions @@ -45,6 +46,7 @@ The PBR establishes the reported timing matrix but does not expose every prompt | Forward dynamics | Initial visual observation and an action trajectory | Future-observation rollout video | | Inverse dynamics | Observed video | Recovered action trajectory; some serving integrations also return video | | Policy | Initial visual observation, instruction, and optional state | Predicted action trajectory and, for general Generator paths, a rollout video | +| Policy-DROID | Multiview wrist/exterior observations for the DROID embodiment | Predicted action chunk from the Policy-DROID checkpoint | The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound-producing modes `t2vs`, `v2vs`, and `i2vs`. See the [audiovisual cookbook](../cookbooks/cosmos3/generator/audiovisual/README.md) for generation inputs and the [action cookbook](../cookbooks/cosmos3/generator/action/README.md) for action representations and output contracts. vLLM-Omni request shapes are maintained in the [Nano recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-Nano.md). Public recipes cover more embodiments than this PBR; the action tables below intentionally use only its measured domain rows. @@ -202,28 +204,28 @@ A text prompt produces synchronized video and sound. | GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | |---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| | **RTX PRO 6000 Blackwell** | PyTorch | 14.13 | 6.47 | 4.89 | 180.98 | 57.21 | 35.77 | 794.74 | 225.73 | 127.53 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 12.25 | 5.32 | 4.19 | 106.02 | 36.11 | 24.02 | 373.10 | 115.06 | 69.19 | | | Diffusers | | | | | | | | | | | **H20** | PyTorch | 30.82 | 11.21 | 7.87 | 256.41 | 80.22 | 50.51 | 922.35 | 268.73 | 158.03 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 29.43 | 10.74 | 7.88 | 260.84 | 78.67 | 48.02 | 937.93 | 262.83 | 148.26 | | | Diffusers | | | | | | | | | | | **H100 NVL** | PyTorch | 9.99 | 4.27 | 3.95 | 82.32 | 28.50 | 21.46 | 297.88 | 94.45 | 60.25 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 9.92 | 3.85 | 3.26 | 81.57 | 27.15 | 18.93 | 291.91 | 90.29 | 55.66 | | | Diffusers | | | | | | | | | | | **H200 NVL** | PyTorch | 8.20 | 3.75 | 3.08 | 69.88 | 24.30 | 15.17 | 250.46 | 78.00 | 45.75 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 7.82 | 3.43 | 2.35 | 65.42 | 21.42 | 12.93 | 244.60 | 70.56 | 39.95 | | | Diffusers | | | | | | | | | | | **H100 80GB HBM3** | PyTorch | 7.60 | 3.43 | 3.18 | 59.09 | 21.18 | 14.32 | 207.87 | 66.74 | 41.71 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 7.34 | 3.54 | 3.43 | 58.65 | 20.10 | 13.47 | 203.92 | 63.22 | 38.45 | | | Diffusers | | | | | | | | | | | **H200 141GB HBM3** | PyTorch | 7.55 | 3.33 | 3.13 | 60.17 | 20.86 | 13.90 | 211.59 | 67.02 | 41.24 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 7.13 | 3.45 | 3.35 | 58.27 | 19.62 | 13.11 | 209.29 | 63.53 | 37.92 | | | Diffusers | | | | | | | | | | | **B200** | PyTorch | 4.63 | 2.81 | 2.81 | 32.60 | 13.17 | 9.64 | 113.89 | 39.59 | 26.33 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 4.29 | 2.51 | 3.47 | 32.20 | 12.70 | 10.08 | 107.74 | 36.10 | 23.83 | | | Diffusers | | | | | | | | | | | **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 5.59 | 4.21 | 5.54 | 33.15 | 14.10 | 11.90 | 104.92 | 36.00 | 24.92 | | | Diffusers | | | | | | | | | | ### Video-to-Audio-and-Video (v2av) @@ -264,61 +266,61 @@ A text prompt and source image produce video with synchronized sound. | GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | |---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| | **RTX PRO 6000 Blackwell** | PyTorch | 14.08 | 6.48 | 4.91 | 181.64 | 57.25 | 35.69 | 787.81 | 226.71 | 127.85 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 12.61 | 5.72 | 4.53 | 108.05 | 38.30 | 26.20 | 378.04 | 120.08 | 74.23 | | | Diffusers | | | | | | | | | | | **H20** | PyTorch | 31.34 | 11.36 | 7.97 | 259.06 | 80.55 | 50.63 | 924.39 | 269.21 | 157.97 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 30.36 | 11.60 | 8.91 | 265.37 | 83.11 | 52.70 | 948.09 | 272.93 | 158.76 | | | Diffusers | | | | | | | | | | | **H100 NVL** | PyTorch | 10.45 | 4.31 | 3.97 | 84.37 | 28.74 | 21.46 | 298.61 | 94.02 | 60.57 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 10.34 | 4.30 | 3.72 | 83.21 | 29.20 | 20.82 | 295.65 | 94.44 | 59.83 | | | Diffusers | | | | | | | | | | | **H200 NVL** | PyTorch | 8.29 | 3.76 | 3.08 | 71.20 | 24.34 | 15.05 | 248.35 | 76.82 | 46.40 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 8.24 | 3.85 | 2.81 | 67.23 | 23.05 | 14.65 | 249.41 | 74.44 | 43.50 | | | Diffusers | | | | | | | | | | | **H100 80GB HBM3** | PyTorch | 7.73 | 3.46 | 3.11 | 59.13 | 21.24 | 14.30 | 207.93 | 67.50 | 41.75 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 7.74 | 3.96 | 4.08 | 60.27 | 21.81 | 15.26 | 207.53 | 66.74 | 41.92 | | | Diffusers | | | | | | | | | | | **H200 141GB HBM3** | PyTorch | 7.66 | 3.35 | 3.14 | 60.55 | 20.98 | 13.99 | 214.82 | 67.35 | 41.21 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 7.55 | 3.90 | 3.77 | 59.73 | 21.35 | 14.67 | 212.73 | 66.78 | 41.03 | | | Diffusers | | | | | | | | | | | **B200** | PyTorch | 4.62 | 2.78 | 2.80 | 32.82 | 13.09 | 9.69 | 113.85 | 39.76 | 26.56 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 4.58 | 2.83 | 3.85 | 33.36 | 13.89 | 11.14 | 110.05 | 38.82 | 26.30 | | | Diffusers | | | | | | | | | | | **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 6.01 | 4.76 | 6.09 | 34.39 | 15.45 | 13.15 | 108.59 | 38.62 | 27.59 | | | Diffusers | | | | | | | | | | ## Action generation -Forward dynamics is reported separately for AV, camera, and robot inputs. Inverse dynamics and policy are reported for AV and robot because those are the only domain rows in this PBR; no camera row is inferred. For each domain, PyTorch is populated from the PBR while vLLM-Omni and Diffusers rows are reserved for future measurements. +Forward dynamics is reported separately for AV, camera, and robot inputs. Inverse dynamics and policy are reported for AV and robot because those are the only domain rows in PBR `#308197`; no camera row is inferred. For each domain, PyTorch is populated from that report. vLLM-Omni currently covers forward dynamics AV and inverse dynamics AV from PBR `#308481`. Camera, robot-FD, robot-ID, policy-AV, and policy-robot vLLM-Omni rows remain reserved. **Policy — DROID** is a separate checkpoint (`nvidia/Cosmos3-Nano-Policy-DROID`) and is not mixed with the policy-robot table. ### Forward Dynamics — Autonomous Vehicle (AV) | GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | |---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| | **RTX PRO 6000 Blackwell** | PyTorch | 16.87 | 12.13 | 8.96 | 16.86 | 12.18 | 8.95 | 16.85 | 12.14 | 8.94 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 2.08 | 1.46 | | 12.94 | 7.06 | | 36.19 | 16.58 | | | | Diffusers | | | | | | | | | | | **H20** | PyTorch | 31.47 | 19.97 | 13.77 | 31.16 | 19.91 | 13.78 | 31.45 | 19.95 | 13.78 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 4.82 | 2.36 | | 31.36 | 12.83 | | 87.55 | 32.09 | | | | Diffusers | | | | | | | | | | | **H100 NVL** | PyTorch | 10.32 | 7.36 | 6.46 | 10.66 | 7.35 | 6.31 | 10.64 | 7.43 | 6.32 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 1.71 | 1.46 | | 10.58 | 5.63 | | 28.26 | 13.61 | | | | Diffusers | | | | | | | | | | | **H200 NVL** | PyTorch | 8.30 | 6.18 | 4.41 | 8.31 | 6.16 | 4.39 | 8.33 | 6.14 | 4.39 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 1.43 | 1.50 | | 8.21 | 3.83 | | 22.78 | 9.37 | | | | Diffusers | | | | | | | | | | | **H100 80GB HBM3** | PyTorch | 7.80 | 5.69 | 4.30 | 7.87 | 5.68 | 4.34 | 7.79 | 5.68 | 4.30 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 1.33 | 1.51 | | 7.74 | 3.77 | | 20.60 | 9.09 | | | | Diffusers | | | | | | | | | | | **H200 141GB HBM3** | PyTorch | 7.77 | 5.51 | 4.16 | 7.76 | 5.56 | 4.18 | 7.81 | 5.54 | 4.16 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 1.33 | 1.49 | | 7.72 | 3.63 | | 20.65 | 8.79 | | | | Diffusers | | | | | | | | | | | **B200** | PyTorch | 4.75 | 3.88 | 3.38 | 4.75 | 3.85 | 3.37 | 4.73 | 3.87 | 3.36 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 0.94 | 1.20 | | 4.93 | 2.84 | | 12.00 | 5.84 | | | | Diffusers | | | | | | | | | | | **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 1.09 | 2.30 | | 4.86 | 3.56 | | 11.70 | 6.31 | | | | Diffusers | | | | | | | | | | ### Forward Dynamics — Camera @@ -384,28 +386,28 @@ Forward dynamics is reported separately for AV, camera, and robot inputs. Invers | GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | |---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| | **RTX PRO 6000 Blackwell** | PyTorch | 16.74 | 12.05 | 8.79 | 16.75 | 12.02 | 8.78 | 16.75 | 12.02 | 8.78 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 2.10 | 1.47 | | 12.96 | 7.09 | | 36.10 | 16.61 | | | | Diffusers | | | | | | | | | | | **H20** | PyTorch | 31.22 | 19.66 | 13.52 | 30.99 | 19.68 | 13.52 | 31.22 | 19.70 | 13.53 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 4.84 | 2.39 | | 31.36 | 12.81 | | 87.50 | 32.08 | | | | Diffusers | | | | | | | | | | | **H100 NVL** | PyTorch | 10.47 | 7.13 | 6.16 | 10.45 | 7.16 | 6.26 | 10.44 | 7.10 | 6.14 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 1.74 | 1.48 | | 10.54 | 5.61 | | 28.16 | | | | | Diffusers | | | | | | | | | | | **H200 NVL** | PyTorch | 8.12 | 5.96 | 4.19 | 8.13 | 5.97 | 4.20 | 8.10 | 5.98 | 4.20 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 1.46 | 1.46 | | 8.20 | 3.80 | | 22.34 | 9.32 | | | | Diffusers | | | | | | | | | | | **H100 80GB HBM3** | PyTorch | 7.62 | 5.54 | 4.15 | 7.63 | 5.52 | 4.16 | 7.68 | 5.53 | 4.14 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 1.39 | 1.50 | | 7.72 | 3.75 | | 20.55 | 9.05 | | | | Diffusers | | | | | | | | | | | **H200 141GB HBM3** | PyTorch | 7.51 | 5.36 | 4.00 | 7.64 | 5.34 | 3.99 | 7.51 | 5.37 | 3.99 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 1.36 | 1.48 | | 7.69 | 3.60 | | 20.65 | 8.71 | | | | Diffusers | | | | | | | | | | | **B200** | PyTorch | 4.60 | 3.74 | 3.20 | 4.63 | 3.73 | 3.19 | 4.58 | 3.76 | 3.23 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 0.92 | 1.55 | | 5.10 | 2.83 | | 12.29 | 5.92 | | | | Diffusers | | | | | | | | | | | **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 1.36 | 2.55 | | 5.75 | 4.04 | | 11.89 | 6.67 | | | | Diffusers | | | | | | | | | | ### Inverse Dynamics — Robot @@ -495,9 +497,43 @@ Forward dynamics is reported separately for AV, camera, and robot inputs. Invers | | vLLM-Omni | | | | | | | | | | | | Diffusers | | | | | | | | | | +### Policy — DROID + +These rows report **Cosmos3-Nano-Policy-DROID**, not the general Cosmos3-Nano checkpoint used in the other action tables. vLLM-Omni only supports this modality at **480p on one GPU** (multi-GPU is disabled, and resolution is fixed for the multiview wrist/exterior inputs). PyTorch and Diffusers cells are reserved. + +| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | +|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | 3.82 | | | | | | +| | Diffusers | | | | | | | | | | +| **H20** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | 8.51 | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 NVL** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | 2.94 | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 NVL** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | 2.34 | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 80GB HBM3** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | 2.23 | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 141GB HBM3** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | 2.21 | | | | | | +| | Diffusers | | | | | | | | | | +| **B200** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | 1.47 | | | | | | +| | Diffusers | | | | | | | | | | +| **B300** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | 2.15 | | | | | | +| | Diffusers | | | | | | | | | | + Additional-modality notes: -1. All reported values are average PyTorch generation (sampling) latency in seconds; lower is better. +1. PyTorch values are average generation (sampling) latency in seconds from PBR `#308197`; lower is better. 2. PyTorch values use `CUDA_GRAPH=No` and the `latency` automatic-sharding preset. -3. The `/1`, `/4`, and `/8` suffixes denote the number of GPUs used by the benchmark run. -4. vLLM-Omni and Diffusers rows are intentionally empty reservations for future benchmark campaigns. -5. Empty cells indicate unmeasured combinations, not unsupported combinations. +3. vLLM-Omni audiovisual values for `t2av`/`t2vs` and `i2av`/`i2vs` are from PBR `#308195`. vLLM-Omni action values are from PBR `#308481`. +4. Action vLLM-Omni `/8` cells are empty because 8-GPU Ulysses runs failed sequence-length divisibility checks. Two documented action misses are also left blank: Super inverse-dynamics AV at 720p/4 on H200 141GB HBM3 (OOM), and Nano inverse-dynamics AV at 720p/4 on H100 NVL (CUDA/NCCL failure). +5. Policy-DROID vLLM-Omni uses the `nvidia/Cosmos3-Nano-Policy-DROID` checkpoint at 480p/1 only. It is reported in its own table and is not comparable to the PyTorch policy-robot row. +6. Diffusers rows are intentionally empty reservations for future benchmark campaigns. +7. The `/1`, `/4`, and `/8` suffixes denote the number of GPUs used by the benchmark run. +8. Empty cells indicate unmeasured combinations, not unsupported combinations. diff --git a/inference_benchmarks/cosmos3-super-generator.md b/inference_benchmarks/cosmos3-super-generator.md index f7164232..3bbaa635 100644 --- a/inference_benchmarks/cosmos3-super-generator.md +++ b/inference_benchmarks/cosmos3-super-generator.md @@ -25,14 +25,15 @@ These tables report **Cosmos3-Super Generator** latency in seconds. Lower is bet - [Inverse Dynamics — Robot](#inverse-dynamics--robot) - [Policy — AV](#policy--autonomous-vehicle-av) - [Policy — Robot](#policy--robot) + - [Policy — DROID](#policy--droid) ## Benchmark methodology The primary t2v, i2v, and t2i tables preserve the previously published benchmark campaigns across PyTorch, vLLM-Omni, Diffusers, and NIM. Those tables use BF16 precision, batch size 1, and matched prompts, seeds, and sampler settings where documented. Video workloads follow the standard Cosmos3 generation profile of 189 frames at 24 FPS unless a resolution tier limits frame count. -The additional audiovisual and action tables come from PBR `#308197`, **Cosmos3-Generator OSS Inference Benchmarking 32B and 8B (189 frames)**. PyTorch values are average generation (sampling) latency from the native OSS path, using **CUDA Graphs disabled** and the **latency** automatic-sharding preset. Columns combine the 256p, 480p, and 720p tiers with 1, 4, or 8 GPUs. Values are rounded to two decimal places. The workbook does not include B300, so B300 is retained as an empty reservation. +The additional audiovisual and action tables come from three internal benchmark reports. PyTorch values are from PBR `#308197`, **Cosmos3-Generator OSS Inference Benchmarking 32B and 8B (189 frames)**: average generation (sampling) latency from the native OSS path, using **CUDA Graphs disabled** and the **latency** automatic-sharding preset. vLLM-Omni values for text-to-audio-and-video (`t2av`/`t2vs`) and image-to-audio-and-video (`i2av`/`i2vs`) are from PBR `#308195`. vLLM-Omni action values are from PBR `#308481`, **Cosmos3-Generator vLLM-Omni Inference Benchmarking (action)**, measured with the `vllm/vllm-omni:cosmos3` image and the official action cookbook samples; `DIFFUSION_ATTENTION_BACKEND=TORCH_SDPA` was not set. Forward-dynamics cells are the mean of `av_forward`, `av_left`, and `av_right`; inverse-dynamics cells are the mean of `av_inverse_0` and `av_inverse_1`. That action sweep reports 1/2/4 GPUs only: 8-GPU Ulysses runs failed a sequence-length divisibility check, so `/8` action cells stay empty. The 2-GPU columns are omitted to keep the published table layout. Policy-DROID is a separate checkpoint and was measured only at 480p on one GPU. Super was not measured on H20, H100 NVL, or H100 80GB HBM3. Values are rounded to two decimal places. -The PBR establishes the reported timing matrix but does not expose every prompt and action payload in this repository. The linked public recipes explain modality behavior and provide representative payloads; their example-specific frame counts and action chunk sizes should not be treated as the exact internal benchmark inputs. +These reports establish the reported timing matrix but do not expose every prompt and action payload in this repository. The linked public recipes explain modality behavior and provide representative payloads; their example-specific frame counts and action chunk sizes should not be treated as the exact internal benchmark inputs. ## Workload definitions @@ -45,6 +46,7 @@ The PBR establishes the reported timing matrix but does not expose every prompt | Forward dynamics | Initial visual observation and an action trajectory | Future-observation rollout video | | Inverse dynamics | Observed video | Recovered action trajectory; some serving integrations also return video | | Policy | Initial visual observation, instruction, and optional state | Predicted action trajectory and, for general Generator paths, a rollout video | +| Policy-DROID | Multiview wrist/exterior observations for the DROID embodiment | Predicted action chunk from the Policy-DROID checkpoint | The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound-producing modes `t2vs`, `v2vs`, and `i2vs`. See the [audiovisual cookbook](../cookbooks/cosmos3/generator/audiovisual/README.md) for generation inputs and the [action cookbook](../cookbooks/cosmos3/generator/action/README.md) for action representations and output contracts. vLLM-Omni request shapes are maintained in the [Super recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-Super.md). Public recipes cover more embodiments than this PBR; the action tables below intentionally use only its measured domain rows. @@ -202,28 +204,28 @@ A text prompt produces synchronized video and sound. | GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | |---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| | **RTX PRO 6000 Blackwell** | PyTorch | | 65.38 | 65.79 | | 202.34 | 118.87 | | 788.88 | 429.78 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | | | Diffusers | | | | | | | | | | | **H20** | PyTorch | | 40.69 | 27.73 | | 277.33 | 152.53 | | 930.63 | 492.38 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | | | Diffusers | | | | | | | | | | | **H100 NVL** | PyTorch | | 20.78 | 16.86 | | 99.28 | 64.17 | | 329.26 | 183.81 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | | | Diffusers | | | | | | | | | | | **H200 NVL** | PyTorch | | 24.88 | 24.79 | | 82.49 | 47.67 | | 267.86 | 142.06 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 29.07 | 9.51 | 5.36 | 260.51 | 70.75 | 36.78 | 916.51 | 249.35 | 127.17 | | | Diffusers | | | | | | | | | | | **H100 80GB HBM3** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | | | Diffusers | | | | | | | | | | | **H200 141GB HBM3** | PyTorch | | 15.05 | 11.74 | | 70.09 | 41.49 | | 223.96 | 123.35 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 27.21 | 9.01 | 6.16 | 221.67 | 64.07 | 35.22 | 763.74 | 212.47 | 119.74 | | | Diffusers | | | | | | | | | | | **B200** | PyTorch | 14.52 | 5.64 | 4.07 | 112.99 | 35.93 | 21.57 | 395.20 | 118.67 | 65.93 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 14.77 | 5.54 | 5.01 | 115.26 | 36.20 | 21.76 | 388.51 | 115.08 | 62.56 | | | Diffusers | | | | | | | | | | | **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 15.26 | 6.89 | 7.09 | 113.34 | 36.79 | 23.39 | 372.74 | 110.75 | 61.99 | | | Diffusers | | | | | | | | | | ### Video-to-Audio-and-Video (v2av) @@ -264,61 +266,61 @@ A text prompt and source image produce video with synchronized sound. | GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | |---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| | **RTX PRO 6000 Blackwell** | PyTorch | | 65.54 | 66.27 | | 202.65 | 119.05 | | 790.09 | 429.26 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | | | Diffusers | | | | | | | | | | | **H20** | PyTorch | | 41.33 | 28.11 | | 277.13 | 153.33 | | 931.18 | 493.51 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | | | Diffusers | | | | | | | | | | | **H100 NVL** | PyTorch | | 20.83 | 16.85 | | 99.62 | 64.19 | | 329.80 | 186.66 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | | | Diffusers | | | | | | | | | | | **H200 NVL** | PyTorch | | 24.81 | 24.69 | | 82.65 | 47.65 | | 269.74 | 142.18 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 29.42 | 9.94 | 5.84 | 262.20 | 72.67 | 38.57 | 922.18 | 252.67 | 131.32 | | | Diffusers | | | | | | | | | | | **H100 80GB HBM3** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | | | Diffusers | | | | | | | | | | | **H200 141GB HBM3** | PyTorch | | 15.09 | 11.82 | | 69.80 | 41.95 | | 224.65 | 123.42 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 27.58 | 9.29 | 6.59 | 220.57 | 65.65 | 36.71 | 774.15 | 217.68 | 7.00 | | | Diffusers | | | | | | | | | | | **B200** | PyTorch | 14.84 | 5.64 | 4.12 | 113.32 | 35.84 | 21.70 | 407.30 | 117.95 | 65.20 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 15.15 | 5.88 | 5.38 | 116.54 | 37.32 | 23.07 | 396.11 | 117.51 | 65.98 | | | Diffusers | | | | | | | | | | | **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 16.46 | 7.22 | 7.38 | 114.02 | 38.10 | 24.69 | 373.17 | 113.46 | 64.64 | | | Diffusers | | | | | | | | | | ## Action generation -Forward dynamics is reported separately for AV, camera, and robot inputs. Inverse dynamics and policy are reported for AV and robot because those are the only domain rows in this PBR; no camera row is inferred. For each domain, PyTorch is populated from the PBR while vLLM-Omni and Diffusers rows are reserved for future measurements. +Forward dynamics is reported separately for AV, camera, and robot inputs. Inverse dynamics and policy are reported for AV and robot because those are the only domain rows in PBR `#308197`; no camera row is inferred. For each domain, PyTorch is populated from that report. vLLM-Omni currently covers forward dynamics AV and inverse dynamics AV from PBR `#308481`. Camera, robot-FD, robot-ID, policy-AV, and policy-robot vLLM-Omni rows remain reserved. **Policy — DROID** is a separate checkpoint and is not mixed with the policy-robot table; this Super page has no Policy-DROID measurements. ### Forward Dynamics — Autonomous Vehicle (AV) | GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | |---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| | **RTX PRO 6000 Blackwell** | PyTorch | | 79.50 | 59.39 | | 61.21 | 59.31 | | 61.15 | 59.35 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | | | Diffusers | | | | | | | | | | | **H20** | PyTorch | | 67.73 | 41.51 | | 67.83 | 41.47 | | 67.85 | 41.42 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | | | Diffusers | | | | | | | | | | | **H100 NVL** | PyTorch | | 28.53 | 21.00 | | 28.69 | 21.00 | | 28.56 | 21.04 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | | | Diffusers | | | | | | | | | | | **H200 NVL** | PyTorch | | 24.40 | 22.66 | | 24.38 | 22.67 | | 24.46 | 22.65 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 4.33 | 2.16 | | 28.78 | 9.47 | | 82.08 | 24.83 | | | | Diffusers | | | | | | | | | | | **H100 80GB HBM3** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | | | Diffusers | | | | | | | | | | | **H200 141GB HBM3** | PyTorch | | 19.88 | 14.41 | | 19.87 | 14.41 | | 19.88 | 14.41 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 3.99 | 2.27 | | 26.04 | 8.83 | | 71.61 | 23.02 | | | | Diffusers | | | | | | | | | | | **B200** | PyTorch | 13.59 | 9.29 | 6.33 | 13.55 | 9.29 | 6.37 | 13.86 | 9.32 | 6.34 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 2.37 | 1.93 | | 14.59 | 5.75 | | 38.41 | 13.46 | | | | Diffusers | | | | | | | | | | | **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 2.85 | 3.20 | | 14.20 | 6.07 | | 36.96 | 13.31 | | | | Diffusers | | | | | | | | | | ### Forward Dynamics — Camera @@ -384,28 +386,28 @@ Forward dynamics is reported separately for AV, camera, and robot inputs. Invers | GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | |---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| | **RTX PRO 6000 Blackwell** | PyTorch | | 60.92 | 59.20 | | 60.87 | 59.14 | | 60.95 | 59.07 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | | | Diffusers | | | | | | | | | | | **H20** | PyTorch | | 67.37 | 41.17 | | 67.09 | 40.96 | | 67.37 | 41.06 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | | | Diffusers | | | | | | | | | | | **H100 NVL** | PyTorch | | 28.42 | 20.76 | | 28.39 | 21.05 | | 28.41 | 20.69 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | | | Diffusers | | | | | | | | | | | **H200 NVL** | PyTorch | | 24.16 | 22.40 | | 24.17 | 22.40 | | 24.18 | 22.42 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 4.36 | 2.17 | | 28.21 | 9.43 | | 80.85 | 25.46 | | | | Diffusers | | | | | | | | | | | **H100 80GB HBM3** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | | | Diffusers | | | | | | | | | | | **H200 141GB HBM3** | PyTorch | | 20.05 | 14.30 | | 19.68 | 14.32 | | 19.91 | 14.16 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 4.03 | 2.29 | | 25.99 | 8.79 | | 71.45 | | | | | Diffusers | | | | | | | | | | | **B200** | PyTorch | 13.38 | 9.08 | 6.11 | 13.37 | 9.13 | 6.15 | 13.36 | 9.08 | 6.11 | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 2.42 | 2.29 | | 14.69 | 5.85 | | 38.74 | 13.54 | | | | Diffusers | | | | | | | | | | | **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | +| | vLLM-Omni | 2.69 | 3.54 | | 14.44 | 6.52 | | 37.70 | 13.88 | | | | Diffusers | | | | | | | | | | ### Inverse Dynamics — Robot @@ -495,9 +497,43 @@ Forward dynamics is reported separately for AV, camera, and robot inputs. Invers | | vLLM-Omni | | | | | | | | | | | | Diffusers | | | | | | | | | | +### Policy — DROID + +PBR `#308481` measured Policy-DROID only for Cosmos3-Nano-Policy-DROID. Super has no Policy-DROID row in that sweep; this table is reserved. + +| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | +|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H20** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 NVL** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 NVL** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H100 80GB HBM3** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **H200 141GB HBM3** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B200** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | +| **B300** | PyTorch | | | | | | | | | | +| | vLLM-Omni | | | | | | | | | | +| | Diffusers | | | | | | | | | | + Additional-modality notes: -1. All reported values are average PyTorch generation (sampling) latency in seconds; lower is better. +1. PyTorch values are average generation (sampling) latency in seconds from PBR `#308197`; lower is better. 2. PyTorch values use `CUDA_GRAPH=No` and the `latency` automatic-sharding preset. -3. The `/1`, `/4`, and `/8` suffixes denote the number of GPUs used by the benchmark run. -4. vLLM-Omni and Diffusers rows are intentionally empty reservations for future benchmark campaigns. -5. Empty cells indicate unmeasured combinations, not unsupported combinations. +3. vLLM-Omni audiovisual values for `t2av`/`t2vs` and `i2av`/`i2vs` are from PBR `#308195`. vLLM-Omni action values are from PBR `#308481`. +4. Action vLLM-Omni `/8` cells are empty because 8-GPU Ulysses runs failed sequence-length divisibility checks. Two documented action misses are also left blank: Super inverse-dynamics AV at 720p/4 on H200 141GB HBM3 (OOM), and Nano inverse-dynamics AV at 720p/4 on H100 NVL (CUDA/NCCL failure). +5. Policy-DROID is reported in its own table. PBR `#308481` has no Super Policy-DROID measurement. +6. Diffusers rows are intentionally empty reservations for future benchmark campaigns. +7. The `/1`, `/4`, and `/8` suffixes denote the number of GPUs used by the benchmark run. +8. Empty cells indicate unmeasured combinations, not unsupported combinations. From 6c89291367eff715235908aababfaf61afa8aecb Mon Sep 17 00:00:00 2001 From: Sergiy Fefilatyev Date: Fri, 2 Oct 2026 13:09:18 -0700 Subject: [PATCH 4/9] Report one GPU configuration per row and fill Diffusers coverage The per-resolution tables previously crossed each input resolution with 1, 4, and 8 GPU counts, which made them wide and left most cells empty. Each row now reports a single configuration: one GPU when the model fits at the model's highest resolution, otherwise eight. A new GPUs column records which configuration a row's values come from. Cosmos3-Edge gains explicit 256p and 480p columns, 480p being its highest resolution; 256p has not been measured yet. Diffusers numbers are filled in from PBR #308202 (all modalities), PBR #308451 (video-to-video), and PBR #308918 (4- and 8-GPU runs, taking the fastest tensor- and context-parallel split at each GPU count). The 1-GPU results in PBR #308918 are not used because that sweep ran a different denoising-step budget. Seventeen B300 values are corrected to the figures from the newer reports. --- .../cosmos3-edge-generator.md | 66 +- .../cosmos3-nano-generator.md | 834 +++++++++--------- .../cosmos3-super-generator.md | 832 ++++++++--------- 3 files changed, 870 insertions(+), 862 deletions(-) diff --git a/inference_benchmarks/cosmos3-edge-generator.md b/inference_benchmarks/cosmos3-edge-generator.md index f0e40cd0..357dc306 100644 --- a/inference_benchmarks/cosmos3-edge-generator.md +++ b/inference_benchmarks/cosmos3-edge-generator.md @@ -2,48 +2,48 @@ [Back to the inference benchmark index](../inference_benchmarks.md) -These tables report **Cosmos3-Edge** Generator latency in seconds for **image-to-video (i2v)**. Measurements use one GPU or one integrated computing platform. Lower latency is better, and empty cells indicate that a run has not been completed. +These tables report **Cosmos3-Edge** Generator latency in seconds for **image-to-video (i2v)**. Lower latency is better, and empty cells indicate that a run has not been completed. -Unless otherwise noted, visual-generation benchmarks use **480p resolution**. Video benchmarks generate **121 frames**. vLLM-Omni values report end-to-end latency, while PyTorch values report average generation latency. +Cosmos3-Edge supports **256p and 480p**, so 480p is its highest resolution. Every row reports one GPU configuration, named in the **GPUs** column: one GPU where the model fits on a single device at 480p, and eight GPUs where it does not. Cosmos3-Edge fits on a single GPU or one integrated computing platform at 480p, so all populated rows report one. Video benchmarks generate **121 frames**. vLLM-Omni values report end-to-end latency, while PyTorch values report average generation latency. ### vLLM-Omni -| GPU or Platform | Image-to-Video | -|---|---:| -| B200 SXM 192 GB | 7.09 | -| B300 | 8.74 | -| H200 SXM 141 GB | 13.13 | -| H200 NVL | 14.15 | -| H100 SXM 80 GB | 13.33 | -| H100 NVL 96 GB | 17.33 | -| H20 SXM 96 GB | 53.94 | -| RTX PRO 6000 Blackwell Server Edition | | -| DGX Station | 7.13 | -| DGX Spark | 89.41 | -| Jetson AGX Thor T5000, 128 GB, MAXN | | -| Jetson T3000, 32 GB, 1100 MHz | | -| Jetson T2000, 16 GB, 702 MHz, THOR_NANO | | +| GPU or Platform | GPUs | 256p | 480p | +|---|:-:|---:|---:| +| B200 SXM 192 GB | 1 | | 7.09 | +| B300 | 1 | | 8.74 | +| H200 SXM 141 GB | 1 | | 13.13 | +| H200 NVL | 1 | | 14.15 | +| H100 SXM 80 GB | 1 | | 13.33 | +| H100 NVL 96 GB | 1 | | 17.33 | +| H20 SXM 96 GB | 1 | | 53.94 | +| RTX PRO 6000 Blackwell Server Edition | — | | | +| DGX Station | 1 | | 7.13 | +| DGX Spark | 1 | | 89.41 | +| Jetson AGX Thor T5000, 128 GB, MAXN | — | | | +| Jetson T3000, 32 GB, 1100 MHz | — | | | +| Jetson T2000, 16 GB, 702 MHz, THOR_NANO | — | | | ### PyTorch -| GPU or Platform | Image-to-Video | -|---|---:| -| B200 SXM 192 GB | 7.45 | -| B300 | 6.84 | -| H200 SXM 141 GB | 12.31 | -| H200 NVL | 14.07 | -| H100 SXM 80 GB | 12.68 | -| H100 NVL 96 GB | 16.42 | -| H20 SXM 96 GB | 52.83 | -| RTX PRO 6000 Blackwell Server Edition | 21.92 | -| DGX Station | 6.31 | -| DGX Spark | 103.36 | -| Jetson AGX Thor T5000, 128 GB, MAXN | | -| Jetson T3000, 32 GB, 1100 MHz | | +| GPU or Platform | GPUs | 256p | 480p | +|---|:-:|---:|---:| +| B200 SXM 192 GB | 1 | | 7.45 | +| B300 | 1 | | 6.84 | +| H200 SXM 141 GB | 1 | | 12.31 | +| H200 NVL | 1 | | 14.07 | +| H100 SXM 80 GB | 1 | | 12.68 | +| H100 NVL 96 GB | 1 | | 16.42 | +| H20 SXM 96 GB | 1 | | 52.83 | +| RTX PRO 6000 Blackwell Server Edition | 1 | | 21.92 | +| DGX Station | 1 | | 6.31 | +| DGX Spark | 1 | | 103.36 | +| Jetson AGX Thor T5000, 128 GB, MAXN | — | | | +| Jetson T3000, 32 GB, 1100 MHz | — | | | Notes: -1. All measurements use one GPU or one integrated computing platform. +1. The **GPUs** column gives the number of GPUs behind every value in that row. All current measurements use one GPU or one integrated computing platform. 2. Values are average end-to-end or generation latency in seconds; lower is better. -3. Unless otherwise specified, visual-generation measurements use **480p resolution**. +3. Cosmos3-Edge supports 256p and 480p. Every benchmark run to date is at 480p, so the 256p column is reserved for a future campaign rather than unsupported. 4. Video measurements (i2v) generate **121 output frames**. 5. PyTorch values report average generation latency rather than diffusion-only latency. diff --git a/inference_benchmarks/cosmos3-nano-generator.md b/inference_benchmarks/cosmos3-nano-generator.md index 45079a47..baaafd1b 100644 --- a/inference_benchmarks/cosmos3-nano-generator.md +++ b/inference_benchmarks/cosmos3-nano-generator.md @@ -2,7 +2,11 @@ [Back to the inference benchmark index](../inference_benchmarks.md) -These tables report **Cosmos3-Nano Generator** latency in seconds. Lower is better. Empty cells mean that a run has not been completed for that GPU, engine, resolution, or GPU count; they do not indicate that a combination is unsupported. +These tables report **Cosmos3-Nano Generator** latency in seconds. Lower is better. + +Every row reports one GPU configuration, named in the **GPUs** column, and all three resolutions in that row come from it. The configuration is one GPU when the model fits on a single device at its highest resolution, 720p, and eight GPUs when it does not. Cosmos3-Nano fits on a single GPU at 720p, so all populated rows here report one GPU. + +Empty cells mean that a run has not been completed for that GPU, engine, or resolution; they do not indicate that a combination is unsupported. ## Table of Contents @@ -31,7 +35,7 @@ These tables report **Cosmos3-Nano Generator** latency in seconds. Lower is bett The primary t2v, i2v, and t2i tables preserve the previously published benchmark campaigns across PyTorch, vLLM-Omni, Diffusers, and NIM. Those tables use BF16 precision, batch size 1, and matched prompts, seeds, and sampler settings where documented. Video workloads follow the standard Cosmos3 generation profile of 189 frames at 24 FPS unless a resolution tier limits frame count. -The additional audiovisual and action tables come from three internal benchmark reports. PyTorch values are from PBR `#308197`, **Cosmos3-Generator OSS Inference Benchmarking 32B and 8B (189 frames)**: average generation (sampling) latency from the native OSS path, using **CUDA Graphs disabled** and the **latency** automatic-sharding preset. vLLM-Omni values for text-to-audio-and-video (`t2av`/`t2vs`) and image-to-audio-and-video (`i2av`/`i2vs`) are from PBR `#308195`. vLLM-Omni action values are from PBR `#308481`, **Cosmos3-Generator vLLM-Omni Inference Benchmarking (action)**, measured with the `vllm/vllm-omni:cosmos3` image and the official action cookbook samples; `DIFFUSION_ATTENTION_BACKEND=TORCH_SDPA` was not set. Forward-dynamics cells are the mean of `av_forward`, `av_left`, and `av_right`; inverse-dynamics cells are the mean of `av_inverse_0` and `av_inverse_1`. That action sweep reports 1/2/4 GPUs only: 8-GPU Ulysses runs failed a sequence-length divisibility check, so `/8` action cells stay empty. The 2-GPU columns are omitted to keep the published table layout. Policy-DROID is a separate checkpoint and was measured only at 480p on one GPU. Super was not measured on H20, H100 NVL, or H100 80GB HBM3. Values are rounded to two decimal places. +The additional audiovisual and action tables come from three internal benchmark reports. PyTorch values are from PBR `#308197`, **Cosmos3-Generator OSS Inference Benchmarking 32B and 8B (189 frames)**: average generation (sampling) latency from the native OSS path, using **CUDA Graphs disabled** and the **latency** automatic-sharding preset. vLLM-Omni values for text-to-audio-and-video (`t2av`/`t2vs`) and image-to-audio-and-video (`i2av`/`i2vs`) are from PBR `#308195`. vLLM-Omni action values are from PBR `#308481`, **Cosmos3-Generator vLLM-Omni Inference Benchmarking (action)**, measured with the `vllm/vllm-omni:cosmos3` image and the official action cookbook samples; `DIFFUSION_ATTENTION_BACKEND=TORCH_SDPA` was not set. Forward-dynamics cells are the mean of `av_forward`, `av_left`, and `av_right`; inverse-dynamics cells are the mean of `av_inverse_0` and `av_inverse_1`. That action sweep covers 1, 2, and 4 GPUs only, because its 8-GPU Ulysses runs failed a sequence-length divisibility check; those rows therefore report the single-GPU configuration. Policy-DROID is a separate checkpoint and was measured only at 480p on one GPU. Super was not measured on H20, H100 NVL, or H100 80GB HBM3. Diffusers values come from three further reports: PBR `#308202` (all modalities) and PBR `#308451` (video-to-video) on one GPU and PBR `#308918` for 4- and 8-GPU runs. Where PBR `#308918` offers several tensor- and context-parallel splits at the same GPU count, the fastest is published. Its 1-GPU numbers are not used, because that sweep ran a different denoising-step budget than the single-GPU reports. Values are rounded to two decimal places. These reports establish the reported timing matrix but do not expose every prompt and action payload in this repository. The linked public recipes explain modality behavior and provide representative payloads; their example-specific frame counts and action chunk sizes should not be treated as the exact internal benchmark inputs. @@ -54,114 +58,114 @@ The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound ### Text-to-Video (t2v) -| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | -|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| **RTX PRO 6000 Blackwell** | PyTorch | 13.95 | 6.46 | 4.90 | 180.81 | 57.23 | 35.71 | 786.37 | 225.45 | 127.57 | -| | vLLM-Omni | 10.65 | 5.06 | 3.78 | 105.61 | 35.93 | 23.76 | 369.67 | 114.30 | 68.66 | -| | Diffusers | 11.20 | | | 112.00 | | | 392.00 | | | -| | NIM | 7.93 | 4.69 | 4.00 | 82.69 | 33.57 | 24.49 | 318.69 | 107.83 | 68.50 | -| **H20** | PyTorch | 30.57 | 11.25 | 7.87 | 257.51 | 80.18 | 50.49 | 931.39 | 268.88 | 157.71 | -| | vLLM-Omni | 28.58 | 10.20 | 7.70 | 256.97 | 77.42 | 47.53 | 929.81 | 260.75 | 148.46 | -| | Diffusers | 30.20 | | | 258.00 | | | 926.00 | | | -| | NIM | 17.81 | 7.84 | 6.14 | 192.07 | 62.80 | 41.67 | 771.37 | 223.24 | 132.68 | -| **H100 NVL** | PyTorch | 10.03 | 4.27 | 3.95 | 84.12 | 29.18 | 21.46 | 297.27 | 94.15 | 61.63 | -| | vLLM-Omni | 9.25 | 3.68 | 3.15 | 80.75 | 27.48 | 18.77 | 311.13 | 88.25(*) | 54.01(*) | -| | Diffusers | 11.00 | | | 90.00 | | | 324.20 | | | -| | NIM | 7.09 | 3.77 | 3.65 | 68.01 | 25.67 | 20.51 | 267.73 | 86.18 | 57.38 | -| **H200 NVL** | PyTorch | 8.17 | 3.75 | 3.14 | 69.79 | 24.23 | 15.10 | 244.39 | 77.35 | 45.70 | -| | vLLM-Omni | 7.44 | 3.27 | 2.33 | 64.58 | 21.31 | 12.92 | 240.05 | 69.63 | 39.17 | -| | Diffusers | 9.00 | | | 74.00 | | | 276.20 | | | -| | NIM | 5.87 | 3.38 | 3.00 | 57.10 | 21.74 | 15.04 | 229.63 | 71.32 | 43.34 | -| **H100 80GB HBM3** | PyTorch | 7.61 | 3.50 | 3.17 | 59.83 | 21.23 | 14.37 | 207.78 | 66.94 | 41.81 | -| | vLLM-Omni | 6.97 | 3.45 | 3.49 | 58.17 | 19.95 | 13.46 | 202.29 | 62.82 | 37.80 | -| | Diffusers | 9.00 | | | 68.00 | | | 240.00 | | | -| | NIM | 5.72 | 3.36 | 3.12 | 51.73 | 20.26 | 14.82 | 199.46 | 65.32 | 41.66 | -| **H200 141GB HBM3** | PyTorch | 7.53 | 3.34 | 3.19 | 60.18 | 20.84 | 13.97 | 214.28 | 67.48 | 41.26 | -| | vLLM-Omni | 6.79 | 3.25 | 3.42 | 58.14 | 19.77 | 12.97 | 208.36 | 63.27 | 37.49 | -| | Diffusers | 9.00 | | | 67.00 | | | 239.60 | | | -| | NIM | 5.72 | 3.26 | 3.10 | 51.90 | 20.09 | 14.61 | 200.63 | 64.57 | 40.70 | -| **B200** | PyTorch | 4.56 | 2.78 | 2.79 | 33.20 | 13.20 | 9.69 | 114.85 | 39.75 | 26.27 | -| | vLLM-Omni | 4.03 | 2.43 | 3.49 | 32.04 | 12.63 | 10.09 | 107.84 | 35.29 | 22.87 | -| | Diffusers | 7.00 | | | 36.80 | | | 117.00 | | | -| | NIM | 3.72 | 2.74 | 2.89 | 26.68 | 12.66 | 10.10 | 93.33 | 35.04 | 24.64 | -| **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | 4.46 | 4.11 | 5.44 | 32.18 | 13.83 | 11.57 | 102.10 | 35.68 | 24.33 | -| | Diffusers | 39.40 | | | 63.40 | | | 139.40 | | | -| | NIM | 4.51 | 3.54 | 4.11 | 27.49 | 14.51 | 11.86 | 90.39 | 37.28 | 26.19 | +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 1 | 13.95 | 180.81 | 786.37 | +| | vLLM-Omni | 1 | 10.65 | 105.61 | 369.67 | +| | Diffusers | 1 | 11.20 | 112.00 | 392.00 | +| | NIM | 1 | 7.93 | 82.69 | 318.69 | +| **H20** | PyTorch | 1 | 30.57 | 257.51 | 931.39 | +| | vLLM-Omni | 1 | 28.58 | 256.97 | 929.81 | +| | Diffusers | 1 | 30.20 | 258.00 | 926.00 | +| | NIM | 1 | 17.81 | 192.07 | 771.37 | +| **H100 NVL** | PyTorch | 1 | 10.03 | 84.12 | 297.27 | +| | vLLM-Omni | 1 | 9.25 | 80.75 | 311.13 | +| | Diffusers | 1 | 11.00 | 90.00 | 324.20 | +| | NIM | 1 | 7.09 | 68.01 | 267.73 | +| **H200 NVL** | PyTorch | 1 | 8.17 | 69.79 | 244.39 | +| | vLLM-Omni | 1 | 7.44 | 64.58 | 240.05 | +| | Diffusers | 1 | 9.00 | 74.00 | 276.20 | +| | NIM | 1 | 5.87 | 57.10 | 229.63 | +| **H100 80GB HBM3** | PyTorch | 1 | 7.61 | 59.83 | 207.78 | +| | vLLM-Omni | 1 | 6.97 | 58.17 | 202.29 | +| | Diffusers | 1 | 9.00 | 68.00 | 240.00 | +| | NIM | 1 | 5.72 | 51.73 | 199.46 | +| **H200 141GB HBM3** | PyTorch | 1 | 7.53 | 60.18 | 214.28 | +| | vLLM-Omni | 1 | 6.79 | 58.14 | 208.36 | +| | Diffusers | 1 | 9.00 | 67.00 | 239.60 | +| | NIM | 1 | 5.72 | 51.90 | 200.63 | +| **B200** | PyTorch | 1 | 4.56 | 33.20 | 114.85 | +| | vLLM-Omni | 1 | 4.03 | 32.04 | 107.84 | +| | Diffusers | 1 | 7.00 | 36.80 | 117.00 | +| | NIM | 1 | 3.72 | 26.68 | 93.33 | +| **B300** | PyTorch | — | | | | +| | vLLM-Omni | 1 | 4.46 | 32.18 | 102.10 | +| | Diffusers | 1 | 6.40 | 39.32 | 121.74 | +| | NIM | 1 | 4.51 | 27.49 | 90.39 | ### Image-to-Video (i2v) -| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | -|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| **RTX PRO 6000 Blackwell** | PyTorch | 14.08 | 6.51 | 4.92 | 182.14 | 57.43 | 35.74 | 788.80 | 226.25 | 127.79 | -| | vLLM-Omni | 11.04 | 5.48 | 4.24 | 107.77 | 38.05 | 25.95 | 375.01 | 119.27 | 73.57 | -| | Diffusers | 12.00 | | | 112.00 | | | 397.00 | | | -| | NIM | 8.23 | 5.28 | 4.64 | 84.58 | 35.74 | 26.63 | 326.96 | 112.27 | 73.31 | -| **H20** | PyTorch | 31.36 | 11.39 | 7.97 | 257.10 | 80.53 | 50.63 | 933.07 | 268.99 | 158.10 | -| | vLLM-Omni | 29.50 | 11.26 | 8.64 | 261.56 | 81.93 | 52.06 | 940.16 | 271.37 | 158.76 | -| | Diffusers | 31.00 | | | 258.00 | | | 925.00 | | | -| | NIM | 18.67 | 9.04 | 7.38 | 195.10 | 67.64 | 46.30 | 774.88 | 233.92 | 143.15 | -| **H100 NVL** | PyTorch | 10.19 | 4.31 | 3.99 | 84.50 | 28.69 | 21.52 | 298.57 | 95.76 | 60.58 | -| | vLLM-Omni | 9.62 | 4.11 | 3.63 | 82.61 | 29.35 | 20.73 | 286.33 | 92.23(*) | 58.02(*) | -| | Diffusers | 11.00 | | | 91.00 | | | 325.20 | | | -| | NIM | 7.39 | 4.36 | 4.28 | 69.39 | 27.67 | 22.43 | 272.29 | 90.26 | 61.55 | -| **H200 NVL** | PyTorch | 8.27 | 3.76 | 3.09 | 69.99 | 24.22 | 15.12 | 246.62 | 77.69 | 45.99 | -| | vLLM-Omni | 7.83 | 3.69 | 2.78 | 66.39 | 22.93 | 14.58 | 243.52 | 73.26 | 42.86 | -| | Diffusers | 9.00 | | | 74.00 | | | 275.20 | | | -| | NIM | 6.25 | 3.97 | 3.60 | 58.64 | 23.33 | 16.75 | 232.47 | 75.13 | 47.22 | -| **H100 80GB HBM3** | PyTorch | 7.64 | 3.47 | 3.21 | 59.95 | 21.40 | 14.43 | 207.87 | 67.52 | 41.66 | -| | vLLM-Omni | 7.37 | 3.81 | 3.97 | 59.77 | 21.68 | 15.12 | 205.97 | 66.52 | 41.51 | -| | Diffusers | 9.00 | | | 68.00 | | | 239.80 | | | -| | NIM | 6.08 | 3.89 | 3.71 | 53.02 | 22.20 | 16.61 | 202.59 | 69.02 | 45.16 | -| **H200 141GB HBM3** | PyTorch | 7.65 | 3.37 | 3.17 | 60.51 | 21.01 | 14.07 | 214.80 | 67.14 | 41.00 | -| | vLLM-Omni | 7.28 | 3.63 | 3.83 | 59.64 | 21.35 | 14.67 | 209.65 | 66.65 | 40.77 | -| | Diffusers | 9.00 | | | 67.20 | | | 240.00 | | | -| | NIM | 6.04 | 3.80 | 3.65 | 53.25 | 21.89 | 16.23 | 203.66 | 68.30 | 44.29 | -| **B200** | PyTorch | 4.60 | 2.77 | 2.81 | 33.08 | 13.07 | 9.66 | 113.90 | 40.01 | 26.58 | -| | vLLM-Omni | 4.33 | 2.77 | 3.84 | 33.09 | 13.79 | 11.39 | 110.19 | 37.76 | 25.68 | -| | Diffusers | | | | | | | 116.00 | | | -| | NIM | 4.05 | 3.43 | 3.60 | 27.72 | 14.02 | 11.43 | 95.57 | 37.76 | 27.29 | -| **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | 5.61 | 4.67 | 5.90 | 33.45 | 15.06 | 13.13 | 104.75 | 38.27 | 26.87 | -| | Diffusers | 28.60 | | | 65.60 | | | 139.60 | | | -| | NIM | 4.50 | 4.96 | 5.03 | 28.90 | 16.06 | 13.25 | 92.59 | 39.49 | 29.00 | +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 1 | 14.08 | 182.14 | 788.80 | +| | vLLM-Omni | 1 | 11.04 | 107.77 | 375.01 | +| | Diffusers | 1 | 12.00 | 112.00 | 397.00 | +| | NIM | 1 | 8.23 | 84.58 | 326.96 | +| **H20** | PyTorch | 1 | 31.36 | 257.10 | 933.07 | +| | vLLM-Omni | 1 | 29.50 | 261.56 | 940.16 | +| | Diffusers | 1 | 31.00 | 258.00 | 925.00 | +| | NIM | 1 | 18.67 | 195.10 | 774.88 | +| **H100 NVL** | PyTorch | 1 | 10.19 | 84.50 | 298.57 | +| | vLLM-Omni | 1 | 9.62 | 82.61 | 286.33 | +| | Diffusers | 1 | 11.00 | 91.00 | 325.20 | +| | NIM | 1 | 7.39 | 69.39 | 272.29 | +| **H200 NVL** | PyTorch | 1 | 8.27 | 69.99 | 246.62 | +| | vLLM-Omni | 1 | 7.83 | 66.39 | 243.52 | +| | Diffusers | 1 | 9.00 | 74.00 | 275.20 | +| | NIM | 1 | 6.25 | 58.64 | 232.47 | +| **H100 80GB HBM3** | PyTorch | 1 | 7.64 | 59.95 | 207.87 | +| | vLLM-Omni | 1 | 7.37 | 59.77 | 205.97 | +| | Diffusers | 1 | 9.00 | 68.00 | 239.80 | +| | NIM | 1 | 6.08 | 53.02 | 202.59 | +| **H200 141GB HBM3** | PyTorch | 1 | 7.65 | 60.51 | 214.80 | +| | vLLM-Omni | 1 | 7.28 | 59.64 | 209.65 | +| | Diffusers | 1 | 9.00 | 67.20 | 240.00 | +| | NIM | 1 | 6.04 | 53.25 | 203.66 | +| **B200** | PyTorch | 1 | 4.60 | 33.08 | 113.90 | +| | vLLM-Omni | 1 | 4.33 | 33.09 | 110.19 | +| | Diffusers | 1 | 7.00 | 37.00 | 116.00 | +| | NIM | 1 | 4.05 | 27.72 | 95.57 | +| **B300** | PyTorch | — | | | | +| | vLLM-Omni | 1 | 5.61 | 33.45 | 104.75 | +| | Diffusers | 1 | 13.40 | 49.59 | 131.25 | +| | NIM | 1 | 4.50 | 28.90 | 92.59 | ### Text-to-Image (t2i) -| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | -|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| **RTX PRO 6000 Blackwell** | PyTorch | 2.99 | 2.36 | 2.38 | 4.51 | 2.46 | 2.48 | 7.12 | 3.18 | 2.70 | -| | vLLM-Omni | 1.59 | 1.54 | 1.59 | 2.87 | 1.55 | 1.81 | 4.99 | 2.32 | 1.96 | -| | Diffusers | 2.00 | | | 4.00 | | | 5.00 | | | -| **H20** | PyTorch | 3.06 | 2.65 | 2.74 | 6.51 | 2.84 | 2.89 | 12.31 | 4.28 | 3.06 | -| | vLLM-Omni | 1.73 | 2.46 | 3.22 | 4.92 | 2.57 | 7.24 | 10.73 | 4.24 | 3.59 | -| | Diffusers | 3.00 | | | 6.00 | | | 10.00 | | | -| **H100 NVL** | PyTorch | 2.77 | 2.45 | 2.57 | 2.83 | 2.56 | 2.51 | 4.21 | 2.57 | 2.64 | -| | vLLM-Omni | 1.55 | 1.75 | 1.91 | 1.92 | 1.81 | 10.82 | 3.44 | 1.83 | 1.90 | -| | Diffusers | 3.00 | | | 3.00 | | | 4.00 | | | -| **H200 NVL** | PyTorch | 2.75 | 2.42 | 2.50 | 2.85 | 2.52 | 2.62 | 3.58 | 2.62 | 2.64 | -| | vLLM-Omni | 1.53 | 2.01 | 1.96 | 1.58 | 1.91 | 17.71 | 2.81 | 1.94 | 1.94 | -| | Diffusers | 3.00 | | | 3.00 | | | 4.00 | | | -| **H100 80GB HBM3** | PyTorch | 3.01 | 2.66 | 2.56 | 3.01 | 2.59 | 2.75 | 3.45 | 2.73 | 2.77 | -| | vLLM-Omni | 1.61 | 2.45 | 3.18 | 1.53 | 2.35 | 7.02 | 2.61 | 2.45 | 3.03 | -| | Diffusers | 3.00 | | | 3.00 | | | 4.00 | | | -| **H200 141GB HBM3** | PyTorch | 2.96 | 2.59 | 2.70 | 3.04 | 2.78 | 2.77 | 3.28 | 2.84 | 2.77 | -| | vLLM-Omni | 1.57 | 2.38 | 3.16 | 1.52 | 2.37 | 7.05 | 2.60 | 2.33 | 3.20 | -| | Diffusers | 3.00 | | | 3.00 | | | 4.00 | | | -| **B200** | PyTorch | 2.68 | 2.39 | 2.59 | 2.75 | 2.43 | 2.56 | 2.87 | 2.58 | 2.62 | -| | vLLM-Omni | 1.49 | 2.21 | 3.27 | 1.20 | 2.05 | 7.58 | 1.77 | 2.20 | 3.41 | -| | Diffusers | | | | | | | 3.00 | | | -| **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | 1.97 | 4.52 | 5.82 | 1.81 | 4.16 | 71.19 | 2.34 | 4.09 | 5.62 | -| | Diffusers | 36.20 | | | | | | 41.00 | | | +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 1 | 2.99 | 4.51 | 7.12 | +| | vLLM-Omni | 1 | 1.59 | 2.87 | 4.99 | +| | Diffusers | 1 | 2.00 | 4.00 | 5.00 | +| **H20** | PyTorch | 1 | 3.06 | 6.51 | 12.31 | +| | vLLM-Omni | 1 | 1.73 | 4.92 | 10.73 | +| | Diffusers | 1 | 3.00 | 6.00 | 10.00 | +| **H100 NVL** | PyTorch | 1 | 2.77 | 2.83 | 4.21 | +| | vLLM-Omni | 1 | 1.55 | 1.92 | 3.44 | +| | Diffusers | 1 | 3.00 | 3.00 | 4.00 | +| **H200 NVL** | PyTorch | 1 | 2.75 | 2.85 | 3.58 | +| | vLLM-Omni | 1 | 1.53 | 1.58 | 2.81 | +| | Diffusers | 1 | 3.00 | 3.00 | 4.00 | +| **H100 80GB HBM3** | PyTorch | 1 | 3.01 | 3.01 | 3.45 | +| | vLLM-Omni | 1 | 1.61 | 1.53 | 2.61 | +| | Diffusers | 1 | 3.00 | 3.00 | 4.00 | +| **H200 141GB HBM3** | PyTorch | 1 | 2.96 | 3.04 | 3.28 | +| | vLLM-Omni | 1 | 1.57 | 1.52 | 2.60 | +| | Diffusers | 1 | 3.00 | 3.00 | 4.00 | +| **B200** | PyTorch | 1 | 2.68 | 2.75 | 2.87 | +| | vLLM-Omni | 1 | 1.49 | 1.20 | 1.77 | +| | Diffusers | 1 | 3.00 | 3.20 | 3.00 | +| **B300** | PyTorch | — | | | | +| | vLLM-Omni | 1 | 1.97 | 1.81 | 2.34 | +| | Diffusers | 1 | 3.86 | 3.86 | 3.99 | Notes: 1. All times measured on identical workloads (same seed, sampler settings, prompt). -2. 4×/8× GPU configurations use tensor parallelism. +2. Multi-GPU configurations use tensor parallelism. 3. vLLM-Omni numbers are for the upcoming public release in the vLLM-Omni repo; subject to change before GA. Values marked with (*) are pre-release vLLM-Omni measurements on H100 NVL and may change before GA. -4. Diffusers numbers use the HuggingFace `diffusers` integration without custom CUDA graphs; reported at 256p/1, 480p/1, and 720p/1 (single-GPU only). +4. Diffusers numbers use the HuggingFace `diffusers` integration without custom CUDA graphs, and were measured on a single GPU only. 5. PyTorch numbers report average generation (sampling) time from OSS inference benchmarking. -6. At 256p, multi-GPU configurations on B300 may underperform single-GPU due to small-workload TP overhead; single-GPU is recommended at this resolution. +6. At 256p, multi-GPU configurations on B300 may underperform single-GPU because of small-workload tensor-parallel overhead, so single-GPU is the recommended deployment at this resolution. 7. NIM numbers use latency profiles with FP8 precision and report end-to-end `Request Latency s`, including request processing, video generation, output encoding, and returning the response. ## Additional audiovisual generation @@ -170,125 +174,125 @@ The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound A text prompt and source video condition a generated continuation or transformation. -| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | -|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| **RTX PRO 6000 Blackwell** | PyTorch | 13.73 | 6.49 | 4.88 | 180.54 | 57.44 | 35.65 | 785.41 | 224.94 | 127.19 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H20** | PyTorch | 30.44 | 11.35 | 7.94 | 257.79 | 80.52 | 50.56 | 930.96 | 268.82 | 157.81 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H100 NVL** | PyTorch | 10.18 | 4.29 | 3.95 | 81.73 | 28.86 | 21.40 | 297.83 | 94.37 | 61.31 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 NVL** | PyTorch | 8.10 | 3.73 | 3.12 | 69.70 | 24.17 | 15.14 | 250.33 | 76.59 | 45.94 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H100 80GB HBM3** | PyTorch | 7.50 | 3.47 | 3.16 | 59.75 | 21.10 | 14.29 | 207.36 | 67.05 | 41.54 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 141GB HBM3** | PyTorch | 7.48 | 3.35 | 3.11 | 60.23 | 20.99 | 13.94 | 214.03 | 66.90 | 41.06 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **B200** | PyTorch | 4.52 | 2.74 | 2.78 | 32.53 | 13.06 | 9.68 | 114.16 | 39.67 | 26.33 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 1 | 13.73 | 180.54 | 785.41 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 24.19 | 135.85 | 432.18 | +| **H20** | PyTorch | 1 | 30.44 | 257.79 | 930.96 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 44.88 | 287.55 | 974.75 | +| **H100 NVL** | PyTorch | 1 | 10.18 | 81.73 | 297.83 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 23.66 | 107.79 | 354.66 | +| **H200 NVL** | PyTorch | 1 | 8.10 | 69.70 | 250.33 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 21.40 | 96.52 | 313.13 | +| **H100 80GB HBM3** | PyTorch | 1 | 7.50 | 59.75 | 207.36 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 19.48 | 86.51 | 267.90 | +| **H200 141GB HBM3** | PyTorch | 1 | 7.48 | 60.23 | 214.03 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 18.93 | 85.52 | 266.84 | +| **B200** | PyTorch | 1 | 4.52 | 32.53 | 114.16 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 16.93 | 52.32 | 137.54 | +| **B300** | PyTorch | — | | | | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 16.14 | 50.74 | 131.11 | ### Text-to-Audio-and-Video (t2av) A text prompt produces synchronized video and sound. -| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | -|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| **RTX PRO 6000 Blackwell** | PyTorch | 14.13 | 6.47 | 4.89 | 180.98 | 57.21 | 35.77 | 794.74 | 225.73 | 127.53 | -| | vLLM-Omni | 12.25 | 5.32 | 4.19 | 106.02 | 36.11 | 24.02 | 373.10 | 115.06 | 69.19 | -| | Diffusers | | | | | | | | | | -| **H20** | PyTorch | 30.82 | 11.21 | 7.87 | 256.41 | 80.22 | 50.51 | 922.35 | 268.73 | 158.03 | -| | vLLM-Omni | 29.43 | 10.74 | 7.88 | 260.84 | 78.67 | 48.02 | 937.93 | 262.83 | 148.26 | -| | Diffusers | | | | | | | | | | -| **H100 NVL** | PyTorch | 9.99 | 4.27 | 3.95 | 82.32 | 28.50 | 21.46 | 297.88 | 94.45 | 60.25 | -| | vLLM-Omni | 9.92 | 3.85 | 3.26 | 81.57 | 27.15 | 18.93 | 291.91 | 90.29 | 55.66 | -| | Diffusers | | | | | | | | | | -| **H200 NVL** | PyTorch | 8.20 | 3.75 | 3.08 | 69.88 | 24.30 | 15.17 | 250.46 | 78.00 | 45.75 | -| | vLLM-Omni | 7.82 | 3.43 | 2.35 | 65.42 | 21.42 | 12.93 | 244.60 | 70.56 | 39.95 | -| | Diffusers | | | | | | | | | | -| **H100 80GB HBM3** | PyTorch | 7.60 | 3.43 | 3.18 | 59.09 | 21.18 | 14.32 | 207.87 | 66.74 | 41.71 | -| | vLLM-Omni | 7.34 | 3.54 | 3.43 | 58.65 | 20.10 | 13.47 | 203.92 | 63.22 | 38.45 | -| | Diffusers | | | | | | | | | | -| **H200 141GB HBM3** | PyTorch | 7.55 | 3.33 | 3.13 | 60.17 | 20.86 | 13.90 | 211.59 | 67.02 | 41.24 | -| | vLLM-Omni | 7.13 | 3.45 | 3.35 | 58.27 | 19.62 | 13.11 | 209.29 | 63.53 | 37.92 | -| | Diffusers | | | | | | | | | | -| **B200** | PyTorch | 4.63 | 2.81 | 2.81 | 32.60 | 13.17 | 9.64 | 113.89 | 39.59 | 26.33 | -| | vLLM-Omni | 4.29 | 2.51 | 3.47 | 32.20 | 12.70 | 10.08 | 107.74 | 36.10 | 23.83 | -| | Diffusers | | | | | | | | | | -| **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | 5.59 | 4.21 | 5.54 | 33.15 | 14.10 | 11.90 | 104.92 | 36.00 | 24.92 | -| | Diffusers | | | | | | | | | | +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 1 | 14.13 | 180.98 | 794.74 | +| | vLLM-Omni | 1 | 12.25 | 106.02 | 373.10 | +| | Diffusers | 1 | 13.00 | 113.00 | 393.00 | +| **H20** | PyTorch | 1 | 30.82 | 256.41 | 922.35 | +| | vLLM-Omni | 1 | 29.43 | 260.84 | 937.93 | +| | Diffusers | 1 | 32.00 | 263.00 | 932.60 | +| **H100 NVL** | PyTorch | 1 | 9.99 | 82.32 | 297.88 | +| | vLLM-Omni | 1 | 9.92 | 81.57 | 291.91 | +| | Diffusers | 1 | 12.00 | 91.00 | 324.60 | +| **H200 NVL** | PyTorch | 1 | 8.20 | 69.88 | 250.46 | +| | vLLM-Omni | 1 | 7.82 | 65.42 | 244.60 | +| | Diffusers | 1 | 10.00 | 75.00 | 277.80 | +| **H100 80GB HBM3** | PyTorch | 1 | 7.60 | 59.09 | 207.87 | +| | vLLM-Omni | 1 | 7.34 | 58.65 | 203.92 | +| | Diffusers | 1 | 9.40 | 69.00 | 242.00 | +| **H200 141GB HBM3** | PyTorch | 1 | 7.55 | 60.17 | 211.59 | +| | vLLM-Omni | 1 | 7.13 | 58.27 | 209.29 | +| | Diffusers | 1 | 9.00 | 69.00 | 237.00 | +| **B200** | PyTorch | 1 | 4.63 | 32.60 | 113.89 | +| | vLLM-Omni | 1 | 4.29 | 32.20 | 107.74 | +| | Diffusers | 1 | 7.00 | 37.00 | 117.00 | +| **B300** | PyTorch | — | | | | +| | vLLM-Omni | 1 | 5.59 | 33.15 | 104.92 | +| | Diffusers | 1 | 6.46 | 39.66 | 122.33 | ### Video-to-Audio-and-Video (v2av) A text prompt and source video produce transformed video with synchronized sound. -| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | -|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| **RTX PRO 6000 Blackwell** | PyTorch | 13.73 | 6.45 | 4.88 | 180.72 | 57.30 | 35.73 | 786.11 | 224.99 | 127.45 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H20** | PyTorch | 30.42 | 11.35 | 7.94 | 257.83 | 80.48 | 50.55 | 921.82 | 268.81 | 158.07 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H100 NVL** | PyTorch | 9.91 | 4.35 | 3.95 | 81.96 | 29.02 | 21.53 | 297.51 | 94.64 | 60.36 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 NVL** | PyTorch | 8.10 | 3.73 | 3.09 | 69.59 | 24.39 | 15.08 | 246.51 | 76.46 | 46.27 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H100 80GB HBM3** | PyTorch | 7.49 | 3.46 | 3.17 | 59.86 | 21.25 | 14.21 | 207.55 | 67.24 | 41.63 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 141GB HBM3** | PyTorch | 7.49 | 3.35 | 3.10 | 60.15 | 21.00 | 13.93 | 214.44 | 67.13 | 41.05 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **B200** | PyTorch | 4.51 | 2.76 | 2.82 | 32.62 | 13.05 | 9.61 | 113.72 | 39.28 | 26.18 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 1 | 13.73 | 180.72 | 786.11 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 26.30 | 135.47 | 429.61 | +| **H20** | PyTorch | 1 | 30.42 | 257.83 | 921.82 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 46.72 | 288.98 | 985.80 | +| **H100 NVL** | PyTorch | 1 | 9.91 | 81.96 | 297.51 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 22.95 | 110.93 | 354.02 | +| **H200 NVL** | PyTorch | 1 | 8.10 | 69.59 | 246.51 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 22.19 | 94.85 | 314.06 | +| **H100 80GB HBM3** | PyTorch | 1 | 7.49 | 59.86 | 207.55 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 21.73 | 88.42 | 268.93 | +| **H200 141GB HBM3** | PyTorch | 1 | 7.49 | 60.15 | 214.44 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 17.98 | 87.84 | 267.54 | +| **B200** | PyTorch | 1 | 4.51 | 32.62 | 113.72 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 15.64 | 50.57 | 138.15 | +| **B300** | PyTorch | — | | | | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 15.93 | 51.87 | 131.52 | ### Image-to-Audio-and-Video (i2av) A text prompt and source image produce video with synchronized sound. -| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | -|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| **RTX PRO 6000 Blackwell** | PyTorch | 14.08 | 6.48 | 4.91 | 181.64 | 57.25 | 35.69 | 787.81 | 226.71 | 127.85 | -| | vLLM-Omni | 12.61 | 5.72 | 4.53 | 108.05 | 38.30 | 26.20 | 378.04 | 120.08 | 74.23 | -| | Diffusers | | | | | | | | | | -| **H20** | PyTorch | 31.34 | 11.36 | 7.97 | 259.06 | 80.55 | 50.63 | 924.39 | 269.21 | 157.97 | -| | vLLM-Omni | 30.36 | 11.60 | 8.91 | 265.37 | 83.11 | 52.70 | 948.09 | 272.93 | 158.76 | -| | Diffusers | | | | | | | | | | -| **H100 NVL** | PyTorch | 10.45 | 4.31 | 3.97 | 84.37 | 28.74 | 21.46 | 298.61 | 94.02 | 60.57 | -| | vLLM-Omni | 10.34 | 4.30 | 3.72 | 83.21 | 29.20 | 20.82 | 295.65 | 94.44 | 59.83 | -| | Diffusers | | | | | | | | | | -| **H200 NVL** | PyTorch | 8.29 | 3.76 | 3.08 | 71.20 | 24.34 | 15.05 | 248.35 | 76.82 | 46.40 | -| | vLLM-Omni | 8.24 | 3.85 | 2.81 | 67.23 | 23.05 | 14.65 | 249.41 | 74.44 | 43.50 | -| | Diffusers | | | | | | | | | | -| **H100 80GB HBM3** | PyTorch | 7.73 | 3.46 | 3.11 | 59.13 | 21.24 | 14.30 | 207.93 | 67.50 | 41.75 | -| | vLLM-Omni | 7.74 | 3.96 | 4.08 | 60.27 | 21.81 | 15.26 | 207.53 | 66.74 | 41.92 | -| | Diffusers | | | | | | | | | | -| **H200 141GB HBM3** | PyTorch | 7.66 | 3.35 | 3.14 | 60.55 | 20.98 | 13.99 | 214.82 | 67.35 | 41.21 | -| | vLLM-Omni | 7.55 | 3.90 | 3.77 | 59.73 | 21.35 | 14.67 | 212.73 | 66.78 | 41.03 | -| | Diffusers | | | | | | | | | | -| **B200** | PyTorch | 4.62 | 2.78 | 2.80 | 32.82 | 13.09 | 9.69 | 113.85 | 39.76 | 26.56 | -| | vLLM-Omni | 4.58 | 2.83 | 3.85 | 33.36 | 13.89 | 11.14 | 110.05 | 38.82 | 26.30 | -| | Diffusers | | | | | | | | | | -| **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | 6.01 | 4.76 | 6.09 | 34.39 | 15.45 | 13.15 | 108.59 | 38.62 | 27.59 | -| | Diffusers | | | | | | | | | | +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 1 | 14.08 | 181.64 | 787.81 | +| | vLLM-Omni | 1 | 12.61 | 108.05 | 378.04 | +| | Diffusers | 1 | 14.00 | 113.00 | 393.00 | +| **H20** | PyTorch | 1 | 31.34 | 259.06 | 924.39 | +| | vLLM-Omni | 1 | 30.36 | 265.37 | 948.09 | +| | Diffusers | 1 | 32.00 | 263.00 | 934.80 | +| **H100 NVL** | PyTorch | 1 | 10.45 | 84.37 | 298.61 | +| | vLLM-Omni | 1 | 10.34 | 83.21 | 295.65 | +| | Diffusers | 1 | 12.00 | 91.00 | 324.40 | +| **H200 NVL** | PyTorch | 1 | 8.29 | 71.20 | 248.35 | +| | vLLM-Omni | 1 | 8.24 | 67.23 | 249.41 | +| | Diffusers | 1 | 10.00 | 75.00 | 277.20 | +| **H100 80GB HBM3** | PyTorch | 1 | 7.73 | 59.13 | 207.93 | +| | vLLM-Omni | 1 | 7.74 | 60.27 | 207.53 | +| | Diffusers | 1 | 10.00 | 69.00 | 241.00 | +| **H200 141GB HBM3** | PyTorch | 1 | 7.66 | 60.55 | 214.82 | +| | vLLM-Omni | 1 | 7.55 | 59.73 | 212.73 | +| | Diffusers | 1 | 9.00 | 68.00 | 241.00 | +| **B200** | PyTorch | 1 | 4.62 | 32.82 | 113.85 | +| | vLLM-Omni | 1 | 4.58 | 33.36 | 110.05 | +| | Diffusers | 1 | 7.20 | 37.00 | 118.00 | +| **B300** | PyTorch | — | | | | +| | vLLM-Omni | 1 | 6.01 | 34.39 | 108.59 | +| | Diffusers | 1 | 15.35 | 50.19 | 131.92 | ## Action generation @@ -296,244 +300,244 @@ Forward dynamics is reported separately for AV, camera, and robot inputs. Invers ### Forward Dynamics — Autonomous Vehicle (AV) -| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | -|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| **RTX PRO 6000 Blackwell** | PyTorch | 16.87 | 12.13 | 8.96 | 16.86 | 12.18 | 8.95 | 16.85 | 12.14 | 8.94 | -| | vLLM-Omni | 2.08 | 1.46 | | 12.94 | 7.06 | | 36.19 | 16.58 | | -| | Diffusers | | | | | | | | | | -| **H20** | PyTorch | 31.47 | 19.97 | 13.77 | 31.16 | 19.91 | 13.78 | 31.45 | 19.95 | 13.78 | -| | vLLM-Omni | 4.82 | 2.36 | | 31.36 | 12.83 | | 87.55 | 32.09 | | -| | Diffusers | | | | | | | | | | -| **H100 NVL** | PyTorch | 10.32 | 7.36 | 6.46 | 10.66 | 7.35 | 6.31 | 10.64 | 7.43 | 6.32 | -| | vLLM-Omni | 1.71 | 1.46 | | 10.58 | 5.63 | | 28.26 | 13.61 | | -| | Diffusers | | | | | | | | | | -| **H200 NVL** | PyTorch | 8.30 | 6.18 | 4.41 | 8.31 | 6.16 | 4.39 | 8.33 | 6.14 | 4.39 | -| | vLLM-Omni | 1.43 | 1.50 | | 8.21 | 3.83 | | 22.78 | 9.37 | | -| | Diffusers | | | | | | | | | | -| **H100 80GB HBM3** | PyTorch | 7.80 | 5.69 | 4.30 | 7.87 | 5.68 | 4.34 | 7.79 | 5.68 | 4.30 | -| | vLLM-Omni | 1.33 | 1.51 | | 7.74 | 3.77 | | 20.60 | 9.09 | | -| | Diffusers | | | | | | | | | | -| **H200 141GB HBM3** | PyTorch | 7.77 | 5.51 | 4.16 | 7.76 | 5.56 | 4.18 | 7.81 | 5.54 | 4.16 | -| | vLLM-Omni | 1.33 | 1.49 | | 7.72 | 3.63 | | 20.65 | 8.79 | | -| | Diffusers | | | | | | | | | | -| **B200** | PyTorch | 4.75 | 3.88 | 3.38 | 4.75 | 3.85 | 3.37 | 4.73 | 3.87 | 3.36 | -| | vLLM-Omni | 0.94 | 1.20 | | 4.93 | 2.84 | | 12.00 | 5.84 | | -| | Diffusers | | | | | | | | | | -| **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | 1.09 | 2.30 | | 4.86 | 3.56 | | 11.70 | 6.31 | | -| | Diffusers | | | | | | | | | | +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 1 | 16.87 | 16.86 | 16.85 | +| | vLLM-Omni | 1 | 2.08 | 12.94 | 36.19 | +| | Diffusers | 1 | 2.00 | 12.00 | 12.00 | +| **H20** | PyTorch | 1 | 31.47 | 31.16 | 31.45 | +| | vLLM-Omni | 1 | 4.82 | 31.36 | 87.55 | +| | Diffusers | 1 | 5.00 | 28.00 | 28.20 | +| **H100 NVL** | PyTorch | 1 | 10.32 | 10.66 | 10.64 | +| | vLLM-Omni | 1 | 1.71 | 10.58 | 28.26 | +| | Diffusers | 1 | 2.00 | 10.00 | 10.00 | +| **H200 NVL** | PyTorch | 1 | 8.30 | 8.31 | 8.33 | +| | vLLM-Omni | 1 | 1.43 | 8.21 | 22.78 | +| | Diffusers | 1 | 2.00 | 8.00 | 8.00 | +| **H100 80GB HBM3** | PyTorch | 1 | 7.80 | 7.87 | 7.79 | +| | vLLM-Omni | 1 | 1.33 | 7.74 | 20.60 | +| | Diffusers | 1 | 2.00 | 8.00 | 8.00 | +| **H200 141GB HBM3** | PyTorch | 1 | 7.77 | 7.76 | 7.81 | +| | vLLM-Omni | 1 | 1.33 | 7.72 | 20.65 | +| | Diffusers | 1 | 2.00 | 8.00 | 8.00 | +| **B200** | PyTorch | 1 | 4.75 | 4.75 | 4.73 | +| | vLLM-Omni | 1 | 0.94 | 4.93 | 12.00 | +| | Diffusers | 1 | 2.00 | 5.00 | 5.00 | +| **B300** | PyTorch | — | | | | +| | vLLM-Omni | 1 | 1.09 | 4.86 | 11.70 | +| | Diffusers | 1 | 20.87 | 24.49 | 26.44 | ### Forward Dynamics — Camera -| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | -|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| **RTX PRO 6000 Blackwell** | PyTorch | 16.82 | 12.01 | 8.80 | 16.82 | 12.01 | 8.80 | 16.81 | 12.01 | 8.81 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H20** | PyTorch | 31.40 | 19.68 | 13.52 | 31.10 | 19.69 | 13.50 | 31.40 | 19.75 | 13.52 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H100 NVL** | PyTorch | 10.61 | 7.35 | 6.19 | 10.32 | 7.27 | 6.31 | 10.57 | 7.35 | 6.25 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 NVL** | PyTorch | 8.24 | 6.08 | 4.32 | 8.26 | 6.09 | 4.32 | 8.26 | 6.09 | 4.33 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H100 80GB HBM3** | PyTorch | 7.78 | 5.64 | 4.24 | 7.83 | 5.61 | 4.21 | 7.75 | 5.60 | 4.21 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 141GB HBM3** | PyTorch | 7.73 | 5.44 | 4.08 | 7.75 | 5.44 | 4.09 | 7.75 | 5.49 | 4.08 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **B200** | PyTorch | 4.70 | 3.79 | 3.23 | 4.74 | 3.78 | 3.27 | 4.71 | 3.78 | 3.23 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 1 | 16.82 | 16.82 | 16.81 | +| | vLLM-Omni | — | | | | +| | Diffusers | — | | | | +| **H20** | PyTorch | 1 | 31.40 | 31.10 | 31.40 | +| | vLLM-Omni | — | | | | +| | Diffusers | — | | | | +| **H100 NVL** | PyTorch | 1 | 10.61 | 10.32 | 10.57 | +| | vLLM-Omni | — | | | | +| | Diffusers | — | | | | +| **H200 NVL** | PyTorch | 1 | 8.24 | 8.26 | 8.26 | +| | vLLM-Omni | — | | | | +| | Diffusers | — | | | | +| **H100 80GB HBM3** | PyTorch | 1 | 7.78 | 7.83 | 7.75 | +| | vLLM-Omni | — | | | | +| | Diffusers | — | | | | +| **H200 141GB HBM3** | PyTorch | 1 | 7.73 | 7.75 | 7.75 | +| | vLLM-Omni | — | | | | +| | Diffusers | — | | | | +| **B200** | PyTorch | 1 | 4.70 | 4.74 | 4.71 | +| | vLLM-Omni | — | | | | +| | Diffusers | — | | | | +| **B300** | PyTorch | — | | | | +| | vLLM-Omni | — | | | | +| | Diffusers | — | | | | ### Forward Dynamics — Robot -| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | -|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| **RTX PRO 6000 Blackwell** | PyTorch | 3.74 | 3.15 | 2.62 | 3.73 | 3.14 | 2.63 | 3.72 | 3.13 | 2.59 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H20** | PyTorch | 7.00 | 4.78 | 3.59 | 7.01 | 4.79 | 3.59 | 7.01 | 4.78 | 3.59 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H100 NVL** | PyTorch | 2.49 | 2.15 | 2.18 | 2.50 | 2.16 | 2.12 | 2.55 | 2.11 | 2.15 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 NVL** | PyTorch | 2.08 | 2.14 | 2.11 | 2.09 | 2.03 | 2.08 | 2.09 | 2.03 | 2.09 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H100 80GB HBM3** | PyTorch | 2.00 | 2.14 | 2.15 | 2.01 | 2.14 | 2.17 | 1.98 | 2.15 | 2.25 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 141GB HBM3** | PyTorch | 1.93 | 2.14 | 2.30 | 1.93 | 2.14 | 2.15 | 1.93 | 2.11 | 2.21 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **B200** | PyTorch | 1.28 | 1.89 | 1.99 | 1.30 | 1.91 | 2.00 | 1.31 | 1.91 | 1.94 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 1 | 3.74 | 3.73 | 3.72 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 2.00 | 3.00 | 3.00 | +| **H20** | PyTorch | 1 | 7.00 | 7.01 | 7.01 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 3.00 | 7.00 | 7.00 | +| **H100 NVL** | PyTorch | 1 | 2.49 | 2.50 | 2.55 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 2.00 | 3.00 | 3.00 | +| **H200 NVL** | PyTorch | 1 | 2.08 | 2.09 | 2.09 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 2.00 | 3.00 | 3.00 | +| **H100 80GB HBM3** | PyTorch | 1 | 2.00 | 2.01 | 1.98 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 2.00 | 3.00 | 3.00 | +| **H200 141GB HBM3** | PyTorch | 1 | 1.93 | 1.93 | 1.93 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 2.00 | 2.00 | 2.00 | +| **B200** | PyTorch | 1 | 1.28 | 1.30 | 1.31 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 2.00 | 2.00 | 2.00 | +| **B300** | PyTorch | — | | | | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 22.65 | 20.75 | 18.10 | ### Inverse Dynamics — Autonomous Vehicle (AV) -| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | -|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| **RTX PRO 6000 Blackwell** | PyTorch | 16.74 | 12.05 | 8.79 | 16.75 | 12.02 | 8.78 | 16.75 | 12.02 | 8.78 | -| | vLLM-Omni | 2.10 | 1.47 | | 12.96 | 7.09 | | 36.10 | 16.61 | | -| | Diffusers | | | | | | | | | | -| **H20** | PyTorch | 31.22 | 19.66 | 13.52 | 30.99 | 19.68 | 13.52 | 31.22 | 19.70 | 13.53 | -| | vLLM-Omni | 4.84 | 2.39 | | 31.36 | 12.81 | | 87.50 | 32.08 | | -| | Diffusers | | | | | | | | | | -| **H100 NVL** | PyTorch | 10.47 | 7.13 | 6.16 | 10.45 | 7.16 | 6.26 | 10.44 | 7.10 | 6.14 | -| | vLLM-Omni | 1.74 | 1.48 | | 10.54 | 5.61 | | 28.16 | | | -| | Diffusers | | | | | | | | | | -| **H200 NVL** | PyTorch | 8.12 | 5.96 | 4.19 | 8.13 | 5.97 | 4.20 | 8.10 | 5.98 | 4.20 | -| | vLLM-Omni | 1.46 | 1.46 | | 8.20 | 3.80 | | 22.34 | 9.32 | | -| | Diffusers | | | | | | | | | | -| **H100 80GB HBM3** | PyTorch | 7.62 | 5.54 | 4.15 | 7.63 | 5.52 | 4.16 | 7.68 | 5.53 | 4.14 | -| | vLLM-Omni | 1.39 | 1.50 | | 7.72 | 3.75 | | 20.55 | 9.05 | | -| | Diffusers | | | | | | | | | | -| **H200 141GB HBM3** | PyTorch | 7.51 | 5.36 | 4.00 | 7.64 | 5.34 | 3.99 | 7.51 | 5.37 | 3.99 | -| | vLLM-Omni | 1.36 | 1.48 | | 7.69 | 3.60 | | 20.65 | 8.71 | | -| | Diffusers | | | | | | | | | | -| **B200** | PyTorch | 4.60 | 3.74 | 3.20 | 4.63 | 3.73 | 3.19 | 4.58 | 3.76 | 3.23 | -| | vLLM-Omni | 0.92 | 1.55 | | 5.10 | 2.83 | | 12.29 | 5.92 | | -| | Diffusers | | | | | | | | | | -| **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | 1.36 | 2.55 | | 5.75 | 4.04 | | 11.89 | 6.67 | | -| | Diffusers | | | | | | | | | | +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 1 | 16.74 | 16.75 | 16.75 | +| | vLLM-Omni | 1 | 2.10 | 12.96 | 36.10 | +| | Diffusers | 1 | 2.00 | 12.00 | 12.00 | +| **H20** | PyTorch | 1 | 31.22 | 30.99 | 31.22 | +| | vLLM-Omni | 1 | 4.84 | 31.36 | 87.50 | +| | Diffusers | 1 | 5.00 | 28.00 | 28.00 | +| **H100 NVL** | PyTorch | 1 | 10.47 | 10.45 | 10.44 | +| | vLLM-Omni | 1 | 1.74 | 10.54 | 28.16 | +| | Diffusers | 1 | 2.00 | 10.00 | 10.00 | +| **H200 NVL** | PyTorch | 1 | 8.12 | 8.13 | 8.10 | +| | vLLM-Omni | 1 | 1.46 | 8.20 | 22.34 | +| | Diffusers | 1 | 2.00 | 8.00 | 8.00 | +| **H100 80GB HBM3** | PyTorch | 1 | 7.62 | 7.63 | 7.68 | +| | vLLM-Omni | 1 | 1.39 | 7.72 | 20.55 | +| | Diffusers | 1 | 2.00 | 8.00 | 8.00 | +| **H200 141GB HBM3** | PyTorch | 1 | 7.51 | 7.64 | 7.51 | +| | vLLM-Omni | 1 | 1.36 | 7.69 | 20.65 | +| | Diffusers | 1 | 2.00 | 8.00 | 8.00 | +| **B200** | PyTorch | 1 | 4.60 | 4.63 | 4.58 | +| | vLLM-Omni | 1 | 0.92 | 5.10 | 12.29 | +| | Diffusers | 1 | 2.00 | 5.00 | 5.00 | +| **B300** | PyTorch | — | | | | +| | vLLM-Omni | 1 | 1.36 | 5.75 | 11.89 | +| | Diffusers | 1 | 8.76 | 17.79 | 17.99 | ### Inverse Dynamics — Robot -| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | -|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| **RTX PRO 6000 Blackwell** | PyTorch | 3.70 | 3.12 | 2.61 | 3.70 | 3.11 | 2.60 | 3.70 | 3.11 | 2.57 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H20** | PyTorch | 6.94 | 4.72 | 3.53 | 6.94 | 4.72 | 3.52 | 6.89 | 4.71 | 3.52 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H100 NVL** | PyTorch | 2.53 | 2.12 | 2.09 | 2.47 | 2.07 | 2.09 | 2.54 | 2.11 | 2.09 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 NVL** | PyTorch | 2.05 | 1.99 | 2.09 | 2.05 | 2.01 | 2.07 | 2.05 | 2.05 | 2.05 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H100 80GB HBM3** | PyTorch | 1.96 | 2.09 | 2.16 | 1.96 | 2.07 | 2.18 | 1.95 | 2.17 | 2.13 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 141GB HBM3** | PyTorch | 1.90 | 2.06 | 2.10 | 1.90 | 2.07 | 2.13 | 1.91 | 2.12 | 2.09 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **B200** | PyTorch | 1.28 | 1.91 | 1.91 | 1.27 | 1.87 | 1.97 | 1.28 | 1.86 | 1.91 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 1 | 3.70 | 3.70 | 3.70 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 2.00 | 3.00 | 3.00 | +| **H20** | PyTorch | 1 | 6.94 | 6.94 | 6.89 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 3.00 | 7.00 | 7.00 | +| **H100 NVL** | PyTorch | 1 | 2.53 | 2.47 | 2.54 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 2.00 | 3.00 | 3.00 | +| **H200 NVL** | PyTorch | 1 | 2.05 | 2.05 | 2.05 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 2.00 | 3.00 | 3.00 | +| **H100 80GB HBM3** | PyTorch | 1 | 1.96 | 1.96 | 1.95 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 2.00 | 3.00 | 3.00 | +| **H200 141GB HBM3** | PyTorch | 1 | 1.90 | 1.90 | 1.91 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 2.00 | 2.20 | 2.00 | +| **B200** | PyTorch | 1 | 1.28 | 1.27 | 1.28 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 2.00 | 2.00 | 2.00 | +| **B300** | PyTorch | — | | | | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 9.28 | 12.40 | 10.86 | ### Policy — Autonomous Vehicle (AV) -| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | -|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| **RTX PRO 6000 Blackwell** | PyTorch | 16.88 | 12.13 | 8.91 | 16.89 | 12.16 | 8.93 | 16.91 | 12.13 | 8.93 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H20** | PyTorch | 31.51 | 19.90 | 13.77 | 31.51 | 19.94 | 13.79 | 31.50 | 19.97 | 13.78 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H100 NVL** | PyTorch | 10.66 | 7.34 | 6.27 | 10.65 | 7.41 | 6.33 | 10.35 | 7.33 | 6.27 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 NVL** | PyTorch | 8.31 | 6.17 | 4.38 | 8.32 | 6.16 | 4.39 | 8.31 | 6.14 | 4.38 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H100 80GB HBM3** | PyTorch | 7.82 | 5.68 | 4.31 | 7.82 | 5.67 | 4.33 | 7.80 | 5.67 | 4.32 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 141GB HBM3** | PyTorch | 7.76 | 5.52 | 4.15 | 7.69 | 5.58 | 4.16 | 7.66 | 5.51 | 4.16 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **B200** | PyTorch | 4.75 | 3.89 | 3.34 | 4.77 | 3.88 | 3.34 | 4.74 | 3.87 | 3.28 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 1 | 16.88 | 16.89 | 16.91 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 2.00 | 12.00 | 12.00 | +| **H20** | PyTorch | 1 | 31.51 | 31.51 | 31.50 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 5.00 | 28.60 | 28.40 | +| **H100 NVL** | PyTorch | 1 | 10.66 | 10.65 | 10.35 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 2.00 | 10.00 | 10.00 | +| **H200 NVL** | PyTorch | 1 | 8.31 | 8.32 | 8.31 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 2.00 | 8.00 | 8.00 | +| **H100 80GB HBM3** | PyTorch | 1 | 7.82 | 7.82 | 7.80 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 2.00 | 8.00 | 8.00 | +| **H200 141GB HBM3** | PyTorch | 1 | 7.76 | 7.69 | 7.66 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 2.00 | 8.00 | 8.00 | +| **B200** | PyTorch | 1 | 4.75 | 4.77 | 4.74 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 2.00 | 5.00 | 5.00 | +| **B300** | PyTorch | — | | | | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 13.11 | 18.21 | 16.49 | ### Policy — Robot -| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | -|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| **RTX PRO 6000 Blackwell** | PyTorch | 3.74 | 3.14 | 2.60 | 3.74 | 3.15 | 2.62 | 3.74 | 3.14 | 2.59 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H20** | PyTorch | 6.95 | 4.79 | 3.60 | 7.00 | 4.78 | 3.59 | 7.01 | 4.78 | 3.60 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H100 NVL** | PyTorch | 2.50 | 2.11 | 2.12 | 2.51 | 2.11 | 2.13 | 2.55 | 2.17 | 2.16 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 NVL** | PyTorch | 2.11 | 2.03 | 2.10 | 2.09 | 2.15 | 2.10 | 2.09 | 2.02 | 2.11 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H100 80GB HBM3** | PyTorch | 1.99 | 2.12 | 2.20 | 1.99 | 2.16 | 2.19 | 1.99 | 2.11 | 2.23 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 141GB HBM3** | PyTorch | 1.93 | 2.10 | 2.21 | 1.93 | 2.10 | 2.16 | 1.94 | 2.11 | 2.10 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **B200** | PyTorch | 1.30 | 1.91 | 1.95 | 1.30 | 1.92 | 1.91 | 1.30 | 1.90 | 2.00 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 1 | 3.74 | 3.74 | 3.74 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 2.00 | 3.00 | 3.00 | +| **H20** | PyTorch | 1 | 6.95 | 7.00 | 7.01 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 3.00 | 7.00 | 7.00 | +| **H100 NVL** | PyTorch | 1 | 2.50 | 2.51 | 2.55 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 2.00 | 3.00 | 3.00 | +| **H200 NVL** | PyTorch | 1 | 2.11 | 2.09 | 2.09 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 2.00 | 3.00 | 3.00 | +| **H100 80GB HBM3** | PyTorch | 1 | 1.99 | 1.99 | 1.99 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 2.00 | 3.00 | 3.00 | +| **H200 141GB HBM3** | PyTorch | 1 | 1.93 | 1.93 | 1.94 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 2.00 | 2.00 | 2.20 | +| **B200** | PyTorch | 1 | 1.30 | 1.30 | 1.30 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 2.00 | 2.00 | 2.00 | +| **B300** | PyTorch | — | | | | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 10.62 | 12.78 | 11.00 | ### Policy — DROID These rows report **Cosmos3-Nano-Policy-DROID**, not the general Cosmos3-Nano checkpoint used in the other action tables. vLLM-Omni only supports this modality at **480p on one GPU** (multi-GPU is disabled, and resolution is fixed for the multiview wrist/exterior inputs). PyTorch and Diffusers cells are reserved. -| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | -|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| **RTX PRO 6000 Blackwell** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | 3.82 | | | | | | -| | Diffusers | | | | | | | | | | -| **H20** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | 8.51 | | | | | | -| | Diffusers | | | | | | | | | | -| **H100 NVL** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | 2.94 | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 NVL** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | 2.34 | | | | | | -| | Diffusers | | | | | | | | | | -| **H100 80GB HBM3** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | 2.23 | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 141GB HBM3** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | 2.21 | | | | | | -| | Diffusers | | | | | | | | | | -| **B200** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | 1.47 | | | | | | -| | Diffusers | | | | | | | | | | -| **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | 2.15 | | | | | | -| | Diffusers | | | | | | | | | | +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | — | | | | +| | vLLM-Omni | 1 | | 3.82 | | +| | Diffusers | — | | | | +| **H20** | PyTorch | — | | | | +| | vLLM-Omni | 1 | | 8.51 | | +| | Diffusers | — | | | | +| **H100 NVL** | PyTorch | — | | | | +| | vLLM-Omni | 1 | | 2.94 | | +| | Diffusers | — | | | | +| **H200 NVL** | PyTorch | — | | | | +| | vLLM-Omni | 1 | | 2.34 | | +| | Diffusers | — | | | | +| **H100 80GB HBM3** | PyTorch | — | | | | +| | vLLM-Omni | 1 | | 2.23 | | +| | Diffusers | — | | | | +| **H200 141GB HBM3** | PyTorch | — | | | | +| | vLLM-Omni | 1 | | 2.21 | | +| | Diffusers | — | | | | +| **B200** | PyTorch | — | | | | +| | vLLM-Omni | 1 | | 1.47 | | +| | Diffusers | — | | | | +| **B300** | PyTorch | — | | | | +| | vLLM-Omni | 1 | | 2.15 | | +| | Diffusers | — | | | | Additional-modality notes: 1. PyTorch values are average generation (sampling) latency in seconds from PBR `#308197`; lower is better. 2. PyTorch values use `CUDA_GRAPH=No` and the `latency` automatic-sharding preset. 3. vLLM-Omni audiovisual values for `t2av`/`t2vs` and `i2av`/`i2vs` are from PBR `#308195`. vLLM-Omni action values are from PBR `#308481`. -4. Action vLLM-Omni `/8` cells are empty because 8-GPU Ulysses runs failed sequence-length divisibility checks. Two documented action misses are also left blank: Super inverse-dynamics AV at 720p/4 on H200 141GB HBM3 (OOM), and Nano inverse-dynamics AV at 720p/4 on H100 NVL (CUDA/NCCL failure). -5. Policy-DROID vLLM-Omni uses the `nvidia/Cosmos3-Nano-Policy-DROID` checkpoint at 480p/1 only. It is reported in its own table and is not comparable to the PyTorch policy-robot row. +4. Action vLLM-Omni rows report one GPU because the 8-GPU Ulysses runs failed sequence-length divisibility checks. +5. Policy-DROID vLLM-Omni uses the `nvidia/Cosmos3-Nano-Policy-DROID` checkpoint at 480p on one GPU only. It is reported in its own table and is not comparable to the PyTorch policy-robot row. 6. Diffusers rows are intentionally empty reservations for future benchmark campaigns. -7. The `/1`, `/4`, and `/8` suffixes denote the number of GPUs used by the benchmark run. -8. Empty cells indicate unmeasured combinations, not unsupported combinations. +7. The **GPUs** column gives the number of GPUs behind every value in that row. +8. Empty cells indicate unmeasured combinations, not unsupported combinations. \ No newline at end of file diff --git a/inference_benchmarks/cosmos3-super-generator.md b/inference_benchmarks/cosmos3-super-generator.md index 3bbaa635..bb02f86b 100644 --- a/inference_benchmarks/cosmos3-super-generator.md +++ b/inference_benchmarks/cosmos3-super-generator.md @@ -2,7 +2,11 @@ [Back to the inference benchmark index](../inference_benchmarks.md) -These tables report **Cosmos3-Super Generator** latency in seconds. Lower is better. Empty cells mean that a run has not been completed for that GPU, engine, resolution, or GPU count; they do not indicate that a combination is unsupported. +These tables report **Cosmos3-Super Generator** latency in seconds. Lower is better. + +Every row reports one GPU configuration, named in the **GPUs** column, and all three resolutions in that row come from it. The configuration is one GPU when the model fits on a single device at its highest resolution, 720p, and eight GPUs when it does not. Cosmos3-Super fits on a single GPU on B200 and B300, and on H200 NVL and H200 141GB HBM3 for every engine except PyTorch; on RTX PRO 6000 Blackwell, H20, H100 NVL, and H100 80GB HBM3 it is reported at eight GPUs. Because the configuration varies between rows, read the **GPUs** column before comparing latencies down a column. + +Empty cells mean that a run has not been completed for that GPU, engine, or resolution; they do not indicate that a combination is unsupported. ## Table of Contents @@ -31,7 +35,7 @@ These tables report **Cosmos3-Super Generator** latency in seconds. Lower is bet The primary t2v, i2v, and t2i tables preserve the previously published benchmark campaigns across PyTorch, vLLM-Omni, Diffusers, and NIM. Those tables use BF16 precision, batch size 1, and matched prompts, seeds, and sampler settings where documented. Video workloads follow the standard Cosmos3 generation profile of 189 frames at 24 FPS unless a resolution tier limits frame count. -The additional audiovisual and action tables come from three internal benchmark reports. PyTorch values are from PBR `#308197`, **Cosmos3-Generator OSS Inference Benchmarking 32B and 8B (189 frames)**: average generation (sampling) latency from the native OSS path, using **CUDA Graphs disabled** and the **latency** automatic-sharding preset. vLLM-Omni values for text-to-audio-and-video (`t2av`/`t2vs`) and image-to-audio-and-video (`i2av`/`i2vs`) are from PBR `#308195`. vLLM-Omni action values are from PBR `#308481`, **Cosmos3-Generator vLLM-Omni Inference Benchmarking (action)**, measured with the `vllm/vllm-omni:cosmos3` image and the official action cookbook samples; `DIFFUSION_ATTENTION_BACKEND=TORCH_SDPA` was not set. Forward-dynamics cells are the mean of `av_forward`, `av_left`, and `av_right`; inverse-dynamics cells are the mean of `av_inverse_0` and `av_inverse_1`. That action sweep reports 1/2/4 GPUs only: 8-GPU Ulysses runs failed a sequence-length divisibility check, so `/8` action cells stay empty. The 2-GPU columns are omitted to keep the published table layout. Policy-DROID is a separate checkpoint and was measured only at 480p on one GPU. Super was not measured on H20, H100 NVL, or H100 80GB HBM3. Values are rounded to two decimal places. +The additional audiovisual and action tables come from three internal benchmark reports. PyTorch values are from PBR `#308197`, **Cosmos3-Generator OSS Inference Benchmarking 32B and 8B (189 frames)**: average generation (sampling) latency from the native OSS path, using **CUDA Graphs disabled** and the **latency** automatic-sharding preset. vLLM-Omni values for text-to-audio-and-video (`t2av`/`t2vs`) and image-to-audio-and-video (`i2av`/`i2vs`) are from PBR `#308195`. vLLM-Omni action values are from PBR `#308481`, **Cosmos3-Generator vLLM-Omni Inference Benchmarking (action)**, measured with the `vllm/vllm-omni:cosmos3` image and the official action cookbook samples; `DIFFUSION_ATTENTION_BACKEND=TORCH_SDPA` was not set. Forward-dynamics cells are the mean of `av_forward`, `av_left`, and `av_right`; inverse-dynamics cells are the mean of `av_inverse_0` and `av_inverse_1`. That action sweep covers 1, 2, and 4 GPUs only, because its 8-GPU Ulysses runs failed a sequence-length divisibility check; those rows therefore report the single-GPU configuration. Policy-DROID is a separate checkpoint and was measured only at 480p on one GPU. Super was not measured on H20, H100 NVL, or H100 80GB HBM3. Diffusers values come from three further reports: PBR `#308202` (all modalities) and PBR `#308451` (video-to-video) on one GPU and PBR `#308918` for 4- and 8-GPU runs. Where PBR `#308918` offers several tensor- and context-parallel splits at the same GPU count, the fastest is published. Its 1-GPU numbers are not used, because that sweep ran a different denoising-step budget than the single-GPU reports. Values are rounded to two decimal places. These reports establish the reported timing matrix but do not expose every prompt and action payload in this repository. The linked public recipes explain modality behavior and provide representative payloads; their example-specific frame counts and action chunk sizes should not be treated as the exact internal benchmark inputs. @@ -54,113 +58,113 @@ The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound ### Text-to-Video (t2v) -| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | -|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| **RTX PRO 6000 Blackwell** | PyTorch | | 65.10 | 66.04 | | 201.16 | 118.90 | | 789.03 | 427.16 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| | NIM | | 12.65 | 13.99 | | 104.25 | 99.05 | | 350.74 | 286.02 | -| **H20** | PyTorch | | 40.74 | 27.72 | | 276.24 | 152.46 | | 930.45 | 492.41 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| | NIM | | 20.07 | 12.95 | | 192.45 | 110.71 | | 734.37 | 395.56 | -| **H100 NVL** | PyTorch | | 20.73 | 16.83 | | 101.27 | 64.14 | | 330.04 | 186.19 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| | NIM | | 8.77 | 12.73 | | 73.37 | 66.07 | | 267.64 | 197.32 | -| **H200 NVL** | PyTorch | | 24.90 | 24.78 | | 82.19 | 47.69 | | 267.66 | 142.35 | -| | vLLM-Omni | 27.54 | | 5.06 | 252.33 | | 36.66 | 911.49 | 245.51 | 123.85 | -| | Diffusers | 33.00 | | | 286.80 | | | 1036.00 | | | -| | NIM | 17.13 | 6.79 | 4.43 | 200.00 | 58.55 | 32.87 | 811.41 | 223.00 | 117.98 | -| **H100 80GB HBM3** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| | NIM | | 6.98 | 5.89 | | 55.10 | 35.52 | | 198.13 | 114.92 | -| **H200 141GB HBM3** | PyTorch | | 14.82 | 11.82 | | 70.27 | 41.78 | | 224.43 | 123.49 | -| | vLLM-Omni | 25.61 | | 5.87 | 219.11 | | 35.26 | 769.63 | 212.30 | 111.94 | -| | Diffusers | 31.00 | | | 251.60 | | | 886.20 | | | -| | NIM | 15.95 | 6.14 | 4.28 | 174.71 | 52.94 | 30.94 | 695.89 | 194.34 | 106.16 | -| **B200** | PyTorch | 14.66 | 5.59 | 4.09 | 114.38 | 35.73 | 21.39 | 407.50 | 118.38 | 65.93 | -| | vLLM-Omni | 13.84 | | 4.76 | 114.08 | | 22.09 | 390.28 | 113.31 | 62.11 | -| | Diffusers | | | | 127.20 | | | 414.40 | | | -| | NIM | 9.09 | 4.26 | 3.38 | 82.39 | 27.83 | 17.74 | 314.68 | 92.25 | 53.43 | -| **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | 14.57 | | 6.68 | 109.03 | | 22.67 | 366.66 | 108.58 | 60.73 | -| | Diffusers | 54.20 | | | 155.40 | | | 424.80 | | | -| | NIM | 9.67 | 5.23 | 5.19 | 79.73 | 28.97 | 18.39 | 292.35 | 92.31 | 54.07 | +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 8 | 66.04 | 118.90 | 427.16 | +| | vLLM-Omni | — | | | | +| | Diffusers | 8 | 25.39 | 154.25 | 437.57 | +| | NIM | 8 | 13.99 | 99.05 | 286.02 | +| **H20** | PyTorch | 8 | 27.72 | 152.46 | 492.41 | +| | vLLM-Omni | — | | | | +| | Diffusers | 8 | 26.66 | 161.49 | 517.37 | +| | NIM | 8 | 12.95 | 110.71 | 395.56 | +| **H100 NVL** | PyTorch | 8 | 16.83 | 64.14 | 186.19 | +| | vLLM-Omni | — | | | | +| | Diffusers | 8 | 24.21 | 108.70 | 322.28 | +| | NIM | 8 | 12.73 | 66.07 | 197.32 | +| **H200 NVL** | PyTorch | 8 | 24.78 | 47.69 | 142.35 | +| | vLLM-Omni | 1 | 27.54 | 252.33 | 911.49 | +| | Diffusers | 1 | 33.00 | 286.80 | 1036.00 | +| | NIM | 1 | 17.13 | 200.00 | 811.41 | +| **H100 80GB HBM3** | PyTorch | — | | | | +| | vLLM-Omni | — | | | | +| | Diffusers | 8 | 19.40 | 56.39 | 174.57 | +| | NIM | 8 | 5.89 | 35.52 | 114.92 | +| **H200 141GB HBM3** | PyTorch | 8 | 11.82 | 41.78 | 123.49 | +| | vLLM-Omni | 1 | 25.61 | 219.11 | 769.63 | +| | Diffusers | 1 | 31.00 | 251.60 | 886.20 | +| | NIM | 1 | 15.95 | 174.71 | 695.89 | +| **B200** | PyTorch | 1 | 14.66 | 114.38 | 407.50 | +| | vLLM-Omni | 1 | 13.84 | 114.08 | 390.28 | +| | Diffusers | 1 | 19.00 | 127.20 | 414.40 | +| | NIM | 1 | 9.09 | 82.39 | 314.68 | +| **B300** | PyTorch | — | | | | +| | vLLM-Omni | 1 | 14.57 | 109.03 | 366.66 | +| | Diffusers | 1 | 17.73 | 125.70 | 398.94 | +| | NIM | 1 | 9.67 | 79.73 | 292.35 | ### Image-to-Video (i2v) -| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | -|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| **RTX PRO 6000 Blackwell** | PyTorch | | 65.42 | 66.54 | | 202.26 | 118.95 | | 795.14 | 427.96 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| | NIM | | 13.17 | 14.48 | | 106.50 | 100.23 | | 356.60 | 289.93 | -| **H20** | PyTorch | | 41.37 | 28.22 | | 278.23 | 153.19 | | 931.74 | 491.93 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| | NIM | | 21.16 | 14.06 | | 196.86 | 114.49 | | 745.12 | 405.68 | -| **H100 NVL** | PyTorch | | 20.85 | 16.96 | | 99.56 | 64.17 | | 331.40 | 186.47 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| | NIM | | 9.32 | 13.30 | | 75.31 | 67.34 | | 271.46 | 201.39 | -| **H200 NVL** | PyTorch | | 24.81 | 24.70 | | 82.51 | 47.60 | | 269.85 | 141.62 | -| | vLLM-Omni | 27.90 | | 5.52 | 254.29 | | 38.51 | 915.05 | 248.89 | 127.32 | -| | Diffusers | 33.00 | | | 287.20 | | | 1034.60 | | | -| | NIM | 17.51 | 7.43 | 5.04 | 201.45 | 60.15 | 34.48 | 817.35 | 226.38 | 121.35 | -| **H100 80GB HBM3** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| | NIM | | 7.50 | 6.49 | | 56.33 | 36.81 | | 200.97 | 118.77 | -| **H200 141GB HBM3** | PyTorch | | 14.87 | 11.80 | | 70.45 | 42.10 | | 224.36 | 123.57 | -| | vLLM-Omni | 25.47 | | 6.32 | 220.70 | | 36.90 | 766.33 | 215.03 | 117.52 | -| | Diffusers | 31.00 | | | 249.20 | | | 879.20 | | | -| | NIM | 16.39 | 6.74 | 4.90 | 175.95 | 54.46 | 32.51 | 699.13 | 197.96 | 109.55 | -| **B200** | PyTorch | 14.71 | 5.63 | 4.12 | 112.40 | 35.70 | 21.25 | 397.31 | 117.98 | 65.91 | -| | vLLM-Omni | 14.13 | | 5.31 | 115.17 | | 23.26 | 393.02 | 115.69 | 64.82 | -| | Diffusers | 19.20 | | | | | | 414.80 | | | -| | NIM | 9.36 | 4.83 | 4.12 | 83.19 | 29.09 | 19.14 | 316.76 | 94.65 | 55.92 | -| **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | 14.14 | | 7.19 | 111.42 | | 23.91 | 368.73 | 111.41 | 63.25 | -| | Diffusers | 54.20 | | | 151.80 | | | 425.00 | | | -| | NIM | 9.73 | 5.58 | 5.94 | 80.51 | 30.17 | 20.62 | 294.11 | 93.77 | 56.76 | +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 8 | 66.54 | 118.95 | 427.96 | +| | vLLM-Omni | — | | | | +| | Diffusers | 8 | 26.29 | 156.16 | 540.65 | +| | NIM | 8 | 14.48 | 100.23 | 289.93 | +| **H20** | PyTorch | 8 | 28.22 | 153.19 | 491.93 | +| | vLLM-Omni | — | | | | +| | Diffusers | 8 | 27.65 | 165.91 | 530.44 | +| | NIM | 8 | 14.06 | 114.49 | 405.68 | +| **H100 NVL** | PyTorch | 8 | 16.96 | 64.17 | 186.47 | +| | vLLM-Omni | — | | | | +| | Diffusers | 8 | 25.45 | 112.36 | 325.56 | +| | NIM | 8 | 13.30 | 67.34 | 201.39 | +| **H200 NVL** | PyTorch | 8 | 24.70 | 47.60 | 141.62 | +| | vLLM-Omni | 1 | 27.90 | 254.29 | 915.05 | +| | Diffusers | 1 | 33.00 | 287.20 | 1034.60 | +| | NIM | 1 | 17.51 | 201.45 | 817.35 | +| **H100 80GB HBM3** | PyTorch | — | | | | +| | vLLM-Omni | — | | | | +| | Diffusers | 8 | 19.98 | 55.63 | 184.81 | +| | NIM | 8 | 6.49 | 36.81 | 118.77 | +| **H200 141GB HBM3** | PyTorch | 8 | 11.80 | 42.10 | 123.57 | +| | vLLM-Omni | 1 | 25.47 | 220.70 | 766.33 | +| | Diffusers | 1 | 31.00 | 249.20 | 879.20 | +| | NIM | 1 | 16.39 | 175.95 | 699.13 | +| **B200** | PyTorch | 1 | 14.71 | 112.40 | 397.31 | +| | vLLM-Omni | 1 | 14.13 | 115.17 | 393.02 | +| | Diffusers | 1 | 19.20 | 127.00 | 414.80 | +| | NIM | 1 | 9.36 | 83.19 | 316.76 | +| **B300** | PyTorch | — | | | | +| | vLLM-Omni | 1 | 14.14 | 111.42 | 368.73 | +| | Diffusers | 1 | 25.21 | 132.23 | 410.81 | +| | NIM | 1 | 9.73 | 80.51 | 294.11 | ### Text-to-Image (t2i) -| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | -|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| **RTX PRO 6000 Blackwell** | PyTorch | | 93.20 | 93.18 | | 92.22 | 93.39 | | 92.89 | 93.47 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H20** | PyTorch | | 14.58 | 14.18 | | 19.93 | 16.46 | | 26.26 | 20.92 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H100 NVL** | PyTorch | | 19.73 | 19.86 | | 19.78 | 19.80 | | 20.68 | 19.87 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 NVL** | PyTorch | | 32.51 | 32.86 | | 32.53 | 33.05 | | 32.64 | 33.16 | -| | vLLM-Omni | 2.73 | | 3.20 | 6.28 | | 17.71 | 11.02 | 4.22 | 3.26 | -| | Diffusers | 5.00 | | | 8.00 | | | 12.00 | | | -| **H100 80GB HBM3** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 141GB HBM3** | PyTorch | | 13.62 | 13.48 | | 13.33 | 13.53 | | 13.78 | 13.50 | -| | vLLM-Omni | 2.83 | | 4.50 | 5.70 | | 7.16 | 10.24 | 4.23 | 4.43 | -| | Diffusers | 5.00 | | | 8.00 | | | 11.00 | | | -| **B200** | PyTorch | 4.51 | 4.10 | 4.27 | 4.78 | 4.13 | 4.48 | 7.25 | 4.28 | 4.65 | -| | vLLM-Omni | 2.32 | | 4.58 | 3.29 | | 9.10 | 6.02 | 3.09 | 4.43 | -| | Diffusers | | | | | | | 8.00 | | | -| **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | 5.05 | | 7.62 | 3.79 | | 72.65 | 7.08 | 5.79 | 7.24 | -| | Diffusers | 38.80 | | | 39.40 | | | 40.40 | | | +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 8 | 93.18 | 93.39 | 93.47 | +| | vLLM-Omni | — | | | | +| | Diffusers | 8 | 16.90 | 17.52 | 16.73 | +| **H20** | PyTorch | 8 | 14.18 | 16.46 | 20.92 | +| | vLLM-Omni | — | | | | +| | Diffusers | 8 | 18.29 | 18.48 | 18.46 | +| **H100 NVL** | PyTorch | 8 | 19.86 | 19.80 | 19.87 | +| | vLLM-Omni | — | | | | +| | Diffusers | 8 | 16.56 | 16.60 | 16.88 | +| **H200 NVL** | PyTorch | 8 | 32.86 | 33.05 | 33.16 | +| | vLLM-Omni | 1 | 2.73 | 6.28 | 11.02 | +| | Diffusers | 1 | 5.00 | 8.00 | 12.00 | +| **H100 80GB HBM3** | PyTorch | — | | | | +| | vLLM-Omni | — | | | | +| | Diffusers | 8 | 17.97 | 18.03 | 19.28 | +| **H200 141GB HBM3** | PyTorch | 8 | 13.48 | 13.53 | 13.50 | +| | vLLM-Omni | 1 | 2.83 | 5.70 | 10.24 | +| | Diffusers | 1 | 5.00 | 8.00 | 11.00 | +| **B200** | PyTorch | 1 | 4.51 | 4.78 | 7.25 | +| | vLLM-Omni | 1 | 2.32 | 3.29 | 6.02 | +| | Diffusers | 1 | 4.40 | 6.00 | 8.00 | +| **B300** | PyTorch | — | | | | +| | vLLM-Omni | 1 | 5.05 | 3.79 | 7.08 | +| | Diffusers | 1 | 6.49 | 6.76 | 7.11 | Notes: 1. All times measured on identical workloads (same seed, sampler settings, prompt). -2. 4×/8× GPU configurations use tensor parallelism. +2. Multi-GPU configurations use tensor parallelism. 3. vLLM-Omni numbers are for the upcoming public release in the vLLM-Omni repo; subject to change before GA. Current vLLM-Omni coverage is B200 at 720p. -4. Diffusers numbers use the HuggingFace `diffusers` integration without custom CUDA graphs; reported at 256p/1, 480p/1, and 720p/1 (single-GPU only). -5. At 256p, multi-GPU configurations on B300 may underperform single-GPU due to small-workload TP overhead; single-GPU is recommended at this resolution. +4. Diffusers numbers use the HuggingFace `diffusers` integration without custom CUDA graphs, and were measured on a single GPU only. +5. At 256p, multi-GPU configurations on B300 may underperform single-GPU because of small-workload tensor-parallel overhead, so single-GPU is the recommended deployment at this resolution. 6. PyTorch numbers report average generation (sampling) time from OSS inference benchmarking. 7. NIM numbers use latency profiles with FP8 precision and report end-to-end `Request Latency s`, including request processing, video generation, output encoding, and returning the response. @@ -170,125 +174,125 @@ The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound A text prompt and source video condition a generated continuation or transformation. -| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | -|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| **RTX PRO 6000 Blackwell** | PyTorch | | 64.80 | 66.31 | | 201.59 | 118.92 | | 787.37 | 427.19 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H20** | PyTorch | | 40.93 | 27.98 | | 277.39 | 153.02 | | 931.12 | 491.75 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H100 NVL** | PyTorch | | 20.75 | 16.79 | | 99.57 | 63.84 | | 328.83 | 183.64 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 NVL** | PyTorch | | 24.87 | 24.74 | | 82.22 | 47.60 | | 268.16 | 142.23 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H100 80GB HBM3** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 141GB HBM3** | PyTorch | | 14.66 | 11.76 | | 70.52 | 41.85 | | 223.86 | 123.09 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **B200** | PyTorch | 14.30 | 5.59 | 4.07 | 111.49 | 35.68 | 21.43 | 395.97 | 117.46 | 65.62 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 8 | 66.31 | 118.92 | 427.19 | +| | vLLM-Omni | — | | | | +| | Diffusers | 8 | 28.39 | 158.14 | 545.45 | +| **H20** | PyTorch | 8 | 27.98 | 153.02 | 491.75 | +| | vLLM-Omni | — | | | | +| | Diffusers | 8 | 28.18 | 167.95 | 532.68 | +| **H100 NVL** | PyTorch | 8 | 16.79 | 63.84 | 183.64 | +| | vLLM-Omni | — | | | | +| | Diffusers | 8 | 25.01 | 112.47 | 348.31 | +| **H200 NVL** | PyTorch | 8 | 24.74 | 47.60 | 142.23 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 40.73 | 316.00 | 1093.16 | +| **H100 80GB HBM3** | PyTorch | — | | | | +| | vLLM-Omni | — | | | | +| | Diffusers | 8 | 21.61 | 57.74 | 188.43 | +| **H200 141GB HBM3** | PyTorch | 8 | 11.76 | 41.85 | 123.09 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 41.17 | 269.44 | 916.41 | +| **B200** | PyTorch | 1 | 14.30 | 111.49 | 395.97 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 26.30 | 142.61 | 437.27 | +| **B300** | PyTorch | — | | | | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 25.86 | 139.04 | 416.98 | ### Text-to-Audio-and-Video (t2av) A text prompt produces synchronized video and sound. -| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | -|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| **RTX PRO 6000 Blackwell** | PyTorch | | 65.38 | 65.79 | | 202.34 | 118.87 | | 788.88 | 429.78 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H20** | PyTorch | | 40.69 | 27.73 | | 277.33 | 152.53 | | 930.63 | 492.38 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H100 NVL** | PyTorch | | 20.78 | 16.86 | | 99.28 | 64.17 | | 329.26 | 183.81 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 NVL** | PyTorch | | 24.88 | 24.79 | | 82.49 | 47.67 | | 267.86 | 142.06 | -| | vLLM-Omni | 29.07 | 9.51 | 5.36 | 260.51 | 70.75 | 36.78 | 916.51 | 249.35 | 127.17 | -| | Diffusers | | | | | | | | | | -| **H100 80GB HBM3** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 141GB HBM3** | PyTorch | | 15.05 | 11.74 | | 70.09 | 41.49 | | 223.96 | 123.35 | -| | vLLM-Omni | 27.21 | 9.01 | 6.16 | 221.67 | 64.07 | 35.22 | 763.74 | 212.47 | 119.74 | -| | Diffusers | | | | | | | | | | -| **B200** | PyTorch | 14.52 | 5.64 | 4.07 | 112.99 | 35.93 | 21.57 | 395.20 | 118.67 | 65.93 | -| | vLLM-Omni | 14.77 | 5.54 | 5.01 | 115.26 | 36.20 | 21.76 | 388.51 | 115.08 | 62.56 | -| | Diffusers | | | | | | | | | | -| **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | 15.26 | 6.89 | 7.09 | 113.34 | 36.79 | 23.39 | 372.74 | 110.75 | 61.99 | -| | Diffusers | | | | | | | | | | +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 8 | 65.79 | 118.87 | 429.78 | +| | vLLM-Omni | — | | | | +| | Diffusers | — | | | | +| **H20** | PyTorch | 8 | 27.73 | 152.53 | 492.38 | +| | vLLM-Omni | — | | | | +| | Diffusers | — | | | | +| **H100 NVL** | PyTorch | 8 | 16.86 | 64.17 | 183.81 | +| | vLLM-Omni | — | | | | +| | Diffusers | — | | | | +| **H200 NVL** | PyTorch | 8 | 24.79 | 47.67 | 142.06 | +| | vLLM-Omni | 1 | 29.07 | 260.51 | 916.51 | +| | Diffusers | 1 | 35.00 | 291.40 | 1034.20 | +| **H100 80GB HBM3** | PyTorch | — | | | | +| | vLLM-Omni | — | | | | +| | Diffusers | — | | | | +| **H200 141GB HBM3** | PyTorch | 8 | 11.74 | 41.49 | 123.35 | +| | vLLM-Omni | 1 | 27.21 | 221.67 | 763.74 | +| | Diffusers | 1 | 33.00 | 253.40 | 874.60 | +| **B200** | PyTorch | 1 | 14.52 | 112.99 | 395.20 | +| | vLLM-Omni | 1 | 14.77 | 115.26 | 388.51 | +| | Diffusers | 1 | 20.00 | 128.00 | 422.00 | +| **B300** | PyTorch | — | | | | +| | vLLM-Omni | 1 | 15.26 | 113.34 | 372.74 | +| | Diffusers | 1 | 18.94 | 126.86 | 402.37 | ### Video-to-Audio-and-Video (v2av) A text prompt and source video produce transformed video with synchronized sound. -| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | -|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| **RTX PRO 6000 Blackwell** | PyTorch | | 64.67 | 66.10 | | 202.31 | 118.88 | | 785.77 | 428.64 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H20** | PyTorch | | 40.89 | 28.00 | | 276.26 | 152.78 | | 930.81 | 492.63 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H100 NVL** | PyTorch | | 20.65 | 16.79 | | 99.90 | 64.69 | | 323.95 | 183.62 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 NVL** | PyTorch | | 24.88 | 24.73 | | 82.36 | 47.67 | | 268.78 | 143.06 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H100 80GB HBM3** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 141GB HBM3** | PyTorch | | 14.85 | 11.74 | | 70.42 | 41.59 | | 223.23 | 123.06 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **B200** | PyTorch | 14.28 | 5.60 | 4.07 | 111.84 | 35.64 | 21.48 | 406.66 | 117.16 | 65.30 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 8 | 66.10 | 118.88 | 428.64 | +| | vLLM-Omni | — | | | | +| | Diffusers | — | | | | +| **H20** | PyTorch | 8 | 28.00 | 152.78 | 492.63 | +| | vLLM-Omni | — | | | | +| | Diffusers | — | | | | +| **H100 NVL** | PyTorch | 8 | 16.79 | 64.69 | 183.62 | +| | vLLM-Omni | — | | | | +| | Diffusers | — | | | | +| **H200 NVL** | PyTorch | 8 | 24.73 | 47.67 | 143.06 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 48.42 | 320.70 | 1093.26 | +| **H100 80GB HBM3** | PyTorch | — | | | | +| | vLLM-Omni | — | | | | +| | Diffusers | — | | | | +| **H200 141GB HBM3** | PyTorch | 8 | 11.74 | 41.59 | 123.06 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 43.23 | 270.19 | 909.99 | +| **B200** | PyTorch | 1 | 14.28 | 111.84 | 406.66 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 30.26 | 141.75 | 446.20 | +| **B300** | PyTorch | — | | | | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 30.36 | 138.17 | 420.63 | ### Image-to-Audio-and-Video (i2av) A text prompt and source image produce video with synchronized sound. -| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | -|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| **RTX PRO 6000 Blackwell** | PyTorch | | 65.54 | 66.27 | | 202.65 | 119.05 | | 790.09 | 429.26 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H20** | PyTorch | | 41.33 | 28.11 | | 277.13 | 153.33 | | 931.18 | 493.51 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H100 NVL** | PyTorch | | 20.83 | 16.85 | | 99.62 | 64.19 | | 329.80 | 186.66 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 NVL** | PyTorch | | 24.81 | 24.69 | | 82.65 | 47.65 | | 269.74 | 142.18 | -| | vLLM-Omni | 29.42 | 9.94 | 5.84 | 262.20 | 72.67 | 38.57 | 922.18 | 252.67 | 131.32 | -| | Diffusers | | | | | | | | | | -| **H100 80GB HBM3** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 141GB HBM3** | PyTorch | | 15.09 | 11.82 | | 69.80 | 41.95 | | 224.65 | 123.42 | -| | vLLM-Omni | 27.58 | 9.29 | 6.59 | 220.57 | 65.65 | 36.71 | 774.15 | 217.68 | 7.00 | -| | Diffusers | | | | | | | | | | -| **B200** | PyTorch | 14.84 | 5.64 | 4.12 | 113.32 | 35.84 | 21.70 | 407.30 | 117.95 | 65.20 | -| | vLLM-Omni | 15.15 | 5.88 | 5.38 | 116.54 | 37.32 | 23.07 | 396.11 | 117.51 | 65.98 | -| | Diffusers | | | | | | | | | | -| **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | 16.46 | 7.22 | 7.38 | 114.02 | 38.10 | 24.69 | 373.17 | 113.46 | 64.64 | -| | Diffusers | | | | | | | | | | +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 8 | 66.27 | 119.05 | 429.26 | +| | vLLM-Omni | — | | | | +| | Diffusers | — | | | | +| **H20** | PyTorch | 8 | 28.11 | 153.33 | 493.51 | +| | vLLM-Omni | — | | | | +| | Diffusers | — | | | | +| **H100 NVL** | PyTorch | 8 | 16.85 | 64.19 | 186.66 | +| | vLLM-Omni | — | | | | +| | Diffusers | — | | | | +| **H200 NVL** | PyTorch | 8 | 24.69 | 47.65 | 142.18 | +| | vLLM-Omni | 1 | 29.42 | 262.20 | 922.18 | +| | Diffusers | 1 | 35.00 | 292.20 | 1037.00 | +| **H100 80GB HBM3** | PyTorch | — | | | | +| | vLLM-Omni | — | | | | +| | Diffusers | — | | | | +| **H200 141GB HBM3** | PyTorch | 8 | 11.82 | 41.95 | 123.42 | +| | vLLM-Omni | 1 | 27.58 | 220.57 | 774.15 | +| | Diffusers | 1 | 33.00 | 256.00 | 880.40 | +| **B200** | PyTorch | 1 | 14.84 | 113.32 | 407.30 | +| | vLLM-Omni | 1 | 15.15 | 116.54 | 396.11 | +| | Diffusers | 1 | 20.00 | 128.00 | 423.00 | +| **B300** | PyTorch | — | | | | +| | vLLM-Omni | 1 | 16.46 | 114.02 | 373.17 | +| | Diffusers | 1 | 29.16 | 134.46 | 414.22 | ## Action generation @@ -296,244 +300,244 @@ Forward dynamics is reported separately for AV, camera, and robot inputs. Invers ### Forward Dynamics — Autonomous Vehicle (AV) -| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | -|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| **RTX PRO 6000 Blackwell** | PyTorch | | 79.50 | 59.39 | | 61.21 | 59.31 | | 61.15 | 59.35 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H20** | PyTorch | | 67.73 | 41.51 | | 67.83 | 41.47 | | 67.85 | 41.42 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H100 NVL** | PyTorch | | 28.53 | 21.00 | | 28.69 | 21.00 | | 28.56 | 21.04 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 NVL** | PyTorch | | 24.40 | 22.66 | | 24.38 | 22.67 | | 24.46 | 22.65 | -| | vLLM-Omni | 4.33 | 2.16 | | 28.78 | 9.47 | | 82.08 | 24.83 | | -| | Diffusers | | | | | | | | | | -| **H100 80GB HBM3** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 141GB HBM3** | PyTorch | | 19.88 | 14.41 | | 19.87 | 14.41 | | 19.88 | 14.41 | -| | vLLM-Omni | 3.99 | 2.27 | | 26.04 | 8.83 | | 71.61 | 23.02 | | -| | Diffusers | | | | | | | | | | -| **B200** | PyTorch | 13.59 | 9.29 | 6.33 | 13.55 | 9.29 | 6.37 | 13.86 | 9.32 | 6.34 | -| | vLLM-Omni | 2.37 | 1.93 | | 14.59 | 5.75 | | 38.41 | 13.46 | | -| | Diffusers | | | | | | | | | | -| **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | 2.85 | 3.20 | | 14.20 | 6.07 | | 36.96 | 13.31 | | -| | Diffusers | | | | | | | | | | +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 8 | 59.39 | 59.31 | 59.35 | +| | vLLM-Omni | — | | | | +| | Diffusers | 8 | 11.99 | 22.93 | 23.68 | +| **H20** | PyTorch | 8 | 41.51 | 41.47 | 41.42 | +| | vLLM-Omni | — | | | | +| | Diffusers | 8 | 11.40 | 26.98 | 28.79 | +| **H100 NVL** | PyTorch | 8 | 21.00 | 21.00 | 21.04 | +| | vLLM-Omni | — | | | | +| | Diffusers | 8 | 10.30 | 19.29 | 20.05 | +| **H200 NVL** | PyTorch | 8 | 22.66 | 22.67 | 22.65 | +| | vLLM-Omni | 1 | 4.33 | 28.78 | 82.08 | +| | Diffusers | 1 | 27.17 | 53.19 | 51.89 | +| **H100 80GB HBM3** | PyTorch | — | | | | +| | vLLM-Omni | — | | | | +| | Diffusers | 8 | 10.34 | 12.89 | 13.45 | +| **H200 141GB HBM3** | PyTorch | 8 | 14.41 | 14.41 | 14.41 | +| | vLLM-Omni | 1 | 3.99 | 26.04 | 71.61 | +| | Diffusers | 1 | 26.51 | 47.34 | 50.20 | +| **B200** | PyTorch | 1 | 13.59 | 13.55 | 13.86 | +| | vLLM-Omni | 1 | 2.37 | 14.59 | 38.41 | +| | Diffusers | 1 | 4.00 | 16.00 | 16.00 | +| **B300** | PyTorch | — | | | | +| | vLLM-Omni | 1 | 2.85 | 14.20 | 36.96 | +| | Diffusers | 1 | 22.18 | 35.41 | 38.82 | ### Forward Dynamics — Camera -| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | -|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| **RTX PRO 6000 Blackwell** | PyTorch | | 60.75 | 59.62 | | 60.69 | 59.39 | | 60.61 | 59.55 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H20** | PyTorch | | 67.86 | 41.13 | | 67.89 | 41.10 | | 67.88 | 41.11 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H100 NVL** | PyTorch | | 28.73 | 21.05 | | 28.76 | 20.98 | | 28.65 | 21.00 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 NVL** | PyTorch | | 24.33 | 22.55 | | 24.35 | 22.54 | | 24.37 | 22.54 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H100 80GB HBM3** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 141GB HBM3** | PyTorch | | 19.85 | 14.30 | | 19.92 | 14.30 | | 20.29 | 14.29 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **B200** | PyTorch | 13.73 | 9.32 | 6.27 | 13.72 | 9.27 | 6.27 | 13.58 | 9.28 | 6.27 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 8 | 59.62 | 59.39 | 59.55 | +| | vLLM-Omni | — | | | | +| | Diffusers | — | | | | +| **H20** | PyTorch | 8 | 41.13 | 41.10 | 41.11 | +| | vLLM-Omni | — | | | | +| | Diffusers | — | | | | +| **H100 NVL** | PyTorch | 8 | 21.05 | 20.98 | 21.00 | +| | vLLM-Omni | — | | | | +| | Diffusers | — | | | | +| **H200 NVL** | PyTorch | 8 | 22.55 | 22.54 | 22.54 | +| | vLLM-Omni | — | | | | +| | Diffusers | — | | | | +| **H100 80GB HBM3** | PyTorch | — | | | | +| | vLLM-Omni | — | | | | +| | Diffusers | — | | | | +| **H200 141GB HBM3** | PyTorch | 8 | 14.30 | 14.30 | 14.29 | +| | vLLM-Omni | — | | | | +| | Diffusers | — | | | | +| **B200** | PyTorch | 1 | 13.73 | 13.72 | 13.58 | +| | vLLM-Omni | — | | | | +| | Diffusers | — | | | | +| **B300** | PyTorch | — | | | | +| | vLLM-Omni | — | | | | +| | Diffusers | — | | | | ### Forward Dynamics — Robot -| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | -|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| **RTX PRO 6000 Blackwell** | PyTorch | | 75.52 | 56.35 | | 55.04 | 56.38 | | 54.88 | 56.01 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H20** | PyTorch | | 20.93 | 15.60 | | 20.91 | 15.61 | | 20.78 | 15.59 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H100 NVL** | PyTorch | | 13.61 | 12.90 | | 13.68 | 12.90 | | 13.57 | 12.90 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 NVL** | PyTorch | | 20.25 | 20.16 | | 20.25 | 20.15 | | 20.25 | 20.15 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H100 80GB HBM3** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 141GB HBM3** | PyTorch | | 9.34 | 8.74 | | 9.34 | 8.74 | | 9.33 | 8.70 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **B200** | PyTorch | 3.50 | 2.91 | 2.98 | 3.49 | 3.06 | 3.00 | 3.47 | 2.96 | 3.00 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 8 | 56.35 | 56.38 | 56.01 | +| | vLLM-Omni | — | | | | +| | Diffusers | 8 | 9.40 | 10.60 | 10.82 | +| **H20** | PyTorch | 8 | 15.60 | 15.61 | 15.59 | +| | vLLM-Omni | — | | | | +| | Diffusers | 8 | 8.95 | 10.24 | 10.65 | +| **H100 NVL** | PyTorch | 8 | 12.90 | 12.90 | 12.90 | +| | vLLM-Omni | — | | | | +| | Diffusers | 8 | 8.03 | 8.49 | 8.61 | +| **H200 NVL** | PyTorch | 8 | 20.16 | 20.15 | 20.15 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 22.21 | 28.53 | 27.35 | +| **H100 80GB HBM3** | PyTorch | — | | | | +| | vLLM-Omni | — | | | | +| | Diffusers | 8 | 10.27 | 9.04 | 9.25 | +| **H200 141GB HBM3** | PyTorch | 8 | 8.74 | 8.74 | 8.70 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 22.02 | 26.45 | 26.38 | +| **B200** | PyTorch | 1 | 3.50 | 3.49 | 3.47 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 3.00 | 5.00 | 5.00 | +| **B300** | PyTorch | — | | | | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 22.21 | 21.81 | 24.51 | ### Inverse Dynamics — Autonomous Vehicle (AV) -| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | -|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| **RTX PRO 6000 Blackwell** | PyTorch | | 60.92 | 59.20 | | 60.87 | 59.14 | | 60.95 | 59.07 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H20** | PyTorch | | 67.37 | 41.17 | | 67.09 | 40.96 | | 67.37 | 41.06 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H100 NVL** | PyTorch | | 28.42 | 20.76 | | 28.39 | 21.05 | | 28.41 | 20.69 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 NVL** | PyTorch | | 24.16 | 22.40 | | 24.17 | 22.40 | | 24.18 | 22.42 | -| | vLLM-Omni | 4.36 | 2.17 | | 28.21 | 9.43 | | 80.85 | 25.46 | | -| | Diffusers | | | | | | | | | | -| **H100 80GB HBM3** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 141GB HBM3** | PyTorch | | 20.05 | 14.30 | | 19.68 | 14.32 | | 19.91 | 14.16 | -| | vLLM-Omni | 4.03 | 2.29 | | 25.99 | 8.79 | | 71.45 | | | -| | Diffusers | | | | | | | | | | -| **B200** | PyTorch | 13.38 | 9.08 | 6.11 | 13.37 | 9.13 | 6.15 | 13.36 | 9.08 | 6.11 | -| | vLLM-Omni | 2.42 | 2.29 | | 14.69 | 5.85 | | 38.74 | 13.54 | | -| | Diffusers | | | | | | | | | | -| **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | 2.69 | 3.54 | | 14.44 | 6.52 | | 37.70 | 13.88 | | -| | Diffusers | | | | | | | | | | +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 8 | 59.20 | 59.14 | 59.07 | +| | vLLM-Omni | — | | | | +| | Diffusers | 8 | 11.15 | 23.33 | 23.28 | +| **H20** | PyTorch | 8 | 41.17 | 40.96 | 41.06 | +| | vLLM-Omni | — | | | | +| | Diffusers | 8 | 11.33 | 26.66 | 28.46 | +| **H100 NVL** | PyTorch | 8 | 20.76 | 21.05 | 20.69 | +| | vLLM-Omni | — | | | | +| | Diffusers | 8 | 9.72 | 18.67 | 20.67 | +| **H200 NVL** | PyTorch | 8 | 22.40 | 22.40 | 22.42 | +| | vLLM-Omni | 1 | 4.36 | 28.21 | 80.85 | +| | Diffusers | 1 | 15.22 | 44.50 | 43.51 | +| **H100 80GB HBM3** | PyTorch | — | | | | +| | vLLM-Omni | — | | | | +| | Diffusers | 8 | 10.49 | 12.73 | 13.30 | +| **H200 141GB HBM3** | PyTorch | 8 | 14.30 | 14.32 | 14.16 | +| | vLLM-Omni | 1 | 4.03 | 25.99 | 71.45 | +| | Diffusers | 1 | 16.58 | 39.75 | 41.71 | +| **B200** | PyTorch | 1 | 13.38 | 13.37 | 13.36 | +| | vLLM-Omni | 1 | 2.42 | 14.69 | 38.74 | +| | Diffusers | 1 | 4.00 | 16.00 | 16.00 | +| **B300** | PyTorch | — | | | | +| | vLLM-Omni | 1 | 2.69 | 14.44 | 37.70 | +| | Diffusers | 1 | 15.13 | 26.24 | 28.51 | ### Inverse Dynamics — Robot -| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | -|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| **RTX PRO 6000 Blackwell** | PyTorch | | 55.15 | 56.28 | | 55.12 | 56.33 | | 54.83 | 56.00 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H20** | PyTorch | | 20.71 | 15.51 | | 20.83 | 15.52 | | 20.84 | 15.51 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H100 NVL** | PyTorch | | 13.51 | 12.51 | | 13.54 | 12.51 | | 13.50 | 12.51 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 NVL** | PyTorch | | 20.21 | 20.11 | | 20.18 | 20.10 | | 20.19 | 20.12 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H100 80GB HBM3** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 141GB HBM3** | PyTorch | | 9.30 | 8.68 | | 9.28 | 8.68 | | 9.29 | 8.65 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **B200** | PyTorch | 3.42 | 2.89 | 2.96 | 3.49 | 3.02 | 2.92 | 3.41 | 2.90 | 3.03 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 8 | 56.28 | 56.33 | 56.00 | +| | vLLM-Omni | — | | | | +| | Diffusers | 8 | 9.59 | 10.90 | 8.71 | +| **H20** | PyTorch | 8 | 15.51 | 15.52 | 15.51 | +| | vLLM-Omni | — | | | | +| | Diffusers | 8 | 9.48 | 10.17 | 10.65 | +| **H100 NVL** | PyTorch | 8 | 12.51 | 12.51 | 12.51 | +| | vLLM-Omni | — | | | | +| | Diffusers | 8 | 7.92 | 8.46 | 8.79 | +| **H200 NVL** | PyTorch | 8 | 20.11 | 20.10 | 20.12 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 12.09 | 16.87 | 18.74 | +| **H100 80GB HBM3** | PyTorch | — | | | | +| | vLLM-Omni | — | | | | +| | Diffusers | 8 | 8.71 | 8.93 | 9.08 | +| **H200 141GB HBM3** | PyTorch | 8 | 8.68 | 8.68 | 8.65 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 25.77 | 16.25 | 16.29 | +| **B200** | PyTorch | 1 | 3.42 | 3.49 | 3.41 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 3.00 | 5.00 | 5.00 | +| **B300** | PyTorch | — | | | | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 9.65 | 15.08 | 15.21 | ### Policy — Autonomous Vehicle (AV) -| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | -|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| **RTX PRO 6000 Blackwell** | PyTorch | | 61.03 | 59.66 | | 61.19 | 59.51 | | 61.04 | 59.49 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H20** | PyTorch | | 67.86 | 41.54 | | 67.56 | 41.51 | | 67.62 | 41.49 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H100 NVL** | PyTorch | | 28.61 | 21.02 | | 28.77 | 21.00 | | 28.54 | 21.31 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 NVL** | PyTorch | | 24.40 | 22.64 | | 24.46 | 22.63 | | 24.47 | 22.64 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H100 80GB HBM3** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 141GB HBM3** | PyTorch | | 20.31 | 14.54 | | 19.87 | 14.41 | | 20.14 | 14.52 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **B200** | PyTorch | 13.71 | 9.33 | 6.36 | 13.73 | 9.31 | 6.34 | 13.60 | 9.32 | 6.34 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 8 | 59.66 | 59.51 | 59.49 | +| | vLLM-Omni | — | | | | +| | Diffusers | 8 | 12.11 | 22.60 | 25.76 | +| **H20** | PyTorch | 8 | 41.54 | 41.51 | 41.49 | +| | vLLM-Omni | — | | | | +| | Diffusers | 8 | 11.35 | 26.61 | 28.81 | +| **H100 NVL** | PyTorch | 8 | 21.02 | 21.00 | 21.31 | +| | vLLM-Omni | — | | | | +| | Diffusers | 8 | 9.77 | 18.80 | 20.12 | +| **H200 NVL** | PyTorch | 8 | 22.64 | 22.63 | 22.64 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 15.35 | 41.89 | 45.40 | +| **H100 80GB HBM3** | PyTorch | — | | | | +| | vLLM-Omni | — | | | | +| | Diffusers | 8 | 10.55 | 13.02 | 13.52 | +| **H200 141GB HBM3** | PyTorch | 8 | 14.54 | 14.41 | 14.52 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 14.71 | 41.30 | 40.35 | +| **B200** | PyTorch | 1 | 13.71 | 13.73 | 13.60 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 4.00 | 16.00 | 16.00 | +| **B300** | PyTorch | — | | | | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 15.29 | 28.62 | 28.87 | ### Policy — Robot -| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | -|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| **RTX PRO 6000 Blackwell** | PyTorch | | 55.13 | 56.31 | | 55.47 | 56.29 | | 54.92 | 55.95 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H20** | PyTorch | | 20.90 | 15.60 | | 20.88 | 15.60 | | 20.92 | 15.62 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H100 NVL** | PyTorch | | 13.57 | 12.56 | | 13.58 | 12.55 | | 13.60 | 12.54 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 NVL** | PyTorch | | 20.25 | 20.16 | | 20.26 | 20.17 | | 20.26 | 20.16 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H100 80GB HBM3** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 141GB HBM3** | PyTorch | | 9.33 | 8.72 | | 9.35 | 8.75 | | 9.34 | 8.70 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **B200** | PyTorch | 3.48 | 2.95 | 3.07 | 3.49 | 3.02 | 2.98 | 3.48 | 2.92 | 3.05 | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 8 | 56.31 | 56.29 | 55.95 | +| | vLLM-Omni | — | | | | +| | Diffusers | 8 | 9.57 | 10.08 | 8.76 | +| **H20** | PyTorch | 8 | 15.60 | 15.60 | 15.62 | +| | vLLM-Omni | — | | | | +| | Diffusers | 8 | 9.03 | 10.19 | 10.73 | +| **H100 NVL** | PyTorch | 8 | 12.56 | 12.55 | 12.54 | +| | vLLM-Omni | — | | | | +| | Diffusers | 8 | 8.05 | 8.60 | 8.89 | +| **H200 NVL** | PyTorch | 8 | 20.16 | 20.17 | 20.16 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 13.82 | 16.92 | 17.03 | +| **H100 80GB HBM3** | PyTorch | — | | | | +| | vLLM-Omni | — | | | | +| | Diffusers | 8 | 8.71 | 9.09 | 9.12 | +| **H200 141GB HBM3** | PyTorch | 8 | 8.72 | 8.75 | 8.70 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 11.90 | 16.13 | 14.72 | +| **B200** | PyTorch | 1 | 3.48 | 3.49 | 3.48 | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 3.00 | 5.00 | 5.00 | +| **B300** | PyTorch | — | | | | +| | vLLM-Omni | — | | | | +| | Diffusers | 1 | 10.57 | 15.31 | 15.33 | ### Policy — DROID PBR `#308481` measured Policy-DROID only for Cosmos3-Nano-Policy-DROID. Super has no Policy-DROID row in that sweep; this table is reserved. -| GPU | Engine | 256p/1 | 256p/4 | 256p/8 | 480p/1 | 480p/4 | 480p/8 | 720p/1 | 720p/4 | 720p/8 | -|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| **RTX PRO 6000 Blackwell** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H20** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H100 NVL** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 NVL** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H100 80GB HBM3** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **H200 141GB HBM3** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **B200** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | -| **B300** | PyTorch | | | | | | | | | | -| | vLLM-Omni | | | | | | | | | | -| | Diffusers | | | | | | | | | | +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | — | | | | +| | vLLM-Omni | — | | | | +| | Diffusers | — | | | | +| **H20** | PyTorch | — | | | | +| | vLLM-Omni | — | | | | +| | Diffusers | — | | | | +| **H100 NVL** | PyTorch | — | | | | +| | vLLM-Omni | — | | | | +| | Diffusers | — | | | | +| **H200 NVL** | PyTorch | — | | | | +| | vLLM-Omni | — | | | | +| | Diffusers | — | | | | +| **H100 80GB HBM3** | PyTorch | — | | | | +| | vLLM-Omni | — | | | | +| | Diffusers | — | | | | +| **H200 141GB HBM3** | PyTorch | — | | | | +| | vLLM-Omni | — | | | | +| | Diffusers | — | | | | +| **B200** | PyTorch | — | | | | +| | vLLM-Omni | — | | | | +| | Diffusers | — | | | | +| **B300** | PyTorch | — | | | | +| | vLLM-Omni | — | | | | +| | Diffusers | — | | | | Additional-modality notes: 1. PyTorch values are average generation (sampling) latency in seconds from PBR `#308197`; lower is better. 2. PyTorch values use `CUDA_GRAPH=No` and the `latency` automatic-sharding preset. 3. vLLM-Omni audiovisual values for `t2av`/`t2vs` and `i2av`/`i2vs` are from PBR `#308195`. vLLM-Omni action values are from PBR `#308481`. -4. Action vLLM-Omni `/8` cells are empty because 8-GPU Ulysses runs failed sequence-length divisibility checks. Two documented action misses are also left blank: Super inverse-dynamics AV at 720p/4 on H200 141GB HBM3 (OOM), and Nano inverse-dynamics AV at 720p/4 on H100 NVL (CUDA/NCCL failure). +4. Action vLLM-Omni rows report one GPU because the 8-GPU Ulysses runs failed sequence-length divisibility checks. 5. Policy-DROID is reported in its own table. PBR `#308481` has no Super Policy-DROID measurement. 6. Diffusers rows are intentionally empty reservations for future benchmark campaigns. -7. The `/1`, `/4`, and `/8` suffixes denote the number of GPUs used by the benchmark run. -8. Empty cells indicate unmeasured combinations, not unsupported combinations. +7. The **GPUs** column gives the number of GPUs behind every value in that row. +8. Empty cells indicate unmeasured combinations, not unsupported combinations. \ No newline at end of file From 70956afe15ed572afd3d53a6b9bc47986aef603d Mon Sep 17 00:00:00 2001 From: Sergiy Fefilatyev Date: Fri, 2 Oct 2026 13:09:52 -0700 Subject: [PATCH 5/9] Add Transfer generation benchmarks for Nano and Super Transfer conditions a video-to-video generation on a structural control hint extracted from the source video. Latencies from PBR #308587 are reported per control (blur, depth, edge, seg, wsm) so the cost of each hint is visible, since the hint determines how much of the frame the model must synthesise. Only Diffusers has transfer coverage so far, all on a single GPU. --- inference_benchmarks.md | 10 ++-- .../cosmos3-nano-generator.md | 59 ++++++++++++++++++- .../cosmos3-super-generator.md | 59 ++++++++++++++++++- 3 files changed, 121 insertions(+), 7 deletions(-) diff --git a/inference_benchmarks.md b/inference_benchmarks.md index 3496c9ba..12961034 100644 --- a/inference_benchmarks.md +++ b/inference_benchmarks.md @@ -8,11 +8,11 @@ Results are published incrementally from internal benchmark runs. **Empty cells Generator results are organized by model so future checkpoints can be added without expanding this page into another wall of tables. Use the section links for modality-oriented navigation. -| Model | Primary vision generation | Additional audiovisual generation | Action generation | -|---|---|---|---| -| Cosmos3-Edge | [i2v](inference_benchmarks/cosmos3-edge-generator.md) | Not reported | Not reported | -| Cosmos3-Nano | [t2v, i2v, t2i](inference_benchmarks/cosmos3-nano-generator.md#primary-vision-generation) | [v2v, t2av, v2av, i2av](inference_benchmarks/cosmos3-nano-generator.md#additional-audiovisual-generation) | [Forward dynamics, inverse dynamics, policy](inference_benchmarks/cosmos3-nano-generator.md#action-generation) | -| Cosmos3-Super | [t2v, i2v, t2i](inference_benchmarks/cosmos3-super-generator.md#primary-vision-generation) | [v2v, t2av, v2av, i2av](inference_benchmarks/cosmos3-super-generator.md#additional-audiovisual-generation) | [Forward dynamics, inverse dynamics, policy](inference_benchmarks/cosmos3-super-generator.md#action-generation) | +| Model | Primary vision generation | Additional audiovisual generation | Transfer generation | Action generation | +|---|---|---|---|---| +| Cosmos3-Edge | [i2v](inference_benchmarks/cosmos3-edge-generator.md) | Not reported | Not reported | Not reported | +| Cosmos3-Nano | [t2v, i2v, t2i](inference_benchmarks/cosmos3-nano-generator.md#primary-vision-generation) | [v2v, t2av, v2av, i2av](inference_benchmarks/cosmos3-nano-generator.md#additional-audiovisual-generation) | [blur, depth, edge, seg, wsm](inference_benchmarks/cosmos3-nano-generator.md#transfer-generation) | [Forward dynamics, inverse dynamics, policy](inference_benchmarks/cosmos3-nano-generator.md#action-generation) | +| Cosmos3-Super | [t2v, i2v, t2i](inference_benchmarks/cosmos3-super-generator.md#primary-vision-generation) | [v2v, t2av, v2av, i2av](inference_benchmarks/cosmos3-super-generator.md#additional-audiovisual-generation) | [blur, depth, edge, seg, wsm](inference_benchmarks/cosmos3-super-generator.md#transfer-generation) | [Forward dynamics, inverse dynamics, policy](inference_benchmarks/cosmos3-super-generator.md#action-generation) | ## Reasoner benchmarks diff --git a/inference_benchmarks/cosmos3-nano-generator.md b/inference_benchmarks/cosmos3-nano-generator.md index baaafd1b..97d0259a 100644 --- a/inference_benchmarks/cosmos3-nano-generator.md +++ b/inference_benchmarks/cosmos3-nano-generator.md @@ -21,6 +21,7 @@ Empty cells mean that a run has not been completed for that GPU, engine, or reso - [Text-to-Audio-and-Video (t2av)](#text-to-audio-and-video-t2av) - [Video-to-Audio-and-Video (v2av)](#video-to-audio-and-video-v2av) - [Image-to-Audio-and-Video (i2av)](#image-to-audio-and-video-i2av) +- [Transfer generation](#transfer-generation) - [Action generation](#action-generation) - [Forward Dynamics — AV](#forward-dynamics--autonomous-vehicle-av) - [Forward Dynamics — Camera](#forward-dynamics--camera) @@ -35,7 +36,7 @@ Empty cells mean that a run has not been completed for that GPU, engine, or reso The primary t2v, i2v, and t2i tables preserve the previously published benchmark campaigns across PyTorch, vLLM-Omni, Diffusers, and NIM. Those tables use BF16 precision, batch size 1, and matched prompts, seeds, and sampler settings where documented. Video workloads follow the standard Cosmos3 generation profile of 189 frames at 24 FPS unless a resolution tier limits frame count. -The additional audiovisual and action tables come from three internal benchmark reports. PyTorch values are from PBR `#308197`, **Cosmos3-Generator OSS Inference Benchmarking 32B and 8B (189 frames)**: average generation (sampling) latency from the native OSS path, using **CUDA Graphs disabled** and the **latency** automatic-sharding preset. vLLM-Omni values for text-to-audio-and-video (`t2av`/`t2vs`) and image-to-audio-and-video (`i2av`/`i2vs`) are from PBR `#308195`. vLLM-Omni action values are from PBR `#308481`, **Cosmos3-Generator vLLM-Omni Inference Benchmarking (action)**, measured with the `vllm/vllm-omni:cosmos3` image and the official action cookbook samples; `DIFFUSION_ATTENTION_BACKEND=TORCH_SDPA` was not set. Forward-dynamics cells are the mean of `av_forward`, `av_left`, and `av_right`; inverse-dynamics cells are the mean of `av_inverse_0` and `av_inverse_1`. That action sweep covers 1, 2, and 4 GPUs only, because its 8-GPU Ulysses runs failed a sequence-length divisibility check; those rows therefore report the single-GPU configuration. Policy-DROID is a separate checkpoint and was measured only at 480p on one GPU. Super was not measured on H20, H100 NVL, or H100 80GB HBM3. Diffusers values come from three further reports: PBR `#308202` (all modalities) and PBR `#308451` (video-to-video) on one GPU and PBR `#308918` for 4- and 8-GPU runs. Where PBR `#308918` offers several tensor- and context-parallel splits at the same GPU count, the fastest is published. Its 1-GPU numbers are not used, because that sweep ran a different denoising-step budget than the single-GPU reports. Values are rounded to two decimal places. +The additional audiovisual and action tables come from three internal benchmark reports. PyTorch values are from PBR `#308197`, **Cosmos3-Generator OSS Inference Benchmarking 32B and 8B (189 frames)**: average generation (sampling) latency from the native OSS path, using **CUDA Graphs disabled** and the **latency** automatic-sharding preset. vLLM-Omni values for text-to-audio-and-video (`t2av`/`t2vs`) and image-to-audio-and-video (`i2av`/`i2vs`) are from PBR `#308195`. vLLM-Omni action values are from PBR `#308481`, **Cosmos3-Generator vLLM-Omni Inference Benchmarking (action)**, measured with the `vllm/vllm-omni:cosmos3` image and the official action cookbook samples; `DIFFUSION_ATTENTION_BACKEND=TORCH_SDPA` was not set. Forward-dynamics cells are the mean of `av_forward`, `av_left`, and `av_right`; inverse-dynamics cells are the mean of `av_inverse_0` and `av_inverse_1`. That action sweep covers 1, 2, and 4 GPUs only, because its 8-GPU Ulysses runs failed a sequence-length divisibility check; those rows therefore report the single-GPU configuration. Policy-DROID is a separate checkpoint and was measured only at 480p on one GPU. Super was not measured on H20, H100 NVL, or H100 80GB HBM3. Diffusers values come from four further reports: PBR `#308202` (all modalities) and PBR `#308451` (video-to-video) on one GPU, PBR `#308918` for 4- and 8-GPU runs, and PBR `#308587` for transfer. Where PBR `#308918` offers several tensor- and context-parallel splits at the same GPU count, the fastest is published. Its 1-GPU numbers are not used, because that sweep ran a different denoising-step budget than the single-GPU reports. Values are rounded to two decimal places. These reports establish the reported timing matrix but do not expose every prompt and action payload in this repository. The linked public recipes explain modality behavior and provide representative payloads; their example-specific frame counts and action chunk sizes should not be treated as the exact internal benchmark inputs. @@ -47,6 +48,7 @@ These reports establish the reported timing matrix but do not expose every promp | Text-to-audio-and-video (`t2av`) | Text prompt | Synchronized video and sound | | Video-to-audio-and-video (`v2av`) | Text prompt and source video | Generated video with synchronized sound | | Image-to-audio-and-video (`i2av`) | Text prompt and source image | Generated video with synchronized sound | +| Transfer video-to-video (`transfer`) | Text prompt, source video, and one control hint | Generated video steered by that hint | | Forward dynamics | Initial visual observation and an action trajectory | Future-observation rollout video | | Inverse dynamics | Observed video | Recovered action trajectory; some serving integrations also return video | | Policy | Initial visual observation, instruction, and optional state | Predicted action trajectory and, for general Generator paths, a rollout video | @@ -294,6 +296,61 @@ A text prompt and source image produce video with synchronized sound. | | vLLM-Omni | 1 | 6.01 | 34.39 | 108.59 | | | Diffusers | 1 | 15.35 | 50.19 | 131.92 | +## Transfer generation + +Transfer conditions a video-to-video generation on a structural control hint +extracted from the source video. Each control is reported separately because +the hint changes how much of the frame the model must synthesise. + +| GPU | Transfer control | Engine | GPUs | 256p | 480p | 720p | +|---|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | blur | Diffusers | 1 | 30.95 | 217.40 | 735.43 | +| | depth | Diffusers | 1 | 36.52 | 225.34 | 745.70 | +| | edge | Diffusers | 1 | 31.25 | 217.90 | 735.35 | +| | seg | Diffusers | 1 | 32.89 | 220.10 | 738.51 | +| | wsm | Diffusers | 1 | 29.84 | 176.06 | 565.32 | +| **H20** | blur | Diffusers | 1 | 73.61 | 491.22 | 1726.61 | +| | depth | Diffusers | 1 | 86.84 | 510.15 | 1753.96 | +| | edge | Diffusers | 1 | 74.68 | 492.78 | 1728.26 | +| | seg | Diffusers | 1 | 77.50 | 496.77 | 1734.77 | +| | wsm | Diffusers | 1 | 70.42 | 398.67 | 1319.24 | +| **H100 NVL** | blur | Diffusers | 1 | 26.97 | 224.61 | 775.61 | +| | depth | Diffusers | 1 | 31.63 | 233.03 | 610.83 | +| | edge | Diffusers | 1 | 34.73 | 173.04 | 602.04 | +| | seg | Diffusers | 1 | 28.55 | 174.78 | 777.58 | +| | wsm | Diffusers | 1 | 26.17 | 139.82 | 594.21 | +| **H100 80GB HBM3** | blur | Diffusers | 1 | 21.06 | 131.86 | 454.46 | +| | depth | Diffusers | 1 | 24.61 | 135.88 | 457.43 | +| | edge | Diffusers | 1 | 21.28 | 132.06 | 447.24 | +| | seg | Diffusers | 1 | 22.54 | 133.24 | 456.55 | +| | wsm | Diffusers | 1 | 20.25 | 106.79 | 342.79 | +| **H200 NVL** | blur | Diffusers | 1 | 22.82 | 152.13 | 535.83 | +| | depth | Diffusers | 1 | 26.80 | 156.98 | 541.71 | +| | edge | Diffusers | 1 | 23.06 | 152.54 | 533.87 | +| | seg | Diffusers | 1 | 24.12 | 153.68 | 537.15 | +| | wsm | Diffusers | 1 | 21.90 | 123.08 | 407.57 | +| **H200 141GB HBM3** | blur | Diffusers | 1 | 20.63 | 128.41 | 450.79 | +| | depth | Diffusers | 1 | 24.01 | 134.51 | 457.91 | +| | edge | Diffusers | 1 | 20.87 | 130.99 | 443.92 | +| | seg | Diffusers | 1 | 21.84 | 130.12 | 453.51 | +| | wsm | Diffusers | 1 | 19.65 | 106.00 | 344.09 | +| **B200** | blur | Diffusers | 1 | 12.69 | 70.78 | 226.51 | +| | depth | Diffusers | 1 | 14.79 | 73.23 | 229.54 | +| | edge | Diffusers | 1 | 12.76 | 71.04 | 223.24 | +| | seg | Diffusers | 1 | 13.54 | 71.21 | 223.41 | +| | wsm | Diffusers | 1 | 12.16 | 59.29 | 172.60 | +| **B300** | blur | Diffusers | 1 | 12.39 | 66.13 | 207.59 | +| | depth | Diffusers | 1 | 14.26 | 70.00 | 211.12 | +| | edge | Diffusers | 1 | 12.48 | 67.68 | 207.95 | +| | seg | Diffusers | 1 | 13.43 | 68.85 | 210.33 | +| | wsm | Diffusers | 1 | 11.95 | 55.41 | 162.28 | + +Transfer notes: +1. Values are average generation latency in seconds from PBR `#308587`; lower is better. +2. All transfer runs use a single GPU; multi-GPU transfer has not been measured. +3. Only Diffusers has transfer coverage so far; other runtimes are unmeasured, not unsupported. +4. Control hints follow the vLLM-Omni `extra_params` names: `blur`, `depth`, `edge`, `seg`, and `wsm`. + ## Action generation Forward dynamics is reported separately for AV, camera, and robot inputs. Inverse dynamics and policy are reported for AV and robot because those are the only domain rows in PBR `#308197`; no camera row is inferred. For each domain, PyTorch is populated from that report. vLLM-Omni currently covers forward dynamics AV and inverse dynamics AV from PBR `#308481`. Camera, robot-FD, robot-ID, policy-AV, and policy-robot vLLM-Omni rows remain reserved. **Policy — DROID** is a separate checkpoint (`nvidia/Cosmos3-Nano-Policy-DROID`) and is not mixed with the policy-robot table. diff --git a/inference_benchmarks/cosmos3-super-generator.md b/inference_benchmarks/cosmos3-super-generator.md index bb02f86b..dc347048 100644 --- a/inference_benchmarks/cosmos3-super-generator.md +++ b/inference_benchmarks/cosmos3-super-generator.md @@ -21,6 +21,7 @@ Empty cells mean that a run has not been completed for that GPU, engine, or reso - [Text-to-Audio-and-Video (t2av)](#text-to-audio-and-video-t2av) - [Video-to-Audio-and-Video (v2av)](#video-to-audio-and-video-v2av) - [Image-to-Audio-and-Video (i2av)](#image-to-audio-and-video-i2av) +- [Transfer generation](#transfer-generation) - [Action generation](#action-generation) - [Forward Dynamics — AV](#forward-dynamics--autonomous-vehicle-av) - [Forward Dynamics — Camera](#forward-dynamics--camera) @@ -35,7 +36,7 @@ Empty cells mean that a run has not been completed for that GPU, engine, or reso The primary t2v, i2v, and t2i tables preserve the previously published benchmark campaigns across PyTorch, vLLM-Omni, Diffusers, and NIM. Those tables use BF16 precision, batch size 1, and matched prompts, seeds, and sampler settings where documented. Video workloads follow the standard Cosmos3 generation profile of 189 frames at 24 FPS unless a resolution tier limits frame count. -The additional audiovisual and action tables come from three internal benchmark reports. PyTorch values are from PBR `#308197`, **Cosmos3-Generator OSS Inference Benchmarking 32B and 8B (189 frames)**: average generation (sampling) latency from the native OSS path, using **CUDA Graphs disabled** and the **latency** automatic-sharding preset. vLLM-Omni values for text-to-audio-and-video (`t2av`/`t2vs`) and image-to-audio-and-video (`i2av`/`i2vs`) are from PBR `#308195`. vLLM-Omni action values are from PBR `#308481`, **Cosmos3-Generator vLLM-Omni Inference Benchmarking (action)**, measured with the `vllm/vllm-omni:cosmos3` image and the official action cookbook samples; `DIFFUSION_ATTENTION_BACKEND=TORCH_SDPA` was not set. Forward-dynamics cells are the mean of `av_forward`, `av_left`, and `av_right`; inverse-dynamics cells are the mean of `av_inverse_0` and `av_inverse_1`. That action sweep covers 1, 2, and 4 GPUs only, because its 8-GPU Ulysses runs failed a sequence-length divisibility check; those rows therefore report the single-GPU configuration. Policy-DROID is a separate checkpoint and was measured only at 480p on one GPU. Super was not measured on H20, H100 NVL, or H100 80GB HBM3. Diffusers values come from three further reports: PBR `#308202` (all modalities) and PBR `#308451` (video-to-video) on one GPU and PBR `#308918` for 4- and 8-GPU runs. Where PBR `#308918` offers several tensor- and context-parallel splits at the same GPU count, the fastest is published. Its 1-GPU numbers are not used, because that sweep ran a different denoising-step budget than the single-GPU reports. Values are rounded to two decimal places. +The additional audiovisual and action tables come from three internal benchmark reports. PyTorch values are from PBR `#308197`, **Cosmos3-Generator OSS Inference Benchmarking 32B and 8B (189 frames)**: average generation (sampling) latency from the native OSS path, using **CUDA Graphs disabled** and the **latency** automatic-sharding preset. vLLM-Omni values for text-to-audio-and-video (`t2av`/`t2vs`) and image-to-audio-and-video (`i2av`/`i2vs`) are from PBR `#308195`. vLLM-Omni action values are from PBR `#308481`, **Cosmos3-Generator vLLM-Omni Inference Benchmarking (action)**, measured with the `vllm/vllm-omni:cosmos3` image and the official action cookbook samples; `DIFFUSION_ATTENTION_BACKEND=TORCH_SDPA` was not set. Forward-dynamics cells are the mean of `av_forward`, `av_left`, and `av_right`; inverse-dynamics cells are the mean of `av_inverse_0` and `av_inverse_1`. That action sweep covers 1, 2, and 4 GPUs only, because its 8-GPU Ulysses runs failed a sequence-length divisibility check; those rows therefore report the single-GPU configuration. Policy-DROID is a separate checkpoint and was measured only at 480p on one GPU. Super was not measured on H20, H100 NVL, or H100 80GB HBM3. Diffusers values come from four further reports: PBR `#308202` (all modalities) and PBR `#308451` (video-to-video) on one GPU, PBR `#308918` for 4- and 8-GPU runs, and PBR `#308587` for transfer. Where PBR `#308918` offers several tensor- and context-parallel splits at the same GPU count, the fastest is published. Its 1-GPU numbers are not used, because that sweep ran a different denoising-step budget than the single-GPU reports. Values are rounded to two decimal places. These reports establish the reported timing matrix but do not expose every prompt and action payload in this repository. The linked public recipes explain modality behavior and provide representative payloads; their example-specific frame counts and action chunk sizes should not be treated as the exact internal benchmark inputs. @@ -47,6 +48,7 @@ These reports establish the reported timing matrix but do not expose every promp | Text-to-audio-and-video (`t2av`) | Text prompt | Synchronized video and sound | | Video-to-audio-and-video (`v2av`) | Text prompt and source video | Generated video with synchronized sound | | Image-to-audio-and-video (`i2av`) | Text prompt and source image | Generated video with synchronized sound | +| Transfer video-to-video (`transfer`) | Text prompt, source video, and one control hint | Generated video steered by that hint | | Forward dynamics | Initial visual observation and an action trajectory | Future-observation rollout video | | Inverse dynamics | Observed video | Recovered action trajectory; some serving integrations also return video | | Policy | Initial visual observation, instruction, and optional state | Predicted action trajectory and, for general Generator paths, a rollout video | @@ -294,6 +296,61 @@ A text prompt and source image produce video with synchronized sound. | | vLLM-Omni | 1 | 16.46 | 114.02 | 373.17 | | | Diffusers | 1 | 29.16 | 134.46 | 414.22 | +## Transfer generation + +Transfer conditions a video-to-video generation on a structural control hint +extracted from the source video. Each control is reported separately because +the hint changes how much of the frame the model must synthesise. + +| GPU | Transfer control | Engine | GPUs | 256p | 480p | 720p | +|---|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | blur | Diffusers | — | | | | +| | depth | Diffusers | — | | | | +| | edge | Diffusers | — | | | | +| | seg | Diffusers | — | | | | +| | wsm | Diffusers | — | | | | +| **H20** | blur | Diffusers | — | | | | +| | depth | Diffusers | — | | | | +| | edge | Diffusers | — | | | | +| | seg | Diffusers | — | | | | +| | wsm | Diffusers | — | | | | +| **H100 NVL** | blur | Diffusers | — | | | | +| | depth | Diffusers | — | | | | +| | edge | Diffusers | — | | | | +| | seg | Diffusers | — | | | | +| | wsm | Diffusers | — | | | | +| **H100 80GB HBM3** | blur | Diffusers | — | | | | +| | depth | Diffusers | — | | | | +| | edge | Diffusers | — | | | | +| | seg | Diffusers | — | | | | +| | wsm | Diffusers | — | | | | +| **H200 NVL** | blur | Diffusers | 1 | 83.43 | 551.97 | 1935.10 | +| | depth | Diffusers | 1 | 98.52 | 572.97 | 1970.31 | +| | edge | Diffusers | 1 | 84.72 | 552.34 | 1940.58 | +| | seg | Diffusers | 1 | 88.16 | 555.93 | 1950.38 | +| | wsm | Diffusers | 1 | 80.77 | 445.36 | 1470.27 | +| **H200 141GB HBM3** | blur | Diffusers | 1 | 71.83 | 463.77 | 1593.34 | +| | depth | Diffusers | 1 | 84.88 | 481.84 | 1618.53 | +| | edge | Diffusers | 1 | 73.22 | 466.10 | 1594.90 | +| | seg | Diffusers | 1 | 77.09 | 476.15 | 1598.31 | +| | wsm | Diffusers | 1 | 70.53 | 382.72 | 1216.43 | +| **B200** | blur | Diffusers | 1 | 42.14 | 241.28 | 789.92 | +| | depth | Diffusers | 1 | 49.41 | 254.68 | 792.02 | +| | edge | Diffusers | 1 | 43.57 | 241.17 | 774.34 | +| | seg | Diffusers | 1 | 45.43 | 243.27 | 792.36 | +| | wsm | Diffusers | 1 | 40.66 | 198.30 | 604.37 | +| **B300** | blur | Diffusers | 1 | 39.64 | 226.08 | 720.04 | +| | depth | Diffusers | 1 | 45.83 | 231.26 | 730.46 | +| | edge | Diffusers | 1 | 39.99 | 226.42 | 719.35 | +| | seg | Diffusers | 1 | 41.64 | 228.18 | 720.63 | +| | wsm | Diffusers | 1 | 38.40 | 186.76 | 555.47 | + +Transfer notes: +1. Values are average generation latency in seconds from PBR `#308587`; lower is better. +2. All transfer runs use a single GPU; multi-GPU transfer has not been measured. +3. Only Diffusers has transfer coverage so far; other runtimes are unmeasured, not unsupported. +4. Control hints follow the vLLM-Omni `extra_params` names: `blur`, `depth`, `edge`, `seg`, and `wsm`. + ## Action generation Forward dynamics is reported separately for AV, camera, and robot inputs. Inverse dynamics and policy are reported for AV and robot because those are the only domain rows in PBR `#308197`; no camera row is inferred. For each domain, PyTorch is populated from that report. vLLM-Omni currently covers forward dynamics AV and inverse dynamics AV from PBR `#308481`. Camera, robot-FD, robot-ID, policy-AV, and policy-robot vLLM-Omni rows remain reserved. **Policy — DROID** is a separate checkpoint and is not mixed with the policy-robot table; this Super page has no Policy-DROID measurements. From 11b96b4e18ea06a4735cea1b5ff34f97b7418a7f Mon Sep 17 00:00:00 2001 From: Sergiy Fefilatyev Date: Fri, 2 Oct 2026 13:29:27 -0700 Subject: [PATCH 6/9] Update vLLM-Omni benchmarks and add vLLM-Omni transfer coverage Four reports refresh and extend the vLLM-Omni numbers: PBR #307999 (t2i, t2v, i2v), PBR #308195 (t2vs, i2vs), PBR #308481 (action), and PBR #308574 (transfer). The transfer table now carries both engines rather than Diffusers alone, and gains a multi_control row group for the combined-hint workload that only vLLM-Omni has measured. Super gains vLLM-Omni i2v and t2i on the four GPUs it previously lacked. Eleven published 720p values are corrected to the figures from these reports. Most differ only in rounding; the largest change is nano t2v on H100 NVL, from 311.13 to 281.99. Under vLLM-Omni, Super fits on a single GPU on every GPU measured, which the Super overview now states, since the other engines still need eight on the smaller-memory parts. --- .../cosmos3-nano-generator.md | 84 ++++++++++++++--- .../cosmos3-super-generator.md | 92 +++++++++++++++---- 2 files changed, 144 insertions(+), 32 deletions(-) diff --git a/inference_benchmarks/cosmos3-nano-generator.md b/inference_benchmarks/cosmos3-nano-generator.md index 97d0259a..6181a224 100644 --- a/inference_benchmarks/cosmos3-nano-generator.md +++ b/inference_benchmarks/cosmos3-nano-generator.md @@ -34,9 +34,9 @@ Empty cells mean that a run has not been completed for that GPU, engine, or reso ## Benchmark methodology -The primary t2v, i2v, and t2i tables preserve the previously published benchmark campaigns across PyTorch, vLLM-Omni, Diffusers, and NIM. Those tables use BF16 precision, batch size 1, and matched prompts, seeds, and sampler settings where documented. Video workloads follow the standard Cosmos3 generation profile of 189 frames at 24 FPS unless a resolution tier limits frame count. +The primary t2v, i2v, and t2i tables cover PyTorch, vLLM-Omni, Diffusers, and NIM. Their vLLM-Omni values are from PBR `#307999`, **Cosmos3-Generator vLLM-Omni Inference Benchmarking (t2i, t2v, i2v)**; the PyTorch and NIM values preserve the previously published campaigns. Those tables use BF16 precision, batch size 1, and matched prompts, seeds, and sampler settings where documented. Video workloads follow the standard Cosmos3 generation profile of 189 frames at 24 FPS unless a resolution tier limits frame count. -The additional audiovisual and action tables come from three internal benchmark reports. PyTorch values are from PBR `#308197`, **Cosmos3-Generator OSS Inference Benchmarking 32B and 8B (189 frames)**: average generation (sampling) latency from the native OSS path, using **CUDA Graphs disabled** and the **latency** automatic-sharding preset. vLLM-Omni values for text-to-audio-and-video (`t2av`/`t2vs`) and image-to-audio-and-video (`i2av`/`i2vs`) are from PBR `#308195`. vLLM-Omni action values are from PBR `#308481`, **Cosmos3-Generator vLLM-Omni Inference Benchmarking (action)**, measured with the `vllm/vllm-omni:cosmos3` image and the official action cookbook samples; `DIFFUSION_ATTENTION_BACKEND=TORCH_SDPA` was not set. Forward-dynamics cells are the mean of `av_forward`, `av_left`, and `av_right`; inverse-dynamics cells are the mean of `av_inverse_0` and `av_inverse_1`. That action sweep covers 1, 2, and 4 GPUs only, because its 8-GPU Ulysses runs failed a sequence-length divisibility check; those rows therefore report the single-GPU configuration. Policy-DROID is a separate checkpoint and was measured only at 480p on one GPU. Super was not measured on H20, H100 NVL, or H100 80GB HBM3. Diffusers values come from four further reports: PBR `#308202` (all modalities) and PBR `#308451` (video-to-video) on one GPU, PBR `#308918` for 4- and 8-GPU runs, and PBR `#308587` for transfer. Where PBR `#308918` offers several tensor- and context-parallel splits at the same GPU count, the fastest is published. Its 1-GPU numbers are not used, because that sweep ran a different denoising-step budget than the single-GPU reports. Values are rounded to two decimal places. +The additional audiovisual and action tables come from three internal benchmark reports. PyTorch values are from PBR `#308197`, **Cosmos3-Generator OSS Inference Benchmarking 32B and 8B (189 frames)**: average generation (sampling) latency from the native OSS path, using **CUDA Graphs disabled** and the **latency** automatic-sharding preset. vLLM-Omni values for text-to-audio-and-video (`t2av`/`t2vs`) and image-to-audio-and-video (`i2av`/`i2vs`) are from PBR `#308195`. vLLM-Omni action values are from PBR `#308481`, **Cosmos3-Generator vLLM-Omni Inference Benchmarking (action)**, measured with the `vllm/vllm-omni:cosmos3` image and the official action cookbook samples; `DIFFUSION_ATTENTION_BACKEND=TORCH_SDPA` was not set. Forward-dynamics cells are the mean of `av_forward`, `av_left`, and `av_right`; inverse-dynamics cells are the mean of `av_inverse_0` and `av_inverse_1`. That action sweep covers 1, 2, and 4 GPUs only, because its 8-GPU Ulysses runs failed a sequence-length divisibility check; those rows therefore report the single-GPU configuration. Policy-DROID is a separate checkpoint and was measured only at 480p on one GPU. That action sweep did not cover Super on H20, H100 NVL, or H100 80GB HBM3. Diffusers values come from four further reports: PBR `#308202` (all modalities) and PBR `#308451` (video-to-video) on one GPU, PBR `#308918` for 4- and 8-GPU runs, and PBR `#308587` for transfer. vLLM-Omni transfer values are from PBR `#308574`. Where PBR `#308918` offers several tensor- and context-parallel splits at the same GPU count, the fastest is published. Its 1-GPU numbers are not used, because that sweep ran a different denoising-step budget than the single-GPU reports. Values are rounded to two decimal places. These reports establish the reported timing matrix but do not expose every prompt and action payload in this repository. The linked public recipes explain modality behavior and provide representative payloads; their example-specific frame counts and action chunk sizes should not be treated as the exact internal benchmark inputs. @@ -63,7 +63,7 @@ The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound | GPU | Engine | GPUs | 256p | 480p | 720p | |---|---|:-:|---:|---:|---:| | **RTX PRO 6000 Blackwell** | PyTorch | 1 | 13.95 | 180.81 | 786.37 | -| | vLLM-Omni | 1 | 10.65 | 105.61 | 369.67 | +| | vLLM-Omni | 1 | 10.65 | 105.61 | 371.53 | | | Diffusers | 1 | 11.20 | 112.00 | 392.00 | | | NIM | 1 | 7.93 | 82.69 | 318.69 | | **H20** | PyTorch | 1 | 30.57 | 257.51 | 931.39 | @@ -71,7 +71,7 @@ The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound | | Diffusers | 1 | 30.20 | 258.00 | 926.00 | | | NIM | 1 | 17.81 | 192.07 | 771.37 | | **H100 NVL** | PyTorch | 1 | 10.03 | 84.12 | 297.27 | -| | vLLM-Omni | 1 | 9.25 | 80.75 | 311.13 | +| | vLLM-Omni | 1 | 9.25 | 80.75 | 281.99 | | | Diffusers | 1 | 11.00 | 90.00 | 324.20 | | | NIM | 1 | 7.09 | 68.01 | 267.73 | | **H200 NVL** | PyTorch | 1 | 8.17 | 69.79 | 244.39 | @@ -87,7 +87,7 @@ The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound | | Diffusers | 1 | 9.00 | 67.00 | 239.60 | | | NIM | 1 | 5.72 | 51.90 | 200.63 | | **B200** | PyTorch | 1 | 4.56 | 33.20 | 114.85 | -| | vLLM-Omni | 1 | 4.03 | 32.04 | 107.84 | +| | vLLM-Omni | 1 | 4.03 | 32.04 | 107.79 | | | Diffusers | 1 | 7.00 | 36.80 | 117.00 | | | NIM | 1 | 3.72 | 26.68 | 93.33 | | **B300** | PyTorch | — | | | | @@ -100,7 +100,7 @@ The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound | GPU | Engine | GPUs | 256p | 480p | 720p | |---|---|:-:|---:|---:|---:| | **RTX PRO 6000 Blackwell** | PyTorch | 1 | 14.08 | 182.14 | 788.80 | -| | vLLM-Omni | 1 | 11.04 | 107.77 | 375.01 | +| | vLLM-Omni | 1 | 11.04 | 107.77 | 376.44 | | | Diffusers | 1 | 12.00 | 112.00 | 397.00 | | | NIM | 1 | 8.23 | 84.58 | 326.96 | | **H20** | PyTorch | 1 | 31.36 | 257.10 | 933.07 | @@ -108,7 +108,7 @@ The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound | | Diffusers | 1 | 31.00 | 258.00 | 925.00 | | | NIM | 1 | 18.67 | 195.10 | 774.88 | | **H100 NVL** | PyTorch | 1 | 10.19 | 84.50 | 298.57 | -| | vLLM-Omni | 1 | 9.62 | 82.61 | 286.33 | +| | vLLM-Omni | 1 | 9.62 | 82.61 | 286.19 | | | Diffusers | 1 | 11.00 | 91.00 | 325.20 | | | NIM | 1 | 7.39 | 69.39 | 272.29 | | **H200 NVL** | PyTorch | 1 | 8.27 | 69.99 | 246.62 | @@ -124,7 +124,7 @@ The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound | | Diffusers | 1 | 9.00 | 67.20 | 240.00 | | | NIM | 1 | 6.04 | 53.25 | 203.66 | | **B200** | PyTorch | 1 | 4.60 | 33.08 | 113.90 | -| | vLLM-Omni | 1 | 4.33 | 33.09 | 110.19 | +| | vLLM-Omni | 1 | 4.33 | 33.09 | 110.08 | | | Diffusers | 1 | 7.00 | 37.00 | 116.00 | | | NIM | 1 | 4.05 | 27.72 | 95.57 | | **B300** | PyTorch | — | | | | @@ -143,7 +143,7 @@ The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound | | vLLM-Omni | 1 | 1.73 | 4.92 | 10.73 | | | Diffusers | 1 | 3.00 | 6.00 | 10.00 | | **H100 NVL** | PyTorch | 1 | 2.77 | 2.83 | 4.21 | -| | vLLM-Omni | 1 | 1.55 | 1.92 | 3.44 | +| | vLLM-Omni | 1 | 1.55 | 1.92 | 3.43 | | | Diffusers | 1 | 3.00 | 3.00 | 4.00 | | **H200 NVL** | PyTorch | 1 | 2.75 | 2.85 | 3.58 | | | vLLM-Omni | 1 | 1.53 | 1.58 | 2.81 | @@ -155,7 +155,7 @@ The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound | | vLLM-Omni | 1 | 1.57 | 1.52 | 2.60 | | | Diffusers | 1 | 3.00 | 3.00 | 4.00 | | **B200** | PyTorch | 1 | 2.68 | 2.75 | 2.87 | -| | vLLM-Omni | 1 | 1.49 | 1.20 | 1.77 | +| | vLLM-Omni | 1 | 1.49 | 1.20 | 1.76 | | | Diffusers | 1 | 3.00 | 3.20 | 3.00 | | **B300** | PyTorch | — | | | | | | vLLM-Omni | 1 | 1.97 | 1.81 | 2.34 | @@ -305,51 +305,107 @@ the hint changes how much of the frame the model must synthesise. | GPU | Transfer control | Engine | GPUs | 256p | 480p | 720p | |---|---|---|:-:|---:|---:|---:| | **RTX PRO 6000 Blackwell** | blur | Diffusers | 1 | 30.95 | 217.40 | 735.43 | +| | | vLLM-Omni | 1 | 40.11 | 288.60 | 1061.44 | | | depth | Diffusers | 1 | 36.52 | 225.34 | 745.70 | +| | | vLLM-Omni | 1 | 31.56 | 291.76 | 1090.33 | | | edge | Diffusers | 1 | 31.25 | 217.90 | 735.35 | +| | | vLLM-Omni | 1 | 30.64 | 287.82 | 1080.82 | +| | multi_control | Diffusers | — | | | | +| | | vLLM-Omni | 1 | 73.36 | 1060.11 | 3988.93 | | | seg | Diffusers | 1 | 32.89 | 220.10 | 738.51 | +| | | vLLM-Omni | 1 | 31.50 | 289.01 | 1083.64 | | | wsm | Diffusers | 1 | 29.84 | 176.06 | 565.32 | +| | | vLLM-Omni | 1 | 26.12 | 225.70 | 814.25 | | **H20** | blur | Diffusers | 1 | 73.61 | 491.22 | 1726.61 | +| | | vLLM-Omni | 1 | 85.93 | 625.60 | 2280.10 | | | depth | Diffusers | 1 | 86.84 | 510.15 | 1753.96 | +| | | vLLM-Omni | 1 | 66.69 | 636.40 | 2367.37 | | | edge | Diffusers | 1 | 74.68 | 492.78 | 1728.26 | +| | | vLLM-Omni | 1 | 64.75 | 626.76 | 2321.00 | +| | multi_control | Diffusers | — | | | | +| | | vLLM-Omni | 8 | 60.40 | | 2622.35 | | | seg | Diffusers | 1 | 77.50 | 496.77 | 1734.77 | +| | | vLLM-Omni | 1 | 65.84 | 623.52 | 2341.06 | | | wsm | Diffusers | 1 | 70.42 | 398.67 | 1319.24 | +| | | vLLM-Omni | 1 | 53.66 | 488.58 | 1753.23 | | **H100 NVL** | blur | Diffusers | 1 | 26.97 | 224.61 | 775.61 | +| | | vLLM-Omni | 1 | 27.53 | 184.34 | 684.44 | | | depth | Diffusers | 1 | 31.63 | 233.03 | 610.83 | +| | | vLLM-Omni | 1 | 20.73 | 191.19 | 683.55 | | | edge | Diffusers | 1 | 34.73 | 173.04 | 602.04 | +| | | vLLM-Omni | 1 | 20.32 | 188.64 | 674.49 | +| | multi_control | Diffusers | — | | | | +| | | vLLM-Omni | 1 | 50.16 | 674.18 | 2999.04 | | | seg | Diffusers | 1 | 28.55 | 174.78 | 777.58 | +| | | vLLM-Omni | 1 | 20.87 | 190.35 | 680.37 | | | wsm | Diffusers | 1 | 26.17 | 139.82 | 594.21 | +| | | vLLM-Omni | 1 | 17.17 | 148.32 | 507.19 | | **H100 80GB HBM3** | blur | Diffusers | 1 | 21.06 | 131.86 | 454.46 | +| | | vLLM-Omni | 1 | 20.40 | 133.27 | 476.11 | | | depth | Diffusers | 1 | 24.61 | 135.88 | 457.43 | +| | | vLLM-Omni | 1 | 15.75 | 135.11 | 488.18 | | | edge | Diffusers | 1 | 21.28 | 132.06 | 447.24 | +| | | vLLM-Omni | 1 | 15.33 | 133.00 | 483.75 | +| | multi_control | Diffusers | — | | | | +| | | vLLM-Omni | 1 | 36.94 | 477.12 | 2122.18 | | | seg | Diffusers | 1 | 22.54 | 133.24 | 456.55 | +| | | vLLM-Omni | 1 | 16.04 | 133.82 | 484.97 | | | wsm | Diffusers | 1 | 20.25 | 106.79 | 342.79 | +| | | vLLM-Omni | 1 | 13.03 | 105.20 | 365.05 | | **H200 NVL** | blur | Diffusers | 1 | 22.82 | 152.13 | 535.83 | +| | | vLLM-Omni | 1 | 21.94 | 156.49 | 586.19 | | | depth | Diffusers | 1 | 26.80 | 156.98 | 541.71 | +| | | vLLM-Omni | 1 | 16.93 | 159.81 | 597.37 | | | edge | Diffusers | 1 | 23.06 | 152.54 | 533.87 | +| | | vLLM-Omni | 1 | 16.46 | 157.29 | 592.05 | +| | multi_control | Diffusers | — | | | | +| | | vLLM-Omni | 1 | 40.28 | 587.45 | 2792.29 | | | seg | Diffusers | 1 | 24.12 | 153.68 | 537.15 | +| | | vLLM-Omni | 1 | 17.13 | 157.81 | 594.45 | | | wsm | Diffusers | 1 | 21.90 | 123.08 | 407.57 | +| | | vLLM-Omni | 1 | 13.97 | 122.69 | 442.62 | | **H200 141GB HBM3** | blur | Diffusers | 1 | 20.63 | 128.41 | 450.79 | +| | | vLLM-Omni | 1 | 20.78 | 134.95 | 492.24 | | | depth | Diffusers | 1 | 24.01 | 134.51 | 457.91 | +| | | vLLM-Omni | 1 | 16.00 | 138.25 | 497.13 | | | edge | Diffusers | 1 | 20.87 | 130.99 | 443.92 | +| | | vLLM-Omni | 1 | 15.26 | 136.41 | 492.75 | +| | multi_control | Diffusers | — | | | | +| | | vLLM-Omni | 1 | 38.09 | 500.91 | 2316.90 | | | seg | Diffusers | 1 | 21.84 | 130.12 | 453.51 | +| | | vLLM-Omni | 1 | 16.03 | 136.90 | 493.89 | | | wsm | Diffusers | 1 | 19.65 | 106.00 | 344.09 | +| | | vLLM-Omni | 1 | 12.95 | 109.39 | 371.24 | | **B200** | blur | Diffusers | 1 | 12.69 | 70.78 | 226.51 | +| | | vLLM-Omni | 1 | 13.26 | 72.95 | 257.39 | | | depth | Diffusers | 1 | 14.79 | 73.23 | 229.54 | +| | | vLLM-Omni | 1 | 10.55 | 73.56 | 264.11 | | | edge | Diffusers | 1 | 12.76 | 71.04 | 223.24 | +| | | vLLM-Omni | 1 | 10.35 | 74.00 | 261.85 | +| | multi_control | Diffusers | — | | | | +| | | vLLM-Omni | 1 | 23.27 | 257.99 | 1111.03 | | | seg | Diffusers | 1 | 13.54 | 71.21 | 223.41 | +| | | vLLM-Omni | 1 | 11.10 | 73.35 | 262.72 | | | wsm | Diffusers | 1 | 12.16 | 59.29 | 172.60 | +| | | vLLM-Omni | 1 | 9.41 | 59.04 | 195.99 | | **B300** | blur | Diffusers | 1 | 12.39 | 66.13 | 207.59 | +| | | vLLM-Omni | 1 | 13.85 | 73.35 | 240.39 | | | depth | Diffusers | 1 | 14.26 | 70.00 | 211.12 | +| | | vLLM-Omni | 1 | 11.11 | 71.59 | 247.16 | | | edge | Diffusers | 1 | 12.48 | 67.68 | 207.95 | +| | | vLLM-Omni | 1 | 13.05 | 72.93 | 245.16 | +| | multi_control | Diffusers | — | | | | +| | | vLLM-Omni | 1 | 26.79 | 243.97 | 1026.91 | | | seg | Diffusers | 1 | 13.43 | 68.85 | 210.33 | +| | | vLLM-Omni | 1 | 14.38 | 71.85 | 246.04 | | | wsm | Diffusers | 1 | 11.95 | 55.41 | 162.28 | +| | | vLLM-Omni | 1 | 10.06 | 58.97 | 185.08 | Transfer notes: -1. Values are average generation latency in seconds from PBR `#308587`; lower is better. -2. All transfer runs use a single GPU; multi-GPU transfer has not been measured. -3. Only Diffusers has transfer coverage so far; other runtimes are unmeasured, not unsupported. -4. Control hints follow the vLLM-Omni `extra_params` names: `blur`, `depth`, `edge`, `seg`, and `wsm`. +1. Values are average generation latency in seconds; lower is better. Diffusers numbers come from PBR `#308587`, vLLM-Omni from PBR `#308574`. +2. The **GPUs** column gives the number of GPUs behind every value in that row. +3. PyTorch and NIM have no transfer coverage yet; those rows are unmeasured, not unsupported. +4. Control hints follow the vLLM-Omni `extra_params` names: `blur`, `depth`, `edge`, `multi_control`, `seg`, and `wsm`. `multi_control` combines several hints in one request. ## Action generation diff --git a/inference_benchmarks/cosmos3-super-generator.md b/inference_benchmarks/cosmos3-super-generator.md index dc347048..81c1d2e1 100644 --- a/inference_benchmarks/cosmos3-super-generator.md +++ b/inference_benchmarks/cosmos3-super-generator.md @@ -4,7 +4,7 @@ These tables report **Cosmos3-Super Generator** latency in seconds. Lower is better. -Every row reports one GPU configuration, named in the **GPUs** column, and all three resolutions in that row come from it. The configuration is one GPU when the model fits on a single device at its highest resolution, 720p, and eight GPUs when it does not. Cosmos3-Super fits on a single GPU on B200 and B300, and on H200 NVL and H200 141GB HBM3 for every engine except PyTorch; on RTX PRO 6000 Blackwell, H20, H100 NVL, and H100 80GB HBM3 it is reported at eight GPUs. Because the configuration varies between rows, read the **GPUs** column before comparing latencies down a column. +Every row reports one GPU configuration, named in the **GPUs** column, and all three resolutions in that row come from it. The configuration is one GPU when the model fits on a single device at its highest resolution, 720p, and eight GPUs when it does not. Under vLLM-Omni, Cosmos3-Super fits on a single GPU on every GPU measured. Under Diffusers and NIM it fits on one GPU on H200 NVL, H200 141GB HBM3, B200, and B300, and is reported at eight GPUs on RTX PRO 6000 Blackwell, H20, H100 NVL, and H100 80GB HBM3. PyTorch needs eight GPUs everywhere except B200. Because the configuration varies between rows, read the **GPUs** column before comparing latencies down a column. Empty cells mean that a run has not been completed for that GPU, engine, or resolution; they do not indicate that a combination is unsupported. @@ -34,9 +34,9 @@ Empty cells mean that a run has not been completed for that GPU, engine, or reso ## Benchmark methodology -The primary t2v, i2v, and t2i tables preserve the previously published benchmark campaigns across PyTorch, vLLM-Omni, Diffusers, and NIM. Those tables use BF16 precision, batch size 1, and matched prompts, seeds, and sampler settings where documented. Video workloads follow the standard Cosmos3 generation profile of 189 frames at 24 FPS unless a resolution tier limits frame count. +The primary t2v, i2v, and t2i tables cover PyTorch, vLLM-Omni, Diffusers, and NIM. Their vLLM-Omni values are from PBR `#307999`, **Cosmos3-Generator vLLM-Omni Inference Benchmarking (t2i, t2v, i2v)**; the PyTorch and NIM values preserve the previously published campaigns. Those tables use BF16 precision, batch size 1, and matched prompts, seeds, and sampler settings where documented. Video workloads follow the standard Cosmos3 generation profile of 189 frames at 24 FPS unless a resolution tier limits frame count. -The additional audiovisual and action tables come from three internal benchmark reports. PyTorch values are from PBR `#308197`, **Cosmos3-Generator OSS Inference Benchmarking 32B and 8B (189 frames)**: average generation (sampling) latency from the native OSS path, using **CUDA Graphs disabled** and the **latency** automatic-sharding preset. vLLM-Omni values for text-to-audio-and-video (`t2av`/`t2vs`) and image-to-audio-and-video (`i2av`/`i2vs`) are from PBR `#308195`. vLLM-Omni action values are from PBR `#308481`, **Cosmos3-Generator vLLM-Omni Inference Benchmarking (action)**, measured with the `vllm/vllm-omni:cosmos3` image and the official action cookbook samples; `DIFFUSION_ATTENTION_BACKEND=TORCH_SDPA` was not set. Forward-dynamics cells are the mean of `av_forward`, `av_left`, and `av_right`; inverse-dynamics cells are the mean of `av_inverse_0` and `av_inverse_1`. That action sweep covers 1, 2, and 4 GPUs only, because its 8-GPU Ulysses runs failed a sequence-length divisibility check; those rows therefore report the single-GPU configuration. Policy-DROID is a separate checkpoint and was measured only at 480p on one GPU. Super was not measured on H20, H100 NVL, or H100 80GB HBM3. Diffusers values come from four further reports: PBR `#308202` (all modalities) and PBR `#308451` (video-to-video) on one GPU, PBR `#308918` for 4- and 8-GPU runs, and PBR `#308587` for transfer. Where PBR `#308918` offers several tensor- and context-parallel splits at the same GPU count, the fastest is published. Its 1-GPU numbers are not used, because that sweep ran a different denoising-step budget than the single-GPU reports. Values are rounded to two decimal places. +The additional audiovisual and action tables come from three internal benchmark reports. PyTorch values are from PBR `#308197`, **Cosmos3-Generator OSS Inference Benchmarking 32B and 8B (189 frames)**: average generation (sampling) latency from the native OSS path, using **CUDA Graphs disabled** and the **latency** automatic-sharding preset. vLLM-Omni values for text-to-audio-and-video (`t2av`/`t2vs`) and image-to-audio-and-video (`i2av`/`i2vs`) are from PBR `#308195`. vLLM-Omni action values are from PBR `#308481`, **Cosmos3-Generator vLLM-Omni Inference Benchmarking (action)**, measured with the `vllm/vllm-omni:cosmos3` image and the official action cookbook samples; `DIFFUSION_ATTENTION_BACKEND=TORCH_SDPA` was not set. Forward-dynamics cells are the mean of `av_forward`, `av_left`, and `av_right`; inverse-dynamics cells are the mean of `av_inverse_0` and `av_inverse_1`. That action sweep covers 1, 2, and 4 GPUs only, because its 8-GPU Ulysses runs failed a sequence-length divisibility check; those rows therefore report the single-GPU configuration. Policy-DROID is a separate checkpoint and was measured only at 480p on one GPU. That action sweep did not cover Super on H20, H100 NVL, or H100 80GB HBM3. Diffusers values come from four further reports: PBR `#308202` (all modalities) and PBR `#308451` (video-to-video) on one GPU, PBR `#308918` for 4- and 8-GPU runs, and PBR `#308587` for transfer. vLLM-Omni transfer values are from PBR `#308574`. Where PBR `#308918` offers several tensor- and context-parallel splits at the same GPU count, the fastest is published. Its 1-GPU numbers are not used, because that sweep ran a different denoising-step budget than the single-GPU reports. Values are rounded to two decimal places. These reports establish the reported timing matrix but do not expose every prompt and action payload in this repository. The linked public recipes explain modality behavior and provide representative payloads; their example-specific frame counts and action chunk sizes should not be treated as the exact internal benchmark inputs. @@ -87,7 +87,7 @@ The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound | | Diffusers | 1 | 31.00 | 251.60 | 886.20 | | | NIM | 1 | 15.95 | 174.71 | 695.89 | | **B200** | PyTorch | 1 | 14.66 | 114.38 | 407.50 | -| | vLLM-Omni | 1 | 13.84 | 114.08 | 390.28 | +| | vLLM-Omni | 1 | 13.84 | 114.08 | 383.84 | | | Diffusers | 1 | 19.00 | 127.20 | 414.40 | | | NIM | 1 | 9.09 | 82.39 | 314.68 | | **B300** | PyTorch | — | | | | @@ -100,15 +100,15 @@ The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound | GPU | Engine | GPUs | 256p | 480p | 720p | |---|---|:-:|---:|---:|---:| | **RTX PRO 6000 Blackwell** | PyTorch | 8 | 66.54 | 118.95 | 427.96 | -| | vLLM-Omni | — | | | | +| | vLLM-Omni | 1 | 81.30 | 401.81 | 1374.84 | | | Diffusers | 8 | 26.29 | 156.16 | 540.65 | | | NIM | 8 | 14.48 | 100.23 | 289.93 | | **H20** | PyTorch | 8 | 28.22 | 153.19 | 491.93 | -| | vLLM-Omni | — | | | | +| | vLLM-Omni | 1 | 110.12 | 1009.04 | 3525.12 | | | Diffusers | 8 | 27.65 | 165.91 | 530.44 | | | NIM | 8 | 14.06 | 114.49 | 405.68 | | **H100 NVL** | PyTorch | 8 | 16.96 | 64.17 | 186.47 | -| | vLLM-Omni | — | | | | +| | vLLM-Omni | 1 | 86.20 | 317.79 | 1075.57 | | | Diffusers | 8 | 25.45 | 112.36 | 325.56 | | | NIM | 8 | 13.30 | 67.34 | 201.39 | | **H200 NVL** | PyTorch | 8 | 24.70 | 47.60 | 141.62 | @@ -116,7 +116,7 @@ The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound | | Diffusers | 1 | 33.00 | 287.20 | 1034.60 | | | NIM | 1 | 17.51 | 201.45 | 817.35 | | **H100 80GB HBM3** | PyTorch | — | | | | -| | vLLM-Omni | — | | | | +| | vLLM-Omni | 1 | 80.39 | 225.74 | 765.95 | | | Diffusers | 8 | 19.98 | 55.63 | 184.81 | | | NIM | 8 | 6.49 | 36.81 | 118.77 | | **H200 141GB HBM3** | PyTorch | 8 | 11.80 | 42.10 | 123.57 | @@ -124,7 +124,7 @@ The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound | | Diffusers | 1 | 31.00 | 249.20 | 879.20 | | | NIM | 1 | 16.39 | 175.95 | 699.13 | | **B200** | PyTorch | 1 | 14.71 | 112.40 | 397.31 | -| | vLLM-Omni | 1 | 14.13 | 115.17 | 393.02 | +| | vLLM-Omni | 1 | 14.13 | 115.17 | 386.35 | | | Diffusers | 1 | 19.20 | 127.00 | 414.80 | | | NIM | 1 | 9.36 | 83.19 | 316.76 | | **B300** | PyTorch | — | | | | @@ -137,25 +137,25 @@ The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound | GPU | Engine | GPUs | 256p | 480p | 720p | |---|---|:-:|---:|---:|---:| | **RTX PRO 6000 Blackwell** | PyTorch | 8 | 93.18 | 93.39 | 93.47 | -| | vLLM-Omni | — | | | | +| | vLLM-Omni | 1 | 105.12 | 105.17 | 105.42 | | | Diffusers | 8 | 16.90 | 17.52 | 16.73 | | **H20** | PyTorch | 8 | 14.18 | 16.46 | 20.92 | -| | vLLM-Omni | — | | | | +| | vLLM-Omni | 1 | 105.03 | 105.28 | 105.64 | | | Diffusers | 8 | 18.29 | 18.48 | 18.46 | | **H100 NVL** | PyTorch | 8 | 19.86 | 19.80 | 19.87 | -| | vLLM-Omni | — | | | | +| | vLLM-Omni | 1 | 112.13 | 112.28 | 112.37 | | | Diffusers | 8 | 16.56 | 16.60 | 16.88 | | **H200 NVL** | PyTorch | 8 | 32.86 | 33.05 | 33.16 | | | vLLM-Omni | 1 | 2.73 | 6.28 | 11.02 | | | Diffusers | 1 | 5.00 | 8.00 | 12.00 | | **H100 80GB HBM3** | PyTorch | — | | | | -| | vLLM-Omni | — | | | | +| | vLLM-Omni | 1 | 104.71 | 104.82 | 104.93 | | | Diffusers | 8 | 17.97 | 18.03 | 19.28 | | **H200 141GB HBM3** | PyTorch | 8 | 13.48 | 13.53 | 13.50 | | | vLLM-Omni | 1 | 2.83 | 5.70 | 10.24 | | | Diffusers | 1 | 5.00 | 8.00 | 11.00 | | **B200** | PyTorch | 1 | 4.51 | 4.78 | 7.25 | -| | vLLM-Omni | 1 | 2.32 | 3.29 | 6.02 | +| | vLLM-Omni | 1 | 2.32 | 3.29 | 5.99 | | | Diffusers | 1 | 4.40 | 6.00 | 8.00 | | **B300** | PyTorch | — | | | | | | vLLM-Omni | 1 | 5.05 | 3.79 | 7.08 | @@ -305,51 +305,107 @@ the hint changes how much of the frame the model must synthesise. | GPU | Transfer control | Engine | GPUs | 256p | 480p | 720p | |---|---|---|:-:|---:|---:|---:| | **RTX PRO 6000 Blackwell** | blur | Diffusers | — | | | | +| | | vLLM-Omni | — | | | | | | depth | Diffusers | — | | | | +| | | vLLM-Omni | — | | | | | | edge | Diffusers | — | | | | +| | | vLLM-Omni | — | | | | +| | multi_control | Diffusers | — | | | | +| | | vLLM-Omni | — | | | | | | seg | Diffusers | — | | | | +| | | vLLM-Omni | — | | | | | | wsm | Diffusers | — | | | | +| | | vLLM-Omni | — | | | | | **H20** | blur | Diffusers | — | | | | +| | | vLLM-Omni | — | | | | | | depth | Diffusers | — | | | | +| | | vLLM-Omni | — | | | | | | edge | Diffusers | — | | | | +| | | vLLM-Omni | — | | | | +| | multi_control | Diffusers | — | | | | +| | | vLLM-Omni | — | | | | | | seg | Diffusers | — | | | | +| | | vLLM-Omni | — | | | | | | wsm | Diffusers | — | | | | +| | | vLLM-Omni | — | | | | | **H100 NVL** | blur | Diffusers | — | | | | +| | | vLLM-Omni | — | | | | | | depth | Diffusers | — | | | | +| | | vLLM-Omni | — | | | | | | edge | Diffusers | — | | | | +| | | vLLM-Omni | — | | | | +| | multi_control | Diffusers | — | | | | +| | | vLLM-Omni | — | | | | | | seg | Diffusers | — | | | | +| | | vLLM-Omni | — | | | | | | wsm | Diffusers | — | | | | +| | | vLLM-Omni | — | | | | | **H100 80GB HBM3** | blur | Diffusers | — | | | | +| | | vLLM-Omni | — | | | | | | depth | Diffusers | — | | | | +| | | vLLM-Omni | — | | | | | | edge | Diffusers | — | | | | +| | | vLLM-Omni | — | | | | +| | multi_control | Diffusers | — | | | | +| | | vLLM-Omni | — | | | | | | seg | Diffusers | — | | | | +| | | vLLM-Omni | — | | | | | | wsm | Diffusers | — | | | | +| | | vLLM-Omni | — | | | | | **H200 NVL** | blur | Diffusers | 1 | 83.43 | 551.97 | 1935.10 | +| | | vLLM-Omni | 1 | 85.61 | 610.21 | 2199.03 | | | depth | Diffusers | 1 | 98.52 | 572.97 | 1970.31 | +| | | vLLM-Omni | 1 | 63.72 | 622.24 | 2255.18 | | | edge | Diffusers | 1 | 84.72 | 552.34 | 1940.58 | +| | | vLLM-Omni | 1 | 61.97 | 612.63 | 2238.40 | +| | multi_control | Diffusers | — | | | | +| | | vLLM-Omni | 8 | 57.19 | | 2432.02 | | | seg | Diffusers | 1 | 88.16 | 555.93 | 1950.38 | +| | | vLLM-Omni | 1 | 62.69 | 614.55 | 2239.94 | | | wsm | Diffusers | 1 | 80.77 | 445.36 | 1470.27 | +| | | vLLM-Omni | 1 | 52.06 | 481.65 | 1670.02 | | **H200 141GB HBM3** | blur | Diffusers | 1 | 71.83 | 463.77 | 1593.34 | +| | | vLLM-Omni | 1 | 77.82 | 515.99 | 1824.12 | | | depth | Diffusers | 1 | 84.88 | 481.84 | 1618.53 | +| | | vLLM-Omni | 1 | 58.38 | 528.43 | 1875.62 | | | edge | Diffusers | 1 | 73.22 | 466.10 | 1594.90 | +| | | vLLM-Omni | 1 | 55.80 | 527.66 | 1825.62 | +| | multi_control | Diffusers | — | | | | +| | | vLLM-Omni | 8 | 58.24 | | | | | seg | Diffusers | 1 | 77.09 | 476.15 | 1598.31 | +| | | vLLM-Omni | 1 | 57.91 | 528.52 | 1826.29 | | | wsm | Diffusers | 1 | 70.53 | 382.72 | 1216.43 | +| | | vLLM-Omni | 1 | 48.37 | 416.84 | 1406.07 | | **B200** | blur | Diffusers | 1 | 42.14 | 241.28 | 789.92 | +| | | vLLM-Omni | 1 | 42.90 | 264.79 | 943.38 | | | depth | Diffusers | 1 | 49.41 | 254.68 | 792.02 | +| | | vLLM-Omni | 1 | 32.81 | 268.01 | 955.14 | | | edge | Diffusers | 1 | 43.57 | 241.17 | 774.34 | +| | | vLLM-Omni | 1 | 31.94 | 263.68 | 946.09 | +| | multi_control | Diffusers | — | | | | +| | | vLLM-Omni | 1 | 73.81 | 934.97 | 4038.87 | | | seg | Diffusers | 1 | 45.43 | 243.27 | 792.36 | +| | | vLLM-Omni | 1 | 33.33 | 264.97 | 959.62 | | | wsm | Diffusers | 1 | 40.66 | 198.30 | 604.37 | +| | | vLLM-Omni | 1 | 27.64 | 209.59 | 718.19 | | **B300** | blur | Diffusers | 1 | 39.64 | 226.08 | 720.04 | +| | | vLLM-Omni | 1 | 44.56 | 251.49 | 863.63 | | | depth | Diffusers | 1 | 45.83 | 231.26 | 730.46 | +| | | vLLM-Omni | 1 | 34.74 | 256.94 | 887.06 | | | edge | Diffusers | 1 | 39.99 | 226.42 | 719.35 | +| | | vLLM-Omni | 1 | 34.13 | 253.19 | 879.84 | +| | multi_control | Diffusers | — | | | | +| | | vLLM-Omni | 1 | 73.12 | 864.66 | 3687.17 | | | seg | Diffusers | 1 | 41.64 | 228.18 | 720.63 | +| | | vLLM-Omni | 1 | 34.88 | 252.30 | 878.95 | | | wsm | Diffusers | 1 | 38.40 | 186.76 | 555.47 | +| | | vLLM-Omni | 1 | 27.58 | 201.45 | 665.09 | Transfer notes: -1. Values are average generation latency in seconds from PBR `#308587`; lower is better. -2. All transfer runs use a single GPU; multi-GPU transfer has not been measured. -3. Only Diffusers has transfer coverage so far; other runtimes are unmeasured, not unsupported. -4. Control hints follow the vLLM-Omni `extra_params` names: `blur`, `depth`, `edge`, `seg`, and `wsm`. +1. Values are average generation latency in seconds; lower is better. Diffusers numbers come from PBR `#308587`, vLLM-Omni from PBR `#308574`. +2. The **GPUs** column gives the number of GPUs behind every value in that row. +3. PyTorch and NIM have no transfer coverage yet; those rows are unmeasured, not unsupported. +4. Control hints follow the vLLM-Omni `extra_params` names: `blur`, `depth`, `edge`, `multi_control`, `seg`, and `wsm`. `multi_control` combines several hints in one request. ## Action generation From 63e6a80d4bad5f1d436c84b88bce454425f584e0 Mon Sep 17 00:00:00 2001 From: Sergiy Fefilatyev Date: Fri, 2 Oct 2026 13:40:06 -0700 Subject: [PATCH 7/9] Report one GPU count per GPU block so runtimes compare directly The tables are read across engines, so a vLLM-Omni row reporting one GPU next to three rows reporting eight invites a false comparison. Every engine in a GPU block now reports the same configuration. This moves eight Super rows, i2v and t2i on RTX PRO 6000 Blackwell, H20, H100 NVL, and H100 80GB HBM3, from one GPU to eight. vLLM-Omni does fit on a single GPU there, but it is the only engine that does, so the eight-GPU figures are the comparable ones. The H200 blocks keep their one-GPU values. PyTorch alone needs eight there, and it appears to be missing a single-GPU run rather than failing to fit, so raising those blocks would retire good numbers from three engines to accommodate one gap. The 720p i2v cell for vLLM-Omni on H100 80GB HBM3 is left empty: PBR #307999 reports 2.65 s at four GPUs and 6.53 s at eight against 38-64 s at 480p, which cannot be right. --- .../cosmos3-nano-generator.md | 6 ++--- .../cosmos3-super-generator.md | 25 ++++++++++--------- 2 files changed, 16 insertions(+), 15 deletions(-) diff --git a/inference_benchmarks/cosmos3-nano-generator.md b/inference_benchmarks/cosmos3-nano-generator.md index 6181a224..e9da99c4 100644 --- a/inference_benchmarks/cosmos3-nano-generator.md +++ b/inference_benchmarks/cosmos3-nano-generator.md @@ -4,7 +4,7 @@ These tables report **Cosmos3-Nano Generator** latency in seconds. Lower is better. -Every row reports one GPU configuration, named in the **GPUs** column, and all three resolutions in that row come from it. The configuration is one GPU when the model fits on a single device at its highest resolution, 720p, and eight GPUs when it does not. Cosmos3-Nano fits on a single GPU at 720p, so all populated rows here report one GPU. +Every row reports one GPU configuration, named in the **GPUs** column, and all three resolutions in that row come from it. The configuration is one GPU when the model fits on a single device at its highest resolution, 720p, and eight GPUs when it does not. Within one GPU's block every engine reports the same configuration, so a column can be read straight across the runtimes. Cosmos3-Nano fits on a single GPU at 720p, so all populated rows here report one GPU. Empty cells mean that a run has not been completed for that GPU, engine, or resolution; they do not indicate that a combination is unsupported. @@ -164,8 +164,8 @@ The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound Notes: 1. All times measured on identical workloads (same seed, sampler settings, prompt). 2. Multi-GPU configurations use tensor parallelism. -3. vLLM-Omni numbers are for the upcoming public release in the vLLM-Omni repo; subject to change before GA. Values marked with (*) are pre-release vLLM-Omni measurements on H100 NVL and may change before GA. -4. Diffusers numbers use the HuggingFace `diffusers` integration without custom CUDA graphs, and were measured on a single GPU only. +3. vLLM-Omni numbers are for the upcoming public release in the vLLM-Omni repo and are subject to change before GA; the H100 NVL values in particular are pre-release. +4. Diffusers numbers use the HuggingFace `diffusers` integration without custom CUDA graphs. Single-GPU values come from PBR `#308202` and PBR `#308451`; multi-GPU values from PBR `#308918`. 5. PyTorch numbers report average generation (sampling) time from OSS inference benchmarking. 6. At 256p, multi-GPU configurations on B300 may underperform single-GPU because of small-workload tensor-parallel overhead, so single-GPU is the recommended deployment at this resolution. 7. NIM numbers use latency profiles with FP8 precision and report end-to-end `Request Latency s`, including request processing, video generation, output encoding, and returning the response. diff --git a/inference_benchmarks/cosmos3-super-generator.md b/inference_benchmarks/cosmos3-super-generator.md index 81c1d2e1..ff4fb187 100644 --- a/inference_benchmarks/cosmos3-super-generator.md +++ b/inference_benchmarks/cosmos3-super-generator.md @@ -4,7 +4,7 @@ These tables report **Cosmos3-Super Generator** latency in seconds. Lower is better. -Every row reports one GPU configuration, named in the **GPUs** column, and all three resolutions in that row come from it. The configuration is one GPU when the model fits on a single device at its highest resolution, 720p, and eight GPUs when it does not. Under vLLM-Omni, Cosmos3-Super fits on a single GPU on every GPU measured. Under Diffusers and NIM it fits on one GPU on H200 NVL, H200 141GB HBM3, B200, and B300, and is reported at eight GPUs on RTX PRO 6000 Blackwell, H20, H100 NVL, and H100 80GB HBM3. PyTorch needs eight GPUs everywhere except B200. Because the configuration varies between rows, read the **GPUs** column before comparing latencies down a column. +Every row reports one GPU configuration, named in the **GPUs** column, and all three resolutions in that row come from it. The configuration is one GPU when the model fits on a single device at its highest resolution, 720p, and eight GPUs when it does not. Within one GPU's block every engine reports the same configuration, so a column can be read straight across the runtimes. Cosmos3-Super is reported at one GPU on H200 NVL, H200 141GB HBM3, B200, and B300, and at eight GPUs on RTX PRO 6000 Blackwell, H20, H100 NVL, and H100 80GB HBM3. vLLM-Omni does fit on a single GPU on those four smaller-memory parts, but is reported at eight alongside the other runtimes; PyTorch is the engine that sets the eight-GPU configuration on the H200 parts. Because the configuration varies between rows, read the **GPUs** column before comparing latencies down a column. Empty cells mean that a run has not been completed for that GPU, engine, or resolution; they do not indicate that a combination is unsupported. @@ -100,15 +100,15 @@ The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound | GPU | Engine | GPUs | 256p | 480p | 720p | |---|---|:-:|---:|---:|---:| | **RTX PRO 6000 Blackwell** | PyTorch | 8 | 66.54 | 118.95 | 427.96 | -| | vLLM-Omni | 1 | 81.30 | 401.81 | 1374.84 | +| | vLLM-Omni | 8 | 11.66 | 80.94 | 240.27 | | | Diffusers | 8 | 26.29 | 156.16 | 540.65 | | | NIM | 8 | 14.48 | 100.23 | 289.93 | | **H20** | PyTorch | 8 | 28.22 | 153.19 | 491.93 | -| | vLLM-Omni | 1 | 110.12 | 1009.04 | 3525.12 | +| | vLLM-Omni | 8 | 19.65 | 148.77 | 488.51 | | | Diffusers | 8 | 27.65 | 165.91 | 530.44 | | | NIM | 8 | 14.06 | 114.49 | 405.68 | | **H100 NVL** | PyTorch | 8 | 16.96 | 64.17 | 186.47 | -| | vLLM-Omni | 1 | 86.20 | 317.79 | 1075.57 | +| | vLLM-Omni | 8 | 8.48 | 59.44 | 212.53 | | | Diffusers | 8 | 25.45 | 112.36 | 325.56 | | | NIM | 8 | 13.30 | 67.34 | 201.39 | | **H200 NVL** | PyTorch | 8 | 24.70 | 47.60 | 141.62 | @@ -116,7 +116,7 @@ The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound | | Diffusers | 1 | 33.00 | 287.20 | 1034.60 | | | NIM | 1 | 17.51 | 201.45 | 817.35 | | **H100 80GB HBM3** | PyTorch | — | | | | -| | vLLM-Omni | 1 | 80.39 | 225.74 | 765.95 | +| | vLLM-Omni | 8 | 6.48 | 38.31 | | | | Diffusers | 8 | 19.98 | 55.63 | 184.81 | | | NIM | 8 | 6.49 | 36.81 | 118.77 | | **H200 141GB HBM3** | PyTorch | 8 | 11.80 | 42.10 | 123.57 | @@ -137,19 +137,19 @@ The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound | GPU | Engine | GPUs | 256p | 480p | 720p | |---|---|:-:|---:|---:|---:| | **RTX PRO 6000 Blackwell** | PyTorch | 8 | 93.18 | 93.39 | 93.47 | -| | vLLM-Omni | 1 | 105.12 | 105.17 | 105.42 | +| | vLLM-Omni | 8 | 3.11 | 3.54 | 5.70 | | | Diffusers | 8 | 16.90 | 17.52 | 16.73 | | **H20** | PyTorch | 8 | 14.18 | 16.46 | 20.92 | -| | vLLM-Omni | 1 | 105.03 | 105.28 | 105.64 | +| | vLLM-Omni | 8 | 5.12 | 5.44 | 8.85 | | | Diffusers | 8 | 18.29 | 18.48 | 18.46 | | **H100 NVL** | PyTorch | 8 | 19.86 | 19.80 | 19.87 | -| | vLLM-Omni | 1 | 112.13 | 112.28 | 112.37 | +| | vLLM-Omni | 8 | 3.54 | 3.48 | 4.11 | | | Diffusers | 8 | 16.56 | 16.60 | 16.88 | | **H200 NVL** | PyTorch | 8 | 32.86 | 33.05 | 33.16 | | | vLLM-Omni | 1 | 2.73 | 6.28 | 11.02 | | | Diffusers | 1 | 5.00 | 8.00 | 12.00 | | **H100 80GB HBM3** | PyTorch | — | | | | -| | vLLM-Omni | 1 | 104.71 | 104.82 | 104.93 | +| | vLLM-Omni | 8 | 5.09 | 5.16 | 4.91 | | | Diffusers | 8 | 17.97 | 18.03 | 19.28 | | **H200 141GB HBM3** | PyTorch | 8 | 13.48 | 13.53 | 13.50 | | | vLLM-Omni | 1 | 2.83 | 5.70 | 10.24 | @@ -164,11 +164,12 @@ The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound Notes: 1. All times measured on identical workloads (same seed, sampler settings, prompt). 2. Multi-GPU configurations use tensor parallelism. -3. vLLM-Omni numbers are for the upcoming public release in the vLLM-Omni repo; subject to change before GA. Current vLLM-Omni coverage is B200 at 720p. -4. Diffusers numbers use the HuggingFace `diffusers` integration without custom CUDA graphs, and were measured on a single GPU only. +3. vLLM-Omni numbers are for the upcoming public release in the vLLM-Omni repo and are subject to change before GA. +4. Diffusers numbers use the HuggingFace `diffusers` integration without custom CUDA graphs. Single-GPU values come from PBR `#308202` and PBR `#308451`; multi-GPU values from PBR `#308918`. 5. At 256p, multi-GPU configurations on B300 may underperform single-GPU because of small-workload tensor-parallel overhead, so single-GPU is the recommended deployment at this resolution. 6. PyTorch numbers report average generation (sampling) time from OSS inference benchmarking. -7. NIM numbers use latency profiles with FP8 precision and report end-to-end `Request Latency s`, including request processing, video generation, output encoding, and returning the response. +7. NIM numbers use latency profiles with FP8 precision and report end-to-end `Request Latency s`, including request processing, video generation, output encoding, and returning the response. +8. The 720p i2v cell for vLLM-Omni on H100 80GB HBM3 is empty because the multi-GPU runs for it returned times far below the 480p result, which is not physically plausible; the measurement is being rerun. ## Additional audiovisual generation From 6111aada71f4d58b2bf93ed8299121eb24932046 Mon Sep 17 00:00:00 2001 From: Sergiy Fefilatyev Date: Fri, 2 Oct 2026 13:52:16 -0700 Subject: [PATCH 8/9] Add TensorRT-LLM as a fourth Generator runtime PBR #308000 covers TensorRT-LLM across all three models. It sweeps tensor-parallel, CFG-parallel, and Ulysses-parallel splits at each GPU count, so the fastest split at a row's count is the one published, the same convention used for the multi-GPU Diffusers report. Nano and Super gain a TensorRT-LLM row in every GPU block, placed between Diffusers and NIM, carrying 216 and 206 values. The campaign covers six GPUs, so the H100 NVL and H200 NVL rows are empty, and it includes no action workloads. The transfer table now lists all five engines instead of only the two that had data, and picks each block's GPU count to surface the most measurements. The Edge page is reorganised around modalities rather than engines, because this report adds t2i, t2v, and v2v to a page that only carried i2v, and adds the first 256p results. Its existing i2v numbers are unchanged. --- inference_benchmarks.md | 4 +- .../cosmos3-edge-generator.md | 234 ++++++++-- .../cosmos3-nano-generator.md | 370 +++++++++++++--- .../cosmos3-super-generator.md | 406 +++++++++++++++--- 4 files changed, 847 insertions(+), 167 deletions(-) diff --git a/inference_benchmarks.md b/inference_benchmarks.md index 12961034..2afbaf61 100644 --- a/inference_benchmarks.md +++ b/inference_benchmarks.md @@ -1,6 +1,6 @@ # Inference Benchmarks -These reports collect inference benchmarks for Cosmos3. **Generator** pages measure image, video, audiovisual, and action-generation latency across PyTorch, vLLM-Omni, Diffusers, and NIM where results are available. **Reasoner** sections below measure VLM serving and token-generation performance for text, image, and video inputs through vLLM and Hugging Face Transformers. +These reports collect inference benchmarks for Cosmos3. **Generator** pages measure image, video, audiovisual, and action-generation latency across PyTorch, vLLM-Omni, Diffusers, TensorRT-LLM, and NIM where results are available. **Reasoner** sections below measure VLM serving and token-generation performance for text, image, and video inputs through vLLM and Hugging Face Transformers. Results are published incrementally from internal benchmark runs. **Empty cells mean that combination has not been measured yet** — not that it is unsupported. See each model page or section for workload details and data-source definitions. @@ -10,7 +10,7 @@ Generator results are organized by model so future checkpoints can be added with | Model | Primary vision generation | Additional audiovisual generation | Transfer generation | Action generation | |---|---|---|---|---| -| Cosmos3-Edge | [i2v](inference_benchmarks/cosmos3-edge-generator.md) | Not reported | Not reported | Not reported | +| Cosmos3-Edge | [t2v, i2v, t2i](inference_benchmarks/cosmos3-edge-generator.md#text-to-video-t2v) | [v2v](inference_benchmarks/cosmos3-edge-generator.md#video-to-video-v2v) | Not reported | Not reported | | Cosmos3-Nano | [t2v, i2v, t2i](inference_benchmarks/cosmos3-nano-generator.md#primary-vision-generation) | [v2v, t2av, v2av, i2av](inference_benchmarks/cosmos3-nano-generator.md#additional-audiovisual-generation) | [blur, depth, edge, seg, wsm](inference_benchmarks/cosmos3-nano-generator.md#transfer-generation) | [Forward dynamics, inverse dynamics, policy](inference_benchmarks/cosmos3-nano-generator.md#action-generation) | | Cosmos3-Super | [t2v, i2v, t2i](inference_benchmarks/cosmos3-super-generator.md#primary-vision-generation) | [v2v, t2av, v2av, i2av](inference_benchmarks/cosmos3-super-generator.md#additional-audiovisual-generation) | [blur, depth, edge, seg, wsm](inference_benchmarks/cosmos3-super-generator.md#transfer-generation) | [Forward dynamics, inverse dynamics, policy](inference_benchmarks/cosmos3-super-generator.md#action-generation) | diff --git a/inference_benchmarks/cosmos3-edge-generator.md b/inference_benchmarks/cosmos3-edge-generator.md index 357dc306..9f48aaa2 100644 --- a/inference_benchmarks/cosmos3-edge-generator.md +++ b/inference_benchmarks/cosmos3-edge-generator.md @@ -2,48 +2,200 @@ [Back to the inference benchmark index](../inference_benchmarks.md) -These tables report **Cosmos3-Edge** Generator latency in seconds for **image-to-video (i2v)**. Lower latency is better, and empty cells indicate that a run has not been completed. - -Cosmos3-Edge supports **256p and 480p**, so 480p is its highest resolution. Every row reports one GPU configuration, named in the **GPUs** column: one GPU where the model fits on a single device at 480p, and eight GPUs where it does not. Cosmos3-Edge fits on a single GPU or one integrated computing platform at 480p, so all populated rows report one. Video benchmarks generate **121 frames**. vLLM-Omni values report end-to-end latency, while PyTorch values report average generation latency. - -### vLLM-Omni - -| GPU or Platform | GPUs | 256p | 480p | -|---|:-:|---:|---:| -| B200 SXM 192 GB | 1 | | 7.09 | -| B300 | 1 | | 8.74 | -| H200 SXM 141 GB | 1 | | 13.13 | -| H200 NVL | 1 | | 14.15 | -| H100 SXM 80 GB | 1 | | 13.33 | -| H100 NVL 96 GB | 1 | | 17.33 | -| H20 SXM 96 GB | 1 | | 53.94 | -| RTX PRO 6000 Blackwell Server Edition | — | | | -| DGX Station | 1 | | 7.13 | -| DGX Spark | 1 | | 89.41 | -| Jetson AGX Thor T5000, 128 GB, MAXN | — | | | -| Jetson T3000, 32 GB, 1100 MHz | — | | | -| Jetson T2000, 16 GB, 702 MHz, THOR_NANO | — | | | - -### PyTorch - -| GPU or Platform | GPUs | 256p | 480p | -|---|:-:|---:|---:| -| B200 SXM 192 GB | 1 | | 7.45 | -| B300 | 1 | | 6.84 | -| H200 SXM 141 GB | 1 | | 12.31 | -| H200 NVL | 1 | | 14.07 | -| H100 SXM 80 GB | 1 | | 12.68 | -| H100 NVL 96 GB | 1 | | 16.42 | -| H20 SXM 96 GB | 1 | | 52.83 | -| RTX PRO 6000 Blackwell Server Edition | 1 | | 21.92 | -| DGX Station | 1 | | 6.31 | -| DGX Spark | 1 | | 103.36 | -| Jetson AGX Thor T5000, 128 GB, MAXN | — | | | -| Jetson T3000, 32 GB, 1100 MHz | — | | | +These tables report **Cosmos3-Edge** Generator latency in seconds. Lower latency is better, and empty cells indicate that a run has not been completed rather than an unsupported combination. + +Cosmos3-Edge supports **256p and 480p**, so 480p is its highest resolution. Every row reports one GPU configuration, named in the **GPUs** column: one GPU where the model fits on a single device at 480p, and eight GPUs where it does not. Cosmos3-Edge fits on a single GPU or one integrated computing platform at 480p, so all populated rows report one. Within a platform's block every engine reports the same configuration, so the rows can be compared directly. + +Video benchmarks generate **121 frames**. vLLM-Omni values report end-to-end latency, PyTorch values report average generation latency, and TensorRT-LLM values report average generation latency from PBR `#308000`. + +## Table of Contents + +- [Text-to-Image (t2i)](#text-to-image-t2i) +- [Text-to-Video (t2v)](#text-to-video-t2v) +- [Image-to-Video (i2v)](#image-to-video-i2v) +- [Video-to-Video (v2v)](#video-to-video-v2v) + +## Text-to-Image (t2i) + +| GPU or Platform | Engine | GPUs | 256p | 480p | +|---|---|:-:|---:|---:| +| **B200 SXM 192 GB** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | 1 | 0.52 | 0.53 | +| **B300** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | 1 | 0.81 | 0.82 | +| **H200 SXM 141 GB** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | 1 | 0.58 | 0.58 | +| **H200 NVL** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | — | | | +| **H100 SXM 80 GB** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | 1 | 0.58 | 0.59 | +| **H100 NVL 96 GB** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | — | | | +| **H20 SXM 96 GB** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | 1 | 0.79 | 1.43 | +| **RTX PRO 6000 Blackwell Server Edition** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | 1 | 0.57 | 0.66 | +| **DGX Station** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | — | | | +| **DGX Spark** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | — | | | +| **Jetson AGX Thor T5000, 128 GB, MAXN** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | — | | | +| **Jetson T3000, 32 GB, 1100 MHz** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | — | | | +| **Jetson T2000, 16 GB, 702 MHz, THOR_NANO** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | — | | | + +## Text-to-Video (t2v) + +| GPU or Platform | Engine | GPUs | 256p | 480p | +|---|---|:-:|---:|---:| +| **B200 SXM 192 GB** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | 1 | 1.05 | 7.84 | +| **B300** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | 1 | 1.00 | 7.22 | +| **H200 SXM 141 GB** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | 1 | 1.77 | 14.89 | +| **H200 NVL** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | — | | | +| **H100 SXM 80 GB** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | 1 | 1.83 | 15.09 | +| **H100 NVL 96 GB** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | — | | | +| **H20 SXM 96 GB** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | 1 | 6.91 | 59.19 | +| **RTX PRO 6000 Blackwell Server Edition** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | 1 | 2.77 | 26.13 | +| **DGX Station** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | — | | | +| **DGX Spark** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | — | | | +| **Jetson AGX Thor T5000, 128 GB, MAXN** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | — | | | +| **Jetson T3000, 32 GB, 1100 MHz** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | — | | | +| **Jetson T2000, 16 GB, 702 MHz, THOR_NANO** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | — | | | + +## Image-to-Video (i2v) + +| GPU or Platform | Engine | GPUs | 256p | 480p | +|---|---|:-:|---:|---:| +| **B200 SXM 192 GB** | PyTorch | 1 | | 7.45 | +| | vLLM-Omni | 1 | | 7.09 | +| | TensorRT-LLM | 1 | 1.23 | 8.55 | +| **B300** | PyTorch | 1 | | 6.84 | +| | vLLM-Omni | 1 | | 8.74 | +| | TensorRT-LLM | 1 | 1.42 | 8.04 | +| **H200 SXM 141 GB** | PyTorch | 1 | | 12.31 | +| | vLLM-Omni | 1 | | 13.13 | +| | TensorRT-LLM | 1 | 1.99 | 15.89 | +| **H200 NVL** | PyTorch | 1 | | 14.07 | +| | vLLM-Omni | 1 | | 14.15 | +| | TensorRT-LLM | — | | | +| **H100 SXM 80 GB** | PyTorch | 1 | | 12.68 | +| | vLLM-Omni | 1 | | 13.33 | +| | TensorRT-LLM | 1 | 2.08 | 15.95 | +| **H100 NVL 96 GB** | PyTorch | 1 | | 16.42 | +| | vLLM-Omni | 1 | | 17.33 | +| | TensorRT-LLM | — | | | +| **H20 SXM 96 GB** | PyTorch | 1 | | 52.83 | +| | vLLM-Omni | 1 | | 53.94 | +| | TensorRT-LLM | 1 | 7.50 | 62.08 | +| **RTX PRO 6000 Blackwell Server Edition** | PyTorch | 1 | | 21.92 | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | 1 | 3.09 | 27.55 | +| **DGX Station** | PyTorch | 1 | | 6.31 | +| | vLLM-Omni | 1 | | 7.13 | +| | TensorRT-LLM | — | | | +| **DGX Spark** | PyTorch | 1 | | 103.36 | +| | vLLM-Omni | 1 | | 89.41 | +| | TensorRT-LLM | — | | | +| **Jetson AGX Thor T5000, 128 GB, MAXN** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | — | | | +| **Jetson T3000, 32 GB, 1100 MHz** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | — | | | +| **Jetson T2000, 16 GB, 702 MHz, THOR_NANO** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | — | | | + +## Video-to-Video (v2v) + +| GPU or Platform | Engine | GPUs | 256p | 480p | +|---|---|:-:|---:|---:| +| **B200 SXM 192 GB** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | 1 | 1.15 | 7.94 | +| **B300** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | 1 | 1.24 | 7.35 | +| **H200 SXM 141 GB** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | 1 | 1.91 | 15.09 | +| **H200 NVL** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | — | | | +| **H100 SXM 80 GB** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | 1 | 1.95 | 15.28 | +| **H100 NVL 96 GB** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | — | | | +| **H20 SXM 96 GB** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | 1 | 7.04 | 60.03 | +| **RTX PRO 6000 Blackwell Server Edition** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | 1 | 2.86 | 26.21 | +| **DGX Station** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | — | | | +| **DGX Spark** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | — | | | +| **Jetson AGX Thor T5000, 128 GB, MAXN** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | — | | | +| **Jetson T3000, 32 GB, 1100 MHz** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | — | | | +| **Jetson T2000, 16 GB, 702 MHz, THOR_NANO** | PyTorch | — | | | +| | vLLM-Omni | — | | | +| | TensorRT-LLM | — | | | Notes: 1. The **GPUs** column gives the number of GPUs behind every value in that row. All current measurements use one GPU or one integrated computing platform. 2. Values are average end-to-end or generation latency in seconds; lower is better. -3. Cosmos3-Edge supports 256p and 480p. Every benchmark run to date is at 480p, so the 256p column is reserved for a future campaign rather than unsupported. -4. Video measurements (i2v) generate **121 output frames**. -5. PyTorch values report average generation latency rather than diffusion-only latency. +3. TensorRT-LLM values come from PBR `#308000`, which covers the six datacenter GPUs only; the DGX and Jetson platforms were not part of that campaign. +4. vLLM-Omni and PyTorch were measured for i2v only, so their rows are empty in the other three sections. +5. Video measurements generate **121 output frames**. +6. PyTorch values report average generation latency rather than diffusion-only latency. +7. Diffusers and NIM have no Cosmos3-Edge coverage yet, so they are not listed. diff --git a/inference_benchmarks/cosmos3-nano-generator.md b/inference_benchmarks/cosmos3-nano-generator.md index e9da99c4..f508f38b 100644 --- a/inference_benchmarks/cosmos3-nano-generator.md +++ b/inference_benchmarks/cosmos3-nano-generator.md @@ -34,9 +34,9 @@ Empty cells mean that a run has not been completed for that GPU, engine, or reso ## Benchmark methodology -The primary t2v, i2v, and t2i tables cover PyTorch, vLLM-Omni, Diffusers, and NIM. Their vLLM-Omni values are from PBR `#307999`, **Cosmos3-Generator vLLM-Omni Inference Benchmarking (t2i, t2v, i2v)**; the PyTorch and NIM values preserve the previously published campaigns. Those tables use BF16 precision, batch size 1, and matched prompts, seeds, and sampler settings where documented. Video workloads follow the standard Cosmos3 generation profile of 189 frames at 24 FPS unless a resolution tier limits frame count. +The primary t2v, i2v, and t2i tables cover PyTorch, vLLM-Omni, Diffusers, TensorRT-LLM, and NIM. Their vLLM-Omni values are from PBR `#307999`, **Cosmos3-Generator vLLM-Omni Inference Benchmarking (t2i, t2v, i2v)**; the PyTorch and NIM values preserve the previously published campaigns. Those tables use BF16 precision, batch size 1, and matched prompts, seeds, and sampler settings where documented. Video workloads follow the standard Cosmos3 generation profile of 189 frames at 24 FPS unless a resolution tier limits frame count. -The additional audiovisual and action tables come from three internal benchmark reports. PyTorch values are from PBR `#308197`, **Cosmos3-Generator OSS Inference Benchmarking 32B and 8B (189 frames)**: average generation (sampling) latency from the native OSS path, using **CUDA Graphs disabled** and the **latency** automatic-sharding preset. vLLM-Omni values for text-to-audio-and-video (`t2av`/`t2vs`) and image-to-audio-and-video (`i2av`/`i2vs`) are from PBR `#308195`. vLLM-Omni action values are from PBR `#308481`, **Cosmos3-Generator vLLM-Omni Inference Benchmarking (action)**, measured with the `vllm/vllm-omni:cosmos3` image and the official action cookbook samples; `DIFFUSION_ATTENTION_BACKEND=TORCH_SDPA` was not set. Forward-dynamics cells are the mean of `av_forward`, `av_left`, and `av_right`; inverse-dynamics cells are the mean of `av_inverse_0` and `av_inverse_1`. That action sweep covers 1, 2, and 4 GPUs only, because its 8-GPU Ulysses runs failed a sequence-length divisibility check; those rows therefore report the single-GPU configuration. Policy-DROID is a separate checkpoint and was measured only at 480p on one GPU. That action sweep did not cover Super on H20, H100 NVL, or H100 80GB HBM3. Diffusers values come from four further reports: PBR `#308202` (all modalities) and PBR `#308451` (video-to-video) on one GPU, PBR `#308918` for 4- and 8-GPU runs, and PBR `#308587` for transfer. vLLM-Omni transfer values are from PBR `#308574`. Where PBR `#308918` offers several tensor- and context-parallel splits at the same GPU count, the fastest is published. Its 1-GPU numbers are not used, because that sweep ran a different denoising-step budget than the single-GPU reports. Values are rounded to two decimal places. +The additional audiovisual and action tables come from three internal benchmark reports. PyTorch values are from PBR `#308197`, **Cosmos3-Generator OSS Inference Benchmarking 32B and 8B (189 frames)**: average generation (sampling) latency from the native OSS path, using **CUDA Graphs disabled** and the **latency** automatic-sharding preset. vLLM-Omni values for text-to-audio-and-video (`t2av`/`t2vs`) and image-to-audio-and-video (`i2av`/`i2vs`) are from PBR `#308195`. vLLM-Omni action values are from PBR `#308481`, **Cosmos3-Generator vLLM-Omni Inference Benchmarking (action)**, measured with the `vllm/vllm-omni:cosmos3` image and the official action cookbook samples; `DIFFUSION_ATTENTION_BACKEND=TORCH_SDPA` was not set. Forward-dynamics cells are the mean of `av_forward`, `av_left`, and `av_right`; inverse-dynamics cells are the mean of `av_inverse_0` and `av_inverse_1`. That action sweep covers 1, 2, and 4 GPUs only, because its 8-GPU Ulysses runs failed a sequence-length divisibility check; those rows therefore report the single-GPU configuration. Policy-DROID is a separate checkpoint and was measured only at 480p on one GPU. That action sweep did not cover Super on H20, H100 NVL, or H100 80GB HBM3. Diffusers values come from four further reports: PBR `#308202` (all modalities) and PBR `#308451` (video-to-video) on one GPU, PBR `#308918` for 4- and 8-GPU runs, and PBR `#308587` for transfer. vLLM-Omni transfer values are from PBR `#308574`. Where PBR `#308918` offers several tensor- and context-parallel splits at the same GPU count, the fastest is published. Its 1-GPU numbers are not used, because that sweep ran a different denoising-step budget than the single-GPU reports. TensorRT-LLM values are from PBR `#308000`, **Cosmos3-Generator TRT-LLM Inference Benchmarking**, which sweeps tensor-parallel, CFG-parallel, and Ulysses-parallel splits at each GPU count; the fastest split at the count a row reports is the one published. That campaign covers six GPUs, so the TensorRT-LLM rows for H100 NVL and H200 NVL are empty, and it includes no action workloads. Values are rounded to two decimal places. These reports establish the reported timing matrix but do not expose every prompt and action payload in this repository. The linked public recipes explain modality behavior and provide representative payloads; their example-specific frame counts and action chunk sizes should not be treated as the exact internal benchmark inputs. @@ -65,34 +65,42 @@ The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound | **RTX PRO 6000 Blackwell** | PyTorch | 1 | 13.95 | 180.81 | 786.37 | | | vLLM-Omni | 1 | 10.65 | 105.61 | 371.53 | | | Diffusers | 1 | 11.20 | 112.00 | 392.00 | +| | TensorRT-LLM | 1 | 10.27 | 102.55 | 369.78 | | | NIM | 1 | 7.93 | 82.69 | 318.69 | | **H20** | PyTorch | 1 | 30.57 | 257.51 | 931.39 | | | vLLM-Omni | 1 | 28.58 | 256.97 | 929.81 | | | Diffusers | 1 | 30.20 | 258.00 | 926.00 | +| | TensorRT-LLM | 1 | 26.92 | 251.79 | 908.90 | | | NIM | 1 | 17.81 | 192.07 | 771.37 | | **H100 NVL** | PyTorch | 1 | 10.03 | 84.12 | 297.27 | | | vLLM-Omni | 1 | 9.25 | 80.75 | 281.99 | | | Diffusers | 1 | 11.00 | 90.00 | 324.20 | +| | TensorRT-LLM | — | | | | | | NIM | 1 | 7.09 | 68.01 | 267.73 | | **H200 NVL** | PyTorch | 1 | 8.17 | 69.79 | 244.39 | | | vLLM-Omni | 1 | 7.44 | 64.58 | 240.05 | | | Diffusers | 1 | 9.00 | 74.00 | 276.20 | +| | TensorRT-LLM | — | | | | | | NIM | 1 | 5.87 | 57.10 | 229.63 | | **H100 80GB HBM3** | PyTorch | 1 | 7.61 | 59.83 | 207.78 | | | vLLM-Omni | 1 | 6.97 | 58.17 | 202.29 | | | Diffusers | 1 | 9.00 | 68.00 | 240.00 | +| | TensorRT-LLM | 1 | 6.50 | 60.10 | 223.51 | | | NIM | 1 | 5.72 | 51.73 | 199.46 | | **H200 141GB HBM3** | PyTorch | 1 | 7.53 | 60.18 | 214.28 | | | vLLM-Omni | 1 | 6.79 | 58.14 | 208.36 | | | Diffusers | 1 | 9.00 | 67.00 | 239.60 | +| | TensorRT-LLM | 1 | 6.42 | 60.57 | 225.06 | | | NIM | 1 | 5.72 | 51.90 | 200.63 | | **B200** | PyTorch | 1 | 4.56 | 33.20 | 114.85 | | | vLLM-Omni | 1 | 4.03 | 32.04 | 107.79 | | | Diffusers | 1 | 7.00 | 36.80 | 117.00 | +| | TensorRT-LLM | 1 | 3.43 | 28.28 | 100.02 | | | NIM | 1 | 3.72 | 26.68 | 93.33 | | **B300** | PyTorch | — | | | | | | vLLM-Omni | 1 | 4.46 | 32.18 | 102.10 | | | Diffusers | 1 | 6.40 | 39.32 | 121.74 | +| | TensorRT-LLM | 1 | 3.31 | 26.52 | 91.15 | | | NIM | 1 | 4.51 | 27.49 | 90.39 | ### Image-to-Video (i2v) @@ -102,34 +110,42 @@ The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound | **RTX PRO 6000 Blackwell** | PyTorch | 1 | 14.08 | 182.14 | 788.80 | | | vLLM-Omni | 1 | 11.04 | 107.77 | 376.44 | | | Diffusers | 1 | 12.00 | 112.00 | 397.00 | +| | TensorRT-LLM | 1 | 10.77 | 105.03 | 374.89 | | | NIM | 1 | 8.23 | 84.58 | 326.96 | | **H20** | PyTorch | 1 | 31.36 | 257.10 | 933.07 | | | vLLM-Omni | 1 | 29.50 | 261.56 | 940.16 | | | Diffusers | 1 | 31.00 | 258.00 | 925.00 | +| | TensorRT-LLM | 1 | 28.07 | 254.79 | 915.71 | | | NIM | 1 | 18.67 | 195.10 | 774.88 | | **H100 NVL** | PyTorch | 1 | 10.19 | 84.50 | 298.57 | | | vLLM-Omni | 1 | 9.62 | 82.61 | 286.19 | | | Diffusers | 1 | 11.00 | 91.00 | 325.20 | +| | TensorRT-LLM | — | | | | | | NIM | 1 | 7.39 | 69.39 | 272.29 | | **H200 NVL** | PyTorch | 1 | 8.27 | 69.99 | 246.62 | | | vLLM-Omni | 1 | 7.83 | 66.39 | 243.52 | | | Diffusers | 1 | 9.00 | 74.00 | 275.20 | +| | TensorRT-LLM | — | | | | | | NIM | 1 | 6.25 | 58.64 | 232.47 | | **H100 80GB HBM3** | PyTorch | 1 | 7.64 | 59.95 | 207.87 | | | vLLM-Omni | 1 | 7.37 | 59.77 | 205.97 | | | Diffusers | 1 | 9.00 | 68.00 | 239.80 | +| | TensorRT-LLM | 1 | 6.90 | 63.24 | 226.15 | | | NIM | 1 | 6.08 | 53.02 | 202.59 | | **H200 141GB HBM3** | PyTorch | 1 | 7.65 | 60.51 | 214.80 | | | vLLM-Omni | 1 | 7.28 | 59.64 | 209.65 | | | Diffusers | 1 | 9.00 | 67.20 | 240.00 | +| | TensorRT-LLM | 1 | 6.74 | 63.54 | 231.15 | | | NIM | 1 | 6.04 | 53.25 | 203.66 | | **B200** | PyTorch | 1 | 4.60 | 33.08 | 113.90 | | | vLLM-Omni | 1 | 4.33 | 33.09 | 110.08 | | | Diffusers | 1 | 7.00 | 37.00 | 116.00 | +| | TensorRT-LLM | 1 | 3.70 | 29.33 | 102.35 | | | NIM | 1 | 4.05 | 27.72 | 95.57 | | **B300** | PyTorch | — | | | | | | vLLM-Omni | 1 | 5.61 | 33.45 | 104.75 | | | Diffusers | 1 | 13.40 | 49.59 | 131.25 | +| | TensorRT-LLM | 1 | 3.70 | 27.73 | 93.88 | | | NIM | 1 | 4.50 | 28.90 | 92.59 | ### Text-to-Image (t2i) @@ -139,31 +155,39 @@ The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound | **RTX PRO 6000 Blackwell** | PyTorch | 1 | 2.99 | 4.51 | 7.12 | | | vLLM-Omni | 1 | 1.59 | 2.87 | 4.99 | | | Diffusers | 1 | 2.00 | 4.00 | 5.00 | +| | TensorRT-LLM | 1 | 1.18 | 2.35 | 4.50 | | **H20** | PyTorch | 1 | 3.06 | 6.51 | 12.31 | | | vLLM-Omni | 1 | 1.73 | 4.92 | 10.73 | | | Diffusers | 1 | 3.00 | 6.00 | 10.00 | +| | TensorRT-LLM | 1 | 1.94 | 5.55 | 11.56 | | **H100 NVL** | PyTorch | 1 | 2.77 | 2.83 | 4.21 | | | vLLM-Omni | 1 | 1.55 | 1.92 | 3.43 | | | Diffusers | 1 | 3.00 | 3.00 | 4.00 | +| | TensorRT-LLM | — | | | | | **H200 NVL** | PyTorch | 1 | 2.75 | 2.85 | 3.58 | | | vLLM-Omni | 1 | 1.53 | 1.58 | 2.81 | | | Diffusers | 1 | 3.00 | 3.00 | 4.00 | +| | TensorRT-LLM | — | | | | | **H100 80GB HBM3** | PyTorch | 1 | 3.01 | 3.01 | 3.45 | | | vLLM-Omni | 1 | 1.61 | 1.53 | 2.61 | | | Diffusers | 1 | 3.00 | 3.00 | 4.00 | +| | TensorRT-LLM | 1 | 1.02 | 1.30 | 2.63 | | **H200 141GB HBM3** | PyTorch | 1 | 2.96 | 3.04 | 3.28 | | | vLLM-Omni | 1 | 1.57 | 1.52 | 2.60 | | | Diffusers | 1 | 3.00 | 3.00 | 4.00 | +| | TensorRT-LLM | 1 | 1.05 | 1.30 | 2.60 | | **B200** | PyTorch | 1 | 2.68 | 2.75 | 2.87 | | | vLLM-Omni | 1 | 1.49 | 1.20 | 1.76 | | | Diffusers | 1 | 3.00 | 3.20 | 3.00 | +| | TensorRT-LLM | 1 | 0.86 | 0.88 | 1.40 | | **B300** | PyTorch | — | | | | | | vLLM-Omni | 1 | 1.97 | 1.81 | 2.34 | | | Diffusers | 1 | 3.86 | 3.86 | 3.99 | +| | TensorRT-LLM | 1 | 1.36 | 1.30 | 1.40 | Notes: 1. All times measured on identical workloads (same seed, sampler settings, prompt). -2. Multi-GPU configurations use tensor parallelism. +2. Multi-GPU configurations use tensor parallelism, except TensorRT-LLM, which may also split across CFG and Ulysses dimensions. 3. vLLM-Omni numbers are for the upcoming public release in the vLLM-Omni repo and are subject to change before GA; the H100 NVL values in particular are pre-release. 4. Diffusers numbers use the HuggingFace `diffusers` integration without custom CUDA graphs. Single-GPU values come from PBR `#308202` and PBR `#308451`; multi-GPU values from PBR `#308918`. 5. PyTorch numbers report average generation (sampling) time from OSS inference benchmarking. @@ -181,27 +205,35 @@ A text prompt and source video condition a generated continuation or transformat | **RTX PRO 6000 Blackwell** | PyTorch | 1 | 13.73 | 180.54 | 785.41 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 24.19 | 135.85 | 432.18 | +| | TensorRT-LLM | 1 | 10.36 | 102.50 | 370.24 | | **H20** | PyTorch | 1 | 30.44 | 257.79 | 930.96 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 44.88 | 287.55 | 974.75 | +| | TensorRT-LLM | 1 | 27.10 | 250.57 | 908.07 | | **H100 NVL** | PyTorch | 1 | 10.18 | 81.73 | 297.83 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 23.66 | 107.79 | 354.66 | +| | TensorRT-LLM | — | | | | | **H200 NVL** | PyTorch | 1 | 8.10 | 69.70 | 250.33 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 21.40 | 96.52 | 313.13 | +| | TensorRT-LLM | — | | | | | **H100 80GB HBM3** | PyTorch | 1 | 7.50 | 59.75 | 207.36 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 19.48 | 86.51 | 267.90 | +| | TensorRT-LLM | 1 | 6.63 | 60.18 | 229.04 | | **H200 141GB HBM3** | PyTorch | 1 | 7.48 | 60.23 | 214.03 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 18.93 | 85.52 | 266.84 | +| | TensorRT-LLM | 1 | 6.59 | 61.40 | 229.48 | | **B200** | PyTorch | 1 | 4.52 | 32.53 | 114.16 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 16.93 | 52.32 | 137.54 | +| | TensorRT-LLM | 1 | 3.54 | 28.37 | 100.29 | | **B300** | PyTorch | — | | | | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 16.14 | 50.74 | 131.11 | +| | TensorRT-LLM | 1 | 3.44 | 26.63 | 91.40 | ### Text-to-Audio-and-Video (t2av) @@ -212,27 +244,35 @@ A text prompt produces synchronized video and sound. | **RTX PRO 6000 Blackwell** | PyTorch | 1 | 14.13 | 180.98 | 794.74 | | | vLLM-Omni | 1 | 12.25 | 106.02 | 373.10 | | | Diffusers | 1 | 13.00 | 113.00 | 393.00 | +| | TensorRT-LLM | 1 | 11.35 | 103.28 | 374.75 | | **H20** | PyTorch | 1 | 30.82 | 256.41 | 922.35 | | | vLLM-Omni | 1 | 29.43 | 260.84 | 937.93 | | | Diffusers | 1 | 32.00 | 263.00 | 932.60 | +| | TensorRT-LLM | 1 | 29.38 | 254.88 | 913.33 | | **H100 NVL** | PyTorch | 1 | 9.99 | 82.32 | 297.88 | | | vLLM-Omni | 1 | 9.92 | 81.57 | 291.91 | | | Diffusers | 1 | 12.00 | 91.00 | 324.60 | +| | TensorRT-LLM | — | | | | | **H200 NVL** | PyTorch | 1 | 8.20 | 69.88 | 250.46 | | | vLLM-Omni | 1 | 7.82 | 65.42 | 244.60 | | | Diffusers | 1 | 10.00 | 75.00 | 277.80 | +| | TensorRT-LLM | — | | | | | **H100 80GB HBM3** | PyTorch | 1 | 7.60 | 59.09 | 207.87 | | | vLLM-Omni | 1 | 7.34 | 58.65 | 203.92 | | | Diffusers | 1 | 9.40 | 69.00 | 242.00 | +| | TensorRT-LLM | 1 | 7.04 | 61.09 | 228.06 | | **H200 141GB HBM3** | PyTorch | 1 | 7.55 | 60.17 | 211.59 | | | vLLM-Omni | 1 | 7.13 | 58.27 | 209.29 | | | Diffusers | 1 | 9.00 | 69.00 | 237.00 | +| | TensorRT-LLM | 1 | 6.93 | 60.77 | 233.06 | | **B200** | PyTorch | 1 | 4.63 | 32.60 | 113.89 | | | vLLM-Omni | 1 | 4.29 | 32.20 | 107.74 | | | Diffusers | 1 | 7.00 | 37.00 | 117.00 | +| | TensorRT-LLM | 1 | 3.77 | 28.64 | 101.28 | | **B300** | PyTorch | — | | | | | | vLLM-Omni | 1 | 5.59 | 33.15 | 104.92 | | | Diffusers | 1 | 6.46 | 39.66 | 122.33 | +| | TensorRT-LLM | 1 | 3.65 | 26.83 | 91.85 | ### Video-to-Audio-and-Video (v2av) @@ -243,27 +283,35 @@ A text prompt and source video produce transformed video with synchronized sound | **RTX PRO 6000 Blackwell** | PyTorch | 1 | 13.73 | 180.72 | 786.11 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 26.30 | 135.47 | 429.61 | +| | TensorRT-LLM | — | | | | | **H20** | PyTorch | 1 | 30.42 | 257.83 | 921.82 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 46.72 | 288.98 | 985.80 | +| | TensorRT-LLM | — | | | | | **H100 NVL** | PyTorch | 1 | 9.91 | 81.96 | 297.51 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 22.95 | 110.93 | 354.02 | +| | TensorRT-LLM | — | | | | | **H200 NVL** | PyTorch | 1 | 8.10 | 69.59 | 246.51 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 22.19 | 94.85 | 314.06 | +| | TensorRT-LLM | — | | | | | **H100 80GB HBM3** | PyTorch | 1 | 7.49 | 59.86 | 207.55 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 21.73 | 88.42 | 268.93 | +| | TensorRT-LLM | — | | | | | **H200 141GB HBM3** | PyTorch | 1 | 7.49 | 60.15 | 214.44 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 17.98 | 87.84 | 267.54 | +| | TensorRT-LLM | — | | | | | **B200** | PyTorch | 1 | 4.51 | 32.62 | 113.72 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 15.64 | 50.57 | 138.15 | +| | TensorRT-LLM | — | | | | | **B300** | PyTorch | — | | | | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 15.93 | 51.87 | 131.52 | +| | TensorRT-LLM | — | | | | ### Image-to-Audio-and-Video (i2av) @@ -274,27 +322,35 @@ A text prompt and source image produce video with synchronized sound. | **RTX PRO 6000 Blackwell** | PyTorch | 1 | 14.08 | 181.64 | 787.81 | | | vLLM-Omni | 1 | 12.61 | 108.05 | 378.04 | | | Diffusers | 1 | 14.00 | 113.00 | 393.00 | +| | TensorRT-LLM | 1 | 11.85 | 105.53 | 379.62 | | **H20** | PyTorch | 1 | 31.34 | 259.06 | 924.39 | | | vLLM-Omni | 1 | 30.36 | 265.37 | 948.09 | | | Diffusers | 1 | 32.00 | 263.00 | 934.80 | +| | TensorRT-LLM | 1 | 29.95 | 259.43 | 924.85 | | **H100 NVL** | PyTorch | 1 | 10.45 | 84.37 | 298.61 | | | vLLM-Omni | 1 | 10.34 | 83.21 | 295.65 | | | Diffusers | 1 | 12.00 | 91.00 | 324.40 | +| | TensorRT-LLM | — | | | | | **H200 NVL** | PyTorch | 1 | 8.29 | 71.20 | 248.35 | | | vLLM-Omni | 1 | 8.24 | 67.23 | 249.41 | | | Diffusers | 1 | 10.00 | 75.00 | 277.20 | +| | TensorRT-LLM | — | | | | | **H100 80GB HBM3** | PyTorch | 1 | 7.73 | 59.13 | 207.93 | | | vLLM-Omni | 1 | 7.74 | 60.27 | 207.53 | | | Diffusers | 1 | 10.00 | 69.00 | 241.00 | +| | TensorRT-LLM | 1 | 7.39 | 62.91 | 227.42 | | **H200 141GB HBM3** | PyTorch | 1 | 7.66 | 60.55 | 214.82 | | | vLLM-Omni | 1 | 7.55 | 59.73 | 212.73 | | | Diffusers | 1 | 9.00 | 68.00 | 241.00 | +| | TensorRT-LLM | 1 | 7.37 | 62.87 | 231.61 | | **B200** | PyTorch | 1 | 4.62 | 32.82 | 113.85 | | | vLLM-Omni | 1 | 4.58 | 33.36 | 110.05 | | | Diffusers | 1 | 7.20 | 37.00 | 118.00 | +| | TensorRT-LLM | 1 | 4.04 | 29.77 | 103.28 | | **B300** | PyTorch | — | | | | | | vLLM-Omni | 1 | 6.01 | 34.39 | 108.59 | | | Diffusers | 1 | 15.35 | 50.19 | 131.92 | +| | TensorRT-LLM | 1 | 4.03 | 28.10 | 94.57 | ## Transfer generation @@ -304,106 +360,250 @@ the hint changes how much of the frame the model must synthesise. | GPU | Transfer control | Engine | GPUs | 256p | 480p | 720p | |---|---|---|:-:|---:|---:|---:| -| **RTX PRO 6000 Blackwell** | blur | Diffusers | 1 | 30.95 | 217.40 | 735.43 | +| **RTX PRO 6000 Blackwell** | blur | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 40.11 | 288.60 | 1061.44 | -| | depth | Diffusers | 1 | 36.52 | 225.34 | 745.70 | +| | | Diffusers | 1 | 30.95 | 217.40 | 735.43 | +| | | TensorRT-LLM | 1 | 25.62 | 249.50 | 935.86 | +| | | NIM | 1 | | | | +| | depth | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 31.56 | 291.76 | 1090.33 | -| | edge | Diffusers | 1 | 31.25 | 217.90 | 735.35 | +| | | Diffusers | 1 | 36.52 | 225.34 | 745.70 | +| | | TensorRT-LLM | 1 | 26.13 | 253.94 | 942.69 | +| | | NIM | 1 | | | | +| | edge | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 30.64 | 287.82 | 1080.82 | -| | multi_control | Diffusers | — | | | | +| | | Diffusers | 1 | 31.25 | 217.90 | 735.35 | +| | | TensorRT-LLM | 1 | 25.67 | 250.54 | 938.79 | +| | | NIM | 1 | | | | +| | multi_control | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 73.36 | 1060.11 | 3988.93 | -| | seg | Diffusers | 1 | 32.89 | 220.10 | 738.51 | +| | | Diffusers | 1 | | | | +| | | TensorRT-LLM | 1 | 35.07 | 423.61 | 1745.86 | +| | | NIM | 1 | | | | +| | seg | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 31.50 | 289.01 | 1083.64 | -| | wsm | Diffusers | 1 | 29.84 | 176.06 | 565.32 | +| | | Diffusers | 1 | 32.89 | 220.10 | 738.51 | +| | | TensorRT-LLM | 1 | 25.85 | 250.47 | 937.18 | +| | | NIM | 1 | | | | +| | wsm | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 26.12 | 225.70 | 814.25 | -| **H20** | blur | Diffusers | 1 | 73.61 | 491.22 | 1726.61 | +| | | Diffusers | 1 | 29.84 | 176.06 | 565.32 | +| | | TensorRT-LLM | 1 | 12.95 | 114.91 | 407.62 | +| | | NIM | 1 | | | | +| **H20** | blur | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 85.93 | 625.60 | 2280.10 | -| | depth | Diffusers | 1 | 86.84 | 510.15 | 1753.96 | +| | | Diffusers | 1 | 73.61 | 491.22 | 1726.61 | +| | | TensorRT-LLM | 1 | 63.10 | 614.14 | 2308.16 | +| | | NIM | 1 | | | | +| | depth | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 66.69 | 636.40 | 2367.37 | -| | edge | Diffusers | 1 | 74.68 | 492.78 | 1728.26 | +| | | Diffusers | 1 | 86.84 | 510.15 | 1753.96 | +| | | TensorRT-LLM | 1 | 64.44 | 622.79 | 2328.65 | +| | | NIM | 1 | | | | +| | edge | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 64.75 | 626.76 | 2321.00 | -| | multi_control | Diffusers | — | | | | +| | | Diffusers | 1 | 74.68 | 492.78 | 1728.26 | +| | | TensorRT-LLM | 1 | 63.92 | 615.09 | 2310.55 | +| | | NIM | 1 | | | | +| | multi_control | PyTorch | 8 | | | | | | | vLLM-Omni | 8 | 60.40 | | 2622.35 | -| | seg | Diffusers | 1 | 77.50 | 496.77 | 1734.77 | +| | | Diffusers | 8 | | | | +| | | TensorRT-LLM | 8 | 0.64 | 0.54 | 0.51 | +| | | NIM | 8 | | | | +| | seg | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 65.84 | 623.52 | 2341.06 | -| | wsm | Diffusers | 1 | 70.42 | 398.67 | 1319.24 | +| | | Diffusers | 1 | 77.50 | 496.77 | 1734.77 | +| | | TensorRT-LLM | 1 | 63.56 | 616.74 | 2314.38 | +| | | NIM | 1 | | | | +| | wsm | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 53.66 | 488.58 | 1753.23 | -| **H100 NVL** | blur | Diffusers | 1 | 26.97 | 224.61 | 775.61 | +| | | Diffusers | 1 | 70.42 | 398.67 | 1319.24 | +| | | TensorRT-LLM | 1 | 32.09 | 284.19 | 1001.59 | +| | | NIM | 1 | | | | +| **H100 NVL** | blur | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 27.53 | 184.34 | 684.44 | -| | depth | Diffusers | 1 | 31.63 | 233.03 | 610.83 | +| | | Diffusers | 1 | 26.97 | 224.61 | 775.61 | +| | | TensorRT-LLM | 1 | | | | +| | | NIM | 1 | | | | +| | depth | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 20.73 | 191.19 | 683.55 | -| | edge | Diffusers | 1 | 34.73 | 173.04 | 602.04 | +| | | Diffusers | 1 | 31.63 | 233.03 | 610.83 | +| | | TensorRT-LLM | 1 | | | | +| | | NIM | 1 | | | | +| | edge | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 20.32 | 188.64 | 674.49 | -| | multi_control | Diffusers | — | | | | +| | | Diffusers | 1 | 34.73 | 173.04 | 602.04 | +| | | TensorRT-LLM | 1 | | | | +| | | NIM | 1 | | | | +| | multi_control | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 50.16 | 674.18 | 2999.04 | -| | seg | Diffusers | 1 | 28.55 | 174.78 | 777.58 | +| | | Diffusers | 1 | | | | +| | | TensorRT-LLM | 1 | | | | +| | | NIM | 1 | | | | +| | seg | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 20.87 | 190.35 | 680.37 | -| | wsm | Diffusers | 1 | 26.17 | 139.82 | 594.21 | +| | | Diffusers | 1 | 28.55 | 174.78 | 777.58 | +| | | TensorRT-LLM | 1 | | | | +| | | NIM | 1 | | | | +| | wsm | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 17.17 | 148.32 | 507.19 | -| **H100 80GB HBM3** | blur | Diffusers | 1 | 21.06 | 131.86 | 454.46 | +| | | Diffusers | 1 | 26.17 | 139.82 | 594.21 | +| | | TensorRT-LLM | 1 | | | | +| | | NIM | 1 | | | | +| **H100 80GB HBM3** | blur | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 20.40 | 133.27 | 476.11 | -| | depth | Diffusers | 1 | 24.61 | 135.88 | 457.43 | +| | | Diffusers | 1 | 21.06 | 131.86 | 454.46 | +| | | TensorRT-LLM | 1 | 15.20 | 149.57 | 575.59 | +| | | NIM | 1 | | | | +| | depth | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 15.75 | 135.11 | 488.18 | -| | edge | Diffusers | 1 | 21.28 | 132.06 | 447.24 | +| | | Diffusers | 1 | 24.61 | 135.88 | 457.43 | +| | | TensorRT-LLM | 1 | 15.45 | 149.96 | 580.66 | +| | | NIM | 1 | | | | +| | edge | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 15.33 | 133.00 | 483.75 | -| | multi_control | Diffusers | — | | | | +| | | Diffusers | 1 | 21.28 | 132.06 | 447.24 | +| | | TensorRT-LLM | 1 | 15.28 | 148.99 | 572.91 | +| | | NIM | 1 | | | | +| | multi_control | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 36.94 | 477.12 | 2122.18 | -| | seg | Diffusers | 1 | 22.54 | 133.24 | 456.55 | +| | | Diffusers | 1 | | | | +| | | TensorRT-LLM | 1 | 21.50 | 254.29 | 1072.98 | +| | | NIM | 1 | | | | +| | seg | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 16.04 | 133.82 | 484.97 | -| | wsm | Diffusers | 1 | 20.25 | 106.79 | 342.79 | +| | | Diffusers | 1 | 22.54 | 133.24 | 456.55 | +| | | TensorRT-LLM | 1 | 15.42 | 145.46 | 576.57 | +| | | NIM | 1 | | | | +| | wsm | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 13.03 | 105.20 | 365.05 | -| **H200 NVL** | blur | Diffusers | 1 | 22.82 | 152.13 | 535.83 | +| | | Diffusers | 1 | 20.25 | 106.79 | 342.79 | +| | | TensorRT-LLM | 1 | 7.96 | 68.97 | 248.27 | +| | | NIM | 1 | | | | +| **H200 NVL** | blur | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 21.94 | 156.49 | 586.19 | -| | depth | Diffusers | 1 | 26.80 | 156.98 | 541.71 | +| | | Diffusers | 1 | 22.82 | 152.13 | 535.83 | +| | | TensorRT-LLM | 1 | | | | +| | | NIM | 1 | | | | +| | depth | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 16.93 | 159.81 | 597.37 | -| | edge | Diffusers | 1 | 23.06 | 152.54 | 533.87 | +| | | Diffusers | 1 | 26.80 | 156.98 | 541.71 | +| | | TensorRT-LLM | 1 | | | | +| | | NIM | 1 | | | | +| | edge | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 16.46 | 157.29 | 592.05 | -| | multi_control | Diffusers | — | | | | +| | | Diffusers | 1 | 23.06 | 152.54 | 533.87 | +| | | TensorRT-LLM | 1 | | | | +| | | NIM | 1 | | | | +| | multi_control | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 40.28 | 587.45 | 2792.29 | -| | seg | Diffusers | 1 | 24.12 | 153.68 | 537.15 | +| | | Diffusers | 1 | | | | +| | | TensorRT-LLM | 1 | | | | +| | | NIM | 1 | | | | +| | seg | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 17.13 | 157.81 | 594.45 | -| | wsm | Diffusers | 1 | 21.90 | 123.08 | 407.57 | +| | | Diffusers | 1 | 24.12 | 153.68 | 537.15 | +| | | TensorRT-LLM | 1 | | | | +| | | NIM | 1 | | | | +| | wsm | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 13.97 | 122.69 | 442.62 | -| **H200 141GB HBM3** | blur | Diffusers | 1 | 20.63 | 128.41 | 450.79 | +| | | Diffusers | 1 | 21.90 | 123.08 | 407.57 | +| | | TensorRT-LLM | 1 | | | | +| | | NIM | 1 | | | | +| **H200 141GB HBM3** | blur | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 20.78 | 134.95 | 492.24 | -| | depth | Diffusers | 1 | 24.01 | 134.51 | 457.91 | +| | | Diffusers | 1 | 20.63 | 128.41 | 450.79 | +| | | TensorRT-LLM | 1 | 15.11 | 149.55 | 580.80 | +| | | NIM | 1 | | | | +| | depth | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 16.00 | 138.25 | 497.13 | -| | edge | Diffusers | 1 | 20.87 | 130.99 | 443.92 | +| | | Diffusers | 1 | 24.01 | 134.51 | 457.91 | +| | | TensorRT-LLM | 1 | 15.42 | 152.64 | 587.03 | +| | | NIM | 1 | | | | +| | edge | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 15.26 | 136.41 | 492.75 | -| | multi_control | Diffusers | — | | | | +| | | Diffusers | 1 | 20.87 | 130.99 | 443.92 | +| | | TensorRT-LLM | 1 | 14.95 | 148.90 | 586.42 | +| | | NIM | 1 | | | | +| | multi_control | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 38.09 | 500.91 | 2316.90 | -| | seg | Diffusers | 1 | 21.84 | 130.12 | 453.51 | +| | | Diffusers | 1 | | | | +| | | TensorRT-LLM | 1 | 21.65 | 259.30 | 1095.79 | +| | | NIM | 1 | | | | +| | seg | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 16.03 | 136.90 | 493.89 | -| | wsm | Diffusers | 1 | 19.65 | 106.00 | 344.09 | +| | | Diffusers | 1 | 21.84 | 130.12 | 453.51 | +| | | TensorRT-LLM | 1 | 15.31 | 148.26 | 583.48 | +| | | NIM | 1 | | | | +| | wsm | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 12.95 | 109.39 | 371.24 | -| **B200** | blur | Diffusers | 1 | 12.69 | 70.78 | 226.51 | +| | | Diffusers | 1 | 19.65 | 106.00 | 344.09 | +| | | TensorRT-LLM | 1 | 7.90 | 68.71 | 252.43 | +| | | NIM | 1 | | | | +| **B200** | blur | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 13.26 | 72.95 | 257.39 | -| | depth | Diffusers | 1 | 14.79 | 73.23 | 229.54 | +| | | Diffusers | 1 | 12.69 | 70.78 | 226.51 | +| | | TensorRT-LLM | 1 | 7.73 | 68.21 | 252.69 | +| | | NIM | 1 | | | | +| | depth | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 10.55 | 73.56 | 264.11 | -| | edge | Diffusers | 1 | 12.76 | 71.04 | 223.24 | +| | | Diffusers | 1 | 14.79 | 73.23 | 229.54 | +| | | TensorRT-LLM | 1 | 7.92 | 68.77 | 248.98 | +| | | NIM | 1 | | | | +| | edge | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 10.35 | 74.00 | 261.85 | -| | multi_control | Diffusers | — | | | | +| | | Diffusers | 1 | 12.76 | 71.04 | 223.24 | +| | | TensorRT-LLM | 1 | 7.78 | 68.29 | 248.51 | +| | | NIM | 1 | | | | +| | multi_control | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 23.27 | 257.99 | 1111.03 | -| | seg | Diffusers | 1 | 13.54 | 71.21 | 223.41 | +| | | Diffusers | 1 | | | | +| | | TensorRT-LLM | 1 | 10.72 | 112.58 | 455.44 | +| | | NIM | 1 | | | | +| | seg | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 11.10 | 73.35 | 262.72 | -| | wsm | Diffusers | 1 | 12.16 | 59.29 | 172.60 | +| | | Diffusers | 1 | 13.54 | 71.21 | 223.41 | +| | | TensorRT-LLM | 1 | 7.99 | 68.45 | 248.75 | +| | | NIM | 1 | | | | +| | wsm | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 9.41 | 59.04 | 195.99 | -| **B300** | blur | Diffusers | 1 | 12.39 | 66.13 | 207.59 | +| | | Diffusers | 1 | 12.16 | 59.29 | 172.60 | +| | | TensorRT-LLM | 1 | 4.25 | 32.24 | 108.75 | +| | | NIM | 1 | | | | +| **B300** | blur | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 13.85 | 73.35 | 240.39 | -| | depth | Diffusers | 1 | 14.26 | 70.00 | 211.12 | +| | | Diffusers | 1 | 12.39 | 66.13 | 207.59 | +| | | TensorRT-LLM | 1 | 7.67 | 62.95 | 228.34 | +| | | NIM | 1 | | | | +| | depth | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 11.11 | 71.59 | 247.16 | -| | edge | Diffusers | 1 | 12.48 | 67.68 | 207.95 | +| | | Diffusers | 1 | 14.26 | 70.00 | 211.12 | +| | | TensorRT-LLM | 1 | 7.67 | 63.68 | 226.26 | +| | | NIM | 1 | | | | +| | edge | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 13.05 | 72.93 | 245.16 | -| | multi_control | Diffusers | — | | | | +| | | Diffusers | 1 | 12.48 | 67.68 | 207.95 | +| | | TensorRT-LLM | 1 | 7.58 | 63.10 | 228.36 | +| | | NIM | 1 | | | | +| | multi_control | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 26.79 | 243.97 | 1026.91 | -| | seg | Diffusers | 1 | 13.43 | 68.85 | 210.33 | +| | | Diffusers | 1 | | | | +| | | TensorRT-LLM | 1 | 10.59 | 105.33 | 414.58 | +| | | NIM | 1 | | | | +| | seg | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 14.38 | 71.85 | 246.04 | -| | wsm | Diffusers | 1 | 11.95 | 55.41 | 162.28 | +| | | Diffusers | 1 | 13.43 | 68.85 | 210.33 | +| | | TensorRT-LLM | 1 | 7.64 | 64.13 | 228.37 | +| | | NIM | 1 | | | | +| | wsm | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 10.06 | 58.97 | 185.08 | +| | | Diffusers | 1 | 11.95 | 55.41 | 162.28 | +| | | TensorRT-LLM | 1 | 4.23 | 30.44 | 101.17 | +| | | NIM | 1 | | | | Transfer notes: -1. Values are average generation latency in seconds; lower is better. Diffusers numbers come from PBR `#308587`, vLLM-Omni from PBR `#308574`. -2. The **GPUs** column gives the number of GPUs behind every value in that row. +1. Values are average generation latency in seconds; lower is better. Diffusers numbers come from PBR `#308587`, vLLM-Omni from PBR `#308574`, and TensorRT-LLM from PBR `#308000`. +2. The **GPUs** column gives the number of GPUs behind every value in that row, and is the same for every engine in a GPU block so the rows can be compared directly. 3. PyTorch and NIM have no transfer coverage yet; those rows are unmeasured, not unsupported. 4. Control hints follow the vLLM-Omni `extra_params` names: `blur`, `depth`, `edge`, `multi_control`, `seg`, and `wsm`. `multi_control` combines several hints in one request. @@ -418,27 +618,35 @@ Forward dynamics is reported separately for AV, camera, and robot inputs. Invers | **RTX PRO 6000 Blackwell** | PyTorch | 1 | 16.87 | 16.86 | 16.85 | | | vLLM-Omni | 1 | 2.08 | 12.94 | 36.19 | | | Diffusers | 1 | 2.00 | 12.00 | 12.00 | +| | TensorRT-LLM | — | | | | | **H20** | PyTorch | 1 | 31.47 | 31.16 | 31.45 | | | vLLM-Omni | 1 | 4.82 | 31.36 | 87.55 | | | Diffusers | 1 | 5.00 | 28.00 | 28.20 | +| | TensorRT-LLM | — | | | | | **H100 NVL** | PyTorch | 1 | 10.32 | 10.66 | 10.64 | | | vLLM-Omni | 1 | 1.71 | 10.58 | 28.26 | | | Diffusers | 1 | 2.00 | 10.00 | 10.00 | +| | TensorRT-LLM | — | | | | | **H200 NVL** | PyTorch | 1 | 8.30 | 8.31 | 8.33 | | | vLLM-Omni | 1 | 1.43 | 8.21 | 22.78 | | | Diffusers | 1 | 2.00 | 8.00 | 8.00 | +| | TensorRT-LLM | — | | | | | **H100 80GB HBM3** | PyTorch | 1 | 7.80 | 7.87 | 7.79 | | | vLLM-Omni | 1 | 1.33 | 7.74 | 20.60 | | | Diffusers | 1 | 2.00 | 8.00 | 8.00 | +| | TensorRT-LLM | — | | | | | **H200 141GB HBM3** | PyTorch | 1 | 7.77 | 7.76 | 7.81 | | | vLLM-Omni | 1 | 1.33 | 7.72 | 20.65 | | | Diffusers | 1 | 2.00 | 8.00 | 8.00 | +| | TensorRT-LLM | — | | | | | **B200** | PyTorch | 1 | 4.75 | 4.75 | 4.73 | | | vLLM-Omni | 1 | 0.94 | 4.93 | 12.00 | | | Diffusers | 1 | 2.00 | 5.00 | 5.00 | +| | TensorRT-LLM | — | | | | | **B300** | PyTorch | — | | | | | | vLLM-Omni | 1 | 1.09 | 4.86 | 11.70 | | | Diffusers | 1 | 20.87 | 24.49 | 26.44 | +| | TensorRT-LLM | — | | | | ### Forward Dynamics — Camera @@ -447,27 +655,35 @@ Forward dynamics is reported separately for AV, camera, and robot inputs. Invers | **RTX PRO 6000 Blackwell** | PyTorch | 1 | 16.82 | 16.82 | 16.81 | | | vLLM-Omni | — | | | | | | Diffusers | — | | | | +| | TensorRT-LLM | — | | | | | **H20** | PyTorch | 1 | 31.40 | 31.10 | 31.40 | | | vLLM-Omni | — | | | | | | Diffusers | — | | | | +| | TensorRT-LLM | — | | | | | **H100 NVL** | PyTorch | 1 | 10.61 | 10.32 | 10.57 | | | vLLM-Omni | — | | | | | | Diffusers | — | | | | +| | TensorRT-LLM | — | | | | | **H200 NVL** | PyTorch | 1 | 8.24 | 8.26 | 8.26 | | | vLLM-Omni | — | | | | | | Diffusers | — | | | | +| | TensorRT-LLM | — | | | | | **H100 80GB HBM3** | PyTorch | 1 | 7.78 | 7.83 | 7.75 | | | vLLM-Omni | — | | | | | | Diffusers | — | | | | +| | TensorRT-LLM | — | | | | | **H200 141GB HBM3** | PyTorch | 1 | 7.73 | 7.75 | 7.75 | | | vLLM-Omni | — | | | | | | Diffusers | — | | | | +| | TensorRT-LLM | — | | | | | **B200** | PyTorch | 1 | 4.70 | 4.74 | 4.71 | | | vLLM-Omni | — | | | | | | Diffusers | — | | | | +| | TensorRT-LLM | — | | | | | **B300** | PyTorch | — | | | | | | vLLM-Omni | — | | | | | | Diffusers | — | | | | +| | TensorRT-LLM | — | | | | ### Forward Dynamics — Robot @@ -476,27 +692,35 @@ Forward dynamics is reported separately for AV, camera, and robot inputs. Invers | **RTX PRO 6000 Blackwell** | PyTorch | 1 | 3.74 | 3.73 | 3.72 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 2.00 | 3.00 | 3.00 | +| | TensorRT-LLM | — | | | | | **H20** | PyTorch | 1 | 7.00 | 7.01 | 7.01 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 3.00 | 7.00 | 7.00 | +| | TensorRT-LLM | — | | | | | **H100 NVL** | PyTorch | 1 | 2.49 | 2.50 | 2.55 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 2.00 | 3.00 | 3.00 | +| | TensorRT-LLM | — | | | | | **H200 NVL** | PyTorch | 1 | 2.08 | 2.09 | 2.09 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 2.00 | 3.00 | 3.00 | +| | TensorRT-LLM | — | | | | | **H100 80GB HBM3** | PyTorch | 1 | 2.00 | 2.01 | 1.98 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 2.00 | 3.00 | 3.00 | +| | TensorRT-LLM | — | | | | | **H200 141GB HBM3** | PyTorch | 1 | 1.93 | 1.93 | 1.93 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 2.00 | 2.00 | 2.00 | +| | TensorRT-LLM | — | | | | | **B200** | PyTorch | 1 | 1.28 | 1.30 | 1.31 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 2.00 | 2.00 | 2.00 | +| | TensorRT-LLM | — | | | | | **B300** | PyTorch | — | | | | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 22.65 | 20.75 | 18.10 | +| | TensorRT-LLM | — | | | | ### Inverse Dynamics — Autonomous Vehicle (AV) @@ -505,27 +729,35 @@ Forward dynamics is reported separately for AV, camera, and robot inputs. Invers | **RTX PRO 6000 Blackwell** | PyTorch | 1 | 16.74 | 16.75 | 16.75 | | | vLLM-Omni | 1 | 2.10 | 12.96 | 36.10 | | | Diffusers | 1 | 2.00 | 12.00 | 12.00 | +| | TensorRT-LLM | — | | | | | **H20** | PyTorch | 1 | 31.22 | 30.99 | 31.22 | | | vLLM-Omni | 1 | 4.84 | 31.36 | 87.50 | | | Diffusers | 1 | 5.00 | 28.00 | 28.00 | +| | TensorRT-LLM | — | | | | | **H100 NVL** | PyTorch | 1 | 10.47 | 10.45 | 10.44 | | | vLLM-Omni | 1 | 1.74 | 10.54 | 28.16 | | | Diffusers | 1 | 2.00 | 10.00 | 10.00 | +| | TensorRT-LLM | — | | | | | **H200 NVL** | PyTorch | 1 | 8.12 | 8.13 | 8.10 | | | vLLM-Omni | 1 | 1.46 | 8.20 | 22.34 | | | Diffusers | 1 | 2.00 | 8.00 | 8.00 | +| | TensorRT-LLM | — | | | | | **H100 80GB HBM3** | PyTorch | 1 | 7.62 | 7.63 | 7.68 | | | vLLM-Omni | 1 | 1.39 | 7.72 | 20.55 | | | Diffusers | 1 | 2.00 | 8.00 | 8.00 | +| | TensorRT-LLM | — | | | | | **H200 141GB HBM3** | PyTorch | 1 | 7.51 | 7.64 | 7.51 | | | vLLM-Omni | 1 | 1.36 | 7.69 | 20.65 | | | Diffusers | 1 | 2.00 | 8.00 | 8.00 | +| | TensorRT-LLM | — | | | | | **B200** | PyTorch | 1 | 4.60 | 4.63 | 4.58 | | | vLLM-Omni | 1 | 0.92 | 5.10 | 12.29 | | | Diffusers | 1 | 2.00 | 5.00 | 5.00 | +| | TensorRT-LLM | — | | | | | **B300** | PyTorch | — | | | | | | vLLM-Omni | 1 | 1.36 | 5.75 | 11.89 | | | Diffusers | 1 | 8.76 | 17.79 | 17.99 | +| | TensorRT-LLM | — | | | | ### Inverse Dynamics — Robot @@ -534,27 +766,35 @@ Forward dynamics is reported separately for AV, camera, and robot inputs. Invers | **RTX PRO 6000 Blackwell** | PyTorch | 1 | 3.70 | 3.70 | 3.70 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 2.00 | 3.00 | 3.00 | +| | TensorRT-LLM | — | | | | | **H20** | PyTorch | 1 | 6.94 | 6.94 | 6.89 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 3.00 | 7.00 | 7.00 | +| | TensorRT-LLM | — | | | | | **H100 NVL** | PyTorch | 1 | 2.53 | 2.47 | 2.54 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 2.00 | 3.00 | 3.00 | +| | TensorRT-LLM | — | | | | | **H200 NVL** | PyTorch | 1 | 2.05 | 2.05 | 2.05 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 2.00 | 3.00 | 3.00 | +| | TensorRT-LLM | — | | | | | **H100 80GB HBM3** | PyTorch | 1 | 1.96 | 1.96 | 1.95 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 2.00 | 3.00 | 3.00 | +| | TensorRT-LLM | — | | | | | **H200 141GB HBM3** | PyTorch | 1 | 1.90 | 1.90 | 1.91 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 2.00 | 2.20 | 2.00 | +| | TensorRT-LLM | — | | | | | **B200** | PyTorch | 1 | 1.28 | 1.27 | 1.28 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 2.00 | 2.00 | 2.00 | +| | TensorRT-LLM | — | | | | | **B300** | PyTorch | — | | | | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 9.28 | 12.40 | 10.86 | +| | TensorRT-LLM | — | | | | ### Policy — Autonomous Vehicle (AV) @@ -563,27 +803,35 @@ Forward dynamics is reported separately for AV, camera, and robot inputs. Invers | **RTX PRO 6000 Blackwell** | PyTorch | 1 | 16.88 | 16.89 | 16.91 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 2.00 | 12.00 | 12.00 | +| | TensorRT-LLM | — | | | | | **H20** | PyTorch | 1 | 31.51 | 31.51 | 31.50 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 5.00 | 28.60 | 28.40 | +| | TensorRT-LLM | — | | | | | **H100 NVL** | PyTorch | 1 | 10.66 | 10.65 | 10.35 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 2.00 | 10.00 | 10.00 | +| | TensorRT-LLM | — | | | | | **H200 NVL** | PyTorch | 1 | 8.31 | 8.32 | 8.31 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 2.00 | 8.00 | 8.00 | +| | TensorRT-LLM | — | | | | | **H100 80GB HBM3** | PyTorch | 1 | 7.82 | 7.82 | 7.80 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 2.00 | 8.00 | 8.00 | +| | TensorRT-LLM | — | | | | | **H200 141GB HBM3** | PyTorch | 1 | 7.76 | 7.69 | 7.66 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 2.00 | 8.00 | 8.00 | +| | TensorRT-LLM | — | | | | | **B200** | PyTorch | 1 | 4.75 | 4.77 | 4.74 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 2.00 | 5.00 | 5.00 | +| | TensorRT-LLM | — | | | | | **B300** | PyTorch | — | | | | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 13.11 | 18.21 | 16.49 | +| | TensorRT-LLM | — | | | | ### Policy — Robot @@ -592,27 +840,35 @@ Forward dynamics is reported separately for AV, camera, and robot inputs. Invers | **RTX PRO 6000 Blackwell** | PyTorch | 1 | 3.74 | 3.74 | 3.74 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 2.00 | 3.00 | 3.00 | +| | TensorRT-LLM | — | | | | | **H20** | PyTorch | 1 | 6.95 | 7.00 | 7.01 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 3.00 | 7.00 | 7.00 | +| | TensorRT-LLM | — | | | | | **H100 NVL** | PyTorch | 1 | 2.50 | 2.51 | 2.55 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 2.00 | 3.00 | 3.00 | +| | TensorRT-LLM | — | | | | | **H200 NVL** | PyTorch | 1 | 2.11 | 2.09 | 2.09 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 2.00 | 3.00 | 3.00 | +| | TensorRT-LLM | — | | | | | **H100 80GB HBM3** | PyTorch | 1 | 1.99 | 1.99 | 1.99 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 2.00 | 3.00 | 3.00 | +| | TensorRT-LLM | — | | | | | **H200 141GB HBM3** | PyTorch | 1 | 1.93 | 1.93 | 1.94 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 2.00 | 2.00 | 2.20 | +| | TensorRT-LLM | — | | | | | **B200** | PyTorch | 1 | 1.30 | 1.30 | 1.30 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 2.00 | 2.00 | 2.00 | +| | TensorRT-LLM | — | | | | | **B300** | PyTorch | — | | | | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 10.62 | 12.78 | 11.00 | +| | TensorRT-LLM | — | | | | ### Policy — DROID @@ -623,27 +879,35 @@ These rows report **Cosmos3-Nano-Policy-DROID**, not the general Cosmos3-Nano ch | **RTX PRO 6000 Blackwell** | PyTorch | — | | | | | | vLLM-Omni | 1 | | 3.82 | | | | Diffusers | — | | | | +| | TensorRT-LLM | — | | | | | **H20** | PyTorch | — | | | | | | vLLM-Omni | 1 | | 8.51 | | | | Diffusers | — | | | | +| | TensorRT-LLM | — | | | | | **H100 NVL** | PyTorch | — | | | | | | vLLM-Omni | 1 | | 2.94 | | | | Diffusers | — | | | | +| | TensorRT-LLM | — | | | | | **H200 NVL** | PyTorch | — | | | | | | vLLM-Omni | 1 | | 2.34 | | | | Diffusers | — | | | | +| | TensorRT-LLM | — | | | | | **H100 80GB HBM3** | PyTorch | — | | | | | | vLLM-Omni | 1 | | 2.23 | | | | Diffusers | — | | | | +| | TensorRT-LLM | — | | | | | **H200 141GB HBM3** | PyTorch | — | | | | | | vLLM-Omni | 1 | | 2.21 | | | | Diffusers | — | | | | +| | TensorRT-LLM | — | | | | | **B200** | PyTorch | — | | | | | | vLLM-Omni | 1 | | 1.47 | | | | Diffusers | — | | | | +| | TensorRT-LLM | — | | | | | **B300** | PyTorch | — | | | | | | vLLM-Omni | 1 | | 2.15 | | | | Diffusers | — | | | | +| | TensorRT-LLM | — | | | | Additional-modality notes: 1. PyTorch values are average generation (sampling) latency in seconds from PBR `#308197`; lower is better. diff --git a/inference_benchmarks/cosmos3-super-generator.md b/inference_benchmarks/cosmos3-super-generator.md index ff4fb187..c96c7490 100644 --- a/inference_benchmarks/cosmos3-super-generator.md +++ b/inference_benchmarks/cosmos3-super-generator.md @@ -34,9 +34,9 @@ Empty cells mean that a run has not been completed for that GPU, engine, or reso ## Benchmark methodology -The primary t2v, i2v, and t2i tables cover PyTorch, vLLM-Omni, Diffusers, and NIM. Their vLLM-Omni values are from PBR `#307999`, **Cosmos3-Generator vLLM-Omni Inference Benchmarking (t2i, t2v, i2v)**; the PyTorch and NIM values preserve the previously published campaigns. Those tables use BF16 precision, batch size 1, and matched prompts, seeds, and sampler settings where documented. Video workloads follow the standard Cosmos3 generation profile of 189 frames at 24 FPS unless a resolution tier limits frame count. +The primary t2v, i2v, and t2i tables cover PyTorch, vLLM-Omni, Diffusers, TensorRT-LLM, and NIM. Their vLLM-Omni values are from PBR `#307999`, **Cosmos3-Generator vLLM-Omni Inference Benchmarking (t2i, t2v, i2v)**; the PyTorch and NIM values preserve the previously published campaigns. Those tables use BF16 precision, batch size 1, and matched prompts, seeds, and sampler settings where documented. Video workloads follow the standard Cosmos3 generation profile of 189 frames at 24 FPS unless a resolution tier limits frame count. -The additional audiovisual and action tables come from three internal benchmark reports. PyTorch values are from PBR `#308197`, **Cosmos3-Generator OSS Inference Benchmarking 32B and 8B (189 frames)**: average generation (sampling) latency from the native OSS path, using **CUDA Graphs disabled** and the **latency** automatic-sharding preset. vLLM-Omni values for text-to-audio-and-video (`t2av`/`t2vs`) and image-to-audio-and-video (`i2av`/`i2vs`) are from PBR `#308195`. vLLM-Omni action values are from PBR `#308481`, **Cosmos3-Generator vLLM-Omni Inference Benchmarking (action)**, measured with the `vllm/vllm-omni:cosmos3` image and the official action cookbook samples; `DIFFUSION_ATTENTION_BACKEND=TORCH_SDPA` was not set. Forward-dynamics cells are the mean of `av_forward`, `av_left`, and `av_right`; inverse-dynamics cells are the mean of `av_inverse_0` and `av_inverse_1`. That action sweep covers 1, 2, and 4 GPUs only, because its 8-GPU Ulysses runs failed a sequence-length divisibility check; those rows therefore report the single-GPU configuration. Policy-DROID is a separate checkpoint and was measured only at 480p on one GPU. That action sweep did not cover Super on H20, H100 NVL, or H100 80GB HBM3. Diffusers values come from four further reports: PBR `#308202` (all modalities) and PBR `#308451` (video-to-video) on one GPU, PBR `#308918` for 4- and 8-GPU runs, and PBR `#308587` for transfer. vLLM-Omni transfer values are from PBR `#308574`. Where PBR `#308918` offers several tensor- and context-parallel splits at the same GPU count, the fastest is published. Its 1-GPU numbers are not used, because that sweep ran a different denoising-step budget than the single-GPU reports. Values are rounded to two decimal places. +The additional audiovisual and action tables come from three internal benchmark reports. PyTorch values are from PBR `#308197`, **Cosmos3-Generator OSS Inference Benchmarking 32B and 8B (189 frames)**: average generation (sampling) latency from the native OSS path, using **CUDA Graphs disabled** and the **latency** automatic-sharding preset. vLLM-Omni values for text-to-audio-and-video (`t2av`/`t2vs`) and image-to-audio-and-video (`i2av`/`i2vs`) are from PBR `#308195`. vLLM-Omni action values are from PBR `#308481`, **Cosmos3-Generator vLLM-Omni Inference Benchmarking (action)**, measured with the `vllm/vllm-omni:cosmos3` image and the official action cookbook samples; `DIFFUSION_ATTENTION_BACKEND=TORCH_SDPA` was not set. Forward-dynamics cells are the mean of `av_forward`, `av_left`, and `av_right`; inverse-dynamics cells are the mean of `av_inverse_0` and `av_inverse_1`. That action sweep covers 1, 2, and 4 GPUs only, because its 8-GPU Ulysses runs failed a sequence-length divisibility check; those rows therefore report the single-GPU configuration. Policy-DROID is a separate checkpoint and was measured only at 480p on one GPU. That action sweep did not cover Super on H20, H100 NVL, or H100 80GB HBM3. Diffusers values come from four further reports: PBR `#308202` (all modalities) and PBR `#308451` (video-to-video) on one GPU, PBR `#308918` for 4- and 8-GPU runs, and PBR `#308587` for transfer. vLLM-Omni transfer values are from PBR `#308574`. Where PBR `#308918` offers several tensor- and context-parallel splits at the same GPU count, the fastest is published. Its 1-GPU numbers are not used, because that sweep ran a different denoising-step budget than the single-GPU reports. TensorRT-LLM values are from PBR `#308000`, **Cosmos3-Generator TRT-LLM Inference Benchmarking**, which sweeps tensor-parallel, CFG-parallel, and Ulysses-parallel splits at each GPU count; the fastest split at the count a row reports is the one published. That campaign covers six GPUs, so the TensorRT-LLM rows for H100 NVL and H200 NVL are empty, and it includes no action workloads. Values are rounded to two decimal places. These reports establish the reported timing matrix but do not expose every prompt and action payload in this repository. The linked public recipes explain modality behavior and provide representative payloads; their example-specific frame counts and action chunk sizes should not be treated as the exact internal benchmark inputs. @@ -65,34 +65,42 @@ The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound | **RTX PRO 6000 Blackwell** | PyTorch | 8 | 66.04 | 118.90 | 427.16 | | | vLLM-Omni | — | | | | | | Diffusers | 8 | 25.39 | 154.25 | 437.57 | +| | TensorRT-LLM | 8 | 0.21 | 0.13 | 0.10 | | | NIM | 8 | 13.99 | 99.05 | 286.02 | | **H20** | PyTorch | 8 | 27.72 | 152.46 | 492.41 | | | vLLM-Omni | — | | | | | | Diffusers | 8 | 26.66 | 161.49 | 517.37 | +| | TensorRT-LLM | 8 | 0.06 | 0.01 | 0.01 | | | NIM | 8 | 12.95 | 110.71 | 395.56 | | **H100 NVL** | PyTorch | 8 | 16.83 | 64.14 | 186.19 | | | vLLM-Omni | — | | | | | | Diffusers | 8 | 24.21 | 108.70 | 322.28 | +| | TensorRT-LLM | — | | | | | | NIM | 8 | 12.73 | 66.07 | 197.32 | | **H200 NVL** | PyTorch | 8 | 24.78 | 47.69 | 142.35 | | | vLLM-Omni | 1 | 27.54 | 252.33 | 911.49 | | | Diffusers | 1 | 33.00 | 286.80 | 1036.00 | +| | TensorRT-LLM | — | | | | | | NIM | 1 | 17.13 | 200.00 | 811.41 | | **H100 80GB HBM3** | PyTorch | — | | | | | | vLLM-Omni | — | | | | | | Diffusers | 8 | 19.40 | 56.39 | 174.57 | +| | TensorRT-LLM | 8 | 0.04 | 0.02 | 0.04 | | | NIM | 8 | 5.89 | 35.52 | 114.92 | | **H200 141GB HBM3** | PyTorch | 8 | 11.82 | 41.78 | 123.49 | | | vLLM-Omni | 1 | 25.61 | 219.11 | 769.63 | | | Diffusers | 1 | 31.00 | 251.60 | 886.20 | +| | TensorRT-LLM | 1 | 24.21 | 234.08 | 853.44 | | | NIM | 1 | 15.95 | 174.71 | 695.89 | | **B200** | PyTorch | 1 | 14.66 | 114.38 | 407.50 | | | vLLM-Omni | 1 | 13.84 | 114.08 | 383.84 | | | Diffusers | 1 | 19.00 | 127.20 | 414.40 | +| | TensorRT-LLM | 1 | 12.32 | 104.63 | 367.12 | | | NIM | 1 | 9.09 | 82.39 | 314.68 | | **B300** | PyTorch | — | | | | | | vLLM-Omni | 1 | 14.57 | 109.03 | 366.66 | | | Diffusers | 1 | 17.73 | 125.70 | 398.94 | +| | TensorRT-LLM | 1 | 11.85 | 97.66 | 333.28 | | | NIM | 1 | 9.67 | 79.73 | 292.35 | ### Image-to-Video (i2v) @@ -102,34 +110,42 @@ The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound | **RTX PRO 6000 Blackwell** | PyTorch | 8 | 66.54 | 118.95 | 427.96 | | | vLLM-Omni | 8 | 11.66 | 80.94 | 240.27 | | | Diffusers | 8 | 26.29 | 156.16 | 540.65 | +| | TensorRT-LLM | 8 | 0.19 | 0.13 | 0.09 | | | NIM | 8 | 14.48 | 100.23 | 289.93 | | **H20** | PyTorch | 8 | 28.22 | 153.19 | 491.93 | | | vLLM-Omni | 8 | 19.65 | 148.77 | 488.51 | | | Diffusers | 8 | 27.65 | 165.91 | 530.44 | +| | TensorRT-LLM | 8 | 0.06 | 0.01 | 0.01 | | | NIM | 8 | 14.06 | 114.49 | 405.68 | | **H100 NVL** | PyTorch | 8 | 16.96 | 64.17 | 186.47 | | | vLLM-Omni | 8 | 8.48 | 59.44 | 212.53 | | | Diffusers | 8 | 25.45 | 112.36 | 325.56 | +| | TensorRT-LLM | — | | | | | | NIM | 8 | 13.30 | 67.34 | 201.39 | | **H200 NVL** | PyTorch | 8 | 24.70 | 47.60 | 141.62 | | | vLLM-Omni | 1 | 27.90 | 254.29 | 915.05 | | | Diffusers | 1 | 33.00 | 287.20 | 1034.60 | +| | TensorRT-LLM | — | | | | | | NIM | 1 | 17.51 | 201.45 | 817.35 | | **H100 80GB HBM3** | PyTorch | — | | | | | | vLLM-Omni | 8 | 6.48 | 38.31 | | | | Diffusers | 8 | 19.98 | 55.63 | 184.81 | +| | TensorRT-LLM | 8 | 0.03 | 0.03 | 0.04 | | | NIM | 8 | 6.49 | 36.81 | 118.77 | | **H200 141GB HBM3** | PyTorch | 8 | 11.80 | 42.10 | 123.57 | | | vLLM-Omni | 1 | 25.47 | 220.70 | 766.33 | | | Diffusers | 1 | 31.00 | 249.20 | 879.20 | +| | TensorRT-LLM | 1 | 24.46 | 230.02 | | | | NIM | 1 | 16.39 | 175.95 | 699.13 | | **B200** | PyTorch | 1 | 14.71 | 112.40 | 397.31 | | | vLLM-Omni | 1 | 14.13 | 115.17 | 386.35 | | | Diffusers | 1 | 19.20 | 127.00 | 414.80 | +| | TensorRT-LLM | 1 | 12.67 | 105.74 | 372.60 | | | NIM | 1 | 9.36 | 83.19 | 316.76 | | **B300** | PyTorch | — | | | | | | vLLM-Omni | 1 | 14.14 | 111.42 | 368.73 | | | Diffusers | 1 | 25.21 | 132.23 | 410.81 | +| | TensorRT-LLM | 1 | 12.15 | 98.90 | 336.20 | | | NIM | 1 | 9.73 | 80.51 | 294.11 | ### Text-to-Image (t2i) @@ -139,31 +155,39 @@ The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound | **RTX PRO 6000 Blackwell** | PyTorch | 8 | 93.18 | 93.39 | 93.47 | | | vLLM-Omni | 8 | 3.11 | 3.54 | 5.70 | | | Diffusers | 8 | 16.90 | 17.52 | 16.73 | +| | TensorRT-LLM | 8 | 0.09 | 0.09 | -0.04 | | **H20** | PyTorch | 8 | 14.18 | 16.46 | 20.92 | | | vLLM-Omni | 8 | 5.12 | 5.44 | 8.85 | | | Diffusers | 8 | 18.29 | 18.48 | 18.46 | +| | TensorRT-LLM | 8 | 0.02 | 0.13 | 0.03 | | **H100 NVL** | PyTorch | 8 | 19.86 | 19.80 | 19.87 | | | vLLM-Omni | 8 | 3.54 | 3.48 | 4.11 | | | Diffusers | 8 | 16.56 | 16.60 | 16.88 | +| | TensorRT-LLM | — | | | | | **H200 NVL** | PyTorch | 8 | 32.86 | 33.05 | 33.16 | | | vLLM-Omni | 1 | 2.73 | 6.28 | 11.02 | | | Diffusers | 1 | 5.00 | 8.00 | 12.00 | +| | TensorRT-LLM | — | | | | | **H100 80GB HBM3** | PyTorch | — | | | | | | vLLM-Omni | 8 | 5.09 | 5.16 | 4.91 | | | Diffusers | 8 | 17.97 | 18.03 | 19.28 | +| | TensorRT-LLM | 8 | 0.06 | 0.02 | 0.02 | | **H200 141GB HBM3** | PyTorch | 8 | 13.48 | 13.53 | 13.50 | | | vLLM-Omni | 1 | 2.83 | 5.70 | 10.24 | | | Diffusers | 1 | 5.00 | 8.00 | 11.00 | +| | TensorRT-LLM | 1 | 2.24 | 5.10 | 10.77 | | **B200** | PyTorch | 1 | 4.51 | 4.78 | 7.25 | | | vLLM-Omni | 1 | 2.32 | 3.29 | 5.99 | | | Diffusers | 1 | 4.40 | 6.00 | 8.00 | +| | TensorRT-LLM | 1 | 1.69 | 2.84 | 5.72 | | **B300** | PyTorch | — | | | | | | vLLM-Omni | 1 | 5.05 | 3.79 | 7.08 | | | Diffusers | 1 | 6.49 | 6.76 | 7.11 | +| | TensorRT-LLM | 1 | 2.40 | 2.77 | 5.43 | Notes: 1. All times measured on identical workloads (same seed, sampler settings, prompt). -2. Multi-GPU configurations use tensor parallelism. +2. Multi-GPU configurations use tensor parallelism, except TensorRT-LLM, which may also split across CFG and Ulysses dimensions. 3. vLLM-Omni numbers are for the upcoming public release in the vLLM-Omni repo and are subject to change before GA. 4. Diffusers numbers use the HuggingFace `diffusers` integration without custom CUDA graphs. Single-GPU values come from PBR `#308202` and PBR `#308451`; multi-GPU values from PBR `#308918`. 5. At 256p, multi-GPU configurations on B300 may underperform single-GPU because of small-workload tensor-parallel overhead, so single-GPU is the recommended deployment at this resolution. @@ -182,27 +206,35 @@ A text prompt and source video condition a generated continuation or transformat | **RTX PRO 6000 Blackwell** | PyTorch | 8 | 66.31 | 118.92 | 427.19 | | | vLLM-Omni | — | | | | | | Diffusers | 8 | 28.39 | 158.14 | 545.45 | +| | TensorRT-LLM | 8 | 0.21 | 0.13 | 0.10 | | **H20** | PyTorch | 8 | 27.98 | 153.02 | 491.75 | | | vLLM-Omni | — | | | | | | Diffusers | 8 | 28.18 | 167.95 | 532.68 | +| | TensorRT-LLM | 8 | 0.06 | 0.02 | 0.01 | | **H100 NVL** | PyTorch | 8 | 16.79 | 63.84 | 183.64 | | | vLLM-Omni | — | | | | | | Diffusers | 8 | 25.01 | 112.47 | 348.31 | +| | TensorRT-LLM | — | | | | | **H200 NVL** | PyTorch | 8 | 24.74 | 47.60 | 142.23 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 40.73 | 316.00 | 1093.16 | +| | TensorRT-LLM | — | | | | | **H100 80GB HBM3** | PyTorch | — | | | | | | vLLM-Omni | — | | | | | | Diffusers | 8 | 21.61 | 57.74 | 188.43 | +| | TensorRT-LLM | 8 | 0.01 | 0.03 | 0.03 | | **H200 141GB HBM3** | PyTorch | 8 | 11.76 | 41.85 | 123.09 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 41.17 | 269.44 | 916.41 | +| | TensorRT-LLM | 1 | 24.19 | 230.98 | | | **B200** | PyTorch | 1 | 14.30 | 111.49 | 395.97 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 26.30 | 142.61 | 437.27 | +| | TensorRT-LLM | 1 | 12.50 | 104.79 | 370.21 | | **B300** | PyTorch | — | | | | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 25.86 | 139.04 | 416.98 | +| | TensorRT-LLM | 1 | 11.95 | 97.75 | 333.72 | ### Text-to-Audio-and-Video (t2av) @@ -213,27 +245,35 @@ A text prompt produces synchronized video and sound. | **RTX PRO 6000 Blackwell** | PyTorch | 8 | 65.79 | 118.87 | 429.78 | | | vLLM-Omni | — | | | | | | Diffusers | — | | | | +| | TensorRT-LLM | 8 | 0.21 | 0.13 | 0.10 | | **H20** | PyTorch | 8 | 27.73 | 152.53 | 492.38 | | | vLLM-Omni | — | | | | | | Diffusers | — | | | | +| | TensorRT-LLM | 8 | 0.05 | 0.01 | 0.01 | | **H100 NVL** | PyTorch | 8 | 16.86 | 64.17 | 183.81 | | | vLLM-Omni | — | | | | | | Diffusers | — | | | | +| | TensorRT-LLM | — | | | | | **H200 NVL** | PyTorch | 8 | 24.79 | 47.67 | 142.06 | | | vLLM-Omni | 1 | 29.07 | 260.51 | 916.51 | | | Diffusers | 1 | 35.00 | 291.40 | 1034.20 | +| | TensorRT-LLM | — | | | | | **H100 80GB HBM3** | PyTorch | — | | | | | | vLLM-Omni | — | | | | | | Diffusers | — | | | | +| | TensorRT-LLM | — | | | | | **H200 141GB HBM3** | PyTorch | 8 | 11.74 | 41.49 | 123.35 | | | vLLM-Omni | 1 | 27.21 | 221.67 | 763.74 | | | Diffusers | 1 | 33.00 | 253.40 | 874.60 | +| | TensorRT-LLM | 1 | 25.89 | 229.12 | | | **B200** | PyTorch | 1 | 14.52 | 112.99 | 395.20 | | | vLLM-Omni | 1 | 14.77 | 115.26 | 388.51 | | | Diffusers | 1 | 20.00 | 128.00 | 422.00 | +| | TensorRT-LLM | 1 | 13.37 | 105.39 | 371.64 | | **B300** | PyTorch | — | | | | | | vLLM-Omni | 1 | 15.26 | 113.34 | 372.74 | | | Diffusers | 1 | 18.94 | 126.86 | 402.37 | +| | TensorRT-LLM | 1 | 12.76 | 98.59 | 335.38 | ### Video-to-Audio-and-Video (v2av) @@ -244,27 +284,35 @@ A text prompt and source video produce transformed video with synchronized sound | **RTX PRO 6000 Blackwell** | PyTorch | 8 | 66.10 | 118.88 | 428.64 | | | vLLM-Omni | — | | | | | | Diffusers | — | | | | +| | TensorRT-LLM | — | | | | | **H20** | PyTorch | 8 | 28.00 | 152.78 | 492.63 | | | vLLM-Omni | — | | | | | | Diffusers | — | | | | +| | TensorRT-LLM | — | | | | | **H100 NVL** | PyTorch | 8 | 16.79 | 64.69 | 183.62 | | | vLLM-Omni | — | | | | | | Diffusers | — | | | | +| | TensorRT-LLM | — | | | | | **H200 NVL** | PyTorch | 8 | 24.73 | 47.67 | 143.06 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 48.42 | 320.70 | 1093.26 | +| | TensorRT-LLM | — | | | | | **H100 80GB HBM3** | PyTorch | — | | | | | | vLLM-Omni | — | | | | | | Diffusers | — | | | | +| | TensorRT-LLM | — | | | | | **H200 141GB HBM3** | PyTorch | 8 | 11.74 | 41.59 | 123.06 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 43.23 | 270.19 | 909.99 | +| | TensorRT-LLM | — | | | | | **B200** | PyTorch | 1 | 14.28 | 111.84 | 406.66 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 30.26 | 141.75 | 446.20 | +| | TensorRT-LLM | — | | | | | **B300** | PyTorch | — | | | | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 30.36 | 138.17 | 420.63 | +| | TensorRT-LLM | — | | | | ### Image-to-Audio-and-Video (i2av) @@ -275,27 +323,35 @@ A text prompt and source image produce video with synchronized sound. | **RTX PRO 6000 Blackwell** | PyTorch | 8 | 66.27 | 119.05 | 429.26 | | | vLLM-Omni | — | | | | | | Diffusers | — | | | | +| | TensorRT-LLM | 8 | 0.18 | 0.13 | 0.10 | | **H20** | PyTorch | 8 | 28.11 | 153.33 | 493.51 | | | vLLM-Omni | — | | | | | | Diffusers | — | | | | +| | TensorRT-LLM | 8 | 0.05 | 0.01 | 0.01 | | **H100 NVL** | PyTorch | 8 | 16.85 | 64.19 | 186.66 | | | vLLM-Omni | — | | | | | | Diffusers | — | | | | +| | TensorRT-LLM | — | | | | | **H200 NVL** | PyTorch | 8 | 24.69 | 47.65 | 142.18 | | | vLLM-Omni | 1 | 29.42 | 262.20 | 922.18 | | | Diffusers | 1 | 35.00 | 292.20 | 1037.00 | +| | TensorRT-LLM | — | | | | | **H100 80GB HBM3** | PyTorch | — | | | | | | vLLM-Omni | — | | | | | | Diffusers | — | | | | +| | TensorRT-LLM | — | | | | | **H200 141GB HBM3** | PyTorch | 8 | 11.82 | 41.95 | 123.42 | | | vLLM-Omni | 1 | 27.58 | 220.57 | 774.15 | | | Diffusers | 1 | 33.00 | 256.00 | 880.40 | +| | TensorRT-LLM | 1 | 26.41 | 233.74 | | | **B200** | PyTorch | 1 | 14.84 | 113.32 | 407.30 | | | vLLM-Omni | 1 | 15.15 | 116.54 | 396.11 | | | Diffusers | 1 | 20.00 | 128.00 | 423.00 | +| | TensorRT-LLM | 1 | 13.65 | 106.62 | 374.24 | | **B300** | PyTorch | — | | | | | | vLLM-Omni | 1 | 16.46 | 114.02 | 373.17 | | | Diffusers | 1 | 29.16 | 134.46 | 414.22 | +| | TensorRT-LLM | 1 | 13.25 | 99.94 | 338.12 | ## Transfer generation @@ -305,106 +361,250 @@ the hint changes how much of the frame the model must synthesise. | GPU | Transfer control | Engine | GPUs | 256p | 480p | 720p | |---|---|---|:-:|---:|---:|---:| -| **RTX PRO 6000 Blackwell** | blur | Diffusers | — | | | | +| **RTX PRO 6000 Blackwell** | blur | PyTorch | 4 | | | | +| | | vLLM-Omni | 4 | | | | +| | | Diffusers | 4 | | | | +| | | TensorRT-LLM | 4 | -0.19 | -0.28 | -0.37 | +| | | NIM | 4 | | | | +| | depth | PyTorch | 4 | | | | +| | | vLLM-Omni | 4 | | | | +| | | Diffusers | 4 | | | | +| | | TensorRT-LLM | 4 | -0.19 | -0.28 | -0.37 | +| | | NIM | 4 | | | | +| | edge | PyTorch | 4 | | | | +| | | vLLM-Omni | 4 | | | | +| | | Diffusers | 4 | | | | +| | | TensorRT-LLM | 4 | -0.19 | -0.28 | -0.37 | +| | | NIM | 4 | | | | +| | multi_control | PyTorch | 4 | | | | +| | | vLLM-Omni | 4 | | | | +| | | Diffusers | 4 | | | | +| | | TensorRT-LLM | 4 | -0.19 | -0.32 | -0.40 | +| | | NIM | 4 | | | | +| | seg | PyTorch | 4 | | | | +| | | vLLM-Omni | 4 | | | | +| | | Diffusers | 4 | | | | +| | | TensorRT-LLM | 4 | -0.19 | -0.28 | -0.37 | +| | | NIM | 4 | | | | +| | wsm | PyTorch | 4 | | | | +| | | vLLM-Omni | 4 | | | | +| | | Diffusers | 4 | | | | +| | | TensorRT-LLM | 4 | -0.15 | -0.27 | -0.34 | +| | | NIM | 4 | | | | +| **H20** | blur | PyTorch | 4 | | | | +| | | vLLM-Omni | 4 | | | | +| | | Diffusers | 4 | | | | +| | | TensorRT-LLM | 4 | -0.46 | -0.49 | -0.49 | +| | | NIM | 4 | | | | +| | depth | PyTorch | 4 | | | | +| | | vLLM-Omni | 4 | | | | +| | | Diffusers | 4 | | | | +| | | TensorRT-LLM | 4 | -0.46 | -0.49 | -0.49 | +| | | NIM | 4 | | | | +| | edge | PyTorch | 4 | | | | +| | | vLLM-Omni | 4 | | | | +| | | Diffusers | 4 | | | | +| | | TensorRT-LLM | 4 | -0.46 | -0.49 | -0.49 | +| | | NIM | 4 | | | | +| | multi_control | PyTorch | 4 | | | | +| | | vLLM-Omni | 4 | | | | +| | | Diffusers | 4 | | | | +| | | TensorRT-LLM | 4 | -0.47 | -0.49 | -0.50 | +| | | NIM | 4 | | | | +| | seg | PyTorch | 4 | | | | +| | | vLLM-Omni | 4 | | | | +| | | Diffusers | 4 | | | | +| | | TensorRT-LLM | 4 | -0.46 | 633.73 | -0.49 | +| | | NIM | 4 | | | | +| | wsm | PyTorch | 4 | | | | +| | | vLLM-Omni | 4 | | | | +| | | Diffusers | 4 | | | | +| | | TensorRT-LLM | 4 | -0.46 | -0.48 | -0.49 | +| | | NIM | 4 | | | | +| **H100 NVL** | blur | PyTorch | — | | | | | | | vLLM-Omni | — | | | | -| | depth | Diffusers | — | | | | +| | | Diffusers | — | | | | +| | | TensorRT-LLM | — | | | | +| | | NIM | — | | | | +| | depth | PyTorch | — | | | | | | | vLLM-Omni | — | | | | -| | edge | Diffusers | — | | | | +| | | Diffusers | — | | | | +| | | TensorRT-LLM | — | | | | +| | | NIM | — | | | | +| | edge | PyTorch | — | | | | | | | vLLM-Omni | — | | | | -| | multi_control | Diffusers | — | | | | +| | | Diffusers | — | | | | +| | | TensorRT-LLM | — | | | | +| | | NIM | — | | | | +| | multi_control | PyTorch | — | | | | | | | vLLM-Omni | — | | | | -| | seg | Diffusers | — | | | | +| | | Diffusers | — | | | | +| | | TensorRT-LLM | — | | | | +| | | NIM | — | | | | +| | seg | PyTorch | — | | | | | | | vLLM-Omni | — | | | | -| | wsm | Diffusers | — | | | | +| | | Diffusers | — | | | | +| | | TensorRT-LLM | — | | | | +| | | NIM | — | | | | +| | wsm | PyTorch | — | | | | | | | vLLM-Omni | — | | | | -| **H20** | blur | Diffusers | — | | | | -| | | vLLM-Omni | — | | | | -| | depth | Diffusers | — | | | | -| | | vLLM-Omni | — | | | | -| | edge | Diffusers | — | | | | -| | | vLLM-Omni | — | | | | -| | multi_control | Diffusers | — | | | | -| | | vLLM-Omni | — | | | | -| | seg | Diffusers | — | | | | -| | | vLLM-Omni | — | | | | -| | wsm | Diffusers | — | | | | -| | | vLLM-Omni | — | | | | -| **H100 NVL** | blur | Diffusers | — | | | | -| | | vLLM-Omni | — | | | | -| | depth | Diffusers | — | | | | -| | | vLLM-Omni | — | | | | -| | edge | Diffusers | — | | | | -| | | vLLM-Omni | — | | | | -| | multi_control | Diffusers | — | | | | -| | | vLLM-Omni | — | | | | -| | seg | Diffusers | — | | | | -| | | vLLM-Omni | — | | | | -| | wsm | Diffusers | — | | | | -| | | vLLM-Omni | — | | | | -| **H100 80GB HBM3** | blur | Diffusers | — | | | | -| | | vLLM-Omni | — | | | | -| | depth | Diffusers | — | | | | -| | | vLLM-Omni | — | | | | -| | edge | Diffusers | — | | | | -| | | vLLM-Omni | — | | | | -| | multi_control | Diffusers | — | | | | -| | | vLLM-Omni | — | | | | -| | seg | Diffusers | — | | | | -| | | vLLM-Omni | — | | | | -| | wsm | Diffusers | — | | | | -| | | vLLM-Omni | — | | | | -| **H200 NVL** | blur | Diffusers | 1 | 83.43 | 551.97 | 1935.10 | +| | | Diffusers | — | | | | +| | | TensorRT-LLM | — | | | | +| | | NIM | — | | | | +| **H100 80GB HBM3** | blur | PyTorch | 4 | | | | +| | | vLLM-Omni | 4 | | | | +| | | Diffusers | 4 | | | | +| | | TensorRT-LLM | 4 | 23.08 | 169.42 | 604.47 | +| | | NIM | 4 | | | | +| | depth | PyTorch | 4 | | | | +| | | vLLM-Omni | 4 | | | | +| | | Diffusers | 4 | | | | +| | | TensorRT-LLM | 4 | 23.27 | 173.32 | 611.69 | +| | | NIM | 4 | | | | +| | edge | PyTorch | 4 | | | | +| | | vLLM-Omni | 4 | | | | +| | | Diffusers | 4 | | | | +| | | TensorRT-LLM | 4 | 23.04 | 170.63 | 603.31 | +| | | NIM | 4 | | | | +| | multi_control | PyTorch | 4 | | | | +| | | vLLM-Omni | 4 | | | | +| | | Diffusers | 4 | | | | +| | | TensorRT-LLM | 4 | 31.56 | 283.95 | 1092.62 | +| | | NIM | 4 | | | | +| | seg | PyTorch | 4 | | | | +| | | vLLM-Omni | 4 | | | | +| | | Diffusers | 4 | | | | +| | | TensorRT-LLM | 4 | 23.04 | 167.31 | 608.35 | +| | | NIM | 4 | | | | +| | wsm | PyTorch | 4 | | | | +| | | vLLM-Omni | 4 | | | | +| | | Diffusers | 4 | | | | +| | | TensorRT-LLM | 4 | 15.04 | 78.98 | 262.98 | +| | | NIM | 4 | | | | +| **H200 NVL** | blur | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 85.61 | 610.21 | 2199.03 | -| | depth | Diffusers | 1 | 98.52 | 572.97 | 1970.31 | +| | | Diffusers | 1 | 83.43 | 551.97 | 1935.10 | +| | | TensorRT-LLM | 1 | | | | +| | | NIM | 1 | | | | +| | depth | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 63.72 | 622.24 | 2255.18 | -| | edge | Diffusers | 1 | 84.72 | 552.34 | 1940.58 | +| | | Diffusers | 1 | 98.52 | 572.97 | 1970.31 | +| | | TensorRT-LLM | 1 | | | | +| | | NIM | 1 | | | | +| | edge | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 61.97 | 612.63 | 2238.40 | -| | multi_control | Diffusers | — | | | | +| | | Diffusers | 1 | 84.72 | 552.34 | 1940.58 | +| | | TensorRT-LLM | 1 | | | | +| | | NIM | 1 | | | | +| | multi_control | PyTorch | 8 | | | | | | | vLLM-Omni | 8 | 57.19 | | 2432.02 | -| | seg | Diffusers | 1 | 88.16 | 555.93 | 1950.38 | +| | | Diffusers | 8 | | | | +| | | TensorRT-LLM | 8 | | | | +| | | NIM | 8 | | | | +| | seg | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 62.69 | 614.55 | 2239.94 | -| | wsm | Diffusers | 1 | 80.77 | 445.36 | 1470.27 | +| | | Diffusers | 1 | 88.16 | 555.93 | 1950.38 | +| | | TensorRT-LLM | 1 | | | | +| | | NIM | 1 | | | | +| | wsm | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 52.06 | 481.65 | 1670.02 | -| **H200 141GB HBM3** | blur | Diffusers | 1 | 71.83 | 463.77 | 1593.34 | +| | | Diffusers | 1 | 80.77 | 445.36 | 1470.27 | +| | | TensorRT-LLM | 1 | | | | +| | | NIM | 1 | | | | +| **H200 141GB HBM3** | blur | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 77.82 | 515.99 | 1824.12 | -| | depth | Diffusers | 1 | 84.88 | 481.84 | 1618.53 | +| | | Diffusers | 1 | 71.83 | 463.77 | 1593.34 | +| | | TensorRT-LLM | 1 | 57.89 | 580.64 | 2176.16 | +| | | NIM | 1 | | | | +| | depth | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 58.38 | 528.43 | 1875.62 | -| | edge | Diffusers | 1 | 73.22 | 466.10 | 1594.90 | +| | | Diffusers | 1 | 84.88 | 481.84 | 1618.53 | +| | | TensorRT-LLM | 1 | 59.25 | 588.29 | 2193.25 | +| | | NIM | 1 | | | | +| | edge | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 55.80 | 527.66 | 1825.62 | -| | multi_control | Diffusers | — | | | | +| | | Diffusers | 1 | 73.22 | 466.10 | 1594.90 | +| | | TensorRT-LLM | 1 | 58.54 | 580.81 | 2176.79 | +| | | NIM | 1 | | | | +| | multi_control | PyTorch | 8 | | | | | | | vLLM-Omni | 8 | 58.24 | | | -| | seg | Diffusers | 1 | 77.09 | 476.15 | 1598.31 | +| | | Diffusers | 8 | | | | +| | | TensorRT-LLM | 8 | -0.40 | -0.48 | -0.49 | +| | | NIM | 8 | | | | +| | seg | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 57.91 | 528.52 | 1826.29 | -| | wsm | Diffusers | 1 | 70.53 | 382.72 | 1216.43 | +| | | Diffusers | 1 | 77.09 | 476.15 | 1598.31 | +| | | TensorRT-LLM | 1 | 58.32 | 571.37 | 2166.54 | +| | | NIM | 1 | | | | +| | wsm | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 48.37 | 416.84 | 1406.07 | -| **B200** | blur | Diffusers | 1 | 42.14 | 241.28 | 789.92 | +| | | Diffusers | 1 | 70.53 | 382.72 | 1216.43 | +| | | TensorRT-LLM | 1 | 29.19 | 260.21 | 929.38 | +| | | NIM | 1 | | | | +| **B200** | blur | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 42.90 | 264.79 | 943.38 | -| | depth | Diffusers | 1 | 49.41 | 254.68 | 792.02 | +| | | Diffusers | 1 | 42.14 | 241.28 | 789.92 | +| | | TensorRT-LLM | 1 | 30.02 | 258.30 | 939.00 | +| | | NIM | 1 | | | | +| | depth | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 32.81 | 268.01 | 955.14 | -| | edge | Diffusers | 1 | 43.57 | 241.17 | 774.34 | +| | | Diffusers | 1 | 49.41 | 254.68 | 792.02 | +| | | TensorRT-LLM | 1 | 30.57 | 261.34 | 926.74 | +| | | NIM | 1 | | | | +| | edge | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 31.94 | 263.68 | 946.09 | -| | multi_control | Diffusers | — | | | | +| | | Diffusers | 1 | 43.57 | 241.17 | 774.34 | +| | | TensorRT-LLM | 1 | 29.67 | 258.50 | 919.21 | +| | | NIM | 1 | | | | +| | multi_control | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 73.81 | 934.97 | 4038.87 | -| | seg | Diffusers | 1 | 45.43 | 243.27 | 792.36 | +| | | Diffusers | 1 | | | | +| | | TensorRT-LLM | 1 | 40.45 | 426.01 | 1670.27 | +| | | NIM | 1 | | | | +| | seg | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 33.33 | 264.97 | 959.62 | -| | wsm | Diffusers | 1 | 40.66 | 198.30 | 604.37 | +| | | Diffusers | 1 | 45.43 | 243.27 | 792.36 | +| | | TensorRT-LLM | 1 | 29.81 | 257.14 | 925.09 | +| | | NIM | 1 | | | | +| | wsm | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 27.64 | 209.59 | 718.19 | -| **B300** | blur | Diffusers | 1 | 39.64 | 226.08 | 720.04 | +| | | Diffusers | 1 | 40.66 | 198.30 | 604.37 | +| | | TensorRT-LLM | 1 | 15.03 | 118.56 | 403.95 | +| | | NIM | 1 | | | | +| **B300** | blur | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 44.56 | 251.49 | 863.63 | -| | depth | Diffusers | 1 | 45.83 | 231.26 | 730.46 | +| | | Diffusers | 1 | 39.64 | 226.08 | 720.04 | +| | | TensorRT-LLM | 1 | 28.84 | 239.51 | 836.14 | +| | | NIM | 1 | | | | +| | depth | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 34.74 | 256.94 | 887.06 | -| | edge | Diffusers | 1 | 39.99 | 226.42 | 719.35 | +| | | Diffusers | 1 | 45.83 | 231.26 | 730.46 | +| | | TensorRT-LLM | 1 | 28.63 | 242.30 | 856.41 | +| | | NIM | 1 | | | | +| | edge | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 34.13 | 253.19 | 879.84 | -| | multi_control | Diffusers | — | | | | +| | | Diffusers | 1 | 39.99 | 226.42 | 719.35 | +| | | TensorRT-LLM | 1 | 28.70 | 243.59 | 835.37 | +| | | NIM | 1 | | | | +| | multi_control | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 73.12 | 864.66 | 3687.17 | -| | seg | Diffusers | 1 | 41.64 | 228.18 | 720.63 | +| | | Diffusers | 1 | | | | +| | | TensorRT-LLM | 1 | 38.82 | 390.38 | 1500.60 | +| | | NIM | 1 | | | | +| | seg | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 34.88 | 252.30 | 878.95 | -| | wsm | Diffusers | 1 | 38.40 | 186.76 | 555.47 | +| | | Diffusers | 1 | 41.64 | 228.18 | 720.63 | +| | | TensorRT-LLM | 1 | 28.66 | 239.26 | 834.49 | +| | | NIM | 1 | | | | +| | wsm | PyTorch | 1 | | | | | | | vLLM-Omni | 1 | 27.58 | 201.45 | 665.09 | +| | | Diffusers | 1 | 38.40 | 186.76 | 555.47 | +| | | TensorRT-LLM | 1 | 14.47 | 112.87 | 372.96 | +| | | NIM | 1 | | | | Transfer notes: -1. Values are average generation latency in seconds; lower is better. Diffusers numbers come from PBR `#308587`, vLLM-Omni from PBR `#308574`. -2. The **GPUs** column gives the number of GPUs behind every value in that row. +1. Values are average generation latency in seconds; lower is better. Diffusers numbers come from PBR `#308587`, vLLM-Omni from PBR `#308574`, and TensorRT-LLM from PBR `#308000`. +2. The **GPUs** column gives the number of GPUs behind every value in that row, and is the same for every engine in a GPU block so the rows can be compared directly. 3. PyTorch and NIM have no transfer coverage yet; those rows are unmeasured, not unsupported. 4. Control hints follow the vLLM-Omni `extra_params` names: `blur`, `depth`, `edge`, `multi_control`, `seg`, and `wsm`. `multi_control` combines several hints in one request. @@ -419,27 +619,35 @@ Forward dynamics is reported separately for AV, camera, and robot inputs. Invers | **RTX PRO 6000 Blackwell** | PyTorch | 8 | 59.39 | 59.31 | 59.35 | | | vLLM-Omni | — | | | | | | Diffusers | 8 | 11.99 | 22.93 | 23.68 | +| | TensorRT-LLM | — | | | | | **H20** | PyTorch | 8 | 41.51 | 41.47 | 41.42 | | | vLLM-Omni | — | | | | | | Diffusers | 8 | 11.40 | 26.98 | 28.79 | +| | TensorRT-LLM | — | | | | | **H100 NVL** | PyTorch | 8 | 21.00 | 21.00 | 21.04 | | | vLLM-Omni | — | | | | | | Diffusers | 8 | 10.30 | 19.29 | 20.05 | +| | TensorRT-LLM | — | | | | | **H200 NVL** | PyTorch | 8 | 22.66 | 22.67 | 22.65 | | | vLLM-Omni | 1 | 4.33 | 28.78 | 82.08 | | | Diffusers | 1 | 27.17 | 53.19 | 51.89 | +| | TensorRT-LLM | — | | | | | **H100 80GB HBM3** | PyTorch | — | | | | | | vLLM-Omni | — | | | | | | Diffusers | 8 | 10.34 | 12.89 | 13.45 | +| | TensorRT-LLM | — | | | | | **H200 141GB HBM3** | PyTorch | 8 | 14.41 | 14.41 | 14.41 | | | vLLM-Omni | 1 | 3.99 | 26.04 | 71.61 | | | Diffusers | 1 | 26.51 | 47.34 | 50.20 | +| | TensorRT-LLM | — | | | | | **B200** | PyTorch | 1 | 13.59 | 13.55 | 13.86 | | | vLLM-Omni | 1 | 2.37 | 14.59 | 38.41 | | | Diffusers | 1 | 4.00 | 16.00 | 16.00 | +| | TensorRT-LLM | — | | | | | **B300** | PyTorch | — | | | | | | vLLM-Omni | 1 | 2.85 | 14.20 | 36.96 | | | Diffusers | 1 | 22.18 | 35.41 | 38.82 | +| | TensorRT-LLM | — | | | | ### Forward Dynamics — Camera @@ -448,27 +656,35 @@ Forward dynamics is reported separately for AV, camera, and robot inputs. Invers | **RTX PRO 6000 Blackwell** | PyTorch | 8 | 59.62 | 59.39 | 59.55 | | | vLLM-Omni | — | | | | | | Diffusers | — | | | | +| | TensorRT-LLM | — | | | | | **H20** | PyTorch | 8 | 41.13 | 41.10 | 41.11 | | | vLLM-Omni | — | | | | | | Diffusers | — | | | | +| | TensorRT-LLM | — | | | | | **H100 NVL** | PyTorch | 8 | 21.05 | 20.98 | 21.00 | | | vLLM-Omni | — | | | | | | Diffusers | — | | | | +| | TensorRT-LLM | — | | | | | **H200 NVL** | PyTorch | 8 | 22.55 | 22.54 | 22.54 | | | vLLM-Omni | — | | | | | | Diffusers | — | | | | +| | TensorRT-LLM | — | | | | | **H100 80GB HBM3** | PyTorch | — | | | | | | vLLM-Omni | — | | | | | | Diffusers | — | | | | +| | TensorRT-LLM | — | | | | | **H200 141GB HBM3** | PyTorch | 8 | 14.30 | 14.30 | 14.29 | | | vLLM-Omni | — | | | | | | Diffusers | — | | | | +| | TensorRT-LLM | — | | | | | **B200** | PyTorch | 1 | 13.73 | 13.72 | 13.58 | | | vLLM-Omni | — | | | | | | Diffusers | — | | | | +| | TensorRT-LLM | — | | | | | **B300** | PyTorch | — | | | | | | vLLM-Omni | — | | | | | | Diffusers | — | | | | +| | TensorRT-LLM | — | | | | ### Forward Dynamics — Robot @@ -477,27 +693,35 @@ Forward dynamics is reported separately for AV, camera, and robot inputs. Invers | **RTX PRO 6000 Blackwell** | PyTorch | 8 | 56.35 | 56.38 | 56.01 | | | vLLM-Omni | — | | | | | | Diffusers | 8 | 9.40 | 10.60 | 10.82 | +| | TensorRT-LLM | — | | | | | **H20** | PyTorch | 8 | 15.60 | 15.61 | 15.59 | | | vLLM-Omni | — | | | | | | Diffusers | 8 | 8.95 | 10.24 | 10.65 | +| | TensorRT-LLM | — | | | | | **H100 NVL** | PyTorch | 8 | 12.90 | 12.90 | 12.90 | | | vLLM-Omni | — | | | | | | Diffusers | 8 | 8.03 | 8.49 | 8.61 | +| | TensorRT-LLM | — | | | | | **H200 NVL** | PyTorch | 8 | 20.16 | 20.15 | 20.15 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 22.21 | 28.53 | 27.35 | +| | TensorRT-LLM | — | | | | | **H100 80GB HBM3** | PyTorch | — | | | | | | vLLM-Omni | — | | | | | | Diffusers | 8 | 10.27 | 9.04 | 9.25 | +| | TensorRT-LLM | — | | | | | **H200 141GB HBM3** | PyTorch | 8 | 8.74 | 8.74 | 8.70 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 22.02 | 26.45 | 26.38 | +| | TensorRT-LLM | — | | | | | **B200** | PyTorch | 1 | 3.50 | 3.49 | 3.47 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 3.00 | 5.00 | 5.00 | +| | TensorRT-LLM | — | | | | | **B300** | PyTorch | — | | | | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 22.21 | 21.81 | 24.51 | +| | TensorRT-LLM | — | | | | ### Inverse Dynamics — Autonomous Vehicle (AV) @@ -506,27 +730,35 @@ Forward dynamics is reported separately for AV, camera, and robot inputs. Invers | **RTX PRO 6000 Blackwell** | PyTorch | 8 | 59.20 | 59.14 | 59.07 | | | vLLM-Omni | — | | | | | | Diffusers | 8 | 11.15 | 23.33 | 23.28 | +| | TensorRT-LLM | — | | | | | **H20** | PyTorch | 8 | 41.17 | 40.96 | 41.06 | | | vLLM-Omni | — | | | | | | Diffusers | 8 | 11.33 | 26.66 | 28.46 | +| | TensorRT-LLM | — | | | | | **H100 NVL** | PyTorch | 8 | 20.76 | 21.05 | 20.69 | | | vLLM-Omni | — | | | | | | Diffusers | 8 | 9.72 | 18.67 | 20.67 | +| | TensorRT-LLM | — | | | | | **H200 NVL** | PyTorch | 8 | 22.40 | 22.40 | 22.42 | | | vLLM-Omni | 1 | 4.36 | 28.21 | 80.85 | | | Diffusers | 1 | 15.22 | 44.50 | 43.51 | +| | TensorRT-LLM | — | | | | | **H100 80GB HBM3** | PyTorch | — | | | | | | vLLM-Omni | — | | | | | | Diffusers | 8 | 10.49 | 12.73 | 13.30 | +| | TensorRT-LLM | — | | | | | **H200 141GB HBM3** | PyTorch | 8 | 14.30 | 14.32 | 14.16 | | | vLLM-Omni | 1 | 4.03 | 25.99 | 71.45 | | | Diffusers | 1 | 16.58 | 39.75 | 41.71 | +| | TensorRT-LLM | — | | | | | **B200** | PyTorch | 1 | 13.38 | 13.37 | 13.36 | | | vLLM-Omni | 1 | 2.42 | 14.69 | 38.74 | | | Diffusers | 1 | 4.00 | 16.00 | 16.00 | +| | TensorRT-LLM | — | | | | | **B300** | PyTorch | — | | | | | | vLLM-Omni | 1 | 2.69 | 14.44 | 37.70 | | | Diffusers | 1 | 15.13 | 26.24 | 28.51 | +| | TensorRT-LLM | — | | | | ### Inverse Dynamics — Robot @@ -535,27 +767,35 @@ Forward dynamics is reported separately for AV, camera, and robot inputs. Invers | **RTX PRO 6000 Blackwell** | PyTorch | 8 | 56.28 | 56.33 | 56.00 | | | vLLM-Omni | — | | | | | | Diffusers | 8 | 9.59 | 10.90 | 8.71 | +| | TensorRT-LLM | — | | | | | **H20** | PyTorch | 8 | 15.51 | 15.52 | 15.51 | | | vLLM-Omni | — | | | | | | Diffusers | 8 | 9.48 | 10.17 | 10.65 | +| | TensorRT-LLM | — | | | | | **H100 NVL** | PyTorch | 8 | 12.51 | 12.51 | 12.51 | | | vLLM-Omni | — | | | | | | Diffusers | 8 | 7.92 | 8.46 | 8.79 | +| | TensorRT-LLM | — | | | | | **H200 NVL** | PyTorch | 8 | 20.11 | 20.10 | 20.12 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 12.09 | 16.87 | 18.74 | +| | TensorRT-LLM | — | | | | | **H100 80GB HBM3** | PyTorch | — | | | | | | vLLM-Omni | — | | | | | | Diffusers | 8 | 8.71 | 8.93 | 9.08 | +| | TensorRT-LLM | — | | | | | **H200 141GB HBM3** | PyTorch | 8 | 8.68 | 8.68 | 8.65 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 25.77 | 16.25 | 16.29 | +| | TensorRT-LLM | — | | | | | **B200** | PyTorch | 1 | 3.42 | 3.49 | 3.41 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 3.00 | 5.00 | 5.00 | +| | TensorRT-LLM | — | | | | | **B300** | PyTorch | — | | | | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 9.65 | 15.08 | 15.21 | +| | TensorRT-LLM | — | | | | ### Policy — Autonomous Vehicle (AV) @@ -564,27 +804,35 @@ Forward dynamics is reported separately for AV, camera, and robot inputs. Invers | **RTX PRO 6000 Blackwell** | PyTorch | 8 | 59.66 | 59.51 | 59.49 | | | vLLM-Omni | — | | | | | | Diffusers | 8 | 12.11 | 22.60 | 25.76 | +| | TensorRT-LLM | — | | | | | **H20** | PyTorch | 8 | 41.54 | 41.51 | 41.49 | | | vLLM-Omni | — | | | | | | Diffusers | 8 | 11.35 | 26.61 | 28.81 | +| | TensorRT-LLM | — | | | | | **H100 NVL** | PyTorch | 8 | 21.02 | 21.00 | 21.31 | | | vLLM-Omni | — | | | | | | Diffusers | 8 | 9.77 | 18.80 | 20.12 | +| | TensorRT-LLM | — | | | | | **H200 NVL** | PyTorch | 8 | 22.64 | 22.63 | 22.64 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 15.35 | 41.89 | 45.40 | +| | TensorRT-LLM | — | | | | | **H100 80GB HBM3** | PyTorch | — | | | | | | vLLM-Omni | — | | | | | | Diffusers | 8 | 10.55 | 13.02 | 13.52 | +| | TensorRT-LLM | — | | | | | **H200 141GB HBM3** | PyTorch | 8 | 14.54 | 14.41 | 14.52 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 14.71 | 41.30 | 40.35 | +| | TensorRT-LLM | — | | | | | **B200** | PyTorch | 1 | 13.71 | 13.73 | 13.60 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 4.00 | 16.00 | 16.00 | +| | TensorRT-LLM | — | | | | | **B300** | PyTorch | — | | | | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 15.29 | 28.62 | 28.87 | +| | TensorRT-LLM | — | | | | ### Policy — Robot @@ -593,27 +841,35 @@ Forward dynamics is reported separately for AV, camera, and robot inputs. Invers | **RTX PRO 6000 Blackwell** | PyTorch | 8 | 56.31 | 56.29 | 55.95 | | | vLLM-Omni | — | | | | | | Diffusers | 8 | 9.57 | 10.08 | 8.76 | +| | TensorRT-LLM | — | | | | | **H20** | PyTorch | 8 | 15.60 | 15.60 | 15.62 | | | vLLM-Omni | — | | | | | | Diffusers | 8 | 9.03 | 10.19 | 10.73 | +| | TensorRT-LLM | — | | | | | **H100 NVL** | PyTorch | 8 | 12.56 | 12.55 | 12.54 | | | vLLM-Omni | — | | | | | | Diffusers | 8 | 8.05 | 8.60 | 8.89 | +| | TensorRT-LLM | — | | | | | **H200 NVL** | PyTorch | 8 | 20.16 | 20.17 | 20.16 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 13.82 | 16.92 | 17.03 | +| | TensorRT-LLM | — | | | | | **H100 80GB HBM3** | PyTorch | — | | | | | | vLLM-Omni | — | | | | | | Diffusers | 8 | 8.71 | 9.09 | 9.12 | +| | TensorRT-LLM | — | | | | | **H200 141GB HBM3** | PyTorch | 8 | 8.72 | 8.75 | 8.70 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 11.90 | 16.13 | 14.72 | +| | TensorRT-LLM | — | | | | | **B200** | PyTorch | 1 | 3.48 | 3.49 | 3.48 | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 3.00 | 5.00 | 5.00 | +| | TensorRT-LLM | — | | | | | **B300** | PyTorch | — | | | | | | vLLM-Omni | — | | | | | | Diffusers | 1 | 10.57 | 15.31 | 15.33 | +| | TensorRT-LLM | — | | | | ### Policy — DROID @@ -624,27 +880,35 @@ PBR `#308481` measured Policy-DROID only for Cosmos3-Nano-Policy-DROID. Super ha | **RTX PRO 6000 Blackwell** | PyTorch | — | | | | | | vLLM-Omni | — | | | | | | Diffusers | — | | | | +| | TensorRT-LLM | — | | | | | **H20** | PyTorch | — | | | | | | vLLM-Omni | — | | | | | | Diffusers | — | | | | +| | TensorRT-LLM | — | | | | | **H100 NVL** | PyTorch | — | | | | | | vLLM-Omni | — | | | | | | Diffusers | — | | | | +| | TensorRT-LLM | — | | | | | **H200 NVL** | PyTorch | — | | | | | | vLLM-Omni | — | | | | | | Diffusers | — | | | | +| | TensorRT-LLM | — | | | | | **H100 80GB HBM3** | PyTorch | — | | | | | | vLLM-Omni | — | | | | | | Diffusers | — | | | | +| | TensorRT-LLM | — | | | | | **H200 141GB HBM3** | PyTorch | — | | | | | | vLLM-Omni | — | | | | | | Diffusers | — | | | | +| | TensorRT-LLM | — | | | | | **B200** | PyTorch | — | | | | | | vLLM-Omni | — | | | | | | Diffusers | — | | | | +| | TensorRT-LLM | — | | | | | **B300** | PyTorch | — | | | | | | vLLM-Omni | — | | | | | | Diffusers | — | | | | +| | TensorRT-LLM | — | | | | Additional-modality notes: 1. PyTorch values are average generation (sampling) latency in seconds from PBR `#308197`; lower is better. From 8b9363e4127f8fb97f29ff23557decce648e2bcb Mon Sep 17 00:00:00 2001 From: Sergiy Fefilatyev Date: Fri, 2 Oct 2026 14:03:18 -0700 Subject: [PATCH 9/9] Add FP8 benchmark pages for Nano and Super The FP8 checkpoints get their own pages rather than extra rows on the BF16 pages, since they are distinct models. PyTorch values come from PBR #309161 and Diffusers values from PBR #309434. Coverage is uneven between the two runtimes, which the methodology records: PyTorch covers 1, 4, and 8 GPUs at all three resolutions, while Diffusers is single-GPU at 480p and 720p only. Neither campaign ran on H100 NVL or H200 NVL, so those GPUs are absent from these pages, and GB300 appears only where the Diffusers campaign measured it. Cosmos3-Super-FP8 needs eight GPUs on H100 80GB HBM3, where its single-GPU 720p runs do not fit; every other block fits on one GPU. One result is worth reading before picking a precision, so it is footnoted: FP8 wins throughout for PyTorch, but for Diffusers it only pays off on long video workloads. Diffusers t2i is about twice as slow in FP8 as in BF16 in every cell measured. --- inference_benchmarks.md | 4 +- .../cosmos3-nano-fp8-generator.md | 433 ++++++++++++++++++ .../cosmos3-super-fp8-generator.md | 433 ++++++++++++++++++ 3 files changed, 869 insertions(+), 1 deletion(-) create mode 100644 inference_benchmarks/cosmos3-nano-fp8-generator.md create mode 100644 inference_benchmarks/cosmos3-super-fp8-generator.md diff --git a/inference_benchmarks.md b/inference_benchmarks.md index 2afbaf61..79487ee5 100644 --- a/inference_benchmarks.md +++ b/inference_benchmarks.md @@ -6,13 +6,15 @@ Results are published incrementally from internal benchmark runs. **Empty cells ## Generator benchmarks -Generator results are organized by model so future checkpoints can be added without expanding this page into another wall of tables. Use the section links for modality-oriented navigation. +Generator results are organized by model so future checkpoints can be added without expanding this page into another wall of tables. Use the section links for modality-oriented navigation. The FP8 pages cover the quantized Nano and Super checkpoints; every other Generator page reports BF16, except NIM rows, which are FP8. | Model | Primary vision generation | Additional audiovisual generation | Transfer generation | Action generation | |---|---|---|---|---| | Cosmos3-Edge | [t2v, i2v, t2i](inference_benchmarks/cosmos3-edge-generator.md#text-to-video-t2v) | [v2v](inference_benchmarks/cosmos3-edge-generator.md#video-to-video-v2v) | Not reported | Not reported | | Cosmos3-Nano | [t2v, i2v, t2i](inference_benchmarks/cosmos3-nano-generator.md#primary-vision-generation) | [v2v, t2av, v2av, i2av](inference_benchmarks/cosmos3-nano-generator.md#additional-audiovisual-generation) | [blur, depth, edge, seg, wsm](inference_benchmarks/cosmos3-nano-generator.md#transfer-generation) | [Forward dynamics, inverse dynamics, policy](inference_benchmarks/cosmos3-nano-generator.md#action-generation) | | Cosmos3-Super | [t2v, i2v, t2i](inference_benchmarks/cosmos3-super-generator.md#primary-vision-generation) | [v2v, t2av, v2av, i2av](inference_benchmarks/cosmos3-super-generator.md#additional-audiovisual-generation) | [blur, depth, edge, seg, wsm](inference_benchmarks/cosmos3-super-generator.md#transfer-generation) | [Forward dynamics, inverse dynamics, policy](inference_benchmarks/cosmos3-super-generator.md#action-generation) | +| Cosmos3-Nano-FP8 | [t2v, i2v, t2i](inference_benchmarks/cosmos3-nano-fp8-generator.md#primary-vision-generation) | [v2v, t2av, v2av, i2av](inference_benchmarks/cosmos3-nano-fp8-generator.md#additional-audiovisual-generation) | [blur, depth, edge, multi_control, seg, wsm](inference_benchmarks/cosmos3-nano-fp8-generator.md#transfer-generation) | [Forward dynamics, inverse dynamics, policy](inference_benchmarks/cosmos3-nano-fp8-generator.md#action-generation) | +| Cosmos3-Super-FP8 | [t2v, i2v, t2i](inference_benchmarks/cosmos3-super-fp8-generator.md#primary-vision-generation) | [v2v, t2av, v2av, i2av](inference_benchmarks/cosmos3-super-fp8-generator.md#additional-audiovisual-generation) | [blur, depth, edge, multi_control, seg, wsm](inference_benchmarks/cosmos3-super-fp8-generator.md#transfer-generation) | [Forward dynamics, inverse dynamics, policy](inference_benchmarks/cosmos3-super-fp8-generator.md#action-generation) | ## Reasoner benchmarks diff --git a/inference_benchmarks/cosmos3-nano-fp8-generator.md b/inference_benchmarks/cosmos3-nano-fp8-generator.md new file mode 100644 index 00000000..9690dc1f --- /dev/null +++ b/inference_benchmarks/cosmos3-nano-fp8-generator.md @@ -0,0 +1,433 @@ +# Cosmos3-Nano-FP8 Generator Inference Benchmarks + +[Back to the inference benchmark index](../inference_benchmarks.md) + +These tables report **Cosmos3-Nano-FP8** Generator latency in seconds, for the FP8 quantized Cosmos3-Nano checkpoint. Lower is better. The BF16 numbers for the same model are on the [Cosmos3-Nano page](cosmos3-nano-generator.md). + +Every row reports one GPU configuration, named in the **GPUs** column, and all three resolutions in that row come from it. The configuration is one GPU when the model fits on a single device at its highest resolution, 720p, and eight GPUs when it does not. Within one GPU's block every engine reports the same configuration, so a column can be read straight across the runtimes. All populated rows report one GPU. + +Empty cells mean that a run has not been completed for that GPU, engine, or resolution; they do not indicate that a combination is unsupported. + +## Table of Contents + +- [Benchmark methodology](#benchmark-methodology) +- [Workload definitions](#workload-definitions) +- [Primary vision generation](#primary-vision-generation) + - [Text-to-Video (t2v)](#text-to-video-t2v) + - [Image-to-Video (i2v)](#image-to-video-i2v) + - [Text-to-Image (t2i)](#text-to-image-t2i) +- [Additional audiovisual generation](#additional-audiovisual-generation) + - [Video-to-Video (v2v)](#video-to-video-v2v) + - [Text-to-Audio-and-Video (t2av)](#text-to-audio-and-video-t2av) + - [Video-to-Audio-and-Video (v2av)](#video-to-audio-and-video-v2av) + - [Image-to-Audio-and-Video (i2av)](#image-to-audio-and-video-i2av) +- [Transfer generation](#transfer-generation) +- [Action generation](#action-generation) + - [Forward Dynamics — Autonomous Vehicle (AV)](#forward-dynamics--autonomous-vehicle-av) + - [Forward Dynamics — Camera](#forward-dynamics--camera) + - [Forward Dynamics — Robot](#forward-dynamics--robot) + - [Inverse Dynamics — Autonomous Vehicle (AV)](#inverse-dynamics--autonomous-vehicle-av) + - [Inverse Dynamics — Robot](#inverse-dynamics--robot) + - [Policy — Autonomous Vehicle (AV)](#policy--autonomous-vehicle-av) + - [Policy — Robot](#policy--robot) + +## Benchmark methodology + +PyTorch values are from PBR `#309161`, **FP8 Cosmos3-Nano/Super (regular non-distilled) Generator PyTorch Inference Benchmarking**, reporting average generation latency over datacenter configurations at 1, 4, and 8 GPUs. Diffusers values are from PBR `#309434`, **FP8 Cosmos3-Nano/Super Generator Diffusers Inference Benchmarking**, measured on a single GPU only and at 480p and 720p only, which is why the 256p column is empty for Diffusers and why its rows are empty wherever a block reports eight GPUs. Neither campaign covered H100 NVL or H200 NVL, so those GPUs are absent from this page. GB300 appears only in the Diffusers campaign. vLLM-Omni, TensorRT-LLM, and NIM have no FP8 results yet. Values are rounded to two decimal places. + +These reports establish the reported timing matrix but do not expose every prompt and action payload in this repository. The linked public recipes explain modality behavior and provide representative payloads; their example-specific frame counts and action chunk sizes should not be treated as the exact internal benchmark inputs. + +## Workload definitions + +| Workload | Input | Output | +|---|---|---| +| Text-to-image (`t2i`) | Text prompt | Generated image | +| Text-to-video (`t2v`) | Text prompt | Generated video | +| Image-to-video (`i2v`) | Text prompt and source image | Generated video | +| Video-to-video (`v2v`) | Text prompt and source video | Generated video | +| Text-to-audio-and-video (`t2av`) | Text prompt | Synchronized video and sound | +| Video-to-audio-and-video (`v2av`) | Text prompt and source video | Generated video with synchronized sound | +| Image-to-audio-and-video (`i2av`) | Text prompt and source image | Generated video with synchronized sound | +| Transfer video-to-video (`transfer`) | Text prompt, source video, and one control hint | Generated video steered by that hint | +| Forward dynamics | Initial visual observation and an action trajectory | Future-observation rollout video | +| Inverse dynamics | Observed video | Recovered action trajectory; some serving integrations also return video | +| Policy | Initial visual observation, instruction, and optional state | Predicted action trajectory and, for general Generator paths, a rollout video | + +The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound-producing modes `t2vs`, `v2vs`, and `i2vs`. See the [audiovisual cookbook](../cookbooks/cosmos3/generator/audiovisual/README.md) for generation inputs and the [action cookbook](../cookbooks/cosmos3/generator/action/README.md) for action representations and output contracts. + +## Primary vision generation + +### Text-to-Video (t2v) + +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 1 | 4.71 | 47.34 | 173.02 | +| | Diffusers | 1 | | | | +| **H20** | PyTorch | 1 | 11.82 | 112.34 | 427.62 | +| | Diffusers | 1 | | | | +| **H100 80GB HBM3** | PyTorch | 1 | 3.12 | 26.01 | 94.01 | +| | Diffusers | 1 | | 69.55 | 237.29 | +| **H200 141GB HBM3** | PyTorch | 1 | 3.09 | 26.59 | 94.57 | +| | Diffusers | 1 | | 68.32 | 237.18 | +| **B200** | PyTorch | 1 | 1.97 | 14.85 | 51.85 | +| | Diffusers | 1 | | 39.70 | 124.10 | +| **B300** | PyTorch | 1 | 1.99 | 13.68 | 46.41 | +| | Diffusers | 1 | | 37.61 | 115.14 | +| **GB300** | PyTorch | 1 | | | | +| | Diffusers | 1 | | 34.53 | 97.30 | + +### Image-to-Video (i2v) + +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 1 | 4.50 | 45.96 | 169.15 | +| | Diffusers | 1 | | | | +| **H20** | PyTorch | 1 | 11.19 | 110.19 | 419.99 | +| | Diffusers | 1 | | | | +| **H100 80GB HBM3** | PyTorch | 1 | 2.96 | 25.04 | 91.65 | +| | Diffusers | 1 | | 71.06 | 241.53 | +| **H200 141GB HBM3** | PyTorch | 1 | 2.94 | 25.56 | 92.44 | +| | Diffusers | 1 | | 69.69 | 237.95 | +| **B200** | PyTorch | 1 | 1.87 | 14.24 | 50.24 | +| | Diffusers | 1 | | 40.69 | 126.07 | +| **B300** | PyTorch | 1 | 1.78 | 13.08 | 44.94 | +| | Diffusers | 1 | | 38.35 | 117.18 | +| **GB300** | PyTorch | 1 | | | | +| | Diffusers | 1 | | 34.96 | 99.37 | + +### Text-to-Image (t2i) + +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 1 | 0.97 | 1.02 | 1.50 | +| | Diffusers | 1 | | 7.29 | | +| **H20** | PyTorch | 1 | 1.27 | 1.74 | 3.25 | +| | Diffusers | 1 | | | | +| **H100 80GB HBM3** | PyTorch | 1 | 1.16 | 1.14 | 1.21 | +| | Diffusers | 1 | | 7.50 | 7.68 | +| **H200 141GB HBM3** | PyTorch | 1 | 1.12 | 1.14 | 1.21 | +| | Diffusers | 1 | | 7.44 | 7.74 | +| **B200** | PyTorch | 1 | 0.96 | 0.92 | 0.97 | +| | Diffusers | 1 | | 6.82 | 6.94 | +| **B300** | PyTorch | 1 | 0.95 | 0.95 | 1.51 | +| | Diffusers | 1 | | 10.28 | 6.71 | +| **GB300** | PyTorch | 1 | | | | +| | Diffusers | 1 | | 11.79 | 13.19 | + +Notes: +1. All times measured on identical workloads (same seed, sampler settings, prompt). +2. Multi-GPU configurations use tensor parallelism. +3. Values are average generation latency in seconds; lower is better. +4. The **GPUs** column gives the number of GPUs behind every value in that row, and is the same for every engine in a GPU block so the rows can be compared directly. +5. Diffusers FP8 was measured on one GPU at 480p and 720p only. +6. FP8 is faster than BF16 for the PyTorch path throughout. For the Diffusers path it only pays off on long video workloads: short ones are slower in FP8, `t2i` consistently so by roughly a factor of two, which suggests a fixed quantization overhead that the short runs cannot amortize. Compare against the BF16 page before choosing a precision for a short workload. + +## Additional audiovisual generation + +### Video-to-Video (v2v) + +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 1 | 4.40 | 47.39 | 175.87 | +| | Diffusers | 1 | | | | +| **H20** | PyTorch | 1 | 10.99 | 112.59 | 437.65 | +| | Diffusers | 1 | | | | +| **H100 80GB HBM3** | PyTorch | 1 | 2.93 | 25.89 | 95.20 | +| | Diffusers | 1 | | 72.01 | 241.38 | +| **H200 141GB HBM3** | PyTorch | 1 | 2.90 | 26.51 | 96.80 | +| | Diffusers | 1 | | 70.17 | 239.24 | +| **B200** | PyTorch | 1 | 1.85 | 14.69 | 52.19 | +| | Diffusers | 1 | | 41.72 | 127.42 | +| **B300** | PyTorch | 1 | 1.92 | 13.61 | 47.32 | +| | Diffusers | 1 | | 39.99 | 118.73 | +| **GB300** | PyTorch | 1 | | | | +| | Diffusers | 1 | | 35.97 | 100.27 | + +### Text-to-Audio-and-Video (t2av) + +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 1 | 7.67 | 50.58 | 178.94 | +| | Diffusers | 1 | | | | +| **H20** | PyTorch | 1 | 17.97 | 120.72 | 442.78 | +| | Diffusers | 1 | | | | +| **H100 80GB HBM3** | PyTorch | 1 | 4.88 | 28.50 | 97.19 | +| | Diffusers | 1 | | 70.46 | 239.14 | +| **H200 141GB HBM3** | PyTorch | 1 | 4.79 | 28.21 | 99.38 | +| | Diffusers | 1 | | 68.99 | 237.11 | +| **B200** | PyTorch | 1 | 3.10 | 16.48 | 54.04 | +| | Diffusers | 1 | | 40.71 | 124.89 | +| **B300** | PyTorch | 1 | 3.12 | 15.17 | 49.50 | +| | Diffusers | 1 | | 38.13 | 116.44 | +| **GB300** | PyTorch | 1 | | | | +| | Diffusers | 1 | | 34.89 | 100.22 | + +### Video-to-Audio-and-Video (v2av) + +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | — | | | | +| | Diffusers | — | | | | +| **H20** | PyTorch | — | | | | +| | Diffusers | — | | | | +| **H100 80GB HBM3** | PyTorch | 1 | | | | +| | Diffusers | 1 | | 73.07 | 243.26 | +| **H200 141GB HBM3** | PyTorch | 1 | | | | +| | Diffusers | 1 | | 71.80 | 241.17 | +| **B200** | PyTorch | 1 | | | | +| | Diffusers | 1 | | 42.28 | 126.29 | +| **B300** | PyTorch | 1 | | | | +| | Diffusers | 1 | | 39.94 | 120.27 | +| **GB300** | PyTorch | 1 | | | | +| | Diffusers | 1 | | 36.41 | 101.35 | + +### Image-to-Audio-and-Video (i2av) + +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 1 | 7.30 | 48.80 | 174.71 | +| | Diffusers | 1 | | | | +| **H20** | PyTorch | 1 | 17.08 | 115.28 | 428.45 | +| | Diffusers | 1 | | | | +| **H100 80GB HBM3** | PyTorch | 1 | 4.63 | 27.33 | 94.44 | +| | Diffusers | 1 | | 72.07 | 243.07 | +| **H200 141GB HBM3** | PyTorch | 1 | 4.54 | 27.44 | 97.66 | +| | Diffusers | 1 | | 70.18 | 241.70 | +| **B200** | PyTorch | 1 | 2.96 | 15.67 | 52.20 | +| | Diffusers | 1 | | 41.66 | 126.91 | +| **B300** | PyTorch | 1 | 2.81 | 14.60 | 47.27 | +| | Diffusers | 1 | | 39.34 | | +| **GB300** | PyTorch | 1 | | | | +| | Diffusers | 1 | | 36.51 | 101.85 | + +## Transfer generation + +Transfer conditions a video-to-video generation on a structural control hint +extracted from the source video. Each control is reported separately because +the hint changes how much of the frame the model must synthesise. + +| GPU | Transfer control | Engine | GPUs | 256p | 480p | 720p | +|---|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | blur | PyTorch | 1 | 15.92 | 99.23 | 369.40 | +| | | Diffusers | 1 | | | | +| | depth | PyTorch | 1 | 15.34 | 102.88 | 379.90 | +| | | Diffusers | 1 | | | | +| | edge | PyTorch | 1 | 13.27 | 99.61 | 374.57 | +| | | Diffusers | 1 | | | | +| | multi_control | PyTorch | — | | | | +| | | Diffusers | — | | | | +| | seg | PyTorch | 1 | 14.54 | 104.96 | 378.47 | +| | | Diffusers | 1 | | | | +| | wsm | PyTorch | 1 | 12.81 | 78.87 | 280.91 | +| | | Diffusers | 1 | | | 430.19 | +| **H20** | blur | PyTorch | 1 | 36.48 | 238.41 | 918.21 | +| | | Diffusers | 1 | | | | +| | depth | PyTorch | 1 | 34.13 | 243.68 | 951.21 | +| | | Diffusers | 1 | | | | +| | edge | PyTorch | 1 | 29.64 | 238.38 | 947.97 | +| | | Diffusers | 1 | | | | +| | multi_control | PyTorch | 1 | 36.57 | 385.10 | 1678.80 | +| | | Diffusers | 1 | | | | +| | seg | PyTorch | 1 | 31.20 | 239.56 | 992.60 | +| | | Diffusers | 1 | | | | +| | wsm | PyTorch | 1 | 28.15 | 186.34 | 707.32 | +| | | Diffusers | 1 | | | | +| **H100 80GB HBM3** | blur | PyTorch | 1 | 10.37 | 55.94 | 198.76 | +| | | Diffusers | 1 | | 171.90 | 600.30 | +| | depth | PyTorch | 1 | 9.83 | 57.44 | 207.16 | +| | | Diffusers | 1 | | 178.89 | 606.60 | +| | edge | PyTorch | 1 | 8.49 | 55.27 | 204.24 | +| | | Diffusers | 1 | | 172.75 | 597.01 | +| | multi_control | PyTorch | 1 | 10.51 | 91.48 | 367.08 | +| | | Diffusers | 1 | | | | +| | seg | PyTorch | 1 | 10.03 | 56.62 | 214.56 | +| | | Diffusers | 1 | | 174.09 | 601.13 | +| | wsm | PyTorch | 1 | 8.33 | 44.52 | 153.26 | +| | | Diffusers | 1 | | 84.48 | 264.67 | +| **H200 141GB HBM3** | blur | PyTorch | 1 | 10.32 | 55.96 | 205.31 | +| | | Diffusers | 1 | | 169.37 | 598.78 | +| | depth | PyTorch | 1 | 9.70 | 58.92 | 211.20 | +| | | Diffusers | 1 | | 177.22 | 603.37 | +| | edge | PyTorch | 1 | 8.60 | 56.78 | 208.28 | +| | | Diffusers | 1 | | 169.21 | 594.04 | +| | multi_control | PyTorch | 1 | 10.46 | 92.81 | 382.20 | +| | | Diffusers | 1 | | | | +| | seg | PyTorch | 1 | 9.68 | 58.18 | 209.54 | +| | | Diffusers | 1 | | 170.62 | 601.64 | +| | wsm | PyTorch | 1 | 8.39 | 45.06 | 154.98 | +| | | Diffusers | 1 | | 82.92 | 263.74 | +| **B200** | blur | PyTorch | 1 | 7.02 | 32.23 | 110.23 | +| | | Diffusers | 1 | | 94.88 | 298.51 | +| | depth | PyTorch | 1 | 6.68 | 33.48 | 113.49 | +| | | Diffusers | 1 | | 98.03 | 307.64 | +| | edge | PyTorch | 1 | 5.79 | 31.53 | 111.67 | +| | | Diffusers | 1 | | 96.11 | 302.90 | +| | multi_control | PyTorch | — | | | | +| | | Diffusers | — | | | | +| | seg | PyTorch | 1 | 6.65 | 33.21 | 114.39 | +| | | Diffusers | 1 | | 97.13 | 301.15 | +| | wsm | PyTorch | 1 | 5.78 | 26.19 | 85.06 | +| | | Diffusers | 1 | | 48.75 | 138.10 | +| **B300** | blur | PyTorch | 1 | 8.30 | 31.29 | 101.38 | +| | | Diffusers | 1 | | | 278.49 | +| | depth | PyTorch | 1 | 7.81 | 32.60 | 105.09 | +| | | Diffusers | 1 | | 93.48 | 282.93 | +| | edge | PyTorch | 1 | 6.08 | 31.68 | 103.18 | +| | | Diffusers | 1 | | 89.60 | 277.25 | +| | multi_control | PyTorch | — | | | | +| | | Diffusers | — | | | | +| | seg | PyTorch | 1 | 7.14 | 32.89 | 106.26 | +| | | Diffusers | 1 | | 89.99 | 280.71 | +| | wsm | PyTorch | 1 | 5.89 | 25.42 | 78.99 | +| | | Diffusers | 1 | | 46.64 | 129.77 | +| **GB300** | blur | PyTorch | 1 | | | | +| | | Diffusers | 1 | | 80.61 | 236.36 | +| | depth | PyTorch | 1 | | | | +| | | Diffusers | 1 | | 81.57 | 238.78 | +| | edge | PyTorch | 1 | | | | +| | | Diffusers | 1 | | 80.54 | 235.31 | +| | multi_control | PyTorch | — | | | | +| | | Diffusers | — | | | | +| | seg | PyTorch | 1 | | | | +| | | Diffusers | 1 | | 81.67 | 237.79 | +| | wsm | PyTorch | 1 | | | | +| | | Diffusers | 1 | | 42.26 | 110.70 | + +Transfer notes: +1. Control hints follow the `extra_params` names: `blur`, `depth`, `edge`, `multi_control`, `seg`, and `wsm`. `multi_control` combines several hints in one request. +2. The PyTorch campaign reports `segmentation`, which is the same control as `seg` in the Diffusers campaign. +3. Diffusers FP8 has no `multi_control` measurements. + +## Action generation + +### Forward Dynamics — Autonomous Vehicle (AV) + +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 1 | 5.80 | 5.78 | 5.80 | +| | Diffusers | 1 | | | | +| **H20** | PyTorch | 1 | 13.44 | 13.32 | 13.44 | +| | Diffusers | 1 | | | | +| **H100 80GB HBM3** | PyTorch | 1 | 3.54 | 3.53 | 3.53 | +| | Diffusers | 1 | | 11.68 | 12.20 | +| **H200 141GB HBM3** | PyTorch | 1 | 3.45 | 3.47 | 3.46 | +| | Diffusers | 1 | | 11.19 | 11.73 | +| **B200** | PyTorch | 1 | 2.16 | 2.16 | 2.16 | +| | Diffusers | 1 | | 8.05 | 8.42 | +| **B300** | PyTorch | 1 | 2.11 | 2.09 | 2.10 | +| | Diffusers | 1 | | 8.83 | 8.96 | +| **GB300** | PyTorch | 1 | | | | +| | Diffusers | 1 | | 7.71 | 7.64 | + +### Forward Dynamics — Camera + +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 1 | 5.47 | 5.49 | 5.48 | +| | Diffusers | 1 | | | | +| **H20** | PyTorch | 1 | 12.69 | 12.57 | 12.58 | +| | Diffusers | 1 | | | | +| **H100 80GB HBM3** | PyTorch | 1 | 3.38 | 3.37 | 3.36 | +| | Diffusers | 1 | | | | +| **H200 141GB HBM3** | PyTorch | 1 | 3.29 | 3.30 | 3.29 | +| | Diffusers | 1 | | | | +| **B200** | PyTorch | 1 | 2.06 | 2.06 | 2.06 | +| | Diffusers | 1 | | | | +| **B300** | PyTorch | 1 | 1.98 | 1.98 | 2.06 | +| | Diffusers | 1 | | | | + +### Forward Dynamics — Robot + +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 1 | 1.28 | 1.28 | 1.28 | +| | Diffusers | 1 | | | 5.63 | +| **H20** | PyTorch | 1 | 2.85 | 2.84 | 2.84 | +| | Diffusers | 1 | | | | +| **H100 80GB HBM3** | PyTorch | 1 | 0.85 | 0.85 | 0.85 | +| | Diffusers | 1 | | 4.63 | 4.79 | +| **H200 141GB HBM3** | PyTorch | 1 | 0.84 | 0.84 | 0.84 | +| | Diffusers | 1 | | 4.58 | 4.69 | +| **B200** | PyTorch | 1 | 0.61 | 0.61 | 0.60 | +| | Diffusers | 1 | | 4.09 | 4.35 | +| **B300** | PyTorch | 1 | 0.87 | 0.88 | 0.60 | +| | Diffusers | 1 | | 3.78 | 5.98 | +| **GB300** | PyTorch | 1 | | | | +| | Diffusers | 1 | | 5.70 | 6.32 | + +### Inverse Dynamics — Autonomous Vehicle (AV) + +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 1 | 9.02 | 9.01 | 9.00 | +| | Diffusers | 1 | | | | +| **H20** | PyTorch | 1 | 21.15 | 21.36 | 21.15 | +| | Diffusers | 1 | | | | +| **H100 80GB HBM3** | PyTorch | 1 | 5.45 | 5.44 | 5.44 | +| | Diffusers | 1 | | 11.61 | 12.21 | +| **H200 141GB HBM3** | PyTorch | 1 | 5.32 | 5.39 | 5.32 | +| | Diffusers | 1 | | 11.10 | 11.65 | +| **B200** | PyTorch | 1 | 3.25 | 3.24 | 3.24 | +| | Diffusers | 1 | | 8.00 | 8.27 | +| **B300** | PyTorch | 1 | 3.15 | 3.14 | 3.13 | +| | Diffusers | 1 | | 8.90 | 8.95 | +| **GB300** | PyTorch | 1 | | | | +| | Diffusers | 1 | | 7.85 | 7.88 | + +### Inverse Dynamics — Robot + +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 1 | 1.86 | 1.85 | 1.85 | +| | Diffusers | 1 | | | 5.65 | +| **H20** | PyTorch | 1 | 4.32 | 4.28 | 4.31 | +| | Diffusers | 1 | | | | +| **H100 80GB HBM3** | PyTorch | 1 | 1.21 | 1.22 | 1.22 | +| | Diffusers | 1 | | 4.75 | 4.79 | +| **H200 141GB HBM3** | PyTorch | 1 | 1.19 | 1.19 | 1.19 | +| | Diffusers | 1 | | 4.64 | 4.72 | +| **B200** | PyTorch | 1 | 0.84 | 0.86 | 0.84 | +| | Diffusers | 1 | | 4.02 | 4.18 | +| **B300** | PyTorch | 1 | 0.87 | 0.84 | 0.84 | +| | Diffusers | 1 | | | 6.03 | +| **GB300** | PyTorch | 1 | | | | +| | Diffusers | 1 | | 6.57 | 6.08 | + +### Policy — Autonomous Vehicle (AV) + +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 1 | 5.80 | 5.79 | 5.80 | +| | Diffusers | 1 | | | | +| **H20** | PyTorch | 1 | 13.44 | 13.44 | 13.44 | +| | Diffusers | 1 | | | | +| **H100 80GB HBM3** | PyTorch | 1 | 3.54 | 3.54 | 3.54 | +| | Diffusers | 1 | | | | +| **H200 141GB HBM3** | PyTorch | 1 | 3.51 | 3.49 | 3.51 | +| | Diffusers | 1 | | | | +| **B200** | PyTorch | 1 | 2.17 | 2.17 | 2.14 | +| | Diffusers | 1 | | | | +| **B300** | PyTorch | 1 | 2.14 | 2.13 | 2.16 | +| | Diffusers | 1 | | | | + +### Policy — Robot + +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 1 | 1.28 | 1.29 | 1.24 | +| | Diffusers | 1 | | | | +| **H20** | PyTorch | 1 | 2.75 | 2.87 | 2.75 | +| | Diffusers | 1 | | | | +| **H100 80GB HBM3** | PyTorch | 1 | 0.84 | 0.84 | 0.85 | +| | Diffusers | 1 | | | | +| **H200 141GB HBM3** | PyTorch | 1 | 0.83 | 0.82 | 0.85 | +| | Diffusers | 1 | | | | +| **B200** | PyTorch | 1 | 0.60 | 0.61 | 0.62 | +| | Diffusers | 1 | | | | +| **B300** | PyTorch | 1 | 0.89 | 0.85 | 0.86 | +| | Diffusers | 1 | | | | + +Action and additional-modality notes: +1. Forward-dynamics cells for the AV domain are the mean of the forward, left, and right trajectories; inverse-dynamics cells are the mean of the two inverse trajectories. +2. The Diffusers FP8 campaign covers forward and inverse dynamics for the AV and Robot domains only, so the Camera and Policy rows carry PyTorch values alone. +3. Policy-DROID is a separate checkpoint and was not part of either FP8 campaign. diff --git a/inference_benchmarks/cosmos3-super-fp8-generator.md b/inference_benchmarks/cosmos3-super-fp8-generator.md new file mode 100644 index 00000000..75b34a6d --- /dev/null +++ b/inference_benchmarks/cosmos3-super-fp8-generator.md @@ -0,0 +1,433 @@ +# Cosmos3-Super-FP8 Generator Inference Benchmarks + +[Back to the inference benchmark index](../inference_benchmarks.md) + +These tables report **Cosmos3-Super-FP8** Generator latency in seconds, for the FP8 quantized Cosmos3-Super checkpoint. Lower is better. The BF16 numbers for the same model are on the [Cosmos3-Super page](cosmos3-super-generator.md). + +Every row reports one GPU configuration, named in the **GPUs** column, and all three resolutions in that row come from it. The configuration is one GPU when the model fits on a single device at its highest resolution, 720p, and eight GPUs when it does not. Within one GPU's block every engine reports the same configuration, so a column can be read straight across the runtimes. Cosmos3-Super-FP8 fits on a single GPU except on H100 80GB HBM3, where the 720p runs need eight. + +Empty cells mean that a run has not been completed for that GPU, engine, or resolution; they do not indicate that a combination is unsupported. + +## Table of Contents + +- [Benchmark methodology](#benchmark-methodology) +- [Workload definitions](#workload-definitions) +- [Primary vision generation](#primary-vision-generation) + - [Text-to-Video (t2v)](#text-to-video-t2v) + - [Image-to-Video (i2v)](#image-to-video-i2v) + - [Text-to-Image (t2i)](#text-to-image-t2i) +- [Additional audiovisual generation](#additional-audiovisual-generation) + - [Video-to-Video (v2v)](#video-to-video-v2v) + - [Text-to-Audio-and-Video (t2av)](#text-to-audio-and-video-t2av) + - [Video-to-Audio-and-Video (v2av)](#video-to-audio-and-video-v2av) + - [Image-to-Audio-and-Video (i2av)](#image-to-audio-and-video-i2av) +- [Transfer generation](#transfer-generation) +- [Action generation](#action-generation) + - [Forward Dynamics — Autonomous Vehicle (AV)](#forward-dynamics--autonomous-vehicle-av) + - [Forward Dynamics — Camera](#forward-dynamics--camera) + - [Forward Dynamics — Robot](#forward-dynamics--robot) + - [Inverse Dynamics — Autonomous Vehicle (AV)](#inverse-dynamics--autonomous-vehicle-av) + - [Inverse Dynamics — Robot](#inverse-dynamics--robot) + - [Policy — Autonomous Vehicle (AV)](#policy--autonomous-vehicle-av) + - [Policy — Robot](#policy--robot) + +## Benchmark methodology + +PyTorch values are from PBR `#309161`, **FP8 Cosmos3-Nano/Super (regular non-distilled) Generator PyTorch Inference Benchmarking**, reporting average generation latency over datacenter configurations at 1, 4, and 8 GPUs. Diffusers values are from PBR `#309434`, **FP8 Cosmos3-Nano/Super Generator Diffusers Inference Benchmarking**, measured on a single GPU only and at 480p and 720p only, which is why the 256p column is empty for Diffusers and why its rows are empty wherever a block reports eight GPUs. Neither campaign covered H100 NVL or H200 NVL, so those GPUs are absent from this page. GB300 appears only in the Diffusers campaign. vLLM-Omni, TensorRT-LLM, and NIM have no FP8 results yet. Values are rounded to two decimal places. + +These reports establish the reported timing matrix but do not expose every prompt and action payload in this repository. The linked public recipes explain modality behavior and provide representative payloads; their example-specific frame counts and action chunk sizes should not be treated as the exact internal benchmark inputs. + +## Workload definitions + +| Workload | Input | Output | +|---|---|---| +| Text-to-image (`t2i`) | Text prompt | Generated image | +| Text-to-video (`t2v`) | Text prompt | Generated video | +| Image-to-video (`i2v`) | Text prompt and source image | Generated video | +| Video-to-video (`v2v`) | Text prompt and source video | Generated video | +| Text-to-audio-and-video (`t2av`) | Text prompt | Synchronized video and sound | +| Video-to-audio-and-video (`v2av`) | Text prompt and source video | Generated video with synchronized sound | +| Image-to-audio-and-video (`i2av`) | Text prompt and source image | Generated video with synchronized sound | +| Transfer video-to-video (`transfer`) | Text prompt, source video, and one control hint | Generated video steered by that hint | +| Forward dynamics | Initial visual observation and an action trajectory | Future-observation rollout video | +| Inverse dynamics | Observed video | Recovered action trajectory; some serving integrations also return video | +| Policy | Initial visual observation, instruction, and optional state | Predicted action trajectory and, for general Generator paths, a rollout video | + +The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound-producing modes `t2vs`, `v2vs`, and `i2vs`. See the [audiovisual cookbook](../cookbooks/cosmos3/generator/audiovisual/README.md) for generation inputs and the [action cookbook](../cookbooks/cosmos3/generator/action/README.md) for action representations and output contracts. + +## Primary vision generation + +### Text-to-Video (t2v) + +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 1 | 15.27 | 154.79 | 589.57 | +| | Diffusers | 1 | | 386.15 | | +| **H20** | PyTorch | 1 | 37.03 | 381.98 | 1505.14 | +| | Diffusers | 1 | | | | +| **H100 80GB HBM3** | PyTorch | 1 | 9.15 | 83.30 | | +| | Diffusers | 1 | | 234.65 | 830.32 | +| **H200 141GB HBM3** | PyTorch | 1 | 9.00 | 85.98 | 322.54 | +| | Diffusers | 1 | | 231.20 | 825.50 | +| **B200** | PyTorch | 1 | 5.25 | 45.67 | 168.91 | +| | Diffusers | 1 | | 130.78 | 405.87 | +| **B300** | PyTorch | 1 | 4.92 | 42.14 | 149.55 | +| | Diffusers | 1 | | 120.69 | 381.36 | +| **GB300** | PyTorch | 1 | | | | +| | Diffusers | 1 | | 109.65 | 325.76 | + +### Image-to-Video (i2v) + +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 1 | 15.21 | 154.24 | 587.19 | +| | Diffusers | 1 | | | | +| **H20** | PyTorch | 1 | 37.09 | 380.01 | 1501.45 | +| | Diffusers | 1 | | | | +| **H100 80GB HBM3** | PyTorch | 8 | 5.75 | 19.08 | 55.33 | +| | Diffusers | 8 | | | | +| **H200 141GB HBM3** | PyTorch | 1 | 9.02 | 85.61 | 318.65 | +| | Diffusers | 1 | | 234.20 | 819.74 | +| **B200** | PyTorch | 1 | 5.18 | 45.24 | 167.93 | +| | Diffusers | 1 | | 129.97 | 408.14 | +| **B300** | PyTorch | 1 | 4.96 | 41.10 | 151.74 | +| | Diffusers | 1 | | 120.90 | | +| **GB300** | PyTorch | 1 | | | | +| | Diffusers | 1 | | 108.89 | 327.89 | + +### Text-to-Image (t2i) + +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 1 | 1.75 | 2.76 | 5.13 | +| | Diffusers | 1 | | | | +| **H20** | PyTorch | 1 | 2.33 | 5.73 | 12.01 | +| | Diffusers | 1 | | | | +| **H100 80GB HBM3** | PyTorch | 1 | 1.84 | 1.92 | 2.94 | +| | Diffusers | 1 | | 21.47 | 24.36 | +| **H200 141GB HBM3** | PyTorch | 1 | 1.88 | 1.85 | 2.98 | +| | Diffusers | 1 | | 19.90 | 23.20 | +| **B200** | PyTorch | 1 | 1.48 | 1.47 | 1.85 | +| | Diffusers | 1 | | 17.38 | 19.21 | +| **B300** | PyTorch | 1 | 2.37 | 2.29 | 2.47 | +| | Diffusers | 1 | | | 18.40 | +| **GB300** | PyTorch | 1 | | | | +| | Diffusers | 1 | | 22.73 | 20.98 | + +Notes: +1. All times measured on identical workloads (same seed, sampler settings, prompt). +2. Multi-GPU configurations use tensor parallelism. +3. Values are average generation latency in seconds; lower is better. +4. The **GPUs** column gives the number of GPUs behind every value in that row, and is the same for every engine in a GPU block so the rows can be compared directly. +5. Diffusers FP8 was measured on one GPU at 480p and 720p only. +6. FP8 is faster than BF16 for the PyTorch path throughout. For the Diffusers path it only pays off on long video workloads: short ones are slower in FP8, `t2i` consistently so by roughly a factor of two, which suggests a fixed quantization overhead that the short runs cannot amortize. Compare against the BF16 page before choosing a precision for a short workload. + +## Additional audiovisual generation + +### Video-to-Video (v2v) + +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 1 | 14.60 | 159.65 | 611.60 | +| | Diffusers | 1 | | | | +| **H20** | PyTorch | 1 | 35.78 | 396.97 | 1551.74 | +| | Diffusers | 1 | | | | +| **H100 80GB HBM3** | PyTorch | 8 | 5.79 | 19.97 | 57.83 | +| | Diffusers | 8 | | | | +| **H200 141GB HBM3** | PyTorch | 1 | 8.74 | 89.08 | 334.16 | +| | Diffusers | 1 | | 233.86 | 829.09 | +| **B200** | PyTorch | 1 | 4.95 | 46.10 | 174.67 | +| | Diffusers | 1 | | 132.66 | 410.66 | +| **B300** | PyTorch | 1 | 4.74 | 43.03 | 157.26 | +| | Diffusers | 1 | | 123.34 | 384.10 | +| **GB300** | PyTorch | 1 | | | | +| | Diffusers | 1 | | 109.88 | 325.10 | + +### Text-to-Audio-and-Video (t2av) + +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 1 | 26.29 | 167.44 | 607.06 | +| | Diffusers | 1 | | | | +| **H20** | PyTorch | 1 | 61.37 | 414.65 | 1539.41 | +| | Diffusers | 1 | | | | +| **H100 80GB HBM3** | PyTorch | 8 | 6.27 | 20.73 | 58.41 | +| | Diffusers | 8 | | | | +| **H200 141GB HBM3** | PyTorch | 1 | 15.05 | 92.13 | 335.54 | +| | Diffusers | 1 | | 237.46 | 821.20 | +| **B200** | PyTorch | 1 | 8.82 | 49.69 | 173.99 | +| | Diffusers | 1 | | 131.90 | 410.84 | +| **B300** | PyTorch | 1 | 8.56 | 44.96 | 157.74 | +| | Diffusers | 1 | | | 382.21 | +| **GB300** | PyTorch | 1 | | | | +| | Diffusers | 1 | | 109.41 | 320.27 | + +### Video-to-Audio-and-Video (v2av) + +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | — | | | | +| | Diffusers | — | | | | +| **H20** | PyTorch | — | | | | +| | Diffusers | — | | | | +| **H100 80GB HBM3** | PyTorch | 1 | | | | +| | Diffusers | 1 | | 240.63 | | +| **H200 141GB HBM3** | PyTorch | 1 | | | | +| | Diffusers | 1 | | 238.03 | 826.01 | +| **B200** | PyTorch | 1 | | | | +| | Diffusers | 1 | | 133.87 | 413.17 | +| **B300** | PyTorch | 1 | | | | +| | Diffusers | 1 | | 123.61 | 386.33 | +| **GB300** | PyTorch | 1 | | | | +| | Diffusers | 1 | | 110.98 | 334.54 | + +### Image-to-Audio-and-Video (i2av) + +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 1 | 25.77 | 165.51 | 602.27 | +| | Diffusers | 1 | | | | +| **H20** | PyTorch | 1 | 59.59 | 408.98 | 1511.82 | +| | Diffusers | 1 | | | | +| **H100 80GB HBM3** | PyTorch | 8 | 6.08 | 19.64 | 55.85 | +| | Diffusers | 8 | | | | +| **H200 141GB HBM3** | PyTorch | 1 | 14.59 | 90.77 | 330.04 | +| | Diffusers | 1 | | 238.95 | 831.19 | +| **B200** | PyTorch | 1 | 8.52 | 48.68 | 172.03 | +| | Diffusers | 1 | | 132.97 | 418.44 | +| **B300** | PyTorch | 1 | 8.30 | 44.71 | 155.68 | +| | Diffusers | 1 | | 122.66 | 384.16 | +| **GB300** | PyTorch | 1 | | | | +| | Diffusers | 1 | | 112.33 | 324.87 | + +## Transfer generation + +Transfer conditions a video-to-video generation on a structural control hint +extracted from the source video. Each control is reported separately because +the hint changes how much of the frame the model must synthesise. + +| GPU | Transfer control | Engine | GPUs | 256p | 480p | 720p | +|---|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | blur | PyTorch | 1 | 56.31 | 348.18 | 1312.06 | +| | | Diffusers | 1 | | | | +| | depth | PyTorch | 1 | 53.67 | 360.23 | 1350.49 | +| | | Diffusers | 1 | | | | +| | edge | PyTorch | 1 | 45.73 | 347.50 | 1329.40 | +| | | Diffusers | 1 | | | | +| | multi_control | PyTorch | — | | | | +| | | Diffusers | — | | | | +| | seg | PyTorch | 1 | 49.86 | 367.16 | 1402.75 | +| | | Diffusers | 1 | | | | +| | wsm | PyTorch | 1 | 43.38 | 275.08 | 999.98 | +| | | Diffusers | 1 | | | | +| **H20** | blur | PyTorch | 1 | 132.90 | 857.75 | 3317.89 | +| | | Diffusers | 1 | | | | +| | depth | PyTorch | 1 | 125.40 | 888.61 | 3434.99 | +| | | Diffusers | 1 | | | | +| | edge | PyTorch | 1 | 107.20 | 867.13 | 3423.93 | +| | | Diffusers | 1 | | | | +| | multi_control | PyTorch | 1 | 129.52 | 1399.05 | 5934.07 | +| | | Diffusers | 1 | | | | +| | seg | PyTorch | 1 | 117.22 | 912.61 | 3473.61 | +| | | Diffusers | 1 | | | | +| | wsm | PyTorch | 1 | 100.62 | 677.55 | 2530.15 | +| | | Diffusers | 1 | | | | +| **H100 80GB HBM3** | blur | PyTorch | 8 | 25.70 | 74.88 | 211.62 | +| | | Diffusers | 8 | | | | +| | depth | PyTorch | 8 | 24.13 | 76.10 | 219.20 | +| | | Diffusers | 8 | | | | +| | edge | PyTorch | 8 | 23.24 | 73.96 | 216.16 | +| | | Diffusers | 8 | | | | +| | multi_control | PyTorch | 8 | 26.52 | 110.00 | 380.70 | +| | | Diffusers | 8 | | | | +| | seg | PyTorch | 8 | 27.41 | 83.47 | 236.85 | +| | | Diffusers | 8 | | | | +| | wsm | PyTorch | 1 | 25.48 | 149.06 | | +| | | Diffusers | 1 | | 289.78 | 923.57 | +| **H200 141GB HBM3** | blur | PyTorch | 1 | 32.51 | 192.22 | 727.48 | +| | | Diffusers | 1 | | 603.23 | 2115.64 | +| | depth | PyTorch | 1 | 30.58 | 200.84 | 741.97 | +| | | Diffusers | 1 | | 621.13 | 2137.67 | +| | edge | PyTorch | 1 | 26.71 | 195.05 | 730.81 | +| | | Diffusers | 1 | | 598.72 | 2100.45 | +| | multi_control | PyTorch | 1 | 31.90 | 316.92 | 1289.45 | +| | | Diffusers | 1 | | | | +| | seg | PyTorch | 1 | 29.32 | 204.51 | 736.27 | +| | | Diffusers | 1 | | 602.62 | 2122.42 | +| | wsm | PyTorch | 1 | 25.08 | 155.47 | 542.31 | +| | | Diffusers | 1 | | 285.48 | 923.68 | +| **B200** | blur | PyTorch | 1 | 19.26 | 103.45 | 374.36 | +| | | Diffusers | 1 | | 326.36 | 1025.97 | +| | depth | PyTorch | 1 | 18.14 | 107.49 | 385.52 | +| | | Diffusers | 1 | | 334.83 | 1044.04 | +| | edge | PyTorch | 1 | 15.76 | 103.47 | 374.81 | +| | | Diffusers | 1 | | 327.94 | 1029.37 | +| | multi_control | PyTorch | — | | | | +| | | Diffusers | — | | | | +| | seg | PyTorch | 1 | 17.60 | 109.95 | 400.23 | +| | | Diffusers | 1 | | 326.72 | 1029.82 | +| | wsm | PyTorch | 1 | 15.36 | 82.54 | 285.69 | +| | | Diffusers | 1 | | 161.39 | 469.48 | +| **B300** | blur | PyTorch | 1 | 18.27 | 94.98 | 331.00 | +| | | Diffusers | 1 | | 301.73 | 956.39 | +| | depth | PyTorch | 1 | 17.29 | 100.31 | 342.30 | +| | | Diffusers | 1 | | 315.60 | 974.65 | +| | edge | PyTorch | 1 | 15.61 | 95.30 | 343.94 | +| | | Diffusers | 1 | | 303.64 | 956.04 | +| | multi_control | PyTorch | — | | | | +| | | Diffusers | — | | | | +| | seg | PyTorch | 1 | 18.14 | 102.72 | 358.43 | +| | | Diffusers | 1 | | 304.87 | 961.79 | +| | wsm | PyTorch | 1 | 15.66 | 76.04 | 258.05 | +| | | Diffusers | 1 | | 150.02 | 430.41 | +| **GB300** | blur | PyTorch | 1 | | | | +| | | Diffusers | 1 | | 268.46 | 806.60 | +| | depth | PyTorch | 1 | | | | +| | | Diffusers | 1 | | 271.61 | 822.16 | +| | edge | PyTorch | 1 | | | | +| | | Diffusers | 1 | | 260.92 | 811.23 | +| | multi_control | PyTorch | — | | | | +| | | Diffusers | — | | | | +| | seg | PyTorch | 1 | | | | +| | | Diffusers | 1 | | 268.54 | 804.36 | +| | wsm | PyTorch | 1 | | | | +| | | Diffusers | 1 | | 134.03 | 365.92 | + +Transfer notes: +1. Control hints follow the `extra_params` names: `blur`, `depth`, `edge`, `multi_control`, `seg`, and `wsm`. `multi_control` combines several hints in one request. +2. The PyTorch campaign reports `segmentation`, which is the same control as `seg` in the Diffusers campaign. +3. Diffusers FP8 has no `multi_control` measurements. + +## Action generation + +### Forward Dynamics — Autonomous Vehicle (AV) + +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 1 | 16.71 | 16.73 | 16.72 | +| | Diffusers | 1 | | | | +| **H20** | PyTorch | 1 | 40.31 | 40.29 | 40.28 | +| | Diffusers | 1 | | | | +| **H100 80GB HBM3** | PyTorch | 1 | 9.82 | 9.80 | 9.83 | +| | Diffusers | 1 | | 33.43 | 33.83 | +| **H200 141GB HBM3** | PyTorch | 1 | 9.48 | 9.63 | 9.48 | +| | Diffusers | 1 | | 32.07 | 32.65 | +| **B200** | PyTorch | 1 | 5.42 | 5.43 | 5.42 | +| | Diffusers | 1 | | 21.96 | 22.05 | +| **B300** | PyTorch | 1 | 5.07 | 5.06 | 5.19 | +| | Diffusers | 1 | | | 21.84 | +| **GB300** | PyTorch | 1 | | | | +| | Diffusers | 1 | | 19.76 | 20.41 | + +### Forward Dynamics — Camera + +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 1 | 15.77 | 15.75 | 15.78 | +| | Diffusers | 1 | | | | +| **H20** | PyTorch | 1 | 38.35 | 37.96 | 37.97 | +| | Diffusers | 1 | | | | +| **H100 80GB HBM3** | PyTorch | 1 | 9.25 | 9.28 | 9.25 | +| | Diffusers | 1 | | | | +| **H200 141GB HBM3** | PyTorch | 1 | 8.99 | 8.99 | 8.99 | +| | Diffusers | 1 | | | | +| **B200** | PyTorch | 1 | 5.14 | 5.15 | 5.12 | +| | Diffusers | 1 | | | | +| **B300** | PyTorch | 1 | 4.82 | 4.86 | 4.80 | +| | Diffusers | 1 | | | | + +### Forward Dynamics — Robot + +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 1 | 3.58 | 3.58 | 3.58 | +| | Diffusers | 1 | | 17.42 | | +| **H20** | PyTorch | 1 | 8.49 | 8.50 | 8.50 | +| | Diffusers | 1 | | | | +| **H100 80GB HBM3** | PyTorch | 1 | 2.14 | 2.14 | 2.14 | +| | Diffusers | 1 | | 13.90 | 14.04 | +| **H200 141GB HBM3** | PyTorch | 1 | 2.09 | 2.10 | 2.10 | +| | Diffusers | 1 | | 13.20 | 13.35 | +| **B200** | PyTorch | 1 | 1.27 | 1.26 | 1.28 | +| | Diffusers | 1 | | 10.60 | 10.71 | +| **B300** | PyTorch | 1 | 1.30 | 1.32 | 1.31 | +| | Diffusers | 1 | | 10.24 | 11.15 | +| **GB300** | PyTorch | 1 | | | | +| | Diffusers | 1 | | 10.71 | 12.12 | + +### Inverse Dynamics — Autonomous Vehicle (AV) + +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 1 | 28.37 | 28.36 | 28.29 | +| | Diffusers | 1 | | | 51.58 | +| **H20** | PyTorch | 1 | 68.74 | 68.70 | 68.72 | +| | Diffusers | 1 | | | | +| **H100 80GB HBM3** | PyTorch | 1 | 16.56 | 16.51 | 16.51 | +| | Diffusers | 1 | | 33.15 | 33.73 | +| **H200 141GB HBM3** | PyTorch | 1 | 15.96 | 15.97 | 15.95 | +| | Diffusers | 1 | | 31.93 | 32.37 | +| **B200** | PyTorch | 1 | 9.01 | 9.03 | 9.02 | +| | Diffusers | 1 | | 21.73 | 21.84 | +| **B300** | PyTorch | 1 | 8.51 | 8.49 | 8.51 | +| | Diffusers | 1 | | 20.85 | 20.99 | +| **GB300** | PyTorch | 1 | | | | +| | Diffusers | 1 | | 19.50 | 20.02 | + +### Inverse Dynamics — Robot + +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 1 | 5.72 | 5.72 | 5.73 | +| | Diffusers | 1 | | | | +| **H20** | PyTorch | 1 | 13.84 | 13.73 | 13.71 | +| | Diffusers | 1 | | | | +| **H100 80GB HBM3** | PyTorch | 1 | 3.39 | 3.39 | 3.39 | +| | Diffusers | 1 | | 13.91 | 14.02 | +| **H200 141GB HBM3** | PyTorch | 1 | 3.37 | 3.32 | 3.32 | +| | Diffusers | 1 | | 13.10 | 13.38 | +| **B200** | PyTorch | 1 | 1.97 | 1.97 | 1.96 | +| | Diffusers | 1 | | 10.57 | 10.70 | +| **B300** | PyTorch | 1 | 1.95 | 1.90 | 1.97 | +| | Diffusers | 1 | | 10.95 | 11.30 | +| **GB300** | PyTorch | 1 | | | | +| | Diffusers | 1 | | 12.74 | 13.76 | + +### Policy — Autonomous Vehicle (AV) + +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 1 | 16.74 | 16.75 | 16.73 | +| | Diffusers | 1 | | | | +| **H20** | PyTorch | 1 | 40.58 | 40.29 | 40.59 | +| | Diffusers | 1 | | | | +| **H100 80GB HBM3** | PyTorch | 1 | 9.80 | 9.83 | 9.81 | +| | Diffusers | 1 | | | | +| **H200 141GB HBM3** | PyTorch | 1 | 9.62 | 9.61 | 9.62 | +| | Diffusers | 1 | | | | +| **B200** | PyTorch | 1 | 5.42 | 5.33 | 5.40 | +| | Diffusers | 1 | | | | +| **B300** | PyTorch | 1 | 5.20 | 5.18 | 5.20 | +| | Diffusers | 1 | | | | + +### Policy — Robot + +| GPU | Engine | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| **RTX PRO 6000 Blackwell** | PyTorch | 1 | 3.42 | 3.41 | 3.42 | +| | Diffusers | 1 | | | | +| **H20** | PyTorch | 1 | 8.17 | 8.11 | 8.10 | +| | Diffusers | 1 | | | | +| **H100 80GB HBM3** | PyTorch | 1 | 2.05 | 2.05 | 2.04 | +| | Diffusers | 1 | | | | +| **H200 141GB HBM3** | PyTorch | 1 | 2.04 | 2.00 | 2.04 | +| | Diffusers | 1 | | | | +| **B200** | PyTorch | 1 | 1.23 | 1.21 | 1.24 | +| | Diffusers | 1 | | | | +| **B300** | PyTorch | 1 | 1.19 | 1.18 | 1.18 | +| | Diffusers | 1 | | | | + +Action and additional-modality notes: +1. Forward-dynamics cells for the AV domain are the mean of the forward, left, and right trajectories; inverse-dynamics cells are the mean of the two inverse trajectories. +2. The Diffusers FP8 campaign covers forward and inverse dynamics for the AV and Robot domains only, so the Camera and Policy rows carry PyTorch values alone. +3. Policy-DROID is a separate checkpoint and was not part of either FP8 campaign.