Repository navigation
Split Cosmos3 Generator benchmarks and add vLLM-Omni audiovisual/action results - #352
Merged
Merged
Conversation
lfengad
approved these changes
Sep 15, 2026
Refactor inference_benchmarks.md into an index that separates Generator and Reasoner results. Generator coverage now points at model-specific pages so audiovisual and action modalities can be reported without collapsing this page, while the Reasoner serving sections stay inline.
Split Edge, Nano, and Super generator results into dedicated pages so audiovisual and action modalities can be reported without collapsing the landing page.
Fill t2av/i2av from PBR 308195 and AV action plus Policy-DROID from PBR 308481 so the new modality tables are no longer empty reservations.
The per-resolution tables previously crossed each input resolution with 1, 4, and 8 GPU counts, which made them wide and left most cells empty. Each row now reports a single configuration: one GPU when the model fits at the model's highest resolution, otherwise eight. A new GPUs column records which configuration a row's values come from. Cosmos3-Edge gains explicit 256p and 480p columns, 480p being its highest resolution; 256p has not been measured yet. Diffusers numbers are filled in from PBR #308202 (all modalities), PBR #308451 (video-to-video), and PBR #308918 (4- and 8-GPU runs, taking the fastest tensor- and context-parallel split at each GPU count). The 1-GPU results in PBR #308918 are not used because that sweep ran a different denoising-step budget. Seventeen B300 values are corrected to the figures from the newer reports.
Transfer conditions a video-to-video generation on a structural control hint extracted from the source video. Latencies from PBR #308587 are reported per control (blur, depth, edge, seg, wsm) so the cost of each hint is visible, since the hint determines how much of the frame the model must synthesise. Only Diffusers has transfer coverage so far, all on a single GPU.
Four reports refresh and extend the vLLM-Omni numbers: PBR #307999 (t2i, t2v, i2v), PBR #308195 (t2vs, i2vs), PBR #308481 (action), and PBR #308574 (transfer). The transfer table now carries both engines rather than Diffusers alone, and gains a multi_control row group for the combined-hint workload that only vLLM-Omni has measured. Super gains vLLM-Omni i2v and t2i on the four GPUs it previously lacked. Eleven published 720p values are corrected to the figures from these reports. Most differ only in rounding; the largest change is nano t2v on H100 NVL, from 311.13 to 281.99. Under vLLM-Omni, Super fits on a single GPU on every GPU measured, which the Super overview now states, since the other engines still need eight on the smaller-memory parts.
The tables are read across engines, so a vLLM-Omni row reporting one GPU next to three rows reporting eight invites a false comparison. Every engine in a GPU block now reports the same configuration. This moves eight Super rows, i2v and t2i on RTX PRO 6000 Blackwell, H20, H100 NVL, and H100 80GB HBM3, from one GPU to eight. vLLM-Omni does fit on a single GPU there, but it is the only engine that does, so the eight-GPU figures are the comparable ones. The H200 blocks keep their one-GPU values. PyTorch alone needs eight there, and it appears to be missing a single-GPU run rather than failing to fit, so raising those blocks would retire good numbers from three engines to accommodate one gap. The 720p i2v cell for vLLM-Omni on H100 80GB HBM3 is left empty: PBR #307999 reports 2.65 s at four GPUs and 6.53 s at eight against 38-64 s at 480p, which cannot be right.
PBR #308000 covers TensorRT-LLM across all three models. It sweeps tensor-parallel, CFG-parallel, and Ulysses-parallel splits at each GPU count, so the fastest split at a row's count is the one published, the same convention used for the multi-GPU Diffusers report. Nano and Super gain a TensorRT-LLM row in every GPU block, placed between Diffusers and NIM, carrying 216 and 206 values. The campaign covers six GPUs, so the H100 NVL and H200 NVL rows are empty, and it includes no action workloads. The transfer table now lists all five engines instead of only the two that had data, and picks each block's GPU count to surface the most measurements. The Edge page is reorganised around modalities rather than engines, because this report adds t2i, t2v, and v2v to a page that only carried i2v, and adds the first 256p results. Its existing i2v numbers are unchanged.
The FP8 checkpoints get their own pages rather than extra rows on the BF16 pages, since they are distinct models. PyTorch values come from PBR #309161 and Diffusers values from PBR #309434. Coverage is uneven between the two runtimes, which the methodology records: PyTorch covers 1, 4, and 8 GPUs at all three resolutions, while Diffusers is single-GPU at 480p and 720p only. Neither campaign ran on H100 NVL or H200 NVL, so those GPUs are absent from these pages, and GB300 appears only where the Diffusers campaign measured it. Cosmos3-Super-FP8 needs eight GPUs on H100 80GB HBM3, where its single-GPU 720p runs do not fit; every other block fits on one GPU. One result is worth reading before picking a precision, so it is footnoted: FP8 wins throughout for PyTorch, but for Diffusers it only pays off on long video workloads. Diffusers t2i is about twice as slow in FP8 as in BF16 in every cell measured.
sfefilatyev
force-pushed
the
2026-09-14-xdvz
branch
from
October 2, 2026 23:51
9bb29b4 to
8b9363e
Compare
lfengad
approved these changes
Oct 6, 2026
xlu451
approved these changes
Oct 6, 2026
yaoxu-crypto
pushed a commit
that referenced
this pull request
Oct 21, 2026
## Summary Makes the contents of `NVIDIA/cosmos` identical to `nvidia-cosmos/cosmos-private` (main @ `b636c91`). After this merges, `git diff` between the two trees is empty. ## What changes - **Deleted (1163 files, ~492k lines):** `evaluation/cosmos3/` (921), `cookbooks/cosmos3/` (239), `inference_benchmarks*`, `RELEASE.md`, `CONTRIBUTING.md`, one `.github/workflows` file - **Added:** `AGENTS.md`, `COOKBOOK.md`, `assets/ASSET_SPECS.md`, brand badge and demo videos under `assets/` - **Modified:** `README.md`, `.gitignore`, `docs/reference/models.md` - **Reverted:** the 19 commits on `NVIDIA/cosmos` main that are not in cosmos-private (e.g. #380, #373, #352) The branch is a single commit on top of current `main`, so history is preserved and this is revertable. ## Please check before merging - The deletions are large and intentional per the request. Confirm nothing removed here is still needed publicly. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
inference_benchmarks.md.#308197, plus vLLM-Omnit2av/i2avfrom#308195and AV action plus Policy-DROID from#308481.Test plan
t2avandi2avvLLM-Omni cells against PBR#308195(1/4/8 GPU columns).#308481; confirm/8action cells stay empty and the two documented misses remain blank.inference_benchmarks.mdlinks resolve to the new model pages.