Add Cosmos3 FP8 quantization cookbook and serving examples by wkutak · Pull Request #298 · NVIDIA/cosmos

wkutak · 2026-07-30T12:55:14Z

New cookbooks/cosmos3/quantization/ ships a self-contained ModelOpt static-scale FP8 recipe:

src/: the Cosmos3 DiT model + full recipe (load, calibrate on real denoising, export a drop-in diffusers checkpoint).
notebooks/: quantize Nano/Super, Super Text-to-Image, and Super Image-to-Video (base + 4-step distilled).
Reproduces the shipped FP8 checkpoints bit-for-bit.

Also add a "Quantized Checkpoints" section to the vLLM-Omni generator notebooks (audiovisual, action, transfer) and the vLLM reasoner notebook: serving the FP8 build is just --revision fp8.

New cookbooks/cosmos3/quantization/ ships a self-contained ModelOpt static-scale FP8 recipe: - src/: the Cosmos3 DiT model + full recipe (load, calibrate on real denoising, export a drop-in diffusers checkpoint). - notebooks/: quantize Nano/Super, Super Text-to-Image, and Super Image-to-Video (base + 4-step distilled). - Reproduces the shipped FP8 checkpoints bit-for-bit. Also add a "Quantized Checkpoints" section to the vLLM-Omni generator notebooks (audiovisual, action, transfer) and the vLLM reasoner notebook: serving the FP8 build is just --revision fp8. Signed-off-by: Wojciech Kutak <wkutak@nvidia.com>

qmiao-hub · 2026-07-30T18:46:10Z

@wkutak , who is the tester validate the inference function and user guide in this PR? The person should be reviewer of this PR as well, approve once tested working with log test result in google doc.

Signed-off-by: Wojciech Kutak <wkutak@nvidia.com>

qmiao-hub · 2026-08-03T21:51:23Z

@KrzysztofTomala - please review and approve this PR once you completed testing.

KrzysztofTomala · 2026-08-04T10:50:59Z

vLLM cookbook

The model download and server startup works fine. I had to remove

 do_sample_frames": True

from the video example as the requests were failing, but the issue is happening also on the non-FP8 model.

KrzysztofTomala · 2026-08-04T10:53:20Z

vLLM-Omni cookbooks
The model download and server startup works fine
Audiovisual - everything works
Forward dynamics - AV works, Robotics - fail, but it is not FP8 released
Inverse dynamics - works
Transfer - I had to disable guardrails, as they block some examples. Without guardrails everything works

KrzysztofTomala · 2026-08-04T10:54:43Z

Quantization cookbooks

Everything works, there are a couple of warning, but nothing crucial.
I have been wondering why do we need 3 notebooks, not just one? Also maybe we should publish the notebooks without outputs, as they make scrolling through them inconvenient.

KrzysztofTomala

Overall everything works as expected, all of the serious problems are not FP8 related

…d99f1254459ef3a4f1bd4263' Signed-off-by: Wojciech Kutak <wkutak@nvidia.com>

… README, small fixes Signed-off-by: Wojciech Kutak <wkutak@nvidia.com>

MaciejBalaNV · 2026-08-06T12:49:05Z

The transfer example issue @KrzysztofTomala mentioned will be fixed here: #299
The forward dynamics issue is fixed here on the dataset sample level, I'll take another look if vllm-omni notebooks needs some fixes as well: #302

MaciejBalaNV · 2026-08-06T12:49:24Z

@wkutak Can you please fix the merge conflict?

Signed-off-by: Wojciech Kutak <wkutak@nvidia.com>

wkutak · 2026-08-07T13:50:45Z

Merge conflicts fixed, @rickzw please review the PR

Remove vendored Cosmos3 model code. Replace model, tokenizer, vae, and scheduler loading with cosmos-framework's API.

* Update dependency. * Use legacy calibration for all notebooks.

Migrate everything under `cookbooks/cosmos3/quantization/src` to `cosmos-framework`. Import paths in the notebooks are modified accordingly. Signed-off-by: Mu-Ti Chung <mutic@nvidia.com>

mutichung · 2026-09-03T15:26:19Z

The recent commits are to achieve the following goals:

Migrate from vendored model code to cosmos-framework implementation. Adds cosmos-framework as dependency.
Remove self-contained quantization helpers and update import paths to cosmos-framework. The helpers are moved into cosmos-framework in feat(quant): move quantization cookbook helpers to framework cosmos-framework#233.

Blocked by NVIDIA/cosmos-framework#233.

Moves all python helpers under `src/` from the quantization cookbooks in NVIDIA/cosmos#298 into `cosmos-framework`. The cookbooks will become thin wrappers that calls the functions to quantize checkpoints. Blocks NVIDIA/cosmos#298. --------- Signed-off-by: Mu-Ti Chung <mutic@nvidia.com> Co-authored-by: lfengad <liangf@nvidia.com> Co-authored-by: pengcuo <pzeren@nvidia.com>

qmiao-hub · 2026-09-05T00:50:15Z

Does it make sense to not have a folder containing 3 notebooks, but take those 3 notebook out of folder and have 3 notebook parallel to readme.md?

…de to cosmos-framework repo. Signed-off-by: Wojciech Kutak <wkutak@nvidia.com>

## Problem The original quantization pipeline did not create the corresponding `modelopt_state.pth` that is presented in the released FP8 checkpoints. The file is required for HuggingFace `transformers`/`diffusers` to load the and run inference on the quantized checkpoints. ## Changes * Export the missing `modelopt_state.pth` files for `transformers` and `diffusers`. * Fix updated functions imported by legacy mode. * Add mixed precision W8A16 policy configuration. Blocks NVIDIA/cosmos#298 --------- Signed-off-by: Mu-Ti Chung <mutic@nvidia.com> Signed-off-by: Wojciech Kutak <wkutak@nvidia.com> Co-authored-by: Wojciech Kutak <wkutak@nvidia.com>

Signed-off-by: Wojciech Kutak <wkutak@nvidia.com>

) Moves all python helpers under `src/` from the quantization cookbooks in NVIDIA/cosmos#298 into `cosmos-framework`. The cookbooks will become thin wrappers that calls the functions to quantize checkpoints. Blocks NVIDIA/cosmos#298. --------- Signed-off-by: Mu-Ti Chung <mutic@nvidia.com> Co-authored-by: lfengad <liangf@nvidia.com> Co-authored-by: pengcuo <pzeren@nvidia.com>

## Problem The original quantization pipeline did not create the corresponding `modelopt_state.pth` that is presented in the released FP8 checkpoints. The file is required for HuggingFace `transformers`/`diffusers` to load the and run inference on the quantized checkpoints. ## Changes * Export the missing `modelopt_state.pth` files for `transformers` and `diffusers`. * Fix updated functions imported by legacy mode. * Add mixed precision W8A16 policy configuration. Blocks NVIDIA/cosmos#298 --------- Signed-off-by: Mu-Ti Chung <mutic@nvidia.com> Signed-off-by: Wojciech Kutak <wkutak@nvidia.com> Co-authored-by: Wojciech Kutak <wkutak@nvidia.com>

Nano and Super FP8 checkpoints now live in dedicated repos instead of the fp8 revision of the bf16 repos. Update the vLLM-Omni generator notebooks (audiovisual, action fd/id, transfer), the vLLM reasoner notebook, and the quantization README. Other Super variants are unchanged. Also fix the stale quantize_nano_super link in the audiovisual notebook. Signed-off-by: Wojciech Kutak <wkutak@nvidia.com>

…P8 repos in README The pinned vllm-openai nightly tag no longer exists. v0.31.0 loads and serves nvidia/Cosmos3-Super-FP8. Replace the FP8 coming-soon entries for Super and Nano in the model table with links to the new FP8 repos. Signed-off-by: Wojciech Kutak <wkutak@nvidia.com>

Adopt the catalog repository layout and remove legacy cookbook copies now maintained in Cosmos Framework. Preserve untracked workspace files.

…cipe Retain the latest main README and tracked Framework links while adding the approved FP8 model table, precision notes, and S/M quantization catalog budgets.

yaoxu-crypto · 2026-10-28T05:45:44Z

@wkutak , in the COOKBOOK.md, can you change the word "Quantization" to "Quantized (FP8)"?

Signed-off-by: Wojciech Kutak <wkutak@nvidia.com>

pengcuo

LGTM. Docs-only; the COOKBOOK.md#inference anchor resolves. The new quantization link targets cosmos-framework main and 404s until cosmos-framework#329 merges, so land that first. See the note there about aligning the fp8-branch availability claim in docs/reference/models.md.

Withdrawn by reviewer pending further checks (tests, checkpoint availability, doc consistency).

pengcuo

Docs-only change; COOKBOOK.md#inference resolves. Two things to fix before this goes onto the public README (re-posting after withdrawing my earlier approve to check properly):

Blocking

README.md and docs/reference/models.md link nvidia/Cosmos3-Nano-FP8 and nvidia/Cosmos3-Super-FP8, which are currently private on Hugging Face (unauthenticated -> 401/404). External readers of this README will get dead links. Please make the repos public before merging, or hold this PR until they are.
The new [quantization] link in COOKBOOK.md points at cosmos-framework/blob/main/.agents/skills/cookbooks/cosmos3/quantization/README.md, which 404s until NVIDIA/cosmos-framework#329 merges. Land that first. (Nit: every other cookbook link in that table is an nvda.ws short link.)

Non-blocking

The four Super fp8 branches in models.md do exist publicly and contain hf_quant_config.json, so that table is fine; cosmos-framework#329's README currently says it does not assume those revisions exist -- worth aligning the two.
PR description says the recipe "reproduces the shipped FP8 checkpoints bit-for-bit" and that serving is "just --revision fp8". The cosmos-framework#329 README says neither mode guarantees reproduction of published scales, and that --revision alone is not honored by the vLLM-Omni Cosmos3 pipeline. The committed docs here are consistent with #329; just flagging the description so nobody quotes it.

…t column - docs/reference/models.md: "Quantization" heading -> "Quantized models". - The quantized-checkpoint table is now an HTML table with `nowrap` on the first column, so each checkpoint name stays on one line (GitHub strips <nobr> and style attributes, but keeps `nowrap`). - models.md and README.md: "quantization cookbook" -> "cookbook". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Yao Xu <yaoxu@nvidia.com>

yaoxu-crypto · 2026-10-29T23:24:37Z

@wkutak , approved, please merge once the models are live

wkutak marked this pull request as ready for review July 30, 2026 13:01

wkutak changed the title ~~[DRAFT] Add Cosmos3 FP8 quantization cookbook and serving examples~~ Add Cosmos3 FP8 quantization cookbook and serving examples Jul 30, 2026

qmiao-hub requested a review from rickzw July 30, 2026 18:06

qmiao-hub self-requested a review July 30, 2026 21:15

Removed --revision argument from vllm-omni commands

a00d3c3

Signed-off-by: Wojciech Kutak <wkutak@nvidia.com>

qmiao-hub requested a review from KrzysztofTomala August 3, 2026 17:05

KrzysztofTomala approved these changes Aug 4, 2026

View reviewed changes

KrzysztofTomala reviewed Aug 4, 2026

View reviewed changes

Comment thread cookbooks/cosmos3/quantization/notebooks/quantize_nano_super.ipynb Resolved

KrzysztofTomala reviewed Aug 4, 2026

View reviewed changes

Comment thread cookbooks/cosmos3/quantization/README.md Resolved

KrzysztofTomala reviewed Aug 4, 2026

View reviewed changes

Comment thread cookbooks/cosmos3/generator/audiovisual/run_with_vllm_omni.ipynb Resolved

wkutak added 2 commits August 4, 2026 06:34

vLLM image version change from 'cosmos3' to 'nightly-cb8104839c141609…

b75431f

…d99f1254459ef3a4f1bd4263' Signed-off-by: Wojciech Kutak <wkutak@nvidia.com>

Convert all quantize paths to quantization, remove dead comments from…

a25fbc6

… README, small fixes Signed-off-by: Wojciech Kutak <wkutak@nvidia.com>

wkutak requested a review from KrzysztofTomala August 4, 2026 13:46

KrzysztofTomala approved these changes Aug 4, 2026

View reviewed changes

wkutak added 2 commits August 7, 2026 06:32

Fixed vllm inference bugs, removed too long notebook output cells

e635690

Signed-off-by: Wojciech Kutak <wkutak@nvidia.com>

Merge branch 'main' into cosmos3-fp8-quantization-recipe

38da028

mutichung added 5 commits September 2, 2026 07:09

feat: migrate model to cosmos-framework

f84223f

Remove vendored Cosmos3 model code. Replace model, tokenizer, vae, and scheduler loading with cosmos-framework's API.

feat: migrate export to support cosmos-framework

e7f7b9b

feat: calibration changes

9de82fe

feat: add legacy calibration option

5b15718

feat: extend legacy calibration to image profiles

c51bc15

mutichung added 3 commits September 3, 2026 13:00

fix: support framework VAE image conditioning

83ae4ac

fix: resolve notebook issues

5de108e

* Update dependency. * Use legacy calibration for all notebooks.

Migrate healpers to cosmos-framework

dfce0fb

Migrate everything under `cookbooks/cosmos3/quantization/src` to `cosmos-framework`. Import paths in the notebooks are modified accordingly. Signed-off-by: Mu-Ti Chung <mutic@nvidia.com>

mutichung mentioned this pull request Sep 3, 2026

feat(quant): move quantization cookbook helpers to framework NVIDIA/cosmos-framework#233

Merged

qmiao-hub requested a review from yaoxu-crypto September 5, 2026 00:45

mutichung mentioned this pull request Sep 14, 2026

fix(quant): export modelopt_state.pth after quantization NVIDIA/cosmos-framework#256

Merged

Update to quantization and inference cookbooks, migrate the source co…

2341d84

…de to cosmos-framework repo. Signed-off-by: Wojciech Kutak <wkutak@nvidia.com>

Fixing commented code.

c1c64dc

Signed-off-by: Wojciech Kutak <wkutak@nvidia.com>

wkutak added 6 commits October 14, 2026 05:42

docs: catalog Cosmos3 FP8 quantization recipes and models

e7eb7b0

docs: list FP8 checkpoints under model family and set GPU budgets

f95f9ae

Merge origin/main into cosmos3-fp8-quantization-recipe

cca3cc8

Adopt the catalog repository layout and remove legacy cookbook copies now maintained in Cosmos Framework. Preserve untracked workspace files.

Merge reviewed FP8 catalog migration into cosmos3-fp8-quantization-re…

a326938

…cipe Retain the latest main README and tracked Framework links while adding the approved FP8 model table, precision notes, and S/M quantization catalog budgets.

docs: rename Quantization to Quantized (FP8) in COOKBOOK.md

decccab

Signed-off-by: Wojciech Kutak <wkutak@nvidia.com>

pengcuo previously approved these changes Oct 29, 2026

View reviewed changes

pengcuo reviewed Oct 29, 2026

View reviewed changes

yaoxu-crypto approved these changes Oct 29, 2026

View reviewed changes

yaoxu-crypto added this to the FP8 Release milestone Oct 30, 2026

Conversation

wkutak commented Jul 30, 2026

Uh oh!

qmiao-hub commented Jul 30, 2026

Uh oh!

qmiao-hub commented Aug 3, 2026

Uh oh!

KrzysztofTomala commented Aug 4, 2026

Uh oh!

KrzysztofTomala commented Aug 4, 2026

Uh oh!

KrzysztofTomala commented Aug 4, 2026

Uh oh!

KrzysztofTomala left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MaciejBalaNV commented Aug 6, 2026

Uh oh!

MaciejBalaNV commented Aug 6, 2026

Uh oh!

wkutak commented Aug 7, 2026

Uh oh!

mutichung commented Sep 3, 2026

Uh oh!

qmiao-hub commented Sep 5, 2026

Uh oh!

yaoxu-crypto commented Oct 28, 2026

Uh oh!

pengcuo left a comment

Choose a reason for hiding this comment

Uh oh!

pengcuo left a comment

Choose a reason for hiding this comment

Uh oh!

yaoxu-crypto commented Oct 29, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants