Skip to content

cookbook(diffusers): add FP8 quantized checkpoint support - #304

Open
ConstBob wants to merge 11 commits into
NVIDIA:mainfrom
ConstBob:feat/diffusers-fp8
Open

ConstBob wants to merge 11 commits into
NVIDIA:mainfrom
ConstBob:feat/diffusers-fp8

Conversation

@ConstBob

@ConstBob ConstBob commented Aug 6, 2026 •

Copy link
Copy Markdown
Collaborator

The cookbook content has moved out of this repo.
Please review the migrated changes here instead:
NVIDIA/cosmos-framework#298

Adds FP8 quantized checkpoint support to the Cosmos3 Diffusers cookbook for FP8 models.

ConstBob and others added 2 commits August 5, 2026 14:02
Enable Nano/Super and Super 4-Step FP8 via COSMOS3_DIFFUSERS_PRECISION
without duplicating request cells. Load public MODEL_IDS with revision=fp8
or a local COSMOS3_FP8_MODEL_PATH.
Comment thread cookbooks/cosmos3/generator/audiovisual/run_with_diffusers.ipynb Outdated
…er preload

Modular load_components passes repo+subfolder, which ModelOpt 0.44 ignores
when looking up modelopt_state.pth. Pre-load transformer from its full local
path, pin it, then load remaining components. Add verify_fp8 helper.
@ConstBob
ConstBob force-pushed the feat/diffusers-fp8 branch from 1831c57 to f4878b6 Compare August 6, 2026 18:29
Load Nano/Super/Edge FP8 from nvidia/Cosmos3-*-FP8 and keep Super 4-Step on revision=fp8. Rebase the notebook onto current audiovisual helpers without documenting mixed precision.
# Conflicts:
#	cookbooks/cosmos3/generator/audiovisual/run_with_diffusers.ipynb
Point users at the Cosmos3 Diffusers docs instead of duplicating generate-time kwargs.
…repos

Point forward dynamics, inverse dynamics, and video transfer at the Nano and Super FP8 model cards without changing their request cells.
@ConstBob
ConstBob requested a review from yzhautouskay October 17, 2026 00:13
Comment thread cookbooks/cosmos3/generator/audiovisual/run_with_diffusers.ipynb Outdated
Comment thread cookbooks/cosmos3/generator/audiovisual/run_with_diffusers.ipynb
Comment thread cookbooks/cosmos3/generator/audiovisual/run_with_diffusers.ipynb Outdated
Comment thread cookbooks/cosmos3/generator/action/run_fd_with_diffusers.ipynb Outdated
@ConstBob

Copy link
Copy Markdown
Collaborator Author

Addressed all review comments in dd7088e:

  • Explicitly fetches transformer/modelopt_state.pth before regular FP8 Hub loads and validates local checkpoint paths.
  • Automatically verifies FP8 weights, restored ModelOpt quantizers, and absence of meta tensors.
  • Fixes the distilled payload summary KeyError.
  • Removes the outdated private-repository notes.
  • Keeps non-transformer modular components on their configured BF16 sources, avoiding the FP8 repository’s root Reasoner state being applied to the VAE or audio tokenizer.

Validation completed:

  • Fresh-cache Nano FP8 restore: weights=504, quantizers=1533, meta=0
  • Nano FP8 35-step 720p T2I
  • Nano FP8 Action FD and ID
  • Nano FP8 Transfer: 121 frames, 50 steps
  • Distilled Super 4-Step FP8: weights=896, quantizers=2709, meta=0, followed by successful T2I generation
  • Notebook schema validation and Ruff checks passed

@yzhautouskay, could you please take another look when convenient?

@ConstBob
ConstBob requested a review from yzhautouskay October 19, 2026 18:47

@yzhautouskay yzhautouskay left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants