Repository navigation
fix(quant): remove nn.Embeddings quantizers - #318
Merged
Merged
Conversation
Remove extra quantizers for `nn.Embeddings` not supported in modelopt v0.44.0. Signed-off-by: Mu-Ti Chung <mutic@nvidia.com>
pengcuo
approved these changes
Oct 27, 2026
lfengad
approved these changes
Oct 27, 2026
erichudev
pushed a commit
to KunlunxinAD/cosmos-framework
that referenced
this pull request
Nov 3, 2026
* Release 2026-10-26 (i4 09e71fb6) (#310)
Automated release from i4.
_source_commit: `09e71fb667f046f9f81f3935f13cd3b2143fd576-dirty`
_dest_commit (base): `4f6e539f98d1e7951e1c0907e61e2ff3ccdbdc92`
Co-authored-by: Liang Feng <liangf@nvidia.com>
* cookbook: add Cosmos3-Edge TensorRT-Edge-LLM reasoner and policy notebooks (#300)
## Summary
Moves the Cosmos3-Edge TensorRT-Edge-LLM user guide from
https://github.com/NVIDIA/cosmos/pull/326 into this repo, under
`.agents/skills/cookbooks/cosmos3/`.
- Shared setup in `.agents/skills/cookbooks/cosmos3/README.md`, pinned
to TensorRT-Edge-LLM v0.11.0. `EDGELLM_ROOT` defaults to
`$HOME/TensorRT-Edge-LLM`.
- Reasoner notebook: export `nvidia/Cosmos3-Edge` with `--task
reasoning`, build engines under `ENGINE_DIR/reasoning`, and run
`llm_inference` on `assets/robot_153.jpg`.
- Policy notebook: on v0.11.0, build `und-prefill`, `gen`, and
`vae-encoder` with `tensorrt-edgellm-build` under `ENGINE_DIR/policy`,
then run `cosmos3_policy_inference`. The C++ runtime needs
`-DBUILD_EXPERIMENTAL_MODELS=ON` and `-DENABLE_CUTE_DSL=ALL`. It uses
the same 640×540 DROID `concat_view` as the vLLM-Omni / SGLang policy
notebooks, with `--viewPoint concat_view` and an 8-value `--state`.
Edge-LLM covers the policy task only.
- Notebooks resolve assets from `.agents/skills/cookbooks/cosmos3`.
On the v0.11.0 tag, `tensorrt-edgellm-export --task policy` writes the
UND input `attention_pos_id` and VAE `height` / `width` only.
`cosmos3_policy_inference` binds `attention_position_ids` and reads
`builder_config.content_height` / `content_width`. The direct builder
writes that contract, so the policy cookbook uses
`tensorrt-edgellm-build`.
Upstream references: [Cosmos3-Edge VLA
guide](https://nvidia.github.io/TensorRT-Edge-LLM/user_guide/examples/vla/cosmos3.html)
for reasoning, and the [direct engine
builder](https://nvidia.github.io/TensorRT-Edge-LLM/user_guide/getting_started/direct-engine-builder.html)
for policy.
## Testing
- Asset paths resolve from both notebook directories: `robot_153.jpg`
and the three DROID camera videos are found.
- Policy, pinned v0.11.0 (`95515c2`), x86 RTX PRO 6000 (SM120):
`tensorrt-edgellm-build --components und-prefill,gen,vae-encoder`, then
`cosmos3_policy_inference --viewPoint concat_view --state
0,0,0,0,0,0,0,0`. `action.json` is `[1, 32, 8]`, `finite=true`. The zero
state is a runtime smoke, not a DROID quality check. v0.11.0
`cosmos3_policy_inference --help` lists `concat_view`.
- Reasoner was not re-run on v0.11.0. The last full export → `llm_build`
+ `visual_build` → `llm_inference` sequence was TensorRT-Edge-LLM
v0.10.0 on the same RTX PRO 6000. After Ever's visual-export fix,
`llm_inference` was re-run on that box for `robot_153.jpg` and
`describe_anything.png`, and the captions matched the images. Ever
confirmed normal outputs on Thor. See the thread on NVIDIA/cosmos#326.
That retest did not name the tag as v0.11.0.
---------
Co-authored-by: pengcuo <pzeren@nvidia.com>
* fix(quant): remove nn.Embeddings quantizers (#318)
## Summary
Remove extra `nn.Embeddings` quantizers that are not supported in
ModelOpt v0.44.0.
## Issue
Originally, edge was quantized with ModelOpt v0.47.0, while nano/super
were quantized with v0.44.0. ModelOpt has `nn.Embeddings` support in
v0.47.0, which leads to restoration failure when loading the checkpoint
with v0.44.0. We decided to revert the ModelOpt version to v0.44.0 to
make all Cosmos3 FP8 checkpoints consistent.
<details>
<summary>Example error message</summary>
```
raise ApplyModeError(f"Unmatched keys in quantizer state_dict: {unmatched_keys}")
modelopt.torch.opt.conversion.ApplyModeError: Unmatched keys in quantizer state_dict: {'embed_tokens.input_quantizer', 'action_proj_in.bias.input_quantizer', 'embed_tokens.weight_quantizer', 'action_proj_out.fc.output_quantizer', 'action_proj_out.bias.weight_quantizer', 'action_proj_in.bias.output_quantizer', 'action_proj_out.fc.input_quantizer', 'action_proj_in.fc.output_quantizer', 'action_proj_in.fc.input_quantizer', 'action_proj_in.bias.weight_quantizer', 'action_proj_out.fc.weight_quantizer', 'action_proj_in.fc.weight_quantizer', 'embed_tokens.output_quantizer', 'action_proj_out.bias.output_quantizer', 'action_proj_out.bias.input_quantizer'}
```
</details>
## Fix
Additional quantizer metadata for `nn.Embeddings` are removed. Those
layers remain bfloat16 regardless of ModelOpt version, so removing the
layer metadata resolves the issue.
Signed-off-by: Mu-Ti Chung <mutic@nvidia.com>
* Add Cosmos3 Diffusers CPU-GPU offloading package (#320)
* docs(auto): clarify recommended inference configuration for 10-FPS inputs (#316)
## Summary
Make the recommended Transfer-Auto configuration for 10-FPS RGB-only
inputs explicit:
- Keep synchronized 10-FPS input frames unchanged.
- Keep model FPS conditioning at 30 (`fps: 30` / `--fps 30`), matching
training.
- Encode the generated frames at 10 FPS for the intended playback speed.
Replace the previous recommendation to match model FPS to source FPS and
the "diagnostic only" framing. Explain that changing the model's FPS
conditioning to 10 has been reported to reduce quality; this PR does not
claim a new model-quality evaluation.
Include a frame-preserving FFmpeg playback conversion, distinguish
caption frame bounds from textual timestamps, and retain the separate
joint RGB/LiDAR clock/alignment requirement. Update the inference
skill's short pointer so it no longer contradicts the reference
document. Preserve the existing FPS section anchor. No runtime changes.
## Validation
- Repository pre-commit checks passed for both edited Markdown files.
- `git diff --check` passed.
- Synthetic FFmpeg smoke test: 297 decoded frames before and after
conversion; 30 FPS / 9.9 seconds becomes 10 FPS / 29.7 seconds.
- Checked the unchanged source-FPS/model-FPS/saver behavior against the
current multiview inference implementation.
- No model inference or quality benchmark was run for this
documentation-only change.
Signed-off-by: Qianli Ma <qianlim@nvidia.com>
* Merge LoRA adapters safely in automatic Hugging Face exports (#308)
Automatic HF checkpoint export wrote LoRA base and adapter parameters
separately, so a plain Hugging Face loader could silently use unmerged
weights. This PR exports merged dense LoRA/RSLoRA weights without
changing live training parameters.
- Merge detached, fully gathered tensors in at least FP32 with
TF32/autocast disabled, restore the caller's CUDA matmul precision, then
cast the completed result to the output dtype. Remove only adapter keys
owned by native `LoraInjectedLinear` modules.
- Validate shared weights before parameter deduplication or tensor
collectives. Reject a base shared between plain and adapted paths, or
distinct adapters, in either registration order. Preserve ordinary
non-LoRA tied weights and aliases of the same LoRA module.
- Gather base/A/B tensors in the same order on every rank. Give CPU
snapshots independent storage so background writes cannot observe
subsequent optimizer updates. Retain wrapper normalization and collision
checks.
- Preserve the pre-PR implementations of training `forward()` and the
existing in-place/offline merge APIs, including their rounding and
ambient-autocast behavior. The new numerical policy applies only to
automatic export.
Validation of `0b43ed1` on H100 with locked `cu128-train` dependencies:
- 47 focused CPU/CUDA tests passed, including local HF reloads,
LoRA/RSLoRA, BF16/FP32 exports, aliases, snapshots, legacy numerical
boundaries, and CUDA precision restoration on success and failure.
- Ten FSDP cases passed on each of two ranks: dense, LoRA and
tied-embedding reloads with TF32 on/off, both ambiguous-alias orders,
and rank-local failure propagation. Original reload tolerances are
unchanged.
- Workflow-selected L0 pytest reported **781 passed, 13 skipped**. Ten
skips are the separately exercised FSDP cases; three are existing
architecture/NATTEN guards. Isolated attention had **1 pass and 1
existing capability skip**.
- CFG parallelism: **3 passed per rank** on two ranks. Context
parallelism: **12 passed per rank** on four ranks, retaining the
workflow's existing deselection.
- Changed-file Ruff/format and Pyrefly (zero errors), both full
pre-commit configurations, source-hash checks, legacy-API AST
comparison, and `git diff --check` passed.
These local runs use H100/Python 3.13.14 rather than hosted H200/Python
3.13.15. The L0 pytest report has zero failures, but its outer local
shell encountered a post-test runner-edit error; subsequent invocations
of the finalized runner exit successfully. See [the CI investigation and
complete validation
limits](https://github.com/NVIDIA/cosmos-framework/pull/308#issuecomment-6426471457).
Hosted CI must still validate the new head.
The focused reloads use a tiny randomly initialized Qwen2 model and
local safetensors. This revision does not claim new full-training,
remote-upload, or downstream execution results.
Backport requirements: corrected LoRA artifacts, independent CPU
snapshots, and explicit rejection of ambiguous aliases are intentional
behavior changes. Preserve unrelated valid workflows, configuration
defaults, training state, and the legacy merge numerical policy. This is
not a zero-impact backport.
`.file_mapping.json` maps the two production files, but neither their
public test files nor the GitHub GPU workflow. Carry equivalent coverage
from `hf_export_test.py` and `lora_test.py`, adapt destination imports,
and preserve existing downstream tests. Include non-LoRA exports, valid
ties, alias-order rejection, snapshot independence, BF16/autocast
boundaries, TF32 on/off, precision-setting restoration, and distributed
gather/rejection/failure-propagation checks. Run the distributed cases
explicitly; single-process pytest skips them.
```shell
python -m pytest cosmos_framework/callbacks/hf_export_test.py cosmos_framework/utils/generator/lora_test.py -o addopts= -m L0
torchrun --nproc_per_node=2 -m pytest cosmos_framework/callbacks/hf_export_test.py -o addopts= -m L0 -k fsdp
```
---------
Signed-off-by: Ramanathan Arunachalam <rarunachalam@nvidia.com>
Co-authored-by: lfengad <liangf@nvidia.com>
* feat(ludus-renderer): add Ludus GPU renderer package from i4 (#322)
## Summary
Adds `packages/ludus-renderer`, a standalone package for the Ludus
GPU-native F-theta renderer (HD-map and obstacle rendering for
autonomous-driving scenes, with an OpenGL mesh-shader backend and a
GL-free CUDA backend). It is ported from imaginaire4
`packages/cosmos-auto/ludus_renderer` at `origin/main` `e87023193d3`.
- **Code:** copied verbatim. `ludus_renderer` is self-contained in i4:
it does not import `cosmos_auto` or any i4 module.
- **Headers:** every file went through the i4 release transform
(`cosmos-framework-release/rewrite_imports.rewrite_text`). Proprietary
headers became the OpenMDW-1.1 SPDX header, and files with no header got
one. The transform also strips TODO comments, which removed one line in
`clipgt.py`. No other body lines changed.
- **Packaging:** `pyproject.toml` has the same runtime dependencies and
the same `cpu` / `cuda11` / `cuda12` / `cuda13` extras as i4's
`cosmos-auto`. The build backend is hatchling, like the other packages
here. `uv.lock` is generated.
- **Scope:** the Ludus README sections, plus `tests/ludus` → `tests/`.
i4's benchmark/example scripts and internal docs are not included.
## ⚠️ License: OSRB review needed before merge
22 C++ files are derived from
[nvdiffrast](https://github.com/NVlabs/nvdiffrast): `_cpp/cudaraster/*`,
`_cpp/common/*`, and `_cpp/bindings/{torch_common.inl,torch_types.h}`.
Several are byte-identical to upstream.
Upstream nvdiffrast is under the **NVIDIA Source Code License (1-Way
Commercial)**, which is non-commercial for third parties. In i4, part of
`cudaraster/` was relabelled Apache-2.0, and a TODO in `CudaRaster.hpp`
says that relicense is still pending. This PR does **not** relicense
them:
- each file keeps its upstream nvdiffrast header verbatim
- the nvdiffrast LICENSE ships at
`ludus_renderer/licenses/nvdiffrast/LICENSE.txt`
- nvdiffrast is listed in `ATTRIBUTIONS.md`
- the README states which parts are under that license
Please confirm with OSRB and the Ludus authors whether these files can
be relicensed (OpenMDW-1.1 or Apache-2.0) before this leaves draft.
## Testing
Ran on GB200 in the NGC `pytorch:26.06` image (torch 2.13, CUDA 13.3):
- `pytest tests`: 13 passed (nvJPEG and CUDA toolchain tests)
- `ludus_renderer.torch.LudusCudaTimestampedContext`: the CUDA-backend
extension JIT-builds and initializes
- GL backend: not verified. It fails to build in this image because the
image has no `EGL/egl.h`. i4's original source fails the same way in the
same image.
- `ruff check`, `rumdl` on the README, and `uv lock --check` all pass.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(auto): separate output FPS from model conditioning (#323)
## Summary
Follow-up to merged #316: make the documented 10-FPS RGB-only Auto
configuration a runtime option rather than a manual FFmpeg
postprocessing step.
Users keep synchronized 10-FPS RGB/WSM input frames unchanged and add:
```bash
--output-fps 10
```
Auto model conditioning defaults to 30 FPS. An explicit different `fps`,
including one in input JSON or a custom defaults file, is rejected with
guidance to use `output_fps`. Existing requests that explicitly use
`fps=30` continue working.
## Implementation and scope
- Identify Auto through existing exported task metadata, with exact
official HF repository IDs as a fallback. No released checkpoint/config
files are modified, and local directory names, view counts and
`base_fps` are not used to guess model identity.
- Attach the inference-only policy to the runtime model config. Keep the
shared argument-builder signature and other models' existing FPS
defaults/validation unchanged. Revalidate already-built samples at the
shared generation entry point.
- Add optional top-level `output_fps` / CLI `--output-fps`. Only the
bidirectional and causal Auto video savers use the resolved playback
rate, consistently for generated RGB and WSM controls. No frame
dropping/duplication/interpolation; model FPS tensors, frame bounds,
prompts and sampling are unchanged.
- Record both FPS values in saved arguments and log the two clocks
explicitly.
- Initially support different playback FPS only for RGB-only Auto. Joint
RGB/LiDAR requires output FPS to match its 30-FPS camera clock; other
models and LiDAR-only requests reject the new option instead of silently
ignoring it.
- Update the shared inference documentation and Transfer-Auto/Sim-Auto
guides. Preserve the timestamped-caption/model-clock and joint-sensor
alignment caveats.
Both current published Auto configs were checked read-only and already
carry `per_view_caption_task=av_joint_camera_lidar_transfer`:
- Transfer-Auto: `ac871d0981046cfd3e7bf99e57c449520b1f28c3`
- Sim-Auto: `6f537e5f07306be5ff3f776f285d69ffe7bc4e2f`
This covers the native `cosmos_framework.scripts.inference` pipeline; it
does not add a new option to independent Diffusers/vLLM entry points.
## Validation
- **414 tests passed**, covering the new Auto FPS tests, caption
metadata, shared arguments, Sim-Auto/Sim-Transfer arguments, inference
construction, causal multiview saving/dispatch and LiDAR transfer.
- New tests exercise CLI/JSON/custom-defaults precedence, 1-camera Auto
recognition, unsupported modes, and unchanged actual model batch
tensors/caption text when only `output_fps` changes.
- Real RGB and WSM MP4 encoding plus ffprobe: **300 frames at 10 FPS =
30 seconds**, with no lost frames. Existing causal saver tests also
cover 10-FPS RGB/WSM output and unchanged joint output.
- Ruff check/format, Pyrefly 0.55.0 on affected Python files (zero
errors), pre-commit hooks and diff whitespace checks passed.
- Updated one existing stale FPS-error assertion to match the current
runtime wording (`trained source-frame clock`).
Two unrelated tests were deselected from the final CPU regression
command: the remote checkpoint-registry download test encountered a
local uv cache installation error; the reasoner-output test requires
CUDA and also fails on the unchanged base in this CPU environment. No
GPU model generation or new 10-FPS visual-quality benchmark was run.
Regression command:
```bash
python -m pytest \
cosmos_framework/inference/auto_fps_test.py \
cosmos_framework/inference/common/multiview_captions_test.py \
cosmos_framework/inference/args_test.py \
cosmos_framework/inference/sim_auto_args_test.py \
cosmos_framework/inference/sim_transfer_args_test.py \
cosmos_framework/inference/inference_test.py \
cosmos_framework/inference/multiview_causal_test.py \
cosmos_framework/inference/sim_auto_dispatch_test.py \
cosmos_framework/inference/lidar_transfer_test.py \
-k 'not test_checkpoints and not test_generate_reasoner_batch_writes_outputs' \
--num-gpus=0 -q
```
Signed-off-by: Qianli Ma <qianlim@nvidia.com>
Co-authored-by: lfengad <liangf@nvidia.com>
* docs: clarify recommended checkpoint and KV-cache precision for streaming (#328)
Streaming quality depends on checkpoint and KV-cache precision. Update
the NIM streaming guide, streaming configuration reference, and
deployment precision guidance to recommend either:
- An FP8 checkpoint with BF16 KV-cache precision.
- A BF16 checkpoint.
NVFP4 checkpoints are currently not recommended for streaming.
* docs: update Cosmos3 TensorRT-LLM guardrail setup to 0.3.2 (#332)
The moved Cosmos3 cookbooks still pin `cosmos_guardrail==0.3.0` and
require manual server-side NLTK setup. Pin `cosmos_guardrail==0.3.2`,
which copies its NLTK data from the Hugging Face cache on first use, and
remove the obsolete setup instructions from the Action and Transfer
READMEs and notebooks. Update the TensorRT-LLM source pin to
`cc63e66daf52644f1441e117bb7ba67d20566016`.
Port of https://github.com/NVIDIA/cosmos/pull/378 to the cookbooks' new
location. Clarify that the empty video-content classifier warning does
not disable the configured text checks or face blurring.
Signed-off-by: Igor Shovkun <ishovkun@nvidia.com>
* Add Cosmos3 FP8 quantization cookbooks and serving examples (#329)
# DO NOT MERGE IF PENDING SIGN-OFF
Move the Cosmos3 FP8 recipes into Framework’s cookbook layout and
integrate FP8 serving instructions into the current audiovisual, forward
dynamics, inverse dynamics, transfer, and reasoner notebooks.
The three quantization notebooks cover six exports: Nano/Super video and
Super base/4-step T2I/I2V. They install from the local Framework
checkout, resolve paths from either the checkout root or notebook
directory, and support DEMO and full calibration. Serving examples
document published checkpoints, locally exported checkpoints, Docker
mounts, and activation precision policies.
GPU validation also exposed two issues fixed here:
- Concatenate native denoiser predictions along their existing batch
dimension, preventing an extra dimension from breaking distilled
calibration.
- Collect model reference cycles and release unused CUDA cache before
loading the next model in each notebook.
Validation:
- All three notebooks executed in DEMO mode on a B200 with 384 GiB host
RAM; all six exports passed checkpoint, FP8 tensor/scale, ModelOpt
state, metadata, and runtime-policy checks.
- Quantization regression tests: 21 passed, 1 skipped.
- All five serving smoke tests passed before the calibration fixes.
- Notebook structure, cell syntax, paths, links, Docker mounts, and
formatting checks passed. Tracked notebook outputs are cleared.
Full calibration and quality reproduction were not run. Installation
cells reused the configured environment, and serving tests were not
repeated after the calibration fixes.
---------
Co-authored-by: pengcuo <pzeren@nvidia.com>
* chore: remove internal references from comments and docs (#331)
## Summary
Removes internal NVIDIA references from **comments, docstrings and
Markdown** across the repo: the Tier 1 ("reword comment/doc, no behavior
change") items from the public-repo release-hygiene audit, plus the
leftovers a broader sweep found after review. 257 files, +926 / −948.
**What changes**
- **Internal code-review provenance**: review and finding ids
(`R149E-03`, `R125-D-08`, …), fix-round labels (`fix round T6j`),
review-thread hashes (`Review of <sha> (thread <id>)`), Greptile / Codex
review notes, "the reviewer's probe", review-round numbers, internal
record ids (`LDCR … mrs-ours V13`) and internal evidence file names
(`review_r134/...`, `fix_r125/...`). The sentences they annotated are
kept.
- **Non-public commit hashes** cited in test docstrings ("On 3b9a465c25a
the preflight …", "FIXTURE: on d8c7213d0a …") → "on the parent", the
wording these docstrings already use for the commit before a fix.
- **Internal monorepo name** (`i4` / `imaginaire4` / `Imaginaire4`):
- → "upstream" where it means the source repo, its CI, its trainer or
its conventions; where it pointed at code that lives in this repo, the
code is named instead (`LazyConfig.save_yaml`, `WandBCallback`,
`FixedStepSamplerConfig`, the Qwen3-VL causal-mask helper, …);
- → "native" where it means the native backend or policy (matching
`NativePolicyBuildSpec`);
- → "training container" for the container.
- **MADS** → "multiview" / "rig". **GA** → "release recipe".
**"prerelease trainer"** → "the trainer".
- **Employee usernames** in `TODO(user)` / `future(user)` tags removed.
- **Internal cluster names, `/lustre` paths and an internal MR number**
in comments replaced with neutral examples; **job ids and run names**
(`job 1881483`, `canM7`, `cw-dfw`) removed; the internal registry and a
wrong path in `docker/ci.Dockerfile`'s build comments fixed.
- **Data-collection "round N"** references in `posttrain/embodied` and
`roles/env` → "an earlier collection run".
- **Small fixes found on the way**: an undefined "Tier-1 caption", "the
the", "to to", three sentences starting in lowercase.
- **Docstrings citing non-public sources or commits** reworded:
`rigrope.py`, `avae.py` (`https://invalid_url`), `docs/automl.md`.
**Deliberately unchanged**
- The literal config value **`model_backend='i4'`** and text that names
that value. It is a public config value, so the docs have to keep
matching the code.
- The **`ludus_renderer`** package name, shipped publicly in #322.
- **Real runtime paths** the code uses: `~/.cache/imaginaire4/...`,
`/tmp/imaginaire4-output`.
- **Runtime strings that carry ids** (error messages such as
`config/sft.py:38`'s `review R138-01`, the `UNIFIED_SERVE_ASSUMPTION`
constant, the test regexes that pin them, fixture JSON), and **endpoint
defaults in code** (`pdx.s8k.io` in the vlmevalkit loaders). Changing
these changes behavior, so they need their owners and a separate PR.
- **References to the posttrain design doc** (`DESIGN.md`, `DESIGN
§x.y`, `docs/design/…`, dated design docs; about 100 files). The doc is
not in this repo, so this needs a decision: publish it or strip the
references.
`posttrain/tests/test_docs_links.py::test_every_docs_link_resolves`
already fails on `main` for this reason.
## Verification
- **No code change.** For every edited `.py` file, the AST with
docstrings stripped is identical to `main`. There are two intended
exceptions:
- `cosmos_framework/posttrain/tests/test_config_sft.py`: its
docstring-pinning assertions follow the reworded
`sft_qwen3_vl_moe_native.py` docstring (dropped expected substrings like
`"review R138-01"`).
- `cosmos_framework/posttrain/tests/test_checkpoint_materialize_hf.py`:
comments inside an embedded helper-script string, and assertion messages
that cited review rounds or commit hashes.
- Assertion messages only: `test_checkpoint_hf_verification.py`,
`test_entrypoints_dpo.py`.
- `test_utils_host_memory.py`: the test constant `CANM7_PEAK_BYTES`
renamed `OBSERVED_PEAK_BYTES`.
- `posttrain/utils/launch_multinode.py`: one comment line inside the
bash script it generates.
- **Tests.** The 28 edited posttrain test files were run on this branch
and on `main` (`dd923db`) in the same CPU-only environment. Both gave
512 passed, 180 failed/errored, 14 skipped, with **0 tests changing
outcome**; the failures are environment-only (no GPU / `triton`) and
identical on `main`.
`test_native_recipe_table_states_the_decay_references_and_the_no_prefill_with_tools`,
whose assertions changed, passes.
- **Tests for the review follow-ups** (`0625b17`, `fc84700`, `4ba38a6`):
the 53 edited or docstring-sensitive test files, run on `4ba38a6` and on
`0625b17`: 1081 passed, 64 failed/errored, 27 skipped on both, **0 tests
changing outcome** (same environment-only failures).
- **pre-commit.** `uvx pre-commit@4.5.1 run --files <changed>` passes,
with the repo's uv 0.11.29; rumdl realigned 8 Markdown table rows whose
cells got shorter. `ci/.pre-commit-config-base.yaml` passes.
- **ruff.** `ruff check` (0.16.4) findings are the same 49 before and
after.
- Merges cleanly with current `main` (`7738e9e`).
## Note for reviewers
About 550 of these lines are in files that the imaginaire4 release
export (`.posttrain_file_mapping.json` / `.file_mapping.json`) copies
over on each "Release (i4 …)" PR. Unless the same wording lands
upstream, or the export strips it, the next export will bring these
references back.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Yao Xu <yaoxu@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: liang.feng <liangf@nvidia.com>
* test: replace internal values in tests and fixtures (#333)
## Summary
Follow-up to #331 (comments and docs). This PR handles the **test values
and fixtures**: internal hostnames, cluster paths, user names, container
names, endpoints and codenames that sit in test data rather than in
comments. Every swap is consistent within its file, so each assertion
still compares like with like. No library code changes; one test helper
script now behaves differently (see **Behavior change** below).
A release-hygiene scan of `main` (1a2f03a) found 74 such rows. This PR
fixes 53 of them; the other 21 are left on purpose (listed below).
| Area | Change |
|---|---|
| Cluster paths (`/lustre/fsw/...`, `/lustre/training/...`, a per-user
HF cache) | `/data/...` or `/shared/...` |
| Hostnames / endpoints (`nvl72d411-T09`, `nvl72D167-T15`, `pdx.s8k.io`,
`inference-api.nvidia.com`) | `<serving-host>`, `rack07-T15`,
`s3.example.com`, `llm.example.com` |
| SLURM account / partition (`edgeai`, `polar`) | `my-account`, `batch`
|
| Personal container default in `run_nixl_raw_bandwidth.sh`
(`yangyangt_dev`) | no default: `CONTAINER` is now required and fails
with a clear message when unset |
| Codenames in test strings (`imaginaire4`, `i4`, `native-i4`,
`/checkpoints/ga`, `Cosmos3-Nano-GA`) | `upstream`, `native`,
`/checkpoints/base`, `Cosmos3-Nano` (the registered checkpoint name) |
| Internal commit / MR ids (`573d465…` fake receipt revision, two
`merge_request` fields in `wsm/PROVENANCE.json`) | neutral 40-hex value;
fields dropped (nothing reads them) |
| Review-round provenance, internal paths and the SLURM job id in
`served_prefix_with_tools.json` | reworded as "live probes …"; only
metadata keys that no test reads |
Also changed, outside the scan:
- the SLURM template test's `--account=edgeai` /
`--partition=polar,polar3` values (same kind of internal name);
- the local DCP path in `inference/common/args_test.py`, neutralized the
same way as its S3 twin;
- `test_camera_defaults_match_imaginaire4` →
`test_camera_defaults_match_upstream` (nothing references the name);
- two comments in `test_config_sft.py` (L492, L494) now use the same
text as #331, so the two PRs merge cleanly in either order.
## Behavior change
`cosmos_framework/scripts/_test/convert_model_to_diffusers.sh` passed
`--checkpoint-path Cosmos3-Nano-GA`, which is not a registered
checkpoint, so the script failed. It now passes `Cosmos3-Nano`, which is
registered (`inference/args.py`), so it downloads and converts the
public Cosmos3-Nano checkpoint. Nothing runs this script automatically:
`scripts/_test.py` runs `convert_model_to_dcp.sh`, not this one.
## Left on purpose (21 rows)
- **Already edited by #331** (would conflict):
`test_checkpoint_materialize_hf.py:2975,2993,3139`,
`test_config_sft.py:431-455`.
- **Pinned to runtime strings** (the test asserts on text that the
library emits, so the library has to change first; needs the code
owner): `test_checkpoint_materialize_hf.py:874` (`R151-05` in
`UNIFIED_SERVE_ASSUMPTION`), `test_config_sft.py:618` (`review R138-01`
in the `config/sft.py` error), `test_validate_sft_rows.py:1981,2072`
(packer error text), `test_control_one_logger.py:277`
(`one_logger_project == "imaginaire4"`).
- **Public names or literal values**: `test_config_precision.py:68`
("its backend is i4" — `i4` is the public `model_backend` value);
`packages/ludus-renderer` test imports (public package name since #322).
- **Checkpoint revisions**:
`tests/bimanual_unified_action_test.py:29,40` are keys of
`_RELEASED_SCHEMA5_DOMAIN_IDS_BY_REVISION` (two 40-hex revisions). They
are only subtest labels in the test, but they are meant to name real
released revisions, and an earlier check found them missing from the
public HF repo. The owner should decide whether to point them at public
revisions or drop them; I didn't invent replacements.
- `conftest.py:260` `imaginaire4-output` tmp dir: it mirrors the real
runtime default `/tmp/imaginaire4-output`.
- **`source_commit` hashes in
`.agents/skills/cookbooks/cosmos3/simulation/auto/wsm/PROVENANCE.json`**
(L3, L509, L524). The file ships via `MANIFEST.in`, and the hashes point
at a non-public repo. Whether provenance files ship with internal hashes
is a release decision covering every provenance file, not a test-value
swap, so it is tracked with those ("Provenance files cite internal
source commits") rather than changed here. Only the two `merge_request`
fields are dropped.
## Testing
- `pre-commit run --files <changed>` (uv 0.11.29): passes.
- Paired pytest on the 16 edited test files plus `test_config_sft.py`
(which reads the fixture), on this branch and on `main`, CPU-only Mac
venv. Re-run at the current head `01c67920` against `main` `7738e9e`:
**459 passed / 28 failed or errored / 12 skipped on both**. The only ID
differences are renamed tests and parametrize IDs
(`test_camera_defaults_match_imaginaire4` → `..._match_upstream`,
`native-i4-1` → `native-1`, `/checkpoints/ga-prefix` →
`/checkpoints/base-prefix`, …), which pass on both sides. Failures are
environment-only (no `triton`, a `ray` API mismatch, a fixture path that
assumes the container layout) and identical on `main`.
- Not exercised in that environment: `bimanual_recipes_test.py` (import
needs `triton`), the fixture-reading test in `test_config_sft.py`
(container path), `test_workers_nixl_raw_bandwidth.py` (GPU),
`qwen35_caption_video_test.py` (needs the 27B model; now defaults to
`/data/.cache/...`, override with `QWEN35_MODEL_PATH`), and the two
shell scripts. The edits there are value swaps checked by reading the
code.
- Merges cleanly with #331 and with current `main`.
## Caveat
Several of these files are generated by the upstream release export
(`.posttrain_file_mapping.json`, `.file_mapping.json`), so the next
export can put the old values back unless the same change is made
upstream.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Yao Xu <yaoxu@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: liang.feng <liangf@nvidia.com>
* cosmos3-generate-media skill (cookbook → skill migration) with eval-derived fixes, cosmos3-skill-authoring guide, opt-in skill-eval CI (#324)
**Includes the eval-derived fixes PR #327**, merged into this branch on
2026-10-29 (`e1a3ff1`): the migration and the fixes that take the eval
matrix from 42/44 to 44/44 first-attempt PASS go to `main` together.
Everything is applied as a diff on top of `main`, not as a copy.
First cookbook→skill migration (**cosmos3-generate-media**), the
**cosmos3-skill-authoring** guide, and an opt-in **skill-eval CI** on
self-hosted H100 runners. No eval (GPU) job runs on push or merge.
### What this PR delivers
- **One skill for the seven Cosmos3 audiovisual serving paths** —
diffusers, cosmos_framework, vLLM-Omni, SGLang, TensorRT-LLM, distilled
TensorRT-LLM, NIM — with the sample prompts, reference images and
negative prompts under `assets/`.
- **The cookbook notebooks migrated 1:1 into `references/`** (one file
per backend plus `payloads.md` for shared content). A format change
only: Jupyter plumbing removed, nothing corrected or rewritten;
`references/vllm-omni.md` follows the vLLM-Omni notebook's last version
on `main` (#329: launch mounts, `HF_TOKEN`, the FP8 "Quantized
Checkpoints" section), since that notebook changed after the migration's
pin and before its retirement here; the six notebooks and the
audiovisual cookbook README are retired from `.agents/skills/cookbooks/`
(links in the cookbook READMEs and the docs site now point at the
skill). Known gap: the video-to-video input clip (an LFS object) is not
vendored.
- **NIM (Variant 7) follows the Cosmos3 NIM 2.1.0 cookbook.**
`run_with_nim.ipynb` was already retired in this repository by #296 in
favour of `.agents/skills/cookbooks/cosmos3/nim/`, so the skill carries
no `references/nim.md` copy of it: Variant 7 delegates to that cookbook
and its `cosmos3-nim-user` skill (pointers, the README's 2.1.0 launch,
the readiness and selection checks of `deployment.md`, and the
`examples/i2v.py` request built from the bundled assets). Record:
`docs/cookbook_to_skill_migration.md`.
- **SKILL.md quotes each backend's smoke path 1:1** from the notebooks,
the cookbook README or the setup guide, and makes the reference files
required reading before the first command. Review requests from the
first review round are applied (the framework example initialises
`COSMOS3_UV_GROUP` / `COSMOS3_UV_ENV` as the notebook does; the live CI
comment uses a sequence guard and per-cell ledger lines).
- **Eval-derived fixes (from #327)**: F1–F18 and F2d — every fatal
failure and repeated self-correction of the 44-cell matrix, fixed as
marked skill notes or declared in-block deviations with the notebook
text kept verbatim; `eval/eval.config` is unchanged, so before and after
are measured on the same prompts (tables below).
- **11 eval cases** (3 Nano, 6 Edge, 1 distilled Super, 1 NIM); the two
framework cases clone this repository's `main`. Known wording leftover
in `eval/eval.config`: the NIM case says the Generator NIM serves
text-to-video and image-to-video only, which was true of the retired
1.0.0 NIM; the pinned 2.1.0 NIM also serves text-to-image and
video-to-video. The case exercises image-to-video, so results are
unaffected; the sentence is corrected in the next `eval.config` revision
together with a re-run (the file is frozen here so the migration head
and the fixes are measured on the same prompts).
- **Skill-eval CI (opt-in)**: `.github/workflows/skill-eval.yml` and the
orchestrator in `.github/scripts/skill_eval/` run a PR's skills on
self-hosted H100 runners — one job per case × agent, one live PR comment
that always shows the newest result per cell, dashboard groups per skill
version, the inference key from a repository secret, a GitHub-hosted job
for the orchestrator's unit tests, and a GitHub-hosted **gate** that
admits only repository members with the write, maintain or admin role
(details below).
- **cosmos3-skill-authoring**: how to write a skill that passes the eval
on the first attempt, how the evaluator judges, how to run the eval in
CI or locally, and the fix loop (a stacked fixes PR, gated by the owner)
that gets every cell to first-attempt PASS
(`references/running-the-eval.md`).
### Evaluation results
#### Migration head `b50b766` (before the fixes)
Head `b50b766`, 11 cases × 4 agents (claude = Claude Opus 5.5, codex =
GPT-6 Astra, cursor = Claude Sonnet 5.5, opencode = Kimi K2.6), [run
40639480322](https://github.com/NVIDIA/cosmos-framework/actions/runs/40639480322)
on the four self-hosted H100 runners, [dashboard group
ci-pr324-b50b766](http://internal-cosmos-skills-eval.nvidia.com/skills/cosmos3-generate-media/groups/ci-pr324-b50b766/):
**42/44 PASS** (14/44 first-attempt, 2 FAIL, 0 no report, 0 errors;
static assessment 100/100)
| Case | claude | codex | cursor | opencode |
|---|---|---|---|---|
| `diffusers-edge-t2i` | ✅ 1.84M ↻ | ✅ 2.89M ↻ | ✅ 1.07M ↻ | ✅ 1.41M ↻ |
| `diffusers-edge-t2v` | ✅ 2.81M ↻ | ✅ 3.32M ↻ | ✅ 594K ↻ | ✅ 1.31M ↻ |
| `diffusers-nano-t2i` | ✅ 2.08M | ✅ 3.28M ↻ | ✅ 529K ↻ | ✅ 869K ↻ |
| `distilled-trtllm-t2i` | ✅ 8.95M ↻ | ✅ 9.73M ↻ | ✅ 2.11M ↻ | ❌ 4.51M ↻
|
| `framework-edge-t2i` | ✅ 2.56M ↻ | ✅ 3.73M | ✅ 465K ↻ | ✅ 2.76M ↻ |
| `framework-nano-t2i` | ✅ 2.71M ↻ | ✅ 4.59M | ✅ 488K ↻ | ✅ 2.32M |
| `nim-generator-i2v` | ✅ 3.46M | ✅ 3.22M | ✅ 390K | ✅ 2.67M ↻ |
| `sglang-edge-t2i` | ✅ 2.93M | ✅ 2.30M ↻ | ✅ 584K ↻ | ✅ 1.13M |
| `trtllm-edge-t2i` | ✅ 6.12M ↻ | ✅ 18.32M ↻ | ✅ 864K ↻ | ❌ 5.53M ↻ |
| `trtllm-nano-t2i` | ✅ 5.43M | ✅ 19.52M ↻ | ✅ 1.46M ↻ | ✅ 7.62M ↻ |
| `vllm-omni-edge-t2i` | ✅ 1.88M | ✅ 1.41M | ✅ 645K | ✅ 2.02M |
Tokens (all cells): 154.42M. ↻ = passed after self-correction.
The two FAILs are the opencode agent on the two TensorRT-LLM wheel cases
— the documentation gap (no notebook or README says how to obtain a
TensorRT-LLM build with Cosmos3 VisualGen) that the fixes below close;
nothing the agents self-corrected was changed by the migration itself.
#### After the fixes (`b7ad087`, the #327 head at validation)
**Baseline (first commit `b7ad087`)** — [run
40660687591](https://github.com/NVIDIA/cosmos-framework/actions/runs/40660687591),
[dashboard group
ci-pr327-b7ad087](http://internal-cosmos-skills-eval.nvidia.com/skills/cosmos3-generate-media/groups/ci-pr327-b7ad087/):
**44/44 PASS** (44/44 first-attempt, 0 FAIL, 0 no report, 0 errors;
static assessment 100/100)
**Baseline vs #324**
([compare](http://internal-cosmos-skills-eval.nvidia.com/skills/cosmos3-generate-media/compare?a=group%3Aci-pr324-b50b766&b=group%3Aci-pr327-b7ad087);
the [branch
compare](http://internal-cosmos-skills-eval.nvidia.com/skills/cosmos3-generate-media/compare?a=branch%3Askills%2Fgenerate-media-migration-and-eval-public&b=branch%3Askills%2Fgenerate-media-fixes-public)
always shows the newest result per cell of each branch):
**A** `group:ci-pr324-b50b766`: 42/44 PASS, 14 first-attempt · **B**
`group:ci-pr327-b7ad087`: 44/44 PASS, 44 first-attempt · improved 2,
regressed 0
| Case | Metric | claude | codex | cursor | opencode |
|---|---|---|---|---|---|
| `diffusers-edge-t2i` | Completion | ✅→✅ | ✅→✅ | ✅→✅ | ✅→✅ |
| | First time success | ❌→✅ | ❌→✅ | ❌→✅ | ❌→✅ |
| `diffusers-edge-t2v` | Completion | ✅→✅ | ✅→✅ | ✅→✅ | ✅→✅ |
| | First time success | ❌→✅ | ❌→✅ | ❌→✅ | ❌→✅ |
| `diffusers-nano-t2i` | Completion | ✅→✅ | ✅→✅ | ✅→✅ | ✅→✅ |
| | First time success | ✅→✅ | ❌→✅ | ❌→✅ | ❌→✅ |
| `distilled-trtllm-t2i` | Completion | ✅→✅ | ✅→✅ | ✅→✅ | ❌→✅ |
| | First time success | ❌→✅ | ❌→✅ | ❌→✅ | ❌→✅ |
| `framework-edge-t2i` | Completion | ✅→✅ | ✅→✅ | ✅→✅ | ✅→✅ |
| | First time success | ❌→✅ | ✅→✅ | ❌→✅ | ❌→✅ |
| `framework-nano-t2i` | Completion | ✅→✅ | ✅→✅ | ✅→✅ | ✅→✅ |
| | First time success | ❌→✅ | ✅→✅ | ❌→✅ | ✅→✅ |
| `nim-generator-i2v` | Completion | ✅→✅ | ✅→✅ | ✅→✅ | ✅→✅ |
| | First time success | ✅→✅ | ✅→✅ | ✅→✅ | ❌→✅ |
| `sglang-edge-t2i` | Completion | ✅→✅ | ✅→✅ | ✅→✅ | ✅→✅ |
| | First time success | ✅→✅ | ❌→✅ | ❌→✅ | ✅→✅ |
| `trtllm-edge-t2i` | Completion | ✅→✅ | ✅→✅ | ✅→✅ | ❌→✅ |
| | First time success | ❌→✅ | ❌→✅ | ❌→✅ | ❌→✅ |
| `trtllm-nano-t2i` | Completion | ✅→✅ | ✅→✅ | ✅→✅ | ✅→✅ |
| | First time success | ✅→✅ | ❌→✅ | ❌→✅ | ❌→✅ |
| `vllm-omni-edge-t2i` | Completion | ✅→✅ | ✅→✅ | ✅→✅ | ✅→✅ |
| | First time success | ✅→✅ | ✅→✅ | ✅→✅ | ✅→✅ |
Completion: overall verdict before → after (✅ PASS, ❌ FAIL). First time
success: passed on the first attempt, without a self-correction, before
→ after.
**Added after the validation run:** F2d (`bad39b1`); four relative-link
path corrections in `references/` (`../cookbooks/…` →
`../../cookbooks/…`); and the vLLM-Omni notebook's final version (#329:
`$PWD` mounts, `-e HF_TOKEN`, the FP8 "Quantized Checkpoints" section in
`vllm-omni.md` § 2 and SKILL.md Variant 3; the FP8 pointer in
`payloads.md` § 1) — validated locally with the four vllm-omni-edge-t2i
cells, 4/4 PASS on the first attempt twice (after the port and after the
merge of the fixes). `git diff b7ad087 e1a3ff1 --
.agents/skills/cosmos3-generate-media` shows exactly these, plus the
review follow-ups below.
### Eval-derived fixes (merged from #327)
Where = `file:lines` at `e1a3ff1` (permalinks; `SKILL.md` =
`.agents/skills/cosmos3-generate-media/SKILL.md`, reference names =
`references/<name>`). Problem → fix, in priority order; F2d was added
after the validation run (see Validation).
| ID | Pri | Problem | Fix applied | Where |
|---|---|---|---|---|
| F1 | P0 | Variant 5 gave only the Docker devel source build (tag left
`<tag>`) plus a dead `TRTLLM_USE_PRECOMPILED` hint; no wheel version
anywhere, so cursor/opencode pip-installed stable 1.2.x, which lacks
VisualGen | Route A (default): prebuilt wheel `tensorrt_llm==1.3.0rc27`
+ `torch==2.13.0+cu130` from pypi.nvidia.com/cu130 (uv `--index-strategy
unsafe-best-match`), VisualGen preflight, pinned checkout for YAMLs
only; Route B = source build; mirrored as skill notes in trt-llm.md /
distilled-trt-llm.md | SKILL.md
[296–299](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L296-L299)
· distilled-trt-llm.md
[61](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L61)
· trt-llm.md
[29](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L29)
[583–589](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L583-L589)
[595–639](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L595-L639)
[656–658](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L656-L658)
[722](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L722)
|
| F2 | P1 | Framework `pyproject.toml` pins uv `==0.11.29`; `uv
venv/pip/sync` abort with uv 0.13 and `uv self update` refuses | Variant
1: `--no-config`; Variant 2: pinned-uv preflight (side-by-side install
or `uvx --from 'uv==0.11.29'`); one note in "Read this first"; skill
notes in diffusers.md, cosmos-framework.md, payloads.md | SKILL.md
[31–35](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L31-L35)
[104–108](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L104-L108)
[120](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L120)
[142–143](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L142-L143)
[177–184](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L177-L184)
· cosmos-framework.md
[202](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/cosmos-framework.md#L202)
[409–410](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/cosmos-framework.md#L409-L410)
· diffusers.md
[136](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/diffusers.md#L136)
[138](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/diffusers.md#L138)
[157](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/diffusers.md#L157)
· payloads.md
[85](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/payloads.md#L85)
[99](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/payloads.md#L99)
|
| F2b | P1 | An agent installed uv 0.11.29 (the Variant 2 preflight) for
the TRT-LLM venv; that uv cannot resolve
`nvidia-cutlass-dsl==4.8.0.dev0` without `--prerelease=allow`, which
then pulls mpmath 1.5.0a1 | "Read this first" / Variant 5 / Variant 2
state which uv each variant uses (stock 0.13+ with `--no-config` for
Variants 1/5/6; 0.11.29 only for `uv sync`) | SKILL.md
[36–37](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L36-L37)
[183](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L183)
· skill-notes.md
[141–145](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/skill-notes.md#L141-L145)
· trt-llm.md
[605](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L605)
|
| F2c | P1 | In a full run of the fixes branch the opencode agent ran
the Variant 1 install block across separate shell calls; the `export`
and `source …/activate` of the first call were gone in the second, so
`uv pip install` went into the evaluator's own venv and the import check
ran against the wrong interpreter (then a needless OpenCV downgrade: the
same block passed first-attempt in the other eleven diffusers cells of
that run) | The three `uv pip` commands of the Variant 1 block and of
diffusers.md § 3 carry `--python "$COSMOS3_DIFFUSERS_VENV"` (declared
in-block addition; with the variable unset uv stops instead of
installing elsewhere), the import check and the generation snippets run
with `"$COSMOS3_DIFFUSERS_VENV/bin/python"`, and the block says it is
one shell call | SKILL.md
[110–126](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L110-L126)
· diffusers.md
[138–151](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/diffusers.md#L138-L151)
[157](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/diffusers.md#L157)
[161](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/diffusers.md#L161)
[453–454](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/diffusers.md#L453-L454)
|
| F3 | P1 | Diffusers venv not headless: `cosmos_guardrail` pulls GUI
opencv; the pipeline imports it even with the safety checker off |
Headless OpenCV swap + import check inside the Variant 1 venv block; apt
fallback reworded (`apt-get update`, root caveat); skill notes in
diffusers.md / payloads.md | SKILL.md
[98–100](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L98-L100)
[116–126](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L116-L126)
· diffusers.md
[74](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/diffusers.md#L74)
[159](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/diffusers.md#L159)
[437–446](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/diffusers.md#L437-L446)
· payloads.md
[81](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/payloads.md#L81)
|
| F3b | P1 | Headless OpenCV swap was a separate command after the
install line; one agent ran the pipeline import first and hit
`libxcb.so.1` | Swap folded into the `uv pip install` command in
SKILL.md Variant 1 and, as a declared in-block deviation, in
diffusers.md § 3 | SKILL.md
[116–126](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L116-L126)
· diffusers.md
[138–151](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/diffusers.md#L138-L151)
[157](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/diffusers.md#L157)
[159](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/diffusers.md#L159)
|
| F4 | P1 | Bare-host prerequisites of the wheel undocumented: OpenMPI,
`CUDA_HOME`, CUDA-13 libs via the cu130 index, seeded venv, ffmpeg |
Prerequisite lines + comments inside the Route A block (Variant 6
inherits); upstream pip-install page cited in the references |
distilled-trt-llm.md
[61](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L61)
· trt-llm.md
[29](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L29)
[601–606](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L601-L606)
[624–635](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L624-L635)
[703–706](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L703-L706)
|
| F5 | P1 | `TRTLLM_USE_PRECOMPILED=1 pip install -e .` resolves an
unpublished `1.3.0rc26+bca6761ab8` wheel; rc26 binaries lack an op the
pinned Python needs | Hint deleted; "do not use the shortcut on this
checkout" paragraph; one-sentence skill notes in both TRT references |
distilled-trt-llm.md
[61](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L61)
· trt-llm.md
[29](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L29)
[669](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L669)
[674–687](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L674-L687)
|
| F6 | P1 | No output-verification step; ffprobe/ffmpeg/Pillow absent on
the host; the NIM container's ffmpeg cannot demux its own VP9 output |
New "Verify the output" section (ffprobe, root-free `av`, PIL; expected
smoke facts) + pointers; NIM headless-check note | SKILL.md
[41–43](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L41-L43)
[454–467](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L454-L467)
· diffusers.md
[437–446](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/diffusers.md#L437-L446)
[482](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/diffusers.md#L482)
· payloads.md
[101](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/payloads.md#L101)
|
| F7 | P1 | Server variants lack a headless start / readiness-poll /
teardown recipe; SGLang `/health` 503 while warming undocumented; NIM
`docker run -d --rm` destroys the logs | Shared named-container +
trtllm-serve patterns (start, poll, logs, teardown) with per-variant
pointers; SGLang readiness documented; NIM `--rm` dropped (recorded
deviation); Nano/Super t2i request body stated in Variant 5 | SKILL.md
[230–231](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L230-L231)
[274–279](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L274-L279)
[319–321](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L319-L321)
[372–374](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L372-L374)
[401](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L401)
[451–452](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L451-L452)
· payloads.md
[216–217](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/payloads.md#L216-L217)
· sglang.md
[21](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/sglang.md#L21)
· skill-notes.md
[58–101](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/skill-notes.md#L58-L101)
· trt-llm.md
[43](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L43)
[733](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L733)
|
| F7b | P2 | Servers started in a backgrounded subshell died with the
shell; SIGINT to the process group did not stop a server; a second
server was started while the first still held the GPU | Headless
pattern: launch from a script with `setsid … &`, SIGINT → SIGTERM →
SIGKILL escalation, leftover check by PID; never start a second server
on a busy GPU (the launch script enforces it) | SKILL.md
[372–377](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L372-L377)
· distilled-trt-llm.md
[465–474](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L465-L474)
· skill-notes.md
[75–94](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/skill-notes.md#L75-L94)
|
| F8 | P1 | Distilled 4-Step students are ~122 GiB bf16, not "32B /
single GPU": the 1-GPU launch OOMs on 80 GB; the verified
`cpu_offload_config` route was undocumented | Model-table note + Variant
6 offload block (measured) + warning that FP8 variants are unverified;
same in distilled-trt-llm.md | SKILL.md
[79–84](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L79-L84)
[339–343](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L339-L343)
· distilled-trt-llm.md
[22](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L22)
[35](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L35)
[84–99](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L84-L99)
[101](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L101)
[103](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L103)
[366–370](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L366-L370)
[431–439](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L431-L439)
[443–447](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L443-L447)
· skill-notes.md
[101](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/skill-notes.md#L101)
|
| F8b | P1 | Distilled launch still OOM-prone in practice: agents ran
the notebook's plain bf16 launch first (OOM on 80–94 GB GPUs, and a
second launch then OOMed against the still-loading server); on 125
GiB-RAM hosts the offload route is OOM-killed | Variant 6 is one gated
launch script: GPU memory → plain route only at ≥ 128,000 MiB; host RAM
(incl. cgroup ceiling) → bf16 + `cpu_offload_config` at ≥ 128 GiB, else
the Hugging Face `fp8` branch with the `embed_tokens` ignore-list patch
(eval-verified); refuses to start on a busy GPU; one `setsid
trtllm-serve` + `/health` poll with a 1800 s deadline | SKILL.md
[345–348](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L345-L348)
[372–377](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L372-L377)
· distilled-trt-llm.md
[63](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L63)
[339–416](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L339-L416)
|
| F9 | P1 | SGLang `docker run` passes no `-e HF_TOKEN` and omits
`SGLANG_DISABLE_COSMOS3_GUARDRAILS=1`; the server exits with
GatedRepoError 401 | Guardrail-table row fixed; `-e
SGLANG_DISABLE_COSMOS3_GUARDRAILS=1 -e HF_TOKEN` in every launch block;
request body `guardrails: false` | SKILL.md
[249–254](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L249-L254)
[262–263](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L262-L263)
[284](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L284)
[291](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L291)
· payloads.md
[67](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/payloads.md#L67)
· sglang.md
[23](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/sglang.md#L23)
[35](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/sglang.md#L35)
[41–42](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/sglang.md#L41-L42)
[80](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/sglang.md#L80)
[371](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/sglang.md#L371)
· skill-notes.md
[42–49](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/skill-notes.md#L42-L49)
[53–57](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/skill-notes.md#L53-L57)
|
| F9b | P1 | vLLM-Omni `docker run` blocks passed no `-e HF_TOKEN`: on a
fresh node with a cold Hugging Face cache the first launch crashed with
HTTP 429 during the anonymous model lookup | Superseded by the
notebook's final version (#329), which passes `-e HF_TOKEN` in every
`vllm serve` launch; the skill carries that text 1:1 (vllm-omni.md § 2,
SKILL.md Variant 3) and `vllm-omni.md` § 7.1 records why the token
matters — export `HF_TOKEN` before the launch | SKILL.md
[213](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L213)
· vllm-omni.md
[602–608](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/vllm-omni.md#L602-L608)
|
| F10 | P1 | NIM profile discovery/pinning (`list-model-profiles`,
`NIM_MODEL_PROFILE`, `/v1/manifest`) and guardrail env switches missing
from the launch block | profile preflight and `/v1/metadata` check in
the Variant 7 launch; env lines in both launch blocks; `/v1/manifest` +
`selectedModelProfileId` in the inspect lists | SKILL.md
[391–397](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L391-L397)
[407–408](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L407-L408)
[417–418](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L417-L418)
[423](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L423)
|
| F11 | P1 | NIM cache bind mount not writable by uid 1000; the
client-side `chmod -R 777` never reaches a daemon-created mount | Cache
preflight in the Variant 7 launch: the write test as the NIM user (uid
1000) runs first; the ownership repair (`docker run --user 0 … chown
1000:1000`) runs only for an empty cache this run created or with
`NIM_CACHE_REPAIR=1`, otherwise the launch stops with guidance (review
follow-up, see below) | SKILL.md
[395–396](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L395-L396)
[409–411](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L409-L411)
· skill-notes.md
[102–108](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/skill-notes.md#L102-L108)
|
| F12 | P1 | rc26 returns HTTP 500 on `/v1/images/generations`;
`/health` 200 does not exercise the image route | rc27 pinned with the
reason inside Route A; "a 200 on /health does not exercise the image
route" sentence; skill note at trt-llm.md § 2 | trt-llm.md
[70](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L70)
[564](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L564)
[612–620](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L612-L620)
[636–637](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L636-L637)
[737–740](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L737-L740)
|
| F13 | P2 | Model table had no measured memory figures, no small-GPU
path | Measured paragraph under the table (Edge/Nano, labelled as eval
measurements) + fp8 pointer; Nano YAML note in Variant 5 / trt-llm.md |
skill-notes.md
[26–38](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/skill-notes.md#L26-L38)
· trt-llm.md
[43](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L43)
[722–733](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L722-L733)
|
| F14 | P2 | Variant 6 returns a one-frame MP4/AVI, the eval asks for a
PNG | `ffmpeg -frames:v 1` extraction documented in Variant 6 and
distilled-trt-llm.md | SKILL.md
[358–361](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L358-L361)
· distilled-trt-llm.md
[267](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L267)
[277–286](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L277-L286)
|
| F14b | P2 | On the fp8 server the one-frame `/v1/videos/generations`
returned HTTP 500 after a successful denoise | Documented fallback:
`/v1/images/generations` with `output_type: "image"` (explicit body;
needs rc27), recorded in result_summary.md | SKILL.md
[361](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L361)
· distilled-trt-llm.md
[279](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L279)
[451–461](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cos…
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Remove extra
nn.Embeddingsquantizers that are not supported in ModelOpt v0.44.0.Issue
Originally, edge was quantized with ModelOpt v0.47.0, while nano/super were quantized with v0.44.0. ModelOpt has
nn.Embeddingssupport in v0.47.0, which leads to restoration failure when loading the checkpoint with v0.44.0. We decided to revert the ModelOpt version to v0.44.0 to make all Cosmos3 FP8 checkpoints consistent.Example error message
Fix
Additional quantizer metadata for
nn.Embeddingsare removed. Those layers remain bfloat16 regardless of ModelOpt version, so removing the layer metadata resolves the issue.