Repository navigation
docs: update Cosmos3 TensorRT-LLM guardrail setup to 0.3.2 - #332
Merged
Merged
Conversation
Port the guardrail setup fix from NVIDIA/cosmos after the Cosmos3 cookbooks moved into cosmos-framework. Pin cosmos_guardrail 0.3.2 and TensorRT-LLM cc63e66d, remove the manual server-side NLTK setup, and drop the obsolete instructions from the Action and Transfer READMEs and client notebooks. Ported-from: e8d79223710074c2d377230aeedc38639b65c676 Signed-off-by: Igor Shovkun <ishovkun@nvidia.com>
lfengad
approved these changes
Oct 29, 2026
lfengad
enabled auto-merge (squash)
October 29, 2026 12:46
Xuanmeng-Zhang
approved these changes
Oct 29, 2026
christinayyw
added a commit
that referenced
this pull request
Oct 29, 2026
…; note the TensorRT-LLM pin drift; point Variant 3 at the FP8 section The shared setup guide on main moved to cosmos_guardrail 0.3.2 (no separate NLTK data step any more) and pins TensorRT-LLM commit cc63e66 (#332). trt-llm.md § 7.6 quotes the current guide; § 7.1 notes that the skill's validated routes stay at bca6761 / 1.3.0rc27 until a re-run moves the pins together. SKILL.md Variant 3 names the vLLM-Omni reference's FP8 "Quantized Checkpoints" section (#329). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Christina Wang <christwang@nvidia.com>
christinayyw
added a commit
that referenced
this pull request
Oct 29, 2026
## Summary **Stacked PR:** based on #324's branch `skills/generate-media-migration-and-eval-public`; merge this PR into that branch first, then #324 carries both to `main`. Skill-only: `.github` is identical to the base. Follow-up to #324: fixes every fatal failure and repeated self-correction of the `cosmos3-generate-media` eval matrix (11 cases × claude, codex, cursor, opencode). Baseline (group `ci-pr324-b50b766`): 42/44 PASS, 14/44 first-attempt. With the fixes (group `ci-pr327-b7ad087`): 44/44 PASS, 44/44 first-attempt. `eval/eval.config` is unchanged from the base branch, so before and after are measured on the same prompts, assertions and checks. Notebook-verbatim text in `references/` stays verbatim; every skill addition is a marked skill note ("Skill note (not in the notebook): …") or a declared in-block deviation. ### Fixes Where = `file:lines` at `55f0f0f` (permalinks; `SKILL.md` = `.agents/skills/cosmos3-generate-media/SKILL.md`, reference names = `references/<name>`). Problem → fix, in priority order; F2d was added after the validation run (see Validation). | ID | Pri | Problem | Fix applied | Where | |---|---|---|---|---| | F1 | P0 | Variant 5 gave only the Docker devel source build (tag left `<tag>`) plus a dead `TRTLLM_USE_PRECOMPILED` hint; no wheel version anywhere, so cursor/opencode pip-installed stable 1.2.x, which lacks VisualGen | Route A (default): prebuilt wheel `tensorrt_llm==1.3.0rc27` + `torch==2.13.0+cu130` from pypi.nvidia.com/cu130 (uv `--index-strategy unsafe-best-match`), VisualGen preflight, pinned checkout for YAMLs only; Route B = source build; mirrored as skill notes in trt-llm.md / distilled-trt-llm.md | SKILL.md [296–299](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/SKILL.md#L296-L299) · distilled-trt-llm.md [61](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L61) · trt-llm.md [29](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L29) [583–589](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L583-L589) [595–639](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L595-L639) [656–658](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L656-L658) [722](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L722) | | F2 | P1 | Framework `pyproject.toml` pins uv `==0.11.29`; `uv venv/pip/sync` abort with uv 0.13 and `uv self update` refuses | Variant 1: `--no-config`; Variant 2: pinned-uv preflight (side-by-side install or `uvx --from 'uv==0.11.29'`); one note in "Read this first"; skill notes in diffusers.md, cosmos-framework.md, payloads.md | SKILL.md [31–35](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/SKILL.md#L31-L35) [104–108](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/SKILL.md#L104-L108) [120](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/SKILL.md#L120) [142–143](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/SKILL.md#L142-L143) [177–184](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/SKILL.md#L177-L184) · cosmos-framework.md [202](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/cosmos-framework.md#L202) [409–410](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/cosmos-framework.md#L409-L410) · diffusers.md [136](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/diffusers.md#L136) [138](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/diffusers.md#L138) [157](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/diffusers.md#L157) · payloads.md [85](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/payloads.md#L85) [99](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/payloads.md#L99) | | F2b | P1 | An agent installed uv 0.11.29 (the Variant 2 preflight) for the TRT-LLM venv; that uv cannot resolve `nvidia-cutlass-dsl==4.8.0.dev0` without `--prerelease=allow`, which then pulls mpmath 1.5.0a1 | "Read this first" / Variant 5 / Variant 2 state which uv each variant uses (stock 0.13+ with `--no-config` for Variants 1/5/6; 0.11.29 only for `uv sync`) | SKILL.md [36–37](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/SKILL.md#L36-L37) [183](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/SKILL.md#L183) · skill-notes.md [141–145](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/skill-notes.md#L141-L145) · trt-llm.md [605](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L605) | | F2c | P1 | In a full run of the fixes branch the opencode agent ran the Variant 1 install block across separate shell calls; the `export` and `source …/activate` of the first call were gone in the second, so `uv pip install` went into the evaluator's own venv and the import check ran against the wrong interpreter (then a needless OpenCV downgrade: the same block passed first-attempt in the other eleven diffusers cells of that run) | The three `uv pip` commands of the Variant 1 block and of diffusers.md § 3 carry `--python "$COSMOS3_DIFFUSERS_VENV"` (declared in-block addition; with the variable unset uv stops instead of installing elsewhere), the import check and the generation snippets run with `"$COSMOS3_DIFFUSERS_VENV/bin/python"`, and the block says it is one shell call | SKILL.md [110–126](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/SKILL.md#L110-L126) · diffusers.md [138–151](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/diffusers.md#L138-L151) [157](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/diffusers.md#L157) [161](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/diffusers.md#L161) [453–454](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/diffusers.md#L453-L454) | | F3 | P1 | Diffusers venv not headless: `cosmos_guardrail` pulls GUI opencv; the pipeline imports it even with the safety checker off | Headless OpenCV swap + import check inside the Variant 1 venv block; apt fallback reworded (`apt-get update`, root caveat); skill notes in diffusers.md / payloads.md | SKILL.md [98–100](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/SKILL.md#L98-L100) [116–126](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/SKILL.md#L116-L126) · diffusers.md [74](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/diffusers.md#L74) [159](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/diffusers.md#L159) [437–446](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/diffusers.md#L437-L446) · payloads.md [81](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/payloads.md#L81) | | F3b | P1 | Headless OpenCV swap was a separate command after the install line; one agent ran the pipeline import first and hit `libxcb.so.1` | Swap folded into the `uv pip install` command in SKILL.md Variant 1 and, as a declared in-block deviation, in diffusers.md § 3 | SKILL.md [116–126](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/SKILL.md#L116-L126) · diffusers.md [138–151](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/diffusers.md#L138-L151) [157](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/diffusers.md#L157) [159](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/diffusers.md#L159) | | F4 | P1 | Bare-host prerequisites of the wheel undocumented: OpenMPI, `CUDA_HOME`, CUDA-13 libs via the cu130 index, seeded venv, ffmpeg | Prerequisite lines + comments inside the Route A block (Variant 6 inherits); upstream pip-install page cited in the references | distilled-trt-llm.md [61](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L61) · trt-llm.md [29](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L29) [601–606](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L601-L606) [624–635](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L624-L635) [703–706](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L703-L706) | | F5 | P1 | `TRTLLM_USE_PRECOMPILED=1 pip install -e .` resolves an unpublished `1.3.0rc26+bca6761ab8` wheel; rc26 binaries lack an op the pinned Python needs | Hint deleted; "do not use the shortcut on this checkout" paragraph; one-sentence skill notes in both TRT references | distilled-trt-llm.md [61](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L61) · trt-llm.md [29](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L29) [669](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L669) [674–687](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L674-L687) | | F6 | P1 | No output-verification step; ffprobe/ffmpeg/Pillow absent on the host; the NIM container's ffmpeg cannot demux its own VP9 output | New "Verify the output" section (ffprobe, root-free `av`, PIL; expected smoke facts) + pointers; NIM headless-check note | SKILL.md [41–43](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/SKILL.md#L41-L43) [454–467](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/SKILL.md#L454-L467) · diffusers.md [437–446](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/diffusers.md#L437-L446) [482](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/diffusers.md#L482) · payloads.md [101](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/payloads.md#L101) | | F7 | P1 | Server variants lack a headless start / readiness-poll / teardown recipe; SGLang `/health` 503 while warming undocumented; NIM `docker run -d --rm` destroys the logs | Shared named-container + trtllm-serve patterns (start, poll, logs, teardown) with per-variant pointers; SGLang readiness documented; NIM `--rm` dropped (recorded deviation); Nano/Super t2i request body stated in Variant 5 | SKILL.md [230–231](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/SKILL.md#L230-L231) [274–279](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/SKILL.md#L274-L279) [319–321](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/SKILL.md#L319-L321) [372–374](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/SKILL.md#L372-L374) [401](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/SKILL.md#L401) [451–452](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/SKILL.md#L451-L452) · payloads.md [216–217](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/payloads.md#L216-L217) · sglang.md [21](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/sglang.md#L21) · skill-notes.md [58–101](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/skill-notes.md#L58-L101) · trt-llm.md [43](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L43) [733](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L733) | | F7b | P2 | Servers started in a backgrounded subshell died with the shell; SIGINT to the process group did not stop a server; a second server was started while the first still held the GPU | Headless pattern: launch from a script with `setsid … &`, SIGINT → SIGTERM → SIGKILL escalation, leftover check by PID; never start a second server on a busy GPU (the launch script enforces it) | SKILL.md [372–377](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/SKILL.md#L372-L377) · distilled-trt-llm.md [465–474](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L465-L474) · skill-notes.md [75–94](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/skill-notes.md#L75-L94) | | F8 | P1 | Distilled 4-Step students are ~122 GiB bf16, not "32B / single GPU": the 1-GPU launch OOMs on 80 GB; the verified `cpu_offload_config` route was undocumented | Model-table note + Variant 6 offload block (measured) + warning that FP8 variants are unverified; same in distilled-trt-llm.md | SKILL.md [79–84](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/SKILL.md#L79-L84) [339–343](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/SKILL.md#L339-L343) · distilled-trt-llm.md [22](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L22) [35](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L35) [84–99](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L84-L99) [101](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L101) [103](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L103) [366–370](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L366-L370) [431–439](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L431-L439) [443–447](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L443-L447) · skill-notes.md [101](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/skill-notes.md#L101) | | F8b | P1 | Distilled launch still OOM-prone in practice: agents ran the notebook's plain bf16 launch first (OOM on 80–94 GB GPUs, and a second launch then OOMed against the still-loading server); on 125 GiB-RAM hosts the offload route is OOM-killed | Variant 6 is one gated launch script: GPU memory → plain route only at ≥ 128,000 MiB; host RAM (incl. cgroup ceiling) → bf16 + `cpu_offload_config` at ≥ 128 GiB, else the Hugging Face `fp8` branch with the `embed_tokens` ignore-list patch (eval-verified); refuses to start on a busy GPU; one `setsid trtllm-serve` + `/health` poll with a 1800 s deadline | SKILL.md [345–348](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/SKILL.md#L345-L348) [372–377](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/SKILL.md#L372-L377) · distilled-trt-llm.md [63](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L63) [339–416](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L339-L416) | | F9 | P1 | SGLang `docker run` passes no `-e HF_TOKEN` and omits `SGLANG_DISABLE_COSMOS3_GUARDRAILS=1`; the server exits with GatedRepoError 401 | Guardrail-table row fixed; `-e SGLANG_DISABLE_COSMOS3_GUARDRAILS=1 -e HF_TOKEN` in every launch block; request body `guardrails: false` | SKILL.md [249–254](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/SKILL.md#L249-L254) [262–263](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/SKILL.md#L262-L263) [284](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/SKILL.md#L284) [291](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/SKILL.md#L291) · payloads.md [67](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/payloads.md#L67) · sglang.md [23](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/sglang.md#L23) [35](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/sglang.md#L35) [41–42](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/sglang.md#L41-L42) [80](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/sglang.md#L80) [371](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/sglang.md#L371) · skill-notes.md [42–49](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/skill-notes.md#L42-L49) [53–57](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/skill-notes.md#L53-L57) | | F9b | P1 | vLLM-Omni `docker run` blocks passed no `-e HF_TOKEN`: on a fresh node with a cold Hugging Face cache the first launch crashed with HTTP 429 during the anonymous model lookup | Superseded by the notebook's final version (#329), which passes `-e HF_TOKEN` in every `vllm serve` launch; the skill carries that text 1:1 (vllm-omni.md § 2, SKILL.md Variant 3) and `vllm-omni.md` § 7.1 records why the token matters — export `HF_TOKEN` before the launch | SKILL.md [213](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/SKILL.md#L213) · vllm-omni.md [602–608](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/vllm-omni.md#L602-L608) | | F10 | P1 | NIM profile discovery/pinning (`list-model-profiles`, `NIM_MODEL_PROFILE`, `/v1/manifest`) and guardrail env switches missing from the launch block | profile preflight and `/v1/metadata` check in the Variant 7 launch; env lines in both launch blocks; `/v1/manifest` + `selectedModelProfileId` in the inspect lists | SKILL.md [391–397](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/SKILL.md#L391-L397) [407–408](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/SKILL.md#L407-L408) [417–418](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/SKILL.md#L417-L418) [423](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/SKILL.md#L423) | | F11 | P1 | NIM cache bind mount not writable by uid 1000; the client-side `chmod -R 777` never reaches a daemon-created mount | Cache preflight in the Variant 7 launch: the write test as the NIM user (uid 1000) runs first; the ownership repair (`docker run --user 0 … chown 1000:1000`) runs only for an empty cache this run created or with `NIM_CACHE_REPAIR=1`, otherwise the launch stops with guidance (review follow-up, see below) | SKILL.md [395–396](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/SKILL.md#L395-L396) [409–411](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/SKILL.md#L409-L411) · skill-notes.md [102–108](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/skill-notes.md#L102-L108) | | F12 | P1 | rc26 returns HTTP 500 on `/v1/images/generations`; `/health` 200 does not exercise the image route | rc27 pinned with the reason inside Route A; "a 200 on /health does not exercise the image route" sentence; skill note at trt-llm.md § 2 | trt-llm.md [70](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L70) [564](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L564) [612–620](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L612-L620) [636–637](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L636-L637) [737–740](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L737-L740) | | F13 | P2 | Model table had no measured memory figures, no small-GPU path | Measured paragraph under the table (Edge/Nano, labelled as eval measurements) + fp8 pointer; Nano YAML note in Variant 5 / trt-llm.md | skill-notes.md [26–38](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/skill-notes.md#L26-L38) · trt-llm.md [43](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L43) [722–733](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L722-L733) | | F14 | P2 | Variant 6 returns a one-frame MP4/AVI, the eval asks for a PNG | `ffmpeg -frames:v 1` extraction documented in Variant 6 and distilled-trt-llm.md | SKILL.md [358–361](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/SKILL.md#L358-L361) · distilled-trt-llm.md [267](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L267) [277–286](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L277-L286) | | F14b | P2 | On the fp8 server the one-frame `/v1/videos/generations` returned HTTP 500 after a successful denoise | Documented fallback: `/v1/images/generations` with `output_type: "image"` (explicit body; needs rc27), recorded in result_summary.md | SKILL.md [361](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/SKILL.md#L361) · distilled-trt-llm.md [279](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L279) [451–461](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L451-L461) | | F15 | P2 | Wheel path only needs `examples/visual_gen/configs/*.yaml`, yet the block prescribed full clone + submodules + LFS | Direct SHA fetch (`git fetch --depth 1 origin <sha>`), LFS skipped; shallow-clone warning; Edge needs no checkout | trt-llm.md [636–642](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L636-L642) [647–652](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L647-L652) [703–706](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L703-L706) | | F16 | P2 | `pip install cosmos_guardrail==0.3.0` unconditional although guardrails are disabled; unresolvable on 1.2.1 | Moved to an optional "only if you keep guardrails on" step quoting the setup guide's current pin, `cosmos_guardrail==0.3.2` (its former NLTK data step is gone, #332), with the transformers>=5 caveat | trt-llm.md [669](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L669) [691–699](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L691-L699) | | F17 | P2 | SKILL.md said the NIM "takes plain-text prompts"; `/v1/infer` accepted the compact-JSON assets | Reworded intro + compact-JSON rule; Variant 7 request guidance; the Variant 7 request block | SKILL.md [428–429](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/SKILL.md#L428-L429) [436–439](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/SKILL.md#L436-L439) · skill-notes.md [8–10](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/skill-notes.md#L8-L10) [13–15](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/skill-notes.md#L13-L15) | | F18 | P2 | Variant 1 had no runnable Edge t2i/t2v snippet, no cwd comment, benign load warnings undocumented | Edge block composed from the README sentence and the notebook branches; cwd comment; two measured bullets | diffusers.md [465](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/diffusers.md#L465) [486–517](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/diffusers.md#L486-L517) · skill-notes.md [122–130](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/skill-notes.md#L122-L130) | | F2d | P1 | Variant 5 Route A: `uv pip install` run in a later shell call than the venv activation landed the wheel set in whichever Python was first on PATH (eval: the evaluator's own venv, where `llist` has no wheel and the source build wanted `clang`) | The § 7.2 install block and its preflights name the venv explicitly: `uv pip install --python "$TRTLLM_VENV/bin/python"`, `"$TRTLLM_VENV/bin/python" -c …`, `"$TRTLLM_VENV/bin/trtllm-serve"` (the F2c mechanism applied to the TensorRT-LLM launch; added after the validation run) | trt-llm.md [608–611](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L608-L611) [621](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L621) [628](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L628) [634–635](https://github.com/NVIDIA/cosmos-framework/blob/55f0f0f/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L634-L635) | ### Review follow-ups (2026-10-29) Three adjustments after the review, all in one commit (`ddb28a2`), no eval re-run: (1) the fp8 route of `distilled-trt-llm.md` § 7.2 no longer deletes anything — the overlay is a fresh `mktemp -d` directory, and a caller-set `FP8_DST` must not exist yet and must not be `FP8_SRC` or a path inside it; (2) the Variant 7 cache preflight tests write access as the NIM user first and repairs ownership only for an empty cache this run created or with `NIM_CACHE_REPAIR=1`, otherwise it fails with guidance (the host-side recursive `chmod` is gone too); (3) `diffusers.md` § 8.4 is a shared setup block plus one block per mode, and Variant 1 asks for the setup block plus only the requested mode's block. ### Validation **Baseline (first commit `b7ad087`)** — [run 40660687591](https://github.com/NVIDIA/cosmos-framework/actions/runs/40660687591), [dashboard group ci-pr327-b7ad087](http://internal-cosmos-skills-eval.nvidia.com/skills/cosmos3-generate-media/groups/ci-pr327-b7ad087/): **44/44 PASS** (44/44 first-attempt, 0 FAIL, 0 no report, 0 errors; static assessment 100/100) **Baseline vs #324** ([compare](http://internal-cosmos-skills-eval.nvidia.com/skills/cosmos3-generate-media/compare?a=group%3Aci-pr324-b50b766&b=group%3Aci-pr327-b7ad087); the [branch compare](http://internal-cosmos-skills-eval.nvidia.com/skills/cosmos3-generate-media/compare?a=branch%3Askills%2Fgenerate-media-migration-and-eval-public&b=branch%3Askills%2Fgenerate-media-fixes-public) always shows the newest result per cell of each branch): **A** `group:ci-pr324-b50b766`: 42/44 PASS, 14 first-attempt · **B** `group:ci-pr327-b7ad087`: 44/44 PASS, 44 first-attempt · improved 2, regressed 0 | Case | Metric | claude | codex | cursor | opencode | |---|---|---|---|---|---| | `diffusers-edge-t2i` | Completion | ✅→✅ | ✅→✅ | ✅→✅ | ✅→✅ | | | First time success | ❌→✅ | ❌→✅ | ❌→✅ | ❌→✅ | | `diffusers-edge-t2v` | Completion | ✅→✅ | ✅→✅ | ✅→✅ | ✅→✅ | | | First time success | ❌→✅ | ❌→✅ | ❌→✅ | ❌→✅ | | `diffusers-nano-t2i` | Completion | ✅→✅ | ✅→✅ | ✅→✅ | ✅→✅ | | | First time success | ✅→✅ | ❌→✅ | ❌→✅ | ❌→✅ | | `distilled-trtllm-t2i` | Completion | ✅→✅ | ✅→✅ | ✅→✅ | ❌→✅ | | | First time success | ❌→✅ | ❌→✅ | ❌→✅ | ❌→✅ | | `framework-edge-t2i` | Completion | ✅→✅ | ✅→✅ | ✅→✅ | ✅→✅ | | | First time success | ❌→✅ | ✅→✅ | ❌→✅ | ❌→✅ | | `framework-nano-t2i` | Completion | ✅→✅ | ✅→✅ | ✅→✅ | ✅→✅ | | | First time success | ❌→✅ | ✅→✅ | ❌→✅ | ✅→✅ | | `nim-generator-i2v` | Completion | ✅→✅ | ✅→✅ | ✅→✅ | ✅→✅ | | | First time success | ✅→✅ | ✅→✅ | ✅→✅ | ❌→✅ | | `sglang-edge-t2i` | Completion | ✅→✅ | ✅→✅ | ✅→✅ | ✅→✅ | | | First time success | ✅→✅ | ❌→✅ | ❌→✅ | ✅→✅ | | `trtllm-edge-t2i` | Completion | ✅→✅ | ✅→✅ | ✅→✅ | ❌→✅ | | | First time success | ❌→✅ | ❌→✅ | ❌→✅ | ❌→✅ | | `trtllm-nano-t2i` | Completion | ✅→✅ | ✅→✅ | ✅→✅ | ✅→✅ | | | First time success | ✅→✅ | ❌→✅ | ❌→✅ | ❌→✅ | | `vllm-omni-edge-t2i` | Completion | ✅→✅ | ✅→✅ | ✅→✅ | ✅→✅ | | | First time success | ✅→✅ | ✅→✅ | ✅→✅ | ✅→✅ | Completion: overall verdict before → after (✅ PASS, ❌ FAIL). First time success: passed on the first attempt, without a self-correction, before → after. **Added after the validation run:** F2d (`bad39b1`); four relative-link path corrections in `references/` (`../cookbooks/…` → `../../cookbooks/…`); and, merged from #324, the vLLM-Omni notebook's final version (#329: `$PWD` mounts, `-e HF_TOKEN`, the FP8 "Quantized Checkpoints" section in `vllm-omni.md` § 2 and SKILL.md Variant 3; the FP8 pointer in `payloads.md` § 1) — validated locally with the four vllm-omni-edge-t2i cells on the base branch, 4/4 PASS first attempt. `git diff b7ad087 55f0f0f -- .agents/skills/cosmos3-generate-media` shows exactly these. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Christina Wang <christwang@nvidia.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
lfengad
added a commit
that referenced
this pull request
Oct 30, 2026
…erived fixes, cosmos3-skill-authoring guide, opt-in skill-eval CI (#324) **Includes the eval-derived fixes PR #327**, merged into this branch on 2026-10-29 (`e1a3ff1`): the migration and the fixes that take the eval matrix from 42/44 to 44/44 first-attempt PASS go to `main` together. Everything is applied as a diff on top of `main`, not as a copy. First cookbook→skill migration (**cosmos3-generate-media**), the **cosmos3-skill-authoring** guide, and an opt-in **skill-eval CI** on self-hosted H100 runners. No eval (GPU) job runs on push or merge. ### What this PR delivers - **One skill for the seven Cosmos3 audiovisual serving paths** — diffusers, cosmos_framework, vLLM-Omni, SGLang, TensorRT-LLM, distilled TensorRT-LLM, NIM — with the sample prompts, reference images and negative prompts under `assets/`. - **The cookbook notebooks migrated 1:1 into `references/`** (one file per backend plus `payloads.md` for shared content). A format change only: Jupyter plumbing removed, nothing corrected or rewritten; `references/vllm-omni.md` follows the vLLM-Omni notebook's last version on `main` (#329: launch mounts, `HF_TOKEN`, the FP8 "Quantized Checkpoints" section), since that notebook changed after the migration's pin and before its retirement here; the six notebooks and the audiovisual cookbook README are retired from `.agents/skills/cookbooks/` (links in the cookbook READMEs and the docs site now point at the skill). Known gap: the video-to-video input clip (an LFS object) is not vendored. - **NIM (Variant 7) follows the Cosmos3 NIM 2.1.0 cookbook.** `run_with_nim.ipynb` was already retired in this repository by #296 in favour of `.agents/skills/cookbooks/cosmos3/nim/`, so the skill carries no `references/nim.md` copy of it: Variant 7 delegates to that cookbook and its `cosmos3-nim-user` skill (pointers, the README's 2.1.0 launch, the readiness and selection checks of `deployment.md`, and the `examples/i2v.py` request built from the bundled assets). Record: `docs/cookbook_to_skill_migration.md`. - **SKILL.md quotes each backend's smoke path 1:1** from the notebooks, the cookbook README or the setup guide, and makes the reference files required reading before the first command. Review requests from the first review round are applied (the framework example initialises `COSMOS3_UV_GROUP` / `COSMOS3_UV_ENV` as the notebook does; the live CI comment uses a sequence guard and per-cell ledger lines). - **Eval-derived fixes (from #327)**: F1–F18 and F2d — every fatal failure and repeated self-correction of the 44-cell matrix, fixed as marked skill notes or declared in-block deviations with the notebook text kept verbatim; `eval/eval.config` is unchanged, so before and after are measured on the same prompts (tables below). - **11 eval cases** (3 Nano, 6 Edge, 1 distilled Super, 1 NIM); the two framework cases clone this repository's `main`. Known wording leftover in `eval/eval.config`: the NIM case says the Generator NIM serves text-to-video and image-to-video only, which was true of the retired 1.0.0 NIM; the pinned 2.1.0 NIM also serves text-to-image and video-to-video. The case exercises image-to-video, so results are unaffected; the sentence is corrected in the next `eval.config` revision together with a re-run (the file is frozen here so the migration head and the fixes are measured on the same prompts). - **Skill-eval CI (opt-in)**: `.github/workflows/skill-eval.yml` and the orchestrator in `.github/scripts/skill_eval/` run a PR's skills on self-hosted H100 runners — one job per case × agent, one live PR comment that always shows the newest result per cell, dashboard groups per skill version, the inference key from a repository secret, a GitHub-hosted job for the orchestrator's unit tests, and a GitHub-hosted **gate** that admits only repository members with the write, maintain or admin role (details below). - **cosmos3-skill-authoring**: how to write a skill that passes the eval on the first attempt, how the evaluator judges, how to run the eval in CI or locally, and the fix loop (a stacked fixes PR, gated by the owner) that gets every cell to first-attempt PASS (`references/running-the-eval.md`). ### Evaluation results #### Migration head `b50b766` (before the fixes) Head `b50b766`, 11 cases × 4 agents (claude = Claude Opus 5.5, codex = GPT-6 Astra, cursor = Claude Sonnet 5.5, opencode = Kimi K2.6), [run 40639480322](https://github.com/NVIDIA/cosmos-framework/actions/runs/40639480322) on the four self-hosted H100 runners, [dashboard group ci-pr324-b50b766](http://internal-cosmos-skills-eval.nvidia.com/skills/cosmos3-generate-media/groups/ci-pr324-b50b766/): **42/44 PASS** (14/44 first-attempt, 2 FAIL, 0 no report, 0 errors; static assessment 100/100) | Case | claude | codex | cursor | opencode | |---|---|---|---|---| | `diffusers-edge-t2i` | ✅ 1.84M ↻ | ✅ 2.89M ↻ | ✅ 1.07M ↻ | ✅ 1.41M ↻ | | `diffusers-edge-t2v` | ✅ 2.81M ↻ | ✅ 3.32M ↻ | ✅ 594K ↻ | ✅ 1.31M ↻ | | `diffusers-nano-t2i` | ✅ 2.08M | ✅ 3.28M ↻ | ✅ 529K ↻ | ✅ 869K ↻ | | `distilled-trtllm-t2i` | ✅ 8.95M ↻ | ✅ 9.73M ↻ | ✅ 2.11M ↻ | ❌ 4.51M ↻ | | `framework-edge-t2i` | ✅ 2.56M ↻ | ✅ 3.73M | ✅ 465K ↻ | ✅ 2.76M ↻ | | `framework-nano-t2i` | ✅ 2.71M ↻ | ✅ 4.59M | ✅ 488K ↻ | ✅ 2.32M | | `nim-generator-i2v` | ✅ 3.46M | ✅ 3.22M | ✅ 390K | ✅ 2.67M ↻ | | `sglang-edge-t2i` | ✅ 2.93M | ✅ 2.30M ↻ | ✅ 584K ↻ | ✅ 1.13M | | `trtllm-edge-t2i` | ✅ 6.12M ↻ | ✅ 18.32M ↻ | ✅ 864K ↻ | ❌ 5.53M ↻ | | `trtllm-nano-t2i` | ✅ 5.43M | ✅ 19.52M ↻ | ✅ 1.46M ↻ | ✅ 7.62M ↻ | | `vllm-omni-edge-t2i` | ✅ 1.88M | ✅ 1.41M | ✅ 645K | ✅ 2.02M | Tokens (all cells): 154.42M. ↻ = passed after self-correction. The two FAILs are the opencode agent on the two TensorRT-LLM wheel cases — the documentation gap (no notebook or README says how to obtain a TensorRT-LLM build with Cosmos3 VisualGen) that the fixes below close; nothing the agents self-corrected was changed by the migration itself. #### After the fixes (`b7ad087`, the #327 head at validation) **Baseline (first commit `b7ad087`)** — [run 40660687591](https://github.com/NVIDIA/cosmos-framework/actions/runs/40660687591), [dashboard group ci-pr327-b7ad087](http://internal-cosmos-skills-eval.nvidia.com/skills/cosmos3-generate-media/groups/ci-pr327-b7ad087/): **44/44 PASS** (44/44 first-attempt, 0 FAIL, 0 no report, 0 errors; static assessment 100/100) **Baseline vs #324** ([compare](http://internal-cosmos-skills-eval.nvidia.com/skills/cosmos3-generate-media/compare?a=group%3Aci-pr324-b50b766&b=group%3Aci-pr327-b7ad087); the [branch compare](http://internal-cosmos-skills-eval.nvidia.com/skills/cosmos3-generate-media/compare?a=branch%3Askills%2Fgenerate-media-migration-and-eval-public&b=branch%3Askills%2Fgenerate-media-fixes-public) always shows the newest result per cell of each branch): **A** `group:ci-pr324-b50b766`: 42/44 PASS, 14 first-attempt · **B** `group:ci-pr327-b7ad087`: 44/44 PASS, 44 first-attempt · improved 2, regressed 0 | Case | Metric | claude | codex | cursor | opencode | |---|---|---|---|---|---| | `diffusers-edge-t2i` | Completion | ✅→✅ | ✅→✅ | ✅→✅ | ✅→✅ | | | First time success | ❌→✅ | ❌→✅ | ❌→✅ | ❌→✅ | | `diffusers-edge-t2v` | Completion | ✅→✅ | ✅→✅ | ✅→✅ | ✅→✅ | | | First time success | ❌→✅ | ❌→✅ | ❌→✅ | ❌→✅ | | `diffusers-nano-t2i` | Completion | ✅→✅ | ✅→✅ | ✅→✅ | ✅→✅ | | | First time success | ✅→✅ | ❌→✅ | ❌→✅ | ❌→✅ | | `distilled-trtllm-t2i` | Completion | ✅→✅ | ✅→✅ | ✅→✅ | ❌→✅ | | | First time success | ❌→✅ | ❌→✅ | ❌→✅ | ❌→✅ | | `framework-edge-t2i` | Completion | ✅→✅ | ✅→✅ | ✅→✅ | ✅→✅ | | | First time success | ❌→✅ | ✅→✅ | ❌→✅ | ❌→✅ | | `framework-nano-t2i` | Completion | ✅→✅ | ✅→✅ | ✅→✅ | ✅→✅ | | | First time success | ❌→✅ | ✅→✅ | ❌→✅ | ✅→✅ | | `nim-generator-i2v` | Completion | ✅→✅ | ✅→✅ | ✅→✅ | ✅→✅ | | | First time success | ✅→✅ | ✅→✅ | ✅→✅ | ❌→✅ | | `sglang-edge-t2i` | Completion | ✅→✅ | ✅→✅ | ✅→✅ | ✅→✅ | | | First time success | ✅→✅ | ❌→✅ | ❌→✅ | ✅→✅ | | `trtllm-edge-t2i` | Completion | ✅→✅ | ✅→✅ | ✅→✅ | ❌→✅ | | | First time success | ❌→✅ | ❌→✅ | ❌→✅ | ❌→✅ | | `trtllm-nano-t2i` | Completion | ✅→✅ | ✅→✅ | ✅→✅ | ✅→✅ | | | First time success | ✅→✅ | ❌→✅ | ❌→✅ | ❌→✅ | | `vllm-omni-edge-t2i` | Completion | ✅→✅ | ✅→✅ | ✅→✅ | ✅→✅ | | | First time success | ✅→✅ | ✅→✅ | ✅→✅ | ✅→✅ | Completion: overall verdict before → after (✅ PASS, ❌ FAIL). First time success: passed on the first attempt, without a self-correction, before → after. **Added after the validation run:** F2d (`bad39b1`); four relative-link path corrections in `references/` (`../cookbooks/…` → `../../cookbooks/…`); and the vLLM-Omni notebook's final version (#329: `$PWD` mounts, `-e HF_TOKEN`, the FP8 "Quantized Checkpoints" section in `vllm-omni.md` § 2 and SKILL.md Variant 3; the FP8 pointer in `payloads.md` § 1) — validated locally with the four vllm-omni-edge-t2i cells, 4/4 PASS on the first attempt twice (after the port and after the merge of the fixes). `git diff b7ad087 e1a3ff1 -- .agents/skills/cosmos3-generate-media` shows exactly these, plus the review follow-ups below. ### Eval-derived fixes (merged from #327) Where = `file:lines` at `e1a3ff1` (permalinks; `SKILL.md` = `.agents/skills/cosmos3-generate-media/SKILL.md`, reference names = `references/<name>`). Problem → fix, in priority order; F2d was added after the validation run (see Validation). | ID | Pri | Problem | Fix applied | Where | |---|---|---|---|---| | F1 | P0 | Variant 5 gave only the Docker devel source build (tag left `<tag>`) plus a dead `TRTLLM_USE_PRECOMPILED` hint; no wheel version anywhere, so cursor/opencode pip-installed stable 1.2.x, which lacks VisualGen | Route A (default): prebuilt wheel `tensorrt_llm==1.3.0rc27` + `torch==2.13.0+cu130` from pypi.nvidia.com/cu130 (uv `--index-strategy unsafe-best-match`), VisualGen preflight, pinned checkout for YAMLs only; Route B = source build; mirrored as skill notes in trt-llm.md / distilled-trt-llm.md | SKILL.md [296–299](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L296-L299) · distilled-trt-llm.md [61](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L61) · trt-llm.md [29](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L29) [583–589](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L583-L589) [595–639](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L595-L639) [656–658](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L656-L658) [722](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L722) | | F2 | P1 | Framework `pyproject.toml` pins uv `==0.11.29`; `uv venv/pip/sync` abort with uv 0.13 and `uv self update` refuses | Variant 1: `--no-config`; Variant 2: pinned-uv preflight (side-by-side install or `uvx --from 'uv==0.11.29'`); one note in "Read this first"; skill notes in diffusers.md, cosmos-framework.md, payloads.md | SKILL.md [31–35](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L31-L35) [104–108](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L104-L108) [120](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L120) [142–143](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L142-L143) [177–184](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L177-L184) · cosmos-framework.md [202](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/cosmos-framework.md#L202) [409–410](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/cosmos-framework.md#L409-L410) · diffusers.md [136](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/diffusers.md#L136) [138](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/diffusers.md#L138) [157](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/diffusers.md#L157) · payloads.md [85](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/payloads.md#L85) [99](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/payloads.md#L99) | | F2b | P1 | An agent installed uv 0.11.29 (the Variant 2 preflight) for the TRT-LLM venv; that uv cannot resolve `nvidia-cutlass-dsl==4.8.0.dev0` without `--prerelease=allow`, which then pulls mpmath 1.5.0a1 | "Read this first" / Variant 5 / Variant 2 state which uv each variant uses (stock 0.13+ with `--no-config` for Variants 1/5/6; 0.11.29 only for `uv sync`) | SKILL.md [36–37](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L36-L37) [183](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L183) · skill-notes.md [141–145](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/skill-notes.md#L141-L145) · trt-llm.md [605](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L605) | | F2c | P1 | In a full run of the fixes branch the opencode agent ran the Variant 1 install block across separate shell calls; the `export` and `source …/activate` of the first call were gone in the second, so `uv pip install` went into the evaluator's own venv and the import check ran against the wrong interpreter (then a needless OpenCV downgrade: the same block passed first-attempt in the other eleven diffusers cells of that run) | The three `uv pip` commands of the Variant 1 block and of diffusers.md § 3 carry `--python "$COSMOS3_DIFFUSERS_VENV"` (declared in-block addition; with the variable unset uv stops instead of installing elsewhere), the import check and the generation snippets run with `"$COSMOS3_DIFFUSERS_VENV/bin/python"`, and the block says it is one shell call | SKILL.md [110–126](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L110-L126) · diffusers.md [138–151](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/diffusers.md#L138-L151) [157](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/diffusers.md#L157) [161](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/diffusers.md#L161) [453–454](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/diffusers.md#L453-L454) | | F3 | P1 | Diffusers venv not headless: `cosmos_guardrail` pulls GUI opencv; the pipeline imports it even with the safety checker off | Headless OpenCV swap + import check inside the Variant 1 venv block; apt fallback reworded (`apt-get update`, root caveat); skill notes in diffusers.md / payloads.md | SKILL.md [98–100](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L98-L100) [116–126](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L116-L126) · diffusers.md [74](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/diffusers.md#L74) [159](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/diffusers.md#L159) [437–446](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/diffusers.md#L437-L446) · payloads.md [81](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/payloads.md#L81) | | F3b | P1 | Headless OpenCV swap was a separate command after the install line; one agent ran the pipeline import first and hit `libxcb.so.1` | Swap folded into the `uv pip install` command in SKILL.md Variant 1 and, as a declared in-block deviation, in diffusers.md § 3 | SKILL.md [116–126](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L116-L126) · diffusers.md [138–151](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/diffusers.md#L138-L151) [157](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/diffusers.md#L157) [159](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/diffusers.md#L159) | | F4 | P1 | Bare-host prerequisites of the wheel undocumented: OpenMPI, `CUDA_HOME`, CUDA-13 libs via the cu130 index, seeded venv, ffmpeg | Prerequisite lines + comments inside the Route A block (Variant 6 inherits); upstream pip-install page cited in the references | distilled-trt-llm.md [61](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L61) · trt-llm.md [29](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L29) [601–606](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L601-L606) [624–635](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L624-L635) [703–706](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L703-L706) | | F5 | P1 | `TRTLLM_USE_PRECOMPILED=1 pip install -e .` resolves an unpublished `1.3.0rc26+bca6761ab8` wheel; rc26 binaries lack an op the pinned Python needs | Hint deleted; "do not use the shortcut on this checkout" paragraph; one-sentence skill notes in both TRT references | distilled-trt-llm.md [61](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L61) · trt-llm.md [29](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L29) [669](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L669) [674–687](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L674-L687) | | F6 | P1 | No output-verification step; ffprobe/ffmpeg/Pillow absent on the host; the NIM container's ffmpeg cannot demux its own VP9 output | New "Verify the output" section (ffprobe, root-free `av`, PIL; expected smoke facts) + pointers; NIM headless-check note | SKILL.md [41–43](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L41-L43) [454–467](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L454-L467) · diffusers.md [437–446](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/diffusers.md#L437-L446) [482](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/diffusers.md#L482) · payloads.md [101](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/payloads.md#L101) | | F7 | P1 | Server variants lack a headless start / readiness-poll / teardown recipe; SGLang `/health` 503 while warming undocumented; NIM `docker run -d --rm` destroys the logs | Shared named-container + trtllm-serve patterns (start, poll, logs, teardown) with per-variant pointers; SGLang readiness documented; NIM `--rm` dropped (recorded deviation); Nano/Super t2i request body stated in Variant 5 | SKILL.md [230–231](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L230-L231) [274–279](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L274-L279) [319–321](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L319-L321) [372–374](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L372-L374) [401](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L401) [451–452](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L451-L452) · payloads.md [216–217](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/payloads.md#L216-L217) · sglang.md [21](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/sglang.md#L21) · skill-notes.md [58–101](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/skill-notes.md#L58-L101) · trt-llm.md [43](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L43) [733](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L733) | | F7b | P2 | Servers started in a backgrounded subshell died with the shell; SIGINT to the process group did not stop a server; a second server was started while the first still held the GPU | Headless pattern: launch from a script with `setsid … &`, SIGINT → SIGTERM → SIGKILL escalation, leftover check by PID; never start a second server on a busy GPU (the launch script enforces it) | SKILL.md [372–377](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L372-L377) · distilled-trt-llm.md [465–474](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L465-L474) · skill-notes.md [75–94](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/skill-notes.md#L75-L94) | | F8 | P1 | Distilled 4-Step students are ~122 GiB bf16, not "32B / single GPU": the 1-GPU launch OOMs on 80 GB; the verified `cpu_offload_config` route was undocumented | Model-table note + Variant 6 offload block (measured) + warning that FP8 variants are unverified; same in distilled-trt-llm.md | SKILL.md [79–84](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L79-L84) [339–343](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L339-L343) · distilled-trt-llm.md [22](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L22) [35](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L35) [84–99](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L84-L99) [101](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L101) [103](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L103) [366–370](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L366-L370) [431–439](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L431-L439) [443–447](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L443-L447) · skill-notes.md [101](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/skill-notes.md#L101) | | F8b | P1 | Distilled launch still OOM-prone in practice: agents ran the notebook's plain bf16 launch first (OOM on 80–94 GB GPUs, and a second launch then OOMed against the still-loading server); on 125 GiB-RAM hosts the offload route is OOM-killed | Variant 6 is one gated launch script: GPU memory → plain route only at ≥ 128,000 MiB; host RAM (incl. cgroup ceiling) → bf16 + `cpu_offload_config` at ≥ 128 GiB, else the Hugging Face `fp8` branch with the `embed_tokens` ignore-list patch (eval-verified); refuses to start on a busy GPU; one `setsid trtllm-serve` + `/health` poll with a 1800 s deadline | SKILL.md [345–348](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L345-L348) [372–377](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L372-L377) · distilled-trt-llm.md [63](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L63) [339–416](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L339-L416) | | F9 | P1 | SGLang `docker run` passes no `-e HF_TOKEN` and omits `SGLANG_DISABLE_COSMOS3_GUARDRAILS=1`; the server exits with GatedRepoError 401 | Guardrail-table row fixed; `-e SGLANG_DISABLE_COSMOS3_GUARDRAILS=1 -e HF_TOKEN` in every launch block; request body `guardrails: false` | SKILL.md [249–254](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L249-L254) [262–263](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L262-L263) [284](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L284) [291](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L291) · payloads.md [67](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/payloads.md#L67) · sglang.md [23](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/sglang.md#L23) [35](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/sglang.md#L35) [41–42](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/sglang.md#L41-L42) [80](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/sglang.md#L80) [371](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/sglang.md#L371) · skill-notes.md [42–49](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/skill-notes.md#L42-L49) [53–57](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/skill-notes.md#L53-L57) | | F9b | P1 | vLLM-Omni `docker run` blocks passed no `-e HF_TOKEN`: on a fresh node with a cold Hugging Face cache the first launch crashed with HTTP 429 during the anonymous model lookup | Superseded by the notebook's final version (#329), which passes `-e HF_TOKEN` in every `vllm serve` launch; the skill carries that text 1:1 (vllm-omni.md § 2, SKILL.md Variant 3) and `vllm-omni.md` § 7.1 records why the token matters — export `HF_TOKEN` before the launch | SKILL.md [213](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L213) · vllm-omni.md [602–608](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/vllm-omni.md#L602-L608) | | F10 | P1 | NIM profile discovery/pinning (`list-model-profiles`, `NIM_MODEL_PROFILE`, `/v1/manifest`) and guardrail env switches missing from the launch block | profile preflight and `/v1/metadata` check in the Variant 7 launch; env lines in both launch blocks; `/v1/manifest` + `selectedModelProfileId` in the inspect lists | SKILL.md [391–397](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L391-L397) [407–408](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L407-L408) [417–418](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L417-L418) [423](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L423) | | F11 | P1 | NIM cache bind mount not writable by uid 1000; the client-side `chmod -R 777` never reaches a daemon-created mount | Cache preflight in the Variant 7 launch: the write test as the NIM user (uid 1000) runs first; the ownership repair (`docker run --user 0 … chown 1000:1000`) runs only for an empty cache this run created or with `NIM_CACHE_REPAIR=1`, otherwise the launch stops with guidance (review follow-up, see below) | SKILL.md [395–396](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L395-L396) [409–411](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L409-L411) · skill-notes.md [102–108](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/skill-notes.md#L102-L108) | | F12 | P1 | rc26 returns HTTP 500 on `/v1/images/generations`; `/health` 200 does not exercise the image route | rc27 pinned with the reason inside Route A; "a 200 on /health does not exercise the image route" sentence; skill note at trt-llm.md § 2 | trt-llm.md [70](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L70) [564](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L564) [612–620](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L612-L620) [636–637](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L636-L637) [737–740](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L737-L740) | | F13 | P2 | Model table had no measured memory figures, no small-GPU path | Measured paragraph under the table (Edge/Nano, labelled as eval measurements) + fp8 pointer; Nano YAML note in Variant 5 / trt-llm.md | skill-notes.md [26–38](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/skill-notes.md#L26-L38) · trt-llm.md [43](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L43) [722–733](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L722-L733) | | F14 | P2 | Variant 6 returns a one-frame MP4/AVI, the eval asks for a PNG | `ffmpeg -frames:v 1` extraction documented in Variant 6 and distilled-trt-llm.md | SKILL.md [358–361](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L358-L361) · distilled-trt-llm.md [267](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L267) [277–286](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L277-L286) | | F14b | P2 | On the fp8 server the one-frame `/v1/videos/generations` returned HTTP 500 after a successful denoise | Documented fallback: `/v1/images/generations` with `output_type: "image"` (explicit body; needs rc27), recorded in result_summary.md | SKILL.md [361](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L361) · distilled-trt-llm.md [279](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L279) [451–461](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/distilled-trt-llm.md#L451-L461) | | F15 | P2 | Wheel path only needs `examples/visual_gen/configs/*.yaml`, yet the block prescribed full clone + submodules + LFS | Direct SHA fetch (`git fetch --depth 1 origin <sha>`), LFS skipped; shallow-clone warning; Edge needs no checkout | trt-llm.md [636–642](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L636-L642) [647–652](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L647-L652) [703–706](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L703-L706) | | F16 | P2 | `pip install cosmos_guardrail==0.3.0` unconditional although guardrails are disabled; unresolvable on 1.2.1 | Moved to an optional "only if you keep guardrails on" step quoting the setup guide's current pin, `cosmos_guardrail==0.3.2` (its former NLTK data step is gone, #332), with the transformers>=5 caveat | trt-llm.md [669](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L669) [691–699](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L691-L699) | | F17 | P2 | SKILL.md said the NIM "takes plain-text prompts"; `/v1/infer` accepted the compact-JSON assets | Reworded intro + compact-JSON rule; Variant 7 request guidance; the Variant 7 request block | SKILL.md [428–429](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L428-L429) [436–439](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/SKILL.md#L436-L439) · skill-notes.md [8–10](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/skill-notes.md#L8-L10) [13–15](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/skill-notes.md#L13-L15) | | F18 | P2 | Variant 1 had no runnable Edge t2i/t2v snippet, no cwd comment, benign load warnings undocumented | Edge block composed from the README sentence and the notebook branches; cwd comment; two measured bullets | diffusers.md [465](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/diffusers.md#L465) [486–517](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/diffusers.md#L486-L517) · skill-notes.md [122–130](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/skill-notes.md#L122-L130) | | F2d | P1 | Variant 5 Route A: `uv pip install` run in a later shell call than the venv activation landed the wheel set in whichever Python was first on PATH (eval: the evaluator's own venv, where `llist` has no wheel and the source build wanted `clang`) | The § 7.2 install block and its preflights name the venv explicitly: `uv pip install --python "$TRTLLM_VENV/bin/python"`, `"$TRTLLM_VENV/bin/python" -c …`, `"$TRTLLM_VENV/bin/trtllm-serve"` (the F2c mechanism applied to the TensorRT-LLM launch; added after the validation run) | trt-llm.md [608–611](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L608-L611) [621](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L621) [628](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L628) [634–635](https://github.com/NVIDIA/cosmos-framework/blob/e1a3ff1/.agents/skills/cosmos3-generate-media/references/trt-llm.md#L634-L635) | ### Review follow-ups (2026-10-29) Three adjustments after the review, all in one commit (`ddb28a2`), no eval re-run: (1) the fp8 route of `distilled-trt-llm.md` § 7.2 no longer deletes anything — the overlay is a fresh `mktemp -d` directory, and a caller-set `FP8_DST` must not exist yet and must not be `FP8_SRC` or a path inside it; (2) the Variant 7 cache preflight tests write access as the NIM user first and repairs ownership only for an empty cache this run created or with `NIM_CACHE_REPAIR=1`, otherwise it fails with guidance (the host-side recursive `chmod` is gone too); (3) `diffusers.md` § 8.4 is a shared setup block plus one block per mode, and Variant 1 asks for the setup block plus only the requested mode's block. A later commit (`55f0f0f`) follows the setup guide's move to `cosmos_guardrail==0.3.2` (trt-llm.md § 7.6; its former NLTK step is gone), notes in § 7.1 that the guide now pins TensorRT-LLM `cc63e66` while the skill's validated routes stay at `bca6761` / 1.3.0rc27, and points Variant 3 at the FP8 "Quantized Checkpoints" section. ### Skill-eval CI (opt-in; every `.github` change lives on this branch) - **Who may start it**: every trigger passes a GitHub-hosted `gate` job first (no checkout). It reads the role of the person who commented, applied the label or dispatched from the collaborators API and accepts `write`, `maintain` or `admin` only — organisation membership, a read/triage role or a PR from a fork is refused before any self-hosted runner is used (`plan` refuses fork heads a second time). - **Trigger**: comment `/run-skill-eval [args]` on the PR (reads the workflow copy on `main`, so it works after this PR merges), apply one of the labels `run-skill-eval` / `run-skill-eval-smoke` / `run-skill-eval-failed` / `run-skill-eval-force` (works before the merge — the PR's own workflow copy runs; the label is removed again at the end), or dispatch manually: `gh workflow run skill-eval.yml --ref <branch> -f pr_number=<N> -f scope=full -f args='…'`. Args: `force`, `rerun-failed`, `smoke-only` (alias `smoke`), `with-smoke`, `cells=<agent>/<case>,…`, `group=<id>`. Never per-commit. - **What runs**: every touched skill with an `eval/eval.config`, one job per case × agent over the idle H100 runners; the CPU canary only on request. One live PR comment per head commit that always shows the newest result per cell (re-runs of the same commit update it; a new commit gets a new comment that lists the PR's earlier runs), plus the dashboard; results are informational. - **Results are keyed by skill content**: PASS / FAIL cells are reused across commits that do not change the skill; a partial rerun replaces only the cells it ran. - **Agents**: Claude Code, Codex CLI, Cursor CLI and OpenCode; the cursor cells run Claude Sonnet 5.5 (`cursor_model` in `.github/skill-eval.yml`) so the four backends exercise four model families. - **Runners**: self-hosted runners registered to this repository with the labels `cosmos-skill-eval` + `h100`; the `gpu`/`h200` runners of `gpu-tests.yml` are not used (disjoint label sets). Deployment values (evaluator image, dashboard URL) are repository variables, the inference key a repository secret. - **Credentials and isolation**: the inference key comes from the repository secret, never from the image; the checkout mounted into the agent container carries no job token. - **Tests**: a GitHub-hosted job runs the orchestrator's 109 unit tests on changes under `.github/scripts/skill_eval/`. ### Authoring-skill revision Rewritten from the generate-media campaign: a description with trigger boundaries; structure and section order; writing commands that pass on the first attempt; how the evaluator judges; eval.config rules; running the eval and reading the dashboard; the fix loop (a stacked fixes PR, gated by the owner); submitting. Generic by design — the generate-media skill is linked as the worked example. Full runbook in `references/running-the-eval.md`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Christina Wang <christwang@nvidia.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: liang.feng <liangf@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The moved Cosmos3 cookbooks still pin
cosmos_guardrail==0.3.0and require manual server-side NLTK setup. Pincosmos_guardrail==0.3.2, which copies its NLTK data from the Hugging Face cache on first use, and remove the obsolete setup instructions from the Action and Transfer READMEs and notebooks. Update the TensorRT-LLM source pin tocc63e66daf52644f1441e117bb7ba67d20566016.Port of NVIDIA/cosmos#378 to the cookbooks' new location. Clarify that the empty video-content classifier warning does not disable the configured text checks or face blurring.