[christinayyw]

Summary

Stacked PR: based on #324's branch skills/generate-media-migration-and-eval-public; merge this PR into that branch first, then #324 carries both to main. Skill-only: .github is identical to the base.

Follow-up to #324: fixes every fatal failure and repeated self-correction of the cosmos3-generate-media eval matrix (11 cases × claude, codex, cursor, opencode). Baseline (group ci-pr324-b50b766): 42/44 PASS, 14/44 first-attempt. With the fixes (group ci-pr327-b7ad087): 44/44 PASS, 44/44 first-attempt.

eval/eval.config is unchanged from the base branch, so before and after are measured on the same prompts, assertions and checks. Notebook-verbatim text in references/ stays verbatim; every skill addition is a marked skill note ("Skill note (not in the notebook): …") or a declared in-block deviation.

Fixes

Where = file:lines at 55f0f0f (permalinks; SKILL.md = .agents/skills/cosmos3-generate-media/SKILL.md, reference names = references/<name>). Problem → fix, in priority order; F2d was added after the validation run (see Validation).

ID	Pri	Problem	Fix applied	Where
F1	P0	Variant 5 gave only the Docker devel source build (tag left <tag>) plus a dead TRTLLM_USE_PRECOMPILED hint; no wheel version anywhere, so cursor/opencode pip-installed stable 1.2.x, which lacks VisualGen	Route A (default): prebuilt wheel tensorrt_llm==1.3.0rc27 + torch==2.13.0+cu130 from pypi.nvidia.com/cu130 (uv --index-strategy unsafe-best-match), VisualGen preflight, pinned checkout for YAMLs only; Route B = source build; mirrored as skill notes in trt-llm.md / distilled-trt-llm.md	SKILL.md 296–299 · distilled-trt-llm.md 61 · trt-llm.md 29 583–589 595–639 656–658 722
F2	P1	Framework pyproject.toml pins uv ==0.11.29; uv venv/pip/sync abort with uv 0.13 and uv self update refuses	Variant 1: --no-config; Variant 2: pinned-uv preflight (side-by-side install or uvx --from 'uv==0.11.29'); one note in "Read this first"; skill notes in diffusers.md, cosmos-framework.md, payloads.md	SKILL.md 31–35 104–108 120 142–143 177–184 · cosmos-framework.md 202 409–410 · diffusers.md 136 138 157 · payloads.md 85 99
F2b	P1	An agent installed uv 0.11.29 (the Variant 2 preflight) for the TRT-LLM venv; that uv cannot resolve nvidia-cutlass-dsl==4.8.0.dev0 without --prerelease=allow, which then pulls mpmath 1.5.0a1	"Read this first" / Variant 5 / Variant 2 state which uv each variant uses (stock 0.13+ with --no-config for Variants 1/5/6; 0.11.29 only for uv sync)	SKILL.md 36–37 183 · skill-notes.md 141–145 · trt-llm.md 605
F2c	P1	In a full run of the fixes branch the opencode agent ran the Variant 1 install block across separate shell calls; the export and source …/activate of the first call were gone in the second, so uv pip install went into the evaluator's own venv and the import check ran against the wrong interpreter (then a needless OpenCV downgrade: the same block passed first-attempt in the other eleven diffusers cells of that run)	The three uv pip commands of the Variant 1 block and of diffusers.md § 3 carry --python "$COSMOS3_DIFFUSERS_VENV" (declared in-block addition; with the variable unset uv stops instead of installing elsewhere), the import check and the generation snippets run with "$COSMOS3_DIFFUSERS_VENV/bin/python", and the block says it is one shell call	SKILL.md 110–126 · diffusers.md 138–151 157 161 453–454
F3	P1	Diffusers venv not headless: cosmos_guardrail pulls GUI opencv; the pipeline imports it even with the safety checker off	Headless OpenCV swap + import check inside the Variant 1 venv block; apt fallback reworded (apt-get update, root caveat); skill notes in diffusers.md / payloads.md	SKILL.md 98–100 116–126 · diffusers.md 74 159 437–446 · payloads.md 81
F3b	P1	Headless OpenCV swap was a separate command after the install line; one agent ran the pipeline import first and hit libxcb.so.1	Swap folded into the uv pip install command in SKILL.md Variant 1 and, as a declared in-block deviation, in diffusers.md § 3	SKILL.md 116–126 · diffusers.md 138–151 157 159
F4	P1	Bare-host prerequisites of the wheel undocumented: OpenMPI, CUDA_HOME, CUDA-13 libs via the cu130 index, seeded venv, ffmpeg	Prerequisite lines + comments inside the Route A block (Variant 6 inherits); upstream pip-install page cited in the references	distilled-trt-llm.md 61 · trt-llm.md 29 601–606 624–635 703–706
F5	P1	TRTLLM_USE_PRECOMPILED=1 pip install -e . resolves an unpublished 1.3.0rc26+bca6761ab8 wheel; rc26 binaries lack an op the pinned Python needs	Hint deleted; "do not use the shortcut on this checkout" paragraph; one-sentence skill notes in both TRT references	distilled-trt-llm.md 61 · trt-llm.md 29 669 674–687
F6	P1	No output-verification step; ffprobe/ffmpeg/Pillow absent on the host; the NIM container's ffmpeg cannot demux its own VP9 output	New "Verify the output" section (ffprobe, root-free av, PIL; expected smoke facts) + pointers; NIM headless-check note	SKILL.md 41–43 454–467 · diffusers.md 437–446 482 · payloads.md 101
F7	P1	Server variants lack a headless start / readiness-poll / teardown recipe; SGLang /health 503 while warming undocumented; NIM docker run -d --rm destroys the logs	Shared named-container + trtllm-serve patterns (start, poll, logs, teardown) with per-variant pointers; SGLang readiness documented; NIM --rm dropped (recorded deviation); Nano/Super t2i request body stated in Variant 5	SKILL.md 230–231 274–279 319–321 372–374 401 451–452 · payloads.md 216–217 · sglang.md 21 · skill-notes.md 58–101 · trt-llm.md 43 733
F7b	P2	Servers started in a backgrounded subshell died with the shell; SIGINT to the process group did not stop a server; a second server was started while the first still held the GPU	Headless pattern: launch from a script with setsid … &, SIGINT → SIGTERM → SIGKILL escalation, leftover check by PID; never start a second server on a busy GPU (the launch script enforces it)	SKILL.md 372–377 · distilled-trt-llm.md 465–474 · skill-notes.md 75–94
F8	P1	Distilled 4-Step students are ~122 GiB bf16, not "32B / single GPU": the 1-GPU launch OOMs on 80 GB; the verified cpu_offload_config route was undocumented	Model-table note + Variant 6 offload block (measured) + warning that FP8 variants are unverified; same in distilled-trt-llm.md	SKILL.md 79–84 339–343 · distilled-trt-llm.md 22 35 84–99 101 103 366–370 431–439 443–447 · skill-notes.md 101
F8b	P1	Distilled launch still OOM-prone in practice: agents ran the notebook's plain bf16 launch first (OOM on 80–94 GB GPUs, and a second launch then OOMed against the still-loading server); on 125 GiB-RAM hosts the offload route is OOM-killed	Variant 6 is one gated launch script: GPU memory → plain route only at ≥ 128,000 MiB; host RAM (incl. cgroup ceiling) → bf16 + cpu_offload_config at ≥ 128 GiB, else the Hugging Face fp8 branch with the embed_tokens ignore-list patch (eval-verified); refuses to start on a busy GPU; one setsid trtllm-serve + /health poll with a 1800 s deadline	SKILL.md 345–348 372–377 · distilled-trt-llm.md 63 339–416
F9	P1	SGLang docker run passes no -e HF_TOKEN and omits SGLANG_DISABLE_COSMOS3_GUARDRAILS=1; the server exits with GatedRepoError 401	Guardrail-table row fixed; -e SGLANG_DISABLE_COSMOS3_GUARDRAILS=1 -e HF_TOKEN in every launch block; request body guardrails: false	SKILL.md 249–254 262–263 284 291 · payloads.md 67 · sglang.md 23 35 41–42 80 371 · skill-notes.md 42–49 53–57
F9b	P1	vLLM-Omni docker run blocks passed no -e HF_TOKEN: on a fresh node with a cold Hugging Face cache the first launch crashed with HTTP 429 during the anonymous model lookup	Superseded by the notebook's final version (#329), which passes -e HF_TOKEN in every vllm serve launch; the skill carries that text 1:1 (vllm-omni.md § 2, SKILL.md Variant 3) and vllm-omni.md § 7.1 records why the token matters — export HF_TOKEN before the launch	SKILL.md 213 · vllm-omni.md 602–608
F10	P1	NIM profile discovery/pinning (list-model-profiles, NIM_MODEL_PROFILE, /v1/manifest) and guardrail env switches missing from the launch block	profile preflight and /v1/metadata check in the Variant 7 launch; env lines in both launch blocks; /v1/manifest + selectedModelProfileId in the inspect lists	SKILL.md 391–397 407–408 417–418 423
F11	P1	NIM cache bind mount not writable by uid 1000; the client-side chmod -R 777 never reaches a daemon-created mount	Cache preflight in the Variant 7 launch: the write test as the NIM user (uid 1000) runs first; the ownership repair (docker run --user 0 … chown 1000:1000) runs only for an empty cache this run created or with NIM_CACHE_REPAIR=1, otherwise the launch stops with guidance (review follow-up, see below)	SKILL.md 395–396 409–411 · skill-notes.md 102–108
F12	P1	rc26 returns HTTP 500 on /v1/images/generations; /health 200 does not exercise the image route	rc27 pinned with the reason inside Route A; "a 200 on /health does not exercise the image route" sentence; skill note at trt-llm.md § 2	trt-llm.md 70 564 612–620 636–637 737–740
F13	P2	Model table had no measured memory figures, no small-GPU path	Measured paragraph under the table (Edge/Nano, labelled as eval measurements) + fp8 pointer; Nano YAML note in Variant 5 / trt-llm.md	skill-notes.md 26–38 · trt-llm.md 43 722–733
F14	P2	Variant 6 returns a one-frame MP4/AVI, the eval asks for a PNG	ffmpeg -frames:v 1 extraction documented in Variant 6 and distilled-trt-llm.md	SKILL.md 358–361 · distilled-trt-llm.md 267 277–286
F14b	P2	On the fp8 server the one-frame /v1/videos/generations returned HTTP 500 after a successful denoise	Documented fallback: /v1/images/generations with output_type: "image" (explicit body; needs rc27), recorded in result_summary.md	SKILL.md 361 · distilled-trt-llm.md 279 451–461
F15	P2	Wheel path only needs examples/visual_gen/configs/*.yaml, yet the block prescribed full clone + submodules + LFS	Direct SHA fetch (git fetch --depth 1 origin <sha>), LFS skipped; shallow-clone warning; Edge needs no checkout	trt-llm.md 636–642 647–652 703–706
F16	P2	pip install cosmos_guardrail==0.3.0 unconditional although guardrails are disabled; unresolvable on 1.2.1	Moved to an optional "only if you keep guardrails on" step quoting the setup guide's current pin, cosmos_guardrail==0.3.2 (its former NLTK data step is gone, #332), with the transformers>=5 caveat	trt-llm.md 669 691–699
F17	P2	SKILL.md said the NIM "takes plain-text prompts"; /v1/infer accepted the compact-JSON assets	Reworded intro + compact-JSON rule; Variant 7 request guidance; the Variant 7 request block	SKILL.md 428–429 436–439 · skill-notes.md 8–10 13–15
F18	P2	Variant 1 had no runnable Edge t2i/t2v snippet, no cwd comment, benign load warnings undocumented	Edge block composed from the README sentence and the notebook branches; cwd comment; two measured bullets	diffusers.md 465 486–517 · skill-notes.md 122–130
F2d	P1	Variant 5 Route A: uv pip install run in a later shell call than the venv activation landed the wheel set in whichever Python was first on PATH (eval: the evaluator's own venv, where llist has no wheel and the source build wanted clang)	The § 7.2 install block and its preflights name the venv explicitly: uv pip install --python "$TRTLLM_VENV/bin/python", "$TRTLLM_VENV/bin/python" -c …, "$TRTLLM_VENV/bin/trtllm-serve" (the F2c mechanism applied to the TensorRT-LLM launch; added after the validation run)	trt-llm.md 608–611 621 628 634–635

Review follow-ups (2026-10-29)

Three adjustments after the review, all in one commit (ddb28a2), no eval re-run: (1) the fp8 route of distilled-trt-llm.md § 7.2 no longer deletes anything — the overlay is a fresh mktemp -d directory, and a caller-set FP8_DST must not exist yet and must not be FP8_SRC or a path inside it; (2) the Variant 7 cache preflight tests write access as the NIM user first and repairs ownership only for an empty cache this run created or with NIM_CACHE_REPAIR=1, otherwise it fails with guidance (the host-side recursive chmod is gone too); (3) diffusers.md § 8.4 is a shared setup block plus one block per mode, and Variant 1 asks for the setup block plus only the requested mode's block.

Validation

Baseline (first commit b7ad087) — run 40660687591, dashboard group ci-pr327-b7ad087:

44/44 PASS (44/44 first-attempt, 0 FAIL, 0 no report, 0 errors; static assessment 100/100)

Baseline vs #324 (compare; the branch compare always shows the newest result per cell of each branch):

A group:ci-pr324-b50b766: 42/44 PASS, 14 first-attempt · B group:ci-pr327-b7ad087: 44/44 PASS, 44 first-attempt · improved 2, regressed 0

Case	Metric	claude	codex	cursor	opencode
diffusers-edge-t2i	Completion	✅→✅	✅→✅	✅→✅	✅→✅
	First time success	❌→✅	❌→✅	❌→✅	❌→✅
diffusers-edge-t2v	Completion	✅→✅	✅→✅	✅→✅	✅→✅
	First time success	❌→✅	❌→✅	❌→✅	❌→✅
diffusers-nano-t2i	Completion	✅→✅	✅→✅	✅→✅	✅→✅
	First time success	✅→✅	❌→✅	❌→✅	❌→✅
distilled-trtllm-t2i	Completion	✅→✅	✅→✅	✅→✅	❌→✅
	First time success	❌→✅	❌→✅	❌→✅	❌→✅
framework-edge-t2i	Completion	✅→✅	✅→✅	✅→✅	✅→✅
	First time success	❌→✅	✅→✅	❌→✅	❌→✅
framework-nano-t2i	Completion	✅→✅	✅→✅	✅→✅	✅→✅
	First time success	❌→✅	✅→✅	❌→✅	✅→✅
nim-generator-i2v	Completion	✅→✅	✅→✅	✅→✅	✅→✅
	First time success	✅→✅	✅→✅	✅→✅	❌→✅
sglang-edge-t2i	Completion	✅→✅	✅→✅	✅→✅	✅→✅
	First time success	✅→✅	❌→✅	❌→✅	✅→✅
trtllm-edge-t2i	Completion	✅→✅	✅→✅	✅→✅	❌→✅
	First time success	❌→✅	❌→✅	❌→✅	❌→✅
trtllm-nano-t2i	Completion	✅→✅	✅→✅	✅→✅	✅→✅
	First time success	✅→✅	❌→✅	❌→✅	❌→✅
vllm-omni-edge-t2i	Completion	✅→✅	✅→✅	✅→✅	✅→✅
	First time success	✅→✅	✅→✅	✅→✅	✅→✅

Completion: overall verdict before → after (✅ PASS, ❌ FAIL). First time success: passed on the first attempt, without a self-correction, before → after.

Added after the validation run: F2d (bad39b1); four relative-link path corrections in references/ (../cookbooks/… → ../../cookbooks/…); and, merged from #324, the vLLM-Omni notebook's final version (#329: $PWD mounts, -e HF_TOKEN, the FP8 "Quantized Checkpoints" section in vllm-omni.md § 2 and SKILL.md Variant 3; the FP8 pointer in payloads.md § 1) — validated locally with the four vllm-omni-edge-t2i cells on the base branch, 4/4 PASS first attempt. git diff b7ad087 55f0f0f -- .agents/skills/cosmos3-generate-media shows exactly these.

🤖 Generated with Claude Code

[github-actions]

Skill eval — b7ad087 · run #40660687591

44 / 44 cells done · 44 passed · 0 failed · 0 no report · 0 errors · 0 reused · started 2026-10-28 08:59:06 UTC · updated 2026-10-28 10:53:31 UTC

cosmos3-generate-media · dashboard overview · static 100/100

Case	claude	codex	cursor	opencode
diffusers-edge-t2i	✅ PASS · report	✅ PASS · report	✅ PASS · report	✅ PASS · report
diffusers-edge-t2v	✅ PASS · report	✅ PASS · report	✅ PASS · report	✅ PASS · report
framework-edge-t2i	✅ PASS · report	✅ PASS · report	✅ PASS · report	✅ PASS · report
vllm-omni-edge-t2i	✅ PASS · report	✅ PASS · report	✅ PASS · report	✅ PASS · report
sglang-edge-t2i	✅ PASS · report	✅ PASS · report	✅ PASS · report	✅ PASS · report
trtllm-edge-t2i	✅ PASS · report	✅ PASS · report	✅ PASS · report	✅ PASS · report
distilled-trtllm-t2i	✅ PASS · report	✅ PASS · report	✅ PASS · report	✅ PASS · report
nim-generator-i2v	✅ PASS · report	✅ PASS · report	✅ PASS · report	✅ PASS · report
diffusers-nano-t2i	✅ PASS · report	✅ PASS · report	✅ PASS · report	✅ PASS · report
framework-nano-t2i	✅ PASS · report	✅ PASS · report	✅ PASS · report	✅ PASS · report
trtllm-nano-t2i	✅ PASS · report	✅ PASS · report	✅ PASS · report	✅ PASS · report

Legend: ⏳ queued · 🟢 running · ✅ PASS · ❌ FAIL · ⚠️ PARTIAL · ❔ NO-REPORT (the agent finished without writing a report) · ⛔ ERROR (model gateway / agent / node error, see the report) · 💥 error (launcher failed before publishing) · reused (progress line) = cells this run did not execute: another run's newest result of this skill version, shown like a fresh one · (not selected) = outside the trigger's cells= with no result of this skill version yet · (not re-run: n report-less attempts) = 3 consecutive NO-REPORT/ERROR results for this skill version, treated as a harness-side failure (force or cells= re-runs it) · not run = the runner lost docker (job stopped)

✅ done — 44/44 passed (0 failed, 0 no report, 0 errors)

[lfengad]

Thanks for working through the evaluation-derived fixes. I have left two inline notes about keeping filesystem changes scoped to task-owned resources, and one small non-blocking suggestion about selecting the requested generation mode.

[christinayyw]

cosmos3-generate-media · local validation run (not published) · head 1ba2181 · static 100/100

Manual run of the four vllm-omni-edge-t2i cells after merging #324 (vLLM-Omni notebook update, F9b superseded): image cosmos-skill-eval:v20261102, the CI models (claude · codex gpt-6-astra · cursor claude-sonnet-5-5-medium · opencode), one cell per runner, 2026-10-29 20:21–20:29 UTC. Results were not uploaded to the dashboard, so there are no report links.

1/1 case · 4/4 PASS (4/4 first-attempt, 0 FAIL, 0 no report, 0 errors)

Case	claude	codex	cursor	opencode
vllm-omni-edge-t2i	✅ PASS · 2.09M · 4m49s	✅ PASS · 1.53M · 4m43s	✅ PASS · 651K · 2m57s	✅ PASS · 971K · 4m59s

Tokens (all cells): 5.24M. Legend: ✅ PASS · ❌ FAIL · ↻ = passed after self-correction (none here) · tokens = the agent's total · duration = the cell's wall time.

🤖 Generated with Claude Code