[christinayyw]

Includes the eval-derived fixes PR #327, merged into this branch on 2026-10-29 (e1a3ff1): the migration and the fixes that take the eval matrix from 42/44 to 44/44 first-attempt PASS go to main together. Everything is applied as a diff on top of main, not as a copy.

First cookbook→skill migration (cosmos3-generate-media), the cosmos3-skill-authoring guide, and an opt-in skill-eval CI on self-hosted H100 runners. No eval (GPU) job runs on push or merge.

What this PR delivers

• One skill for the seven Cosmos3 audiovisual serving paths — diffusers, cosmos_framework, vLLM-Omni, SGLang, TensorRT-LLM, distilled TensorRT-LLM, NIM — with the sample prompts, reference images and negative prompts under assets/.
• The cookbook notebooks migrated 1:1 into references/ (one file per backend plus payloads.md for shared content). A format change only: Jupyter plumbing removed, nothing corrected or rewritten; references/vllm-omni.md follows the vLLM-Omni notebook's last version on main (Add Cosmos3 FP8 quantization cookbooks and serving examples #329: launch mounts, HF_TOKEN, the FP8 "Quantized Checkpoints" section), since that notebook changed after the migration's pin and before its retirement here; the six notebooks and the audiovisual cookbook README are retired from .agents/skills/cookbooks/ (links in the cookbook READMEs and the docs site now point at the skill). Known gap: the video-to-video input clip (an LFS object) is not vendored.
• NIM (Variant 7) follows the Cosmos3 NIM 2.1.0 cookbook. run_with_nim.ipynb was already retired in this repository by docs(cosmos3/nim): update Cosmos3 NIM cookbook to 2.1.0 #296 in favour of .agents/skills/cookbooks/cosmos3/nim/, so the skill carries no references/nim.md copy of it: Variant 7 delegates to that cookbook and its cosmos3-nim-user skill (pointers, the README's 2.1.0 launch, the readiness and selection checks of deployment.md, and the examples/i2v.py request built from the bundled assets). Record: docs/cookbook_to_skill_migration.md.
• SKILL.md quotes each backend's smoke path 1:1 from the notebooks, the cookbook README or the setup guide, and makes the reference files required reading before the first command. Review requests from the first review round are applied (the framework example initialises COSMOS3_UV_GROUP / COSMOS3_UV_ENV as the notebook does; the live CI comment uses a sequence guard and per-cell ledger lines).
• Eval-derived fixes (from cosmos3-generate-media: eval-derived fixes (stacked on #324) #327): F1–F18 and F2d — every fatal failure and repeated self-correction of the 44-cell matrix, fixed as marked skill notes or declared in-block deviations with the notebook text kept verbatim; eval/eval.config is unchanged, so before and after are measured on the same prompts (tables below).
• 11 eval cases (3 Nano, 6 Edge, 1 distilled Super, 1 NIM); the two framework cases clone this repository's main. Known wording leftover in eval/eval.config: the NIM case says the Generator NIM serves text-to-video and image-to-video only, which was true of the retired 1.0.0 NIM; the pinned 2.1.0 NIM also serves text-to-image and video-to-video. The case exercises image-to-video, so results are unaffected; the sentence is corrected in the next eval.config revision together with a re-run (the file is frozen here so the migration head and the fixes are measured on the same prompts).
• Skill-eval CI (opt-in): .github/workflows/skill-eval.yml and the orchestrator in .github/scripts/skill_eval/ run a PR's skills on self-hosted H100 runners — one job per case × agent, one live PR comment that always shows the newest result per cell, dashboard groups per skill version, the inference key from a repository secret, a GitHub-hosted job for the orchestrator's unit tests, and a GitHub-hosted gate that admits only repository members with the write, maintain or admin role (details below).
• cosmos3-skill-authoring: how to write a skill that passes the eval on the first attempt, how the evaluator judges, how to run the eval in CI or locally, and the fix loop (a stacked fixes PR, gated by the owner) that gets every cell to first-attempt PASS (references/running-the-eval.md).

Evaluation results

Migration head b50b766 (before the fixes)

Head b50b766, 11 cases × 4 agents (claude = Claude Opus 5.5, codex = GPT-6 Astra, cursor = Claude Sonnet 5.5, opencode = Kimi K2.6), run 40639480322 on the four self-hosted H100 runners, dashboard group ci-pr324-b50b766:

42/44 PASS (14/44 first-attempt, 2 FAIL, 0 no report, 0 errors; static assessment 100/100)

Case	claude	codex	cursor	opencode
diffusers-edge-t2i	✅ 1.84M ↻	✅ 2.89M ↻	✅ 1.07M ↻	✅ 1.41M ↻
diffusers-edge-t2v	✅ 2.81M ↻	✅ 3.32M ↻	✅ 594K ↻	✅ 1.31M ↻
diffusers-nano-t2i	✅ 2.08M	✅ 3.28M ↻	✅ 529K ↻	✅ 869K ↻
distilled-trtllm-t2i	✅ 8.95M ↻	✅ 9.73M ↻	✅ 2.11M ↻	❌ 4.51M ↻
framework-edge-t2i	✅ 2.56M ↻	✅ 3.73M	✅ 465K ↻	✅ 2.76M ↻
framework-nano-t2i	✅ 2.71M ↻	✅ 4.59M	✅ 488K ↻	✅ 2.32M
nim-generator-i2v	✅ 3.46M	✅ 3.22M	✅ 390K	✅ 2.67M ↻
sglang-edge-t2i	✅ 2.93M	✅ 2.30M ↻	✅ 584K ↻	✅ 1.13M
trtllm-edge-t2i	✅ 6.12M ↻	✅ 18.32M ↻	✅ 864K ↻	❌ 5.53M ↻
trtllm-nano-t2i	✅ 5.43M	✅ 19.52M ↻	✅ 1.46M ↻	✅ 7.62M ↻
vllm-omni-edge-t2i	✅ 1.88M	✅ 1.41M	✅ 645K	✅ 2.02M

Tokens (all cells): 154.42M. ↻ = passed after self-correction.

The two FAILs are the opencode agent on the two TensorRT-LLM wheel cases — the documentation gap (no notebook or README says how to obtain a TensorRT-LLM build with Cosmos3 VisualGen) that the fixes below close; nothing the agents self-corrected was changed by the migration itself.

After the fixes (b7ad087, the #327 head at validation)

Baseline (first commit b7ad087) — run 40660687591, dashboard group ci-pr327-b7ad087:

44/44 PASS (44/44 first-attempt, 0 FAIL, 0 no report, 0 errors; static assessment 100/100)

Baseline vs #324 (compare; the branch compare always shows the newest result per cell of each branch):

A group:ci-pr324-b50b766: 42/44 PASS, 14 first-attempt · B group:ci-pr327-b7ad087: 44/44 PASS, 44 first-attempt · improved 2, regressed 0

Case	Metric	claude	codex	cursor	opencode
diffusers-edge-t2i	Completion	✅→✅	✅→✅	✅→✅	✅→✅
	First time success	❌→✅	❌→✅	❌→✅	❌→✅
diffusers-edge-t2v	Completion	✅→✅	✅→✅	✅→✅	✅→✅
	First time success	❌→✅	❌→✅	❌→✅	❌→✅
diffusers-nano-t2i	Completion	✅→✅	✅→✅	✅→✅	✅→✅
	First time success	✅→✅	❌→✅	❌→✅	❌→✅
distilled-trtllm-t2i	Completion	✅→✅	✅→✅	✅→✅	❌→✅
	First time success	❌→✅	❌→✅	❌→✅	❌→✅
framework-edge-t2i	Completion	✅→✅	✅→✅	✅→✅	✅→✅
	First time success	❌→✅	✅→✅	❌→✅	❌→✅
framework-nano-t2i	Completion	✅→✅	✅→✅	✅→✅	✅→✅
	First time success	❌→✅	✅→✅	❌→✅	✅→✅
nim-generator-i2v	Completion	✅→✅	✅→✅	✅→✅	✅→✅
	First time success	✅→✅	✅→✅	✅→✅	❌→✅
sglang-edge-t2i	Completion	✅→✅	✅→✅	✅→✅	✅→✅
	First time success	✅→✅	❌→✅	❌→✅	✅→✅
trtllm-edge-t2i	Completion	✅→✅	✅→✅	✅→✅	❌→✅
	First time success	❌→✅	❌→✅	❌→✅	❌→✅
trtllm-nano-t2i	Completion	✅→✅	✅→✅	✅→✅	✅→✅
	First time success	✅→✅	❌→✅	❌→✅	❌→✅
vllm-omni-edge-t2i	Completion	✅→✅	✅→✅	✅→✅	✅→✅
	First time success	✅→✅	✅→✅	✅→✅	✅→✅

Completion: overall verdict before → after (✅ PASS, ❌ FAIL). First time success: passed on the first attempt, without a self-correction, before → after.

Added after the validation run: F2d (bad39b1); four relative-link path corrections in references/ (../cookbooks/… → ../../cookbooks/…); and the vLLM-Omni notebook's final version (#329: $PWD mounts, -e HF_TOKEN, the FP8 "Quantized Checkpoints" section in vllm-omni.md § 2 and SKILL.md Variant 3; the FP8 pointer in payloads.md § 1) — validated locally with the four vllm-omni-edge-t2i cells, 4/4 PASS on the first attempt twice (after the port and after the merge of the fixes). git diff b7ad087 e1a3ff1 -- .agents/skills/cosmos3-generate-media shows exactly these, plus the review follow-ups below.

Eval-derived fixes (merged from #327)

Where = file:lines at e1a3ff1 (permalinks; SKILL.md = .agents/skills/cosmos3-generate-media/SKILL.md, reference names = references/<name>). Problem → fix, in priority order; F2d was added after the validation run (see Validation).

ID	Pri	Problem	Fix applied	Where
F1	P0	Variant 5 gave only the Docker devel source build (tag left <tag>) plus a dead TRTLLM_USE_PRECOMPILED hint; no wheel version anywhere, so cursor/opencode pip-installed stable 1.2.x, which lacks VisualGen	Route A (default): prebuilt wheel tensorrt_llm==1.3.0rc27 + torch==2.13.0+cu130 from pypi.nvidia.com/cu130 (uv --index-strategy unsafe-best-match), VisualGen preflight, pinned checkout for YAMLs only; Route B = source build; mirrored as skill notes in trt-llm.md / distilled-trt-llm.md	SKILL.md 296–299 · distilled-trt-llm.md 61 · trt-llm.md 29 583–589 595–639 656–658 722
F2	P1	Framework pyproject.toml pins uv ==0.11.29; uv venv/pip/sync abort with uv 0.13 and uv self update refuses	Variant 1: --no-config; Variant 2: pinned-uv preflight (side-by-side install or uvx --from 'uv==0.11.29'); one note in "Read this first"; skill notes in diffusers.md, cosmos-framework.md, payloads.md	SKILL.md 31–35 104–108 120 142–143 177–184 · cosmos-framework.md 202 409–410 · diffusers.md 136 138 157 · payloads.md 85 99
F2b	P1	An agent installed uv 0.11.29 (the Variant 2 preflight) for the TRT-LLM venv; that uv cannot resolve nvidia-cutlass-dsl==4.8.0.dev0 without --prerelease=allow, which then pulls mpmath 1.5.0a1	"Read this first" / Variant 5 / Variant 2 state which uv each variant uses (stock 0.13+ with --no-config for Variants 1/5/6; 0.11.29 only for uv sync)	SKILL.md 36–37 183 · skill-notes.md 141–145 · trt-llm.md 605
F2c	P1	In a full run of the fixes branch the opencode agent ran the Variant 1 install block across separate shell calls; the export and source …/activate of the first call were gone in the second, so uv pip install went into the evaluator's own venv and the import check ran against the wrong interpreter (then a needless OpenCV downgrade: the same block passed first-attempt in the other eleven diffusers cells of that run)	The three uv pip commands of the Variant 1 block and of diffusers.md § 3 carry --python "$COSMOS3_DIFFUSERS_VENV" (declared in-block addition; with the variable unset uv stops instead of installing elsewhere), the import check and the generation snippets run with "$COSMOS3_DIFFUSERS_VENV/bin/python", and the block says it is one shell call	SKILL.md 110–126 · diffusers.md 138–151 157 161 453–454
F3	P1	Diffusers venv not headless: cosmos_guardrail pulls GUI opencv; the pipeline imports it even with the safety checker off	Headless OpenCV swap + import check inside the Variant 1 venv block; apt fallback reworded (apt-get update, root caveat); skill notes in diffusers.md / payloads.md	SKILL.md 98–100 116–126 · diffusers.md 74 159 437–446 · payloads.md 81
F3b	P1	Headless OpenCV swap was a separate command after the install line; one agent ran the pipeline import first and hit libxcb.so.1	Swap folded into the uv pip install command in SKILL.md Variant 1 and, as a declared in-block deviation, in diffusers.md § 3	SKILL.md 116–126 · diffusers.md 138–151 157 159
F4	P1	Bare-host prerequisites of the wheel undocumented: OpenMPI, CUDA_HOME, CUDA-13 libs via the cu130 index, seeded venv, ffmpeg	Prerequisite lines + comments inside the Route A block (Variant 6 inherits); upstream pip-install page cited in the references	distilled-trt-llm.md 61 · trt-llm.md 29 601–606 624–635 703–706
F5	P1	TRTLLM_USE_PRECOMPILED=1 pip install -e . resolves an unpublished 1.3.0rc26+bca6761ab8 wheel; rc26 binaries lack an op the pinned Python needs	Hint deleted; "do not use the shortcut on this checkout" paragraph; one-sentence skill notes in both TRT references	distilled-trt-llm.md 61 · trt-llm.md 29 669 674–687
F6	P1	No output-verification step; ffprobe/ffmpeg/Pillow absent on the host; the NIM container's ffmpeg cannot demux its own VP9 output	New "Verify the output" section (ffprobe, root-free av, PIL; expected smoke facts) + pointers; NIM headless-check note	SKILL.md 41–43 454–467 · diffusers.md 437–446 482 · payloads.md 101
F7	P1	Server variants lack a headless start / readiness-poll / teardown recipe; SGLang /health 503 while warming undocumented; NIM docker run -d --rm destroys the logs	Shared named-container + trtllm-serve patterns (start, poll, logs, teardown) with per-variant pointers; SGLang readiness documented; NIM --rm dropped (recorded deviation); Nano/Super t2i request body stated in Variant 5	SKILL.md 230–231 274–279 319–321 372–374 401 451–452 · payloads.md 216–217 · sglang.md 21 · skill-notes.md 58–101 · trt-llm.md 43 733
F7b	P2	Servers started in a backgrounded subshell died with the shell; SIGINT to the process group did not stop a server; a second server was started while the first still held the GPU	Headless pattern: launch from a script with setsid … &, SIGINT → SIGTERM → SIGKILL escalation, leftover check by PID; never start a second server on a busy GPU (the launch script enforces it)	SKILL.md 372–377 · distilled-trt-llm.md 465–474 · skill-notes.md 75–94
F8	P1	Distilled 4-Step students are ~122 GiB bf16, not "32B / single GPU": the 1-GPU launch OOMs on 80 GB; the verified cpu_offload_config route was undocumented	Model-table note + Variant 6 offload block (measured) + warning that FP8 variants are unverified; same in distilled-trt-llm.md	SKILL.md 79–84 339–343 · distilled-trt-llm.md 22 35 84–99 101 103 366–370 431–439 443–447 · skill-notes.md 101
F8b	P1	Distilled launch still OOM-prone in practice: agents ran the notebook's plain bf16 launch first (OOM on 80–94 GB GPUs, and a second launch then OOMed against the still-loading server); on 125 GiB-RAM hosts the offload route is OOM-killed	Variant 6 is one gated launch script: GPU memory → plain route only at ≥ 128,000 MiB; host RAM (incl. cgroup ceiling) → bf16 + cpu_offload_config at ≥ 128 GiB, else the Hugging Face fp8 branch with the embed_tokens ignore-list patch (eval-verified); refuses to start on a busy GPU; one setsid trtllm-serve + /health poll with a 1800 s deadline	SKILL.md 345–348 372–377 · distilled-trt-llm.md 63 339–416
F9	P1	SGLang docker run passes no -e HF_TOKEN and omits SGLANG_DISABLE_COSMOS3_GUARDRAILS=1; the server exits with GatedRepoError 401	Guardrail-table row fixed; -e SGLANG_DISABLE_COSMOS3_GUARDRAILS=1 -e HF_TOKEN in every launch block; request body guardrails: false	SKILL.md 249–254 262–263 284 291 · payloads.md 67 · sglang.md 23 35 41–42 80 371 · skill-notes.md 42–49 53–57
F9b	P1	vLLM-Omni docker run blocks passed no -e HF_TOKEN: on a fresh node with a cold Hugging Face cache the first launch crashed with HTTP 429 during the anonymous model lookup	Superseded by the notebook's final version (#329), which passes -e HF_TOKEN in every vllm serve launch; the skill carries that text 1:1 (vllm-omni.md § 2, SKILL.md Variant 3) and vllm-omni.md § 7.1 records why the token matters — export HF_TOKEN before the launch	SKILL.md 213 · vllm-omni.md 602–608
F10	P1	NIM profile discovery/pinning (list-model-profiles, NIM_MODEL_PROFILE, /v1/manifest) and guardrail env switches missing from the launch block	profile preflight and /v1/metadata check in the Variant 7 launch; env lines in both launch blocks; /v1/manifest + selectedModelProfileId in the inspect lists	SKILL.md 391–397 407–408 417–418 423
F11	P1	NIM cache bind mount not writable by uid 1000; the client-side chmod -R 777 never reaches a daemon-created mount	Cache preflight in the Variant 7 launch: the write test as the NIM user (uid 1000) runs first; the ownership repair (docker run --user 0 … chown 1000:1000) runs only for an empty cache this run created or with NIM_CACHE_REPAIR=1, otherwise the launch stops with guidance (review follow-up, see below)	SKILL.md 395–396 409–411 · skill-notes.md 102–108
F12	P1	rc26 returns HTTP 500 on /v1/images/generations; /health 200 does not exercise the image route	rc27 pinned with the reason inside Route A; "a 200 on /health does not exercise the image route" sentence; skill note at trt-llm.md § 2	trt-llm.md 70 564 612–620 636–637 737–740
F13	P2	Model table had no measured memory figures, no small-GPU path	Measured paragraph under the table (Edge/Nano, labelled as eval measurements) + fp8 pointer; Nano YAML note in Variant 5 / trt-llm.md	skill-notes.md 26–38 · trt-llm.md 43 722–733
F14	P2	Variant 6 returns a one-frame MP4/AVI, the eval asks for a PNG	ffmpeg -frames:v 1 extraction documented in Variant 6 and distilled-trt-llm.md	SKILL.md 358–361 · distilled-trt-llm.md 267 277–286
F14b	P2	On the fp8 server the one-frame /v1/videos/generations returned HTTP 500 after a successful denoise	Documented fallback: /v1/images/generations with output_type: "image" (explicit body; needs rc27), recorded in result_summary.md	SKILL.md 361 · distilled-trt-llm.md 279 451–461
F15	P2	Wheel path only needs examples/visual_gen/configs/*.yaml, yet the block prescribed full clone + submodules + LFS	Direct SHA fetch (git fetch --depth 1 origin <sha>), LFS skipped; shallow-clone warning; Edge needs no checkout	trt-llm.md 636–642 647–652 703–706
F16	P2	pip install cosmos_guardrail==0.3.0 unconditional although guardrails are disabled; unresolvable on 1.2.1	Moved to an optional "only if you keep guardrails on" step quoting the setup guide's current pin, cosmos_guardrail==0.3.2 (its former NLTK data step is gone, #332), with the transformers>=5 caveat	trt-llm.md 669 691–699
F17	P2	SKILL.md said the NIM "takes plain-text prompts"; /v1/infer accepted the compact-JSON assets	Reworded intro + compact-JSON rule; Variant 7 request guidance; the Variant 7 request block	SKILL.md 428–429 436–439 · skill-notes.md 8–10 13–15
F18	P2	Variant 1 had no runnable Edge t2i/t2v snippet, no cwd comment, benign load warnings undocumented	Edge block composed from the README sentence and the notebook branches; cwd comment; two measured bullets	diffusers.md 465 486–517 · skill-notes.md 122–130
F2d	P1	Variant 5 Route A: uv pip install run in a later shell call than the venv activation landed the wheel set in whichever Python was first on PATH (eval: the evaluator's own venv, where llist has no wheel and the source build wanted clang)	The § 7.2 install block and its preflights name the venv explicitly: uv pip install --python "$TRTLLM_VENV/bin/python", "$TRTLLM_VENV/bin/python" -c …, "$TRTLLM_VENV/bin/trtllm-serve" (the F2c mechanism applied to the TensorRT-LLM launch; added after the validation run)	trt-llm.md 608–611 621 628 634–635

Review follow-ups (2026-10-29)

Three adjustments after the review, all in one commit (ddb28a2), no eval re-run: (1) the fp8 route of distilled-trt-llm.md § 7.2 no longer deletes anything — the overlay is a fresh mktemp -d directory, and a caller-set FP8_DST must not exist yet and must not be FP8_SRC or a path inside it; (2) the Variant 7 cache preflight tests write access as the NIM user first and repairs ownership only for an empty cache this run created or with NIM_CACHE_REPAIR=1, otherwise it fails with guidance (the host-side recursive chmod is gone too); (3) diffusers.md § 8.4 is a shared setup block plus one block per mode, and Variant 1 asks for the setup block plus only the requested mode's block. A later commit (55f0f0f) follows the setup guide's move to cosmos_guardrail==0.3.2 (trt-llm.md § 7.6; its former NLTK step is gone), notes in § 7.1 that the guide now pins TensorRT-LLM cc63e66 while the skill's validated routes stay at bca6761 / 1.3.0rc27, and points Variant 3 at the FP8 "Quantized Checkpoints" section.

Skill-eval CI (opt-in; every .github change lives on this branch)

• Who may start it: every trigger passes a GitHub-hosted gate job first (no checkout). It reads the role of the person who commented, applied the label or dispatched from the collaborators API and accepts write, maintain or admin only — organisation membership, a read/triage role or a PR from a fork is refused before any self-hosted runner is used (plan refuses fork heads a second time).
• Trigger: comment /run-skill-eval [args] on the PR (reads the workflow copy on main, so it works after this PR merges), apply one of the labels run-skill-eval / run-skill-eval-smoke / run-skill-eval-failed / run-skill-eval-force (works before the merge — the PR's own workflow copy runs; the label is removed again at the end), or dispatch manually: gh workflow run skill-eval.yml --ref <branch> -f pr_number=<N> -f scope=full -f args='…'. Args: force, rerun-failed, smoke-only (alias smoke), with-smoke, cells=<agent>/<case>,…, group=<id>. Never per-commit.
• What runs: every touched skill with an eval/eval.config, one job per case × agent over the idle H100 runners; the CPU canary only on request. One live PR comment per head commit that always shows the newest result per cell (re-runs of the same commit update it; a new commit gets a new comment that lists the PR's earlier runs), plus the dashboard; results are informational.
• Results are keyed by skill content: PASS / FAIL cells are reused across commits that do not change the skill; a partial rerun replaces only the cells it ran.
• Agents: Claude Code, Codex CLI, Cursor CLI and OpenCode; the cursor cells run Claude Sonnet 5.5 (cursor_model in .github/skill-eval.yml) so the four backends exercise four model families.
• Runners: self-hosted runners registered to this repository with the labels cosmos-skill-eval + h100; the gpu/h200 runners of gpu-tests.yml are not used (disjoint label sets). Deployment values (evaluator image, dashboard URL) are repository variables, the inference key a repository secret.
• Credentials and isolation: the inference key comes from the repository secret, never from the image; the checkout mounted into the agent container carries no job token.
• Tests: a GitHub-hosted job runs the orchestrator's 109 unit tests on changes under .github/scripts/skill_eval/.

Authoring-skill revision

Rewritten from the generate-media campaign: a description with trigger boundaries; structure and section order; writing commands that pass on the first attempt; how the evaluator judges; eval.config rules; running the eval and reading the dashboard; the fix loop (a stacked fixes PR, gated by the owner); submitting. Generic by design — the generate-media skill is linked as the worked example. Full runbook in references/running-the-eval.md.

🤖 Generated with Claude Code

[github-actions]

Skill eval — b50b766 · run #40639480322

44 / 44 cells done · 42 passed · 2 failed · 0 no report · 0 errors · 0 reused · started 2026-10-28 05:42:05 UTC · updated 2026-10-28 08:56:13 UTC

cosmos3-generate-media · dashboard overview · static 100/100

Case	claude	codex	cursor	opencode
diffusers-edge-t2i	✅ PASS · report	✅ PASS · report	✅ PASS · report	✅ PASS · report
diffusers-edge-t2v	✅ PASS · report	✅ PASS · report	✅ PASS · report	✅ PASS · report
framework-edge-t2i	✅ PASS · report	✅ PASS · report	✅ PASS · report	✅ PASS · report
vllm-omni-edge-t2i	✅ PASS · report	✅ PASS · report	✅ PASS · report	✅ PASS · report
sglang-edge-t2i	✅ PASS · report	✅ PASS · report	✅ PASS · report	✅ PASS · report
trtllm-edge-t2i	✅ PASS · report	✅ PASS · report	✅ PASS · report	❌ FAIL · report
distilled-trtllm-t2i	✅ PASS · report	✅ PASS · report	✅ PASS · report	❌ FAIL · report
nim-generator-i2v	✅ PASS · report	✅ PASS · report	✅ PASS · report	✅ PASS · report
diffusers-nano-t2i	✅ PASS · report	✅ PASS · report	✅ PASS · report	✅ PASS · report
framework-nano-t2i	✅ PASS · report	✅ PASS · report	✅ PASS · report	✅ PASS · report
trtllm-nano-t2i	✅ PASS · report	✅ PASS · report	✅ PASS · report	✅ PASS · report

Not evaluated

• cookbooks — eval.config not found at .agents/skills/cookbooks/eval/eval.config
• cosmos3-skill-authoring — eval.config not found at .agents/skills/cosmos3-skill-authoring/eval/eval.config

Legend: ⏳ queued · 🟢 running · ✅ PASS · ❌ FAIL · ⚠️ PARTIAL · ❔ NO-REPORT (the agent finished without writing a report) · ⛔ ERROR (model gateway / agent / node error, see the report) · 💥 error (launcher failed before publishing) · reused (progress line) = cells this run did not execute: another run's newest result of this skill version, shown like a fresh one · (not selected) = outside the trigger's cells= with no result of this skill version yet · (not re-run: n report-less attempts) = 3 consecutive NO-REPORT/ERROR results for this skill version, treated as a harness-side failure (force or cells= re-runs it) · not run = the runner lost docker (job stopped)

❌ done — 42/44 passed (2 failed, 0 no report, 0 errors)

[lfengad]

Thanks for putting this migration and the opt-in evaluation workflow together. I have left a few inline notes on preserving the existing public entry point and bounding selective runs, plus a non-blocking cache suggestion.

[lfengad]

Could we please retain a small migration README at the previous audiovisual README path, or coordinate the public shortcut update before removing it? The public Cosmos COOKBOOK.md still uses https://nvda.ws/4z3OmbC for Generation; I checked that it currently redirects to cosmos-framework/blob/main/.agents/skills/cookbooks/cosmos3/generator/audiovisual/README.md. Updating this navigation entry does not preserve that external entry point, so removing the old README would leave those users with a broken link. A short pointer to the new skill would preserve compatibility without restoring the notebooks.

[christinayyw]

Thanks for catching, we will update the redirect from cosmos cookbook once this merges.

[christinayyw]

PR to update cosmos cookbook: NVIDIA/cosmos#390

[lfengad]

Could we please stop planning when an explicitly supplied cell selection is invalid, rather than dropping it and continuing? For example, force cells=codex records a syntax error but leaves options.cells empty; selects() treats an empty list as selecting every cell, and force disables cache reuse. A typo in an intended selective rerun can therefore launch the entire matrix. Without force, the same expansion affects uncached cells. Rejecting the invalid selection before scheduling jobs would keep the default full run for an intentionally argument-free request, without unexpectedly widening an explicit request.

[christinayyw]

Done in f8041fe: an empty or malformed cells=/rerun selection now fails the plan with the syntax error posted on the PR, and no cell job is scheduled; a trigger without arguments still runs the full matrix. Test added.

[lfengad]

Non-blocking suggestion: could we consider including an evaluation-environment fingerprint, or an explicit cache epoch, in result compatibility? This match currently uses skill content/equivalent commits, with the case and backend matched separately, but the evaluator image and configured agent models can change independently in .github/skill-eval.yml or repository variables. The same skill could then reuse a PASS/FAIL from a different evaluation environment. I understand that reuse is disclosed and force is available, so I am not treating this as a blocker; an image/model fingerprint or a documented cache-invalidation step on environment changes would make the scope of cached evidence clearer.

[christinayyw]

Done in f8041fe: a published result is reused only if its recorded evaluator image and, where the config sets one, the backend model match the current config; otherwise the comment shows "♻️ Cache: n published results … not reused — environment changed (…)" and the cell runs again. Manifests without env.image (evaluator < 0.3.3) are matched on content as before. Test added, README updated.

[christinayyw]

cosmos3-generate-media · local validation run (not published) · head d701679 · static 100/100

Manual run of the four vllm-omni-edge-t2i cells after the vLLM-Omni notebook update (#329 port): image cosmos-skill-eval:v20261102, the CI models (claude · codex gpt-6-astra · cursor claude-sonnet-5-5-medium · opencode), one cell per runner, 2026-10-29 19:09–19:18 UTC. Results were not uploaded to the dashboard, so there are no report links.

1/1 case · 4/4 PASS (4/4 first-attempt, 0 FAIL, 0 no report, 0 errors)

Case	claude	codex	cursor	opencode
vllm-omni-edge-t2i	✅ PASS · 2.17M · 4m43s	✅ PASS · 1.71M · 4m07s	✅ PASS · 468K · 2m45s	✅ PASS · 1.77M · 7m18s

Tokens (all cells): 6.11M. Legend: ✅ PASS · ❌ FAIL · ↻ = passed after self-correction (none here) · tokens = the agent's total · duration = the cell's wall time.

🤖 Generated with Claude Code

[lfengad]

Checks are green and the skill content I spot-checked matches the tree: cosmos_framework.scripts.inference, the parallelism flags, the -4Step checkpoints, and the sibling skills it links to. The workflow trigger model looks right. There is no pull_request_target, a write-role gate runs before any checkout, forks are refused twice, every checkout uses persist-credentials: false, and GH_TOKEN is dropped before the agent containers start.

One broken link: .agents/skills/cookbooks/cosmos3/quantization/README.md:220 pointed at generator/audiovisual/run_with_vllm_omni.ipynb, which this PR deletes. It came in with the merge of #329. I pushed ae910f7, which points it at cosmos3-generate-media/references/vllm-omni.md#quantized-checkpoints.

Please look at these before merge:

• Internal dashboard links in public bot comments. The skill-eval bot comments on this PR carry many links to an internal *.nvidia.com dashboard. The URL itself stays out of the tree, but the workflow publishes it on a public PR. Consider posting it only to the job summary or an internal channel.
• Review threads. Three of my threads are still unresolved. If the run_matrix.py ones are done in f8041fe, please resolve them. For tools/docs/nav.yml:41, NVIDIA/cosmos#390 has to merge right after this PR, or the public COOKBOOK.md link stays broken.

Nits:

• The workflow uses actions/*@v4, while the rest of the repo is on checkout@v6, setup-python@v6 and upload-artifact@v7.
• The skill-authoring guide and skill_eval/README.md describe internal-only infrastructure: lease hosts, the evaluator runbook, ~/run-case.sh. Outside contributors can't follow those parts, so a sentence saying so would help.