[christinayyw]

Summary

Stacked on #343: based on #343's head d94b180 (mirrored here as branch skills/qwen3-vl-robovqa-sft-pr343-base so the diff shows only the fixes). Skill-only: scripts/run.sh and SKILL.md of cosmos3-qwen3-vl-robovqa-sft; no framework files, no eval/eval.config change, so before and after are measured on the same cases, assertions and checks. Once reviewed, the two commits go onto #343's branch (maintainer push) and this PR closes.

Follow-up to the skill eval of #343 (comment, dashboard group ci-pr343-affbdac): 8/8 cells PASS, but 0/8 on the first attempt. Every first attempt failed on the same line, and the agents' remaining friction points are the LV2 suggestions of the eval reports.

Fixes

Where = file:lines at 1ad7352ea (permalinks; SKILL.md = .agents/skills/cosmos3-qwen3-vl-robovqa-sft/SKILL.md, run.sh = scripts/run.sh). Problem → fix, in priority order.

ID	Pri	Problem (eval evidence)	Fix applied	Where
F1	P0	check_host parsed the nvidia-smi banner for CUDA Version: only; R615 drivers print CUDA UMD Version: 13.4 (old label deprecated), so the version came back empty and setup stopped at stage 1 with "the driver supports CUDA ;" on a supported host. First-attempt failure of all 8 cells (LV1 in every report); each agent patched the regex and then passed.	Accept both labels; when neither parses, fail naming the banner line instead of a blank version	run.sh 86–95
F2	P1	setup is one 12-minute command; Claude Code's 10-minute shell cap killed it once in each of its two cells (re-run recovered, apex reused). SKILL.md told agents to detach only train (LV2, claude ×2).	Step 1 tells agents to run setup as one long-timeout call or detached like train, wait for its last lines, and re-run the same command if a tool stops it early (environments and apex build are kept)	SKILL.md 116–119
F3	P2	A re-run of a stage overwrote the previous attempt's log, losing the failure evidence (LV2, claude rows)	stage() keeps the previous log as <name>.log.prev; documented in "Verify the output"	run.sh 70 · SKILL.md 199
F4	P2	setup ended with "logs: … apex-build.log" but the smoke variant writes apex-build-smoke.log (LV2, claude zero-shot)	Hint and step 1 name the $TAG variant	run.sh 141 · SKILL.md 114
F5	P2	The host check apt-installs missing tools when run as root, which agents in containers saw without warning (LV2, cursor rows)	Stated in step 1	SKILL.md 109–110
F6	P2	No troubleshooting entry for the new host-check message	"Things not obvious": what the banner labels mean and what to check	SKILL.md 222–225

Left out on purpose: an evaluation-only setup path (three reports asked; a larger change, separate follow-up), and the review items unrelated to the eval (MD060 formatting, LFS clips, docs/posttrain revert, license text, reviewer comments on #343).

Validation

Baseline (#343 @ affbdac) — dashboard group ci-pr343-affbdac: 8/8 PASS, 0/8 first-attempt (every cell passed after one self-correction, F1).

This branch (1ad7352) — manual CI-equivalent run on the eval runners (the eval workflow refuses fork PRs; same orchestrator, image and runners as CI), 2026-10-30 17:36–18:14 UTC: results comment; dashboard groups ci-pr346-1ad7352 + ci-pr346-1ad7352-run2 (the two ⛔ entries dated 17:28 UTC in the first group are a double-launch artifact, not a skill failure).

8/8 PASS, 8/8 first-attempt (0 FAIL, 0 no report, 0 errors; static assessment 95/100). Mean overall score 87.8 → 97.4, correctness 80 → 100 on every cell, critical self-corrections 8 → 0, median cell time 30 → 15 minutes.

A group:ci-pr343-affbdac (#343 head affbdac): 8/8 PASS, 0 first-attempt · B this PR (1ad7352): 8/8 PASS, 8 first-attempt · dashboard compare, A vs this branch (branch scope = newest result per cell across both groups, all 8 cells; the run2 group alone lacks codex)

Case	Metric	claude	codex	cursor	opencode
smoke-rows-and-preflight	Completion	✅→✅	✅→✅	✅→✅	✅→✅
	First time success	❌→✅	❌→✅	❌→✅	❌→✅
smoke-zero-shot-score	Completion	✅→✅	✅→✅	✅→✅	✅→✅
	First time success	❌→✅	❌→✅	❌→✅	❌→✅

🤖 Generated with Claude Code

[christinayyw]

Skill eval — 1ad7352 · manual run (CI-equivalent)

8 / 8 cells done · 8 passed · 0 failed · 0 no report · 0 errors · 0 passed after self-correction · started 2026-10-30 17:36 UTC · updated 2026-10-30 18:14 UTC

cosmos3-qwen3-vl-robovqa-sft · dashboard groups ci-pr346-1ad7352 + ci-pr346-1ad7352-run2

Case	claude	codex	cursor	opencode
smoke-rows-and-preflight	✅ PASS · report	✅ PASS · report	✅ PASS · report	✅ PASS · report
smoke-zero-shot-score	✅ PASS · report	✅ PASS · report	✅ PASS · report	✅ PASS · report

Legend: 🟢 running · ✅ PASS · ❌ FAIL · ⚠️ PARTIAL · ❔ NO-REPORT (the agent finished without writing a report) · ⛔ ERROR (model gateway / agent / node error, see the report) · ↻ = passed after self-correction (the first attempt of the documented workflow failed; the agent fixed it and re-ran)

⛔ The 2 ERROR entries dated 17:28 UTC in group ci-pr346-1ad7352 are not skill failures: two orchestrators were launched for the same cells by mistake and raced on one working directory (harbor RewardFileNotFoundError / chmod instruction.md); they were stopped and the cells re-run. They are kept as history and excluded here.

Before → after

A group:ci-pr343-affbdac (#343 head affbdac): 8/8 PASS, 0 first-attempt · B this PR (1ad7352): 8/8 PASS, 8 first-attempt · dashboard compare, A vs this branch (branch scope = newest result per cell across both groups, all 8 cells; the run2 group alone lacks codex)

Case	Metric	claude	codex	cursor	opencode
smoke-rows-and-preflight	Completion	✅→✅	✅→✅	✅→✅	✅→✅
	First time success	❌→✅	❌→✅	❌→✅	❌→✅
smoke-zero-shot-score	Completion	✅→✅	✅→✅	✅→✅	✅→✅
	First time success	❌→✅	❌→✅	❌→✅	❌→✅

Run details: fork PRs cannot start the self-hosted eval workflow, so the same matrix was run by hand on the eval runners with the CI orchestrator (main) and image (cosmos-skill-eval:v20261102, evaluator 0.3.4), one backend per host, 1× H100 80GB (x86_64, driver 615.71.10); the cells are split over two dashboard groups because two sessions shared the run. Static tier: 95/100.

✅ done — 8/8 passed (0 failed, 0 no report, 0 errors; 0 after self-correction)