From f1aee57d45bdc49aa96b8b3438e7f2f70ccd2806 Mon Sep 17 00:00:00 2001 From: sbabamohamad Date: Thu, 29 Oct 2026 10:55:02 -0700 Subject: [PATCH 01/14] skills: add cosmos3-qwen3-vl-robovqa-sft (Qwen3-VL-8B SFT on RoboVQA, one GPU, scored before and after) A worked post-training example for a model outside the Cosmos family: full-parameter SFT of Qwen/Qwen3-VL-8B-Instruct on RoboVQA (Apache-2.0) through the posttrain SFT path on the i4-native seat, on one GPU, with the same evaluator before and after training. Written to the cosmos3-skill-authoring rules. - .agents/skills/cosmos3-qwen3-vl-robovqa-sft/: SKILL.md (steps 0 to 8; every block self-contained through scripts/env.sh), scripts/ prepare_robovqa.py (split, frames, verify), scripts/eval_robovqa.py (evaluator and scorer), references/measured-run.md, six bundled RoboVQA clips (1.8 MB, Git LFS) for the smoke variant, and eval/eval.config with two smoke cases that fit one 80 GB GPU. - cosmos_framework/posttrain/configs/sft_qwen3_vl_8b_robovqa.py: the recipe (one epoch of 125 updates of 64 clips, fusedadam lr 2e-5 with a cosine schedule, max_length 10240, one consolidated model save) and its test. - docs/posttrain/guides/sft.md: a pointer to the skill. Measured on one GB300 (aarch64, on the host, framework 9a70e64): exact match 0.548 zero-shot, 0.632 and 0.620 after one epoch in two runs; training 2 h 05 to 2 h 14, torch peak 143,566 MiB reserved. The skill's split reproduces the seed-0 split of those runs byte for byte, and its scorer reproduces their recorded scores exactly. Signed-off-by: sbabamohamad --- .../cosmos3-qwen3-vl-robovqa-sft/SKILL.md | 396 ++++++++++++++++ .../assets/robovqa/LICENSE.txt | 202 +++++++++ .../assets/robovqa/json/train/sample.json | 6 + .../robovqa/videos/10015443583468041553.mp4 | 3 + .../robovqa/videos/14989971876239671643.mp4 | 3 + .../robovqa/videos/215018504990484899.mp4 | 3 + .../robovqa/videos/294863854345401650.mp4 | 3 + .../robovqa/videos/7344627454494368553.mp4 | 3 + .../robovqa/videos/8217664127105486720.mp4 | 3 + .../eval/eval.config | 50 +++ .../references/measured-run.md | 93 ++++ .../scripts/env.sh | 23 + .../scripts/eval_robovqa.py | 239 ++++++++++ .../scripts/prepare_robovqa.py | 422 ++++++++++++++++++ .../configs/sft_qwen3_vl_8b_robovqa.py | 116 +++++ .../test_config_sft_qwen3_vl_8b_robovqa.py | 64 +++ docs/posttrain/guides/sft.md | 4 +- 17 files changed, 1632 insertions(+), 1 deletion(-) create mode 100644 .agents/skills/cosmos3-qwen3-vl-robovqa-sft/SKILL.md create mode 100644 .agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/LICENSE.txt create mode 100644 .agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/json/train/sample.json create mode 100644 .agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/10015443583468041553.mp4 create mode 100644 .agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/14989971876239671643.mp4 create mode 100644 .agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/215018504990484899.mp4 create mode 100644 .agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/294863854345401650.mp4 create mode 100644 .agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/7344627454494368553.mp4 create mode 100644 .agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/8217664127105486720.mp4 create mode 100644 .agents/skills/cosmos3-qwen3-vl-robovqa-sft/eval/eval.config create mode 100644 .agents/skills/cosmos3-qwen3-vl-robovqa-sft/references/measured-run.md create mode 100755 .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh create mode 100755 .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/eval_robovqa.py create mode 100755 .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/prepare_robovqa.py create mode 100644 cosmos_framework/posttrain/configs/sft_qwen3_vl_8b_robovqa.py create mode 100644 cosmos_framework/posttrain/tests/test_config_sft_qwen3_vl_8b_robovqa.py diff --git a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/SKILL.md b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/SKILL.md new file mode 100644 index 000000000..045bc0fdd --- /dev/null +++ b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/SKILL.md @@ -0,0 +1,396 @@ +--- +name: cosmos3-qwen3-vl-robovqa-sft +description: > + Fine-tune Qwen3-VL-8B-Instruct, a vision-language model from outside the + Cosmos family, with the Cosmos Framework posttrain SFT path on robot-video + question answering (RoboVQA), on one GPU, and score it before and after with + the same evaluator. Use when the user asks to "fine-tune Qwen3-VL with Cosmos + Framework", "train a non-Cosmos model or a VLM in the framework", "run SFT on + robot videos / RoboVQA", "reproduce the Qwen3-VL RoboVQA result", or wants a + worked post-training example with a measured before and after (environments, + data preparation, preflight, training, Hugging Face export, evaluation). Not + for Cosmos3 checkpoints (use cosmos3-post-training) and not for LoRA (the SFT + path trains every parameter). +--- + +# Qwen3-VL-8B SFT on RoboVQA (one GPU) + +One job: full-parameter SFT of `Qwen/Qwen3-VL-8B-Instruct` on the RoboVQA training clips through +`cosmos_framework.posttrain.entrypoints.sft`, then the same scorer on 1,000 held-out clips before and after. +Example from the held-out set: the clip of a robot arm by a table, "put the box on the table Q: possible right +now?", reference `no`: the base model answers `yes`, the fine-tuned model `no`. + +Measured on one NVIDIA GB300 (aarch64, driver 595, CUDA 13.0), two runs at seed 0 (training is not +bit-reproducible): exact match 0.548 before, 0.632 and 0.620 after. Details: +[`references/measured-run.md`](references/measured-run.md). + +## Read this first + +1. **Path convention.** Run every block from the repository root (the directory with `pyproject.toml`). Every + block starts with `source .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh`, which sets the paths, + the uv pin and the artifact directory `$ART`; no block relies on shell state from an earlier block. +2. **Pick the variant from the GPU** (table below): the full run needs one GPU with about 150 GB free; the smoke + variant (steps 0 to 5 on the bundled clips) runs on any GPU that holds the 8B model in bfloat16. +3. **Two Python environments, never mixed**: the SFT environment (framework, Transformers 5.8.1, TorchCodec 0.11) + prepares the training files, checks and trains; the evaluation environment (plain Transformers 4.57.6, + TorchCodec 0.10) decodes the evaluation frames and scores. Each command names its interpreter. +4. **Check each step's expected output** (given under the step and in "Verify the output") before the next step. +5. **After a failure**, read "Things not obvious from the docs" and + [`references/measured-run.md`](references/measured-run.md) § Troubleshooting before changing a command. + +## When to use this skill + +- Fine-tune Qwen3-VL-8B on RoboVQA with Cosmos Framework, or reproduce the before-and-after numbers above +- A worked, measured example of the posttrain SFT path for a Hugging Face VLM on video question answering +- Score any Hugging Face Qwen3-VL directory on the RoboVQA held-out clips (step 5 / step 8) +- Not for Cosmos3 models: `../cosmos3-post-training/SKILL.md`. Not for LoRA: the SFT path does not support it + (`docs/posttrain/guides/sft.md` section 1). Environment errors: `../cosmos3-env-troubleshoot/SKILL.md` + +## Variants and hardware + +| Variant | GPU | Steps | Time | Status | +| --- | --- | --- | --- | --- | +| Full run | one GPU with at least 150,000 MiB free; torch peaks at 143,566 MiB reserved, so 80 to 141 GB cards do not fit | 0 to 8 | about 3.5 h (training 2 h 05 to 2 h 14) | verified twice on one GB300, October 2026 | +| Smoke | any GPU for step 5 (the 8B model in bfloat16, 16.3 GiB of weights); steps 3 and 4 run on the CPU | 0 to 5 on `assets/robovqa` | environments plus a few minutes | the same commands on 6 bundled clips; measured by this skill's eval only | + +Disk: about 110 GB for the full run (frames 33 GB, model, checkpoint and export 17 GB each), about 40 GB for the +smoke variant (environments and the model). The x86_64 branches below (`nvidia-npp` from PyPI) are unverified. +To score a model without training: steps 0 and 1c, the model download of step 2, the `split` line of step 3 and +step 5. + +## Bundled assets + +| Asset | Path | Used by | +| --- | --- | --- | +| Six RoboVQA clips with their source rows (Tianli/robovqa, Apache-2.0, revision `eb58194`) | `assets/robovqa/videos/`, `assets/robovqa/json/train/sample.json`, `assets/robovqa/LICENSE.txt` | smoke variant, steps 3 to 5 | + +The bundled directory has the layout of the full download, so the smoke commands differ from the full ones in +their `--src` / `--out` paths only. Six clips split 4 / 0 / 2 (train / val / test). + +## Step 0 — prerequisites + +Check the GPU, the driver's CUDA version (the wheels are CUDA 13.0), the system tools and uv: + +```bash +source .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh +nvidia-smi --query-gpu=name,memory.total,memory.free,driver_version,compute_cap --format=csv,noheader +CUDA_MAJOR=$(nvidia-smi | sed -n 's/.*CUDA Version: \([0-9]*\)\..*/\1/p' | head -1) +[ "${CUDA_MAJOR:-0}" -ge 13 ] || { echo "driver supports CUDA ${CUDA_MAJOR}; this recipe uses CUDA 13.0 wheels (a CUDA 12 route is unverified)"; exit 1; } +MISSING="" +for tool in git ffmpeg g++ curl; do command -v "$tool" >/dev/null || MISSING="$MISSING $tool"; done +if [ -n "$MISSING" ]; then + if [ "$(id -u)" = 0 ]; then apt-get update -q && apt-get install -y -q git ffmpeg g++ curl + else echo "missing:$MISSING (sudo apt-get install -y git ffmpeg g++ curl)"; exit 1; fi +fi +command -v uvx >/dev/null || curl -LsSf https://astral.sh/uv/install.sh | sh +$UV --version +ffmpeg -version | head -1 +df -h "$ART" | tail -1 +``` + +Continue when the CUDA check passes, `uv 0.11.29` prints, `ffmpeg` is 4 to 8, and the free disk covers the +variant (above). + +## Step 1 — environments + +**1a. SFT environment** — `docs/posttrain/guides/sft.md` section 2 plus `--python 3.13` (with `--project`, uv does +not read the checkout's `.python-version` and can pick a system 3.12), the aarch64 group that adds `nvidia-npp` +(TorchCodec needs `libnppicc.so.13`) and `--inexact` (an exact sync uninstalls apex, built in 1b): + +```bash +source .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh +ARCH_GROUP=""; [ "$(uname -m)" = aarch64 ] && ARCH_GROUP="--group cu130-aarch64" +UV_PROJECT_ENVIRONMENT="$SFT_VENV" $UV sync \ + --project docker/posttrain/environment \ + --frozen \ + --python 3.13 \ + --no-group transformers4 \ + --group sft \ + --group cu130-runtime \ + $ARCH_GROUP \ + --inexact +[ "$(uname -m)" = aarch64 ] || $UV pip install --python "$SFT_PY" "nvidia-npp==13.0.0.50" +LD_LIBRARY_PATH="$SFT_LIBS" "$SFT_PY" -c "import torch, transformers, torchcodec.decoders; print(torch.__version__, transformers.__version__)" +``` + +Expected: `2.11.0+cu130 5.8.1`. + +**1b. apex** — the native Qwen3-VL seat trains with apex `FusedAdam`, and the SFT preflight refuses `fusedadam` +without apex. NVIDIA training containers ship it; otherwise this builds it from source with the CUDA 13.0 compiler +and headers from PyPI (about 10 minutes on a GB300), for the GPU's own architecture, with one compile job per +4 GiB of free host memory: + +```bash +source .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh +if "$SFT_PY" -c "import torch, amp_C" 2>/dev/null; then + echo "apex already importable" +else + TC="$ART/venvs/cuda13-toolchain" + $UV venv --allow-existing --python 3.13 "$TC" + $UV pip install --python "$TC/bin/python" ninja \ + "nvidia-cuda-nvcc==13.0.88" "nvidia-cuda-cccl==13.0.85" "nvidia-cuda-crt==13.0.88" "nvidia-nvvm==13.0.88" \ + "nvidia-cuda-runtime==13.0.96" "nvidia-cuda-profiler-api==13.0.85" "nvidia-cublas==13.1.0.3" \ + "nvidia-cusparse==12.6.3.3" "nvidia-cusolver==12.0.4.66" "nvidia-curand==10.4.0.35" \ + "nvidia-cufft==12.0.0.61" "nvidia-nvjitlink==13.0.88" + export CUDA_HOME="$TC/lib/python3.13/site-packages/nvidia/cu13" + (cd "$CUDA_HOME/lib" && for f in libcudart libcublas libcublasLt libcusparse libcusolver libcurand libcufft libnvJitLink; do + ln -sfn "$(ls $f.so.* | head -1)" "$f.so"; done) + ln -sfn lib "$CUDA_HOME/lib64" + export PATH="$CUDA_HOME/bin:$TC/bin:$PATH" + [ -d "$ART/apex/.git" ] || git clone https://github.com/NVIDIA/apex.git "$ART/apex" + git -C "$ART/apex" checkout --detach 575968bc1f9127ccd61003a681472a83af4ff1a1 + JOBS=$(awk -v n="$(nproc)" '/MemAvailable/ {m = int($2 / 4194304); print (m < n ? m : n)}' /proc/meminfo) + TORCH_CUDA_ARCH_LIST="$(nvidia-smi --query-gpu=compute_cap --format=csv,noheader | head -1)" \ + APEX_CPP_EXT=1 APEX_CUDA_EXT=1 MAX_JOBS="$JOBS" \ + $UV pip install --python "$SFT_PY" --no-build-isolation "$ART/apex" +fi +LD_LIBRARY_PATH="$SFT_LIBS" "$SFT_PY" -c "import torch, transformers, torchcodec.decoders, amp_C; print('apex ok')" +``` + +Expected: `apex ok`. (The GB300 runs built with 64 jobs; the memory-based cap is a recommendation for smaller hosts.) + +**1c. Evaluation environment** — plain Transformers, the stack the scores were measured with: + +```bash +source .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh +$UV venv --allow-existing --python 3.12 "$EVAL_VENV" +$UV pip install --python "$EVAL_PY" "torch==2.10.0" "torchvision==0.25.0" "torchcodec==0.10.0" \ + --index-url https://download.pytorch.org/whl/cu130 +$UV pip install --python "$EVAL_PY" "transformers==4.57.6" "accelerate==1.15.0" "av==19.0.1" "numpy==2.2.6" \ + pillow safetensors "huggingface_hub==0.36.2" "nvidia-npp==13.0.0.50" +LD_LIBRARY_PATH="$EVAL_LIBS" "$EVAL_PY" -c "import torch, transformers, torchcodec.decoders; print(torch.__version__, transformers.__version__)" +``` + +Expected: `2.10.0+cu130 4.57.6`. + +## Step 2 — model and data + +Both pinned, downloaded with the evaluation environment's `hf` (huggingface_hub 0.36.2; newer `hf` releases read +the patterns after the first `--include` as file names and skip the videos folder). No token is needed. + +```bash +source .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh +"$EVAL_VENV/bin/hf" download Qwen/Qwen3-VL-8B-Instruct \ + --revision 0c351dd01ed87e9c1b53cbc748cba10e6187ff3b \ + --local-dir "$SFT_MODEL_PATH" +``` + +**Full run only** (3.5 GB; the smoke variant uses the bundled clips instead): + +```bash +source .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh +"$EVAL_VENV/bin/hf" download Tianli/robovqa --repo-type dataset \ + --revision eb5819490f0d9ec18a79774233d4b1c9ebd52eea \ + --include "videos/*" "json/*" "LICENSE.txt" \ + --local-dir "$ART/robovqa" +``` + +## Step 3 — training rows + +[`scripts/prepare_robovqa.py`](scripts/prepare_robovqa.py): `split` (80/10/10 by clip, one question per clip, +byte-identical clips kept in one split), `frames` (each training clip's evaluation frames, every round(fps/2)-th +frame up to 32, written losslessly so training sees exactly what the evaluator decodes; evaluation environment), +`verify` (the training stack decodes the same frames and the processor gives the same tokens and pixels; SFT +environment). Full run, about 25 minutes with 48 CPU processes: + +```bash +source .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh +"$EVAL_PY" "$SKILL_DIR/scripts/prepare_robovqa.py" split --src "$ART/robovqa" --out "$ART/data" +CUDA_VISIBLE_DEVICES= LD_LIBRARY_PATH="$EVAL_LIBS" \ + "$EVAL_PY" "$SKILL_DIR/scripts/prepare_robovqa.py" frames --out "$ART/data" +CUDA_VISIBLE_DEVICES= HF_HUB_OFFLINE=1 LD_LIBRARY_PATH="$SFT_LIBS" \ + "$SFT_PY" "$SKILL_DIR/scripts/prepare_robovqa.py" verify --out "$ART/data" --model "$SFT_MODEL_PATH" +``` + +Expected: counts `train 7998, val 1001, test 1000`; the md5 lines equal their seed-0 references +(`0b3739a5…` train, `1a9b1924…` test); `frames` ends with `"rows": 7998` and `"T_total": 119827`; `verify` ends with +`"all_ok": true, "bad": 0` and `"len_max": 4113`. + +**Smoke variant** (the bundled clips; the md5 references apply to the full download only): + +```bash +source .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh +"$EVAL_PY" "$SKILL_DIR/scripts/prepare_robovqa.py" split --src "$SKILL_DIR/assets/robovqa" --out "$ART/smoke" +CUDA_VISIBLE_DEVICES= LD_LIBRARY_PATH="$EVAL_LIBS" \ + "$EVAL_PY" "$SKILL_DIR/scripts/prepare_robovqa.py" frames --out "$ART/smoke" +CUDA_VISIBLE_DEVICES= HF_HUB_OFFLINE=1 LD_LIBRARY_PATH="$SFT_LIBS" \ + "$SFT_PY" "$SKILL_DIR/scripts/prepare_robovqa.py" verify --out "$ART/smoke" --model "$SFT_MODEL_PATH" +``` + +Expected: counts `train 4, val 0, test 2`; `frames` `"rows": 4`; `verify` `"all_ok": true, "bad": 0`. + +## Step 4 — preflight + +The two CPU checks of `docs/posttrain/guides/sft.md` section 2 with the recipe that trains, +`cosmos_framework/posttrain/configs/sft_qwen3_vl_8b_robovqa.py` (it reads `SFT_MODEL_PATH` and `SFT_DATA_ROOT`). +Full run, about 13 minutes: + +```bash +source .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh +for check in --validate-manifest --validate-rows; do + CUDA_VISIBLE_DEVICES= PYTHONPATH=. SFT_DATA_ROOT="$ART/data" IMAGINAIRE_OUTPUT_ROOT="$ART/runs/preflight" \ + LD_LIBRARY_PATH="$SFT_LIBS" \ + "$SFT_PY" -m cosmos_framework.posttrain.entrypoints.sft --config "$CONFIG" "$check" || exit 1 +done 2>&1 | tee "$ART/logs/preflight.log" +``` + +Expected gate line: +`train: rows 7998 | loss_tokens == assistant_tokens on 7998/7998 | longest 4113 <= 10240 | failed 0`; the next +line, `train: dp_world not checked`, is expected outside torchrun. + +**Smoke variant:** + +```bash +source .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh +for check in --validate-manifest --validate-rows; do + CUDA_VISIBLE_DEVICES= PYTHONPATH=. SFT_DATA_ROOT="$ART/smoke" IMAGINAIRE_OUTPUT_ROOT="$ART/runs/preflight-smoke" \ + LD_LIBRARY_PATH="$SFT_LIBS" \ + "$SFT_PY" -m cosmos_framework.posttrain.entrypoints.sft --config "$CONFIG" "$check" || exit 1 +done 2>&1 | tee "$ART/logs/preflight-smoke.log" +``` + +Expected gate line: `train: rows 4 | loss_tokens == assistant_tokens on 4/4 | ... | failed 0`. + +## Step 5 — before: score the untouched model + +[`scripts/eval_robovqa.py`](scripts/eval_robovqa.py) answers greedily (at most 64 new tokens, batches of 4, +bfloat16) and scores normalised exact match, token F1 and yes/no accuracy. Full run, about 14 minutes: + +```bash +source .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh +LD_LIBRARY_PATH="$EVAL_LIBS" "$EVAL_PY" "$SKILL_DIR/scripts/eval_robovqa.py" \ + --model "$SFT_MODEL_PATH" \ + --data "$ART/data/hf/test.jsonl" \ + --ref "$ART/data/samples_test.jsonl" \ + --out "$ART/eval/zeroshot" +``` + +Expected last line: `exact 0.548 token-F1 0.676 yes/no 0.918 (n=1000)` (deterministic). + +**Smoke variant** (2 held-out clips; a pipeline check, not a quality measure): + +```bash +source .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh +LD_LIBRARY_PATH="$EVAL_LIBS" "$EVAL_PY" "$SKILL_DIR/scripts/eval_robovqa.py" \ + --model "$SFT_MODEL_PATH" \ + --data "$ART/smoke/hf/test.jsonl" \ + --ref "$ART/smoke/samples_test.jsonl" \ + --out "$ART/eval/zeroshot-smoke" +``` + +Expected: `$ART/eval/zeroshot-smoke/preds.jsonl` with 2 rows, `$ART/eval/zeroshot-smoke/scores.json`, and a last line +`exact … token-F1 … yes/no … (n=2)`. + +## Step 6 — train (full run only) + +One GPU through torchrun, as `docs/posttrain/guides/sft.md` requires; started detached with its log kept. The block +stops before launching when the GPU has less than 150,000 MiB free: + +```bash +source .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh +FREE=$(nvidia-smi --query-gpu=memory.free --format=csv,noheader,nounits | head -1) +[ "$FREE" -ge 150000 ] || { echo "${FREE} MiB free; full SFT of the 8B model needs about 150 GB (measured peak 143,566 MiB reserved)"; exit 1; } +PYTHONPATH=. SFT_DATA_ROOT="$ART/data" IMAGINAIRE_OUTPUT_ROOT="$ART/runs" LD_LIBRARY_PATH="$SFT_LIBS" \ + setsid nohup "$SFT_PY" -m torch.distributed.run \ + --standalone \ + --nproc_per_node=1 \ + -m cosmos_framework.posttrain.entrypoints.sft \ + --config "$CONFIG" \ + > "$ART/logs/train.log" 2>&1 < /dev/null & +echo "training started; log: $ART/logs/train.log" +``` + +Wait for it (about three minutes to update 1, then one `step N/125` line about every minute; deadline 3 h): + +```bash +source .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh +for minute in $(seq 1 180); do + grep -q "SFT DONE" "$ART/logs/train.log" && break + pgrep -f "entrypoints.sft --config $CONFIG" >/dev/null || { echo "training exited"; tail -n 40 "$ART/logs/train.log"; exit 1; } + sleep 60 +done +grep -E "step +(1|125)/125 |SFT DONE" "$ART/logs/train.log" | cut -c1-120 +``` + +Expected: the logged training loss (`loss_avg`) near 1.0 at update 1 and near 0.5 at update 125 (single updates +move with each batch of 64 clips), then `SFT DONE {'start': 0, 'final_step': 125, ...}`. A second seed: append `-- rl.seed=1 job.name=seed1` +to the launch command. + +## Step 7 — export to Hugging Face format + +The framework's HF materializer, on the CPU (about 2 minutes); it writes 5 GB shards and verifies all 750 tensors: + +```bash +source .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh +CKPT=$(ls -d "$ART"/runs/posttrain_sft/qwen3_vl_8b_robovqa/seed0/*/checkpoint_artifacts/iter_* | sort | tail -1) +CUDA_VISIBLE_DEVICES= PYTHONPATH=. LD_LIBRARY_PATH="$SFT_LIBS" \ + "$SFT_PY" -m cosmos_framework.posttrain.checkpoint.tools.materialize_hf \ + --ckpt-dir "$CKPT" \ + --base "$SFT_MODEL_PATH" \ + --out "$ART/export" \ + --expect-tensors 750 \ + --min-tensors 750 +``` + +Expected: `[materialize] OK -> .../export (16.3 GiB, 750 tensors in 4 shards, 8 sidecars identical to base, ...)`. + +## Step 8 — after: score the fine-tuned model + +```bash +source .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh +LD_LIBRARY_PATH="$EVAL_LIBS" "$EVAL_PY" "$SKILL_DIR/scripts/eval_robovqa.py" \ + --model "$ART/export" \ + --data "$ART/data/hf/test.jsonl" \ + --ref "$ART/data/samples_test.jsonl" \ + --out "$ART/eval/sft" +``` + +Expected: exact match near 0.62 to 0.63 (0.632 and 0.620 in the two measured runs). A score near 0.55 means the +fine-tuned weights were not loaded. + +## Verify the output + +| Step | File | Correct when | +| --- | --- | --- | +| 3 | `$ART/data/split_manifest.json` | counts 7998 / 1001 / 1000; every `leakage_check` value is 0 | +| 3 | `$ART/data/train.jsonl.verify.jsonl` | 7,998 lines, all `sha_ok`, `ids_equal`, `grid_equal`, `pix_equal` true | +| 4 | `$ART/logs/preflight.log` | the gate line ends in `failed 0` | +| 5, 8 | `$ART/eval//scores.json` | `__all__.n` is 1000 and `__meta__.missing_predictions` is 0 | +| 6 | `$ART/logs/train.log` | 125 `step N/125` lines and `SFT DONE` | +| 7 | `$ART/export/model.safetensors.index.json` | 750 entries in `weight_map` | + +Smoke variant: the same files under `$ART/smoke`, `$ART/logs/preflight-smoke.log` and `$ART/eval/zeroshot-smoke`, +with 4 training rows and `n` = 2. + +## Reference files + +| File | What it holds | +| --- | --- | +| [`references/measured-run.md`](references/measured-run.md) | the two measured runs: versions, timings, memory, loss, example answers, troubleshooting | +| `cosmos_framework/posttrain/configs/sft_qwen3_vl_8b_robovqa.py` | the recipe, each field with its reason | +| [`scripts/prepare_robovqa.py`](scripts/prepare_robovqa.py) | split, frames, verify | +| [`scripts/eval_robovqa.py`](scripts/eval_robovqa.py) | evaluator, scorer and the shared frame decoder | +| [`scripts/env.sh`](scripts/env.sh) | paths and pins every block sources | + +## Things not obvious from the docs + +- **`libnppicc.so.13: cannot open shared object file`**: TorchCodec needs NVIDIA NPP. Every command that decodes + video sets `LD_LIBRARY_PATH` to its environment's NPP library directory (`$SFT_LIBS` or `$EVAL_LIBS`). Do not clear + `LD_LIBRARY_PATH`. +- **`ImportError: libc10.so` importing `amp_C`**: import `torch` first; apex's extensions link against torch. +- **apex gone after a second `uv sync`**: an exact sync uninstalls packages the lock does not list; keep `--inexact`. +- **SFT environment on Python 3.12**: `$SFT_LIBS` then points at a missing `python3.13` path; keep `--python 3.13`. +- **`[materialize] FATAL ... model.safetensors.index.json missing`**: a checkout whose materializer still writes one + 50 GB shard; Transformers 5 writes a model under 50 GB as one file without the index the verifier requires. The + materializer now writes 5 GB shards. +- **`hf download` fetched no videos**: a newer `hf`; use `$EVAL_VENV/bin/hf` (huggingface_hub 0.36.2). +- **Training needs one GPU with about 150 GB**: 80 GB and 141 GB cards run every step except step 6. +- **The first update takes about two minutes and prints nothing before it**: model load and warm-up, not a hang. + +## Related skills + +| Skill | When to use | +| --- | --- | +| `../cosmos3-setup/SKILL.md` | framework installation and checkpoint downloads in general | +| `../cosmos3-post-training/SKILL.md` | post-training Cosmos3 models | +| `../cosmos3-env-troubleshoot/SKILL.md` | import, CUDA, shared-library and download errors | diff --git a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/LICENSE.txt b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/LICENSE.txt new file mode 100644 index 000000000..4bbb8433f --- /dev/null +++ b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/LICENSE.txt @@ -0,0 +1,202 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright 2023 Google LLC + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/json/train/sample.json b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/json/train/sample.json new file mode 100644 index 000000000..60d2da9ea --- /dev/null +++ b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/json/train/sample.json @@ -0,0 +1,6 @@ +{"uid":"12991928966725870442","text":"\u003ctask:affordance:discriminative:discrete:False\u003e\nput the box on the table Q: possible right now? \u003cPRED\u003eA: \u003cPRED:ANSWER\u003e\u003cPRED:DISCRETE\u003e\u003cPRED:BINARY\u003eno\u003c/PRED:BINARY\u003e\u003c/PRED:DISCRETE\u003e\u003c/PRED:ANSWER\u003e\n\u003c/PRED\u003e","video":"10015443583468041553.mp4"} +{"uid":"12974193841542607726","text":"\u003ctask:remaining5_planning_with_context20:freeform\u003e\ncurrent goal is: Please place the Peas and strawberries into the bowl. last 20 steps: 1- place peas in the bowl 2- place pea in the bowl 3- place pea in the bowl 4- place pea in the bowl 5- place pea in the bowl 6- place pea in the bowl 7- place pea in the bowl 8- place peas in the bowl 9- place pea in the bowl 10- place pea in the bowl 11- place peas in the bowl 12- place pea in the bowl 13- place pea in the bowl 14- place strawberry in the bowl Q: next 5 steps? \u003cPRED\u003eA: 1- place strawberry in the bowl 2- place strawberry in the bowl 3- place strawberry in the bowl 4- place strawberry in the bowl 5- place strawberry in the bowl\n\u003c/PRED\u003e\u003ctask:affordance:generative:positive:freeform\u003e\nQ: what action is possible right now? \u003cPRED\u003eA: place strawberry in the bowl\n\u003c/PRED\u003e\u003ctask:planning:freeform\u003e\ncurrent goal is: Please place the Peas and strawberries into the bowl Q: immediate next step? \u003cPRED\u003eA: place strawberry in the bowl\n\u003c/PRED\u003e\u003ctask:immediate_planning_with_context20:freeform\u003e\ncurrent goal is: Please place the Peas and strawberries into the bowl. last 20 steps: 1- place peas in the bowl 2- place pea in the bowl 3- place pea in the bowl 4- place pea in the bowl 5- place pea in the bowl 6- place pea in the bowl 7- place pea in the bowl 8- place peas in the bowl 9- place pea in the bowl 10- place pea in the bowl 11- place peas in the bowl 12- place pea in the bowl 13- place pea in the bowl 14- place strawberry in the bowl Q: immediate next step? \u003cPRED\u003eA: place strawberry in the bowl\n\u003c/PRED\u003e\u003ctask:future_prediction:freeform\u003e\nQ: what is likely to happen next? \u003cPRED\u003eA: place strawberry in the bowl\n\u003c/PRED\u003e\u003ctask:affordance:discriminative:discrete:True\u003e\nplace strawberry in the bowl Q: possible right now? \u003cPRED\u003eA: \u003cPRED:ANSWER\u003e\u003cPRED:DISCRETE\u003e\u003cPRED:BINARY\u003eyes\u003c/PRED:BINARY\u003e\u003c/PRED:DISCRETE\u003e\u003c/PRED:ANSWER\u003e\n\u003c/PRED\u003e","video":"215018504990484899.mp4"} +{"uid":"17235415102297408195","text":"\u003ctask:immediate_planning_with_context20:freeform\u003e\ncurrent goal is: please use the brush to sweep everything off the desks. last 20 steps: 1- pick the brush from the table 2- sweep the table 3- move towards the popcorn 4- move all the objects to left 5- sweep the blue chip packet 6- move your arm upwards 7- drop sweep the table 8- sweep the candy's on the floor Q: immediate next step? \u003cPRED\u003eA: done\n\u003c/PRED\u003e\u003ctask:planning:freeform\u003e\ncurrent goal is: please use the brush to sweep everything off the desks. Q: immediate next step? \u003cPRED\u003eA: done\n\u003c/PRED\u003e\u003ctask:success:discrete:True\u003e\nsweep the candy's on the floor Q: satisfied? \u003cPRED\u003eA: \u003cPRED:ANSWER\u003e\u003cPRED:DISCRETE\u003e\u003cPRED:BINARY\u003eyes\u003c/PRED:BINARY\u003e\u003c/PRED:DISCRETE\u003e\u003c/PRED:ANSWER\u003e\n\u003c/PRED\u003e\u003ctask:remaining5_planning_with_context20:freeform\u003e\ncurrent goal is: please use the brush to sweep everything off the desks. last 20 steps: 1- pick the brush from the table 2- sweep the table 3- move towards the popcorn 4- move all the objects to left 5- sweep the blue chip packet 6- move your arm upwards 7- drop sweep the table 8- sweep the candy's on the floor Q: next 5 steps? \u003cPRED\u003eA: 1- done\n\u003c/PRED\u003e\u003ctask:past_description:freeform\u003e\nQ: what just happened? \u003cPRED\u003eA: sweep the candy's on the floor\n\u003c/PRED\u003e","video":"294863854345401650.mp4"} +{"uid":"5447814328030553394","text":"\u003ctask:past_description:freeform\u003e\nQ: what just happened? \u003cPRED\u003eA: Put the orange pen in the penholder\n\u003c/PRED\u003e\u003ctask:success:discrete:True\u003e\nPut the orange pen in the penholder Q: satisfied? \u003cPRED\u003eA: \u003cPRED:ANSWER\u003e\u003cPRED:DISCRETE\u003e\u003cPRED:BINARY\u003eyes\u003c/PRED:BINARY\u003e\u003c/PRED:DISCRETE\u003e\u003c/PRED:ANSWER\u003e\n\u003c/PRED\u003e","video":"8217664127105486720.mp4"} +{"uid":"8365152957462210433","text":"\u003ctask:affordance:generative:positive:freeform\u003e\nQ: what action is possible right now? \u003cPRED\u003eA: drop the brown sachet in the plastic glass\n\u003c/PRED\u003e\u003ctask:future_prediction:freeform\u003e\nQ: what is likely to happen next? \u003cPRED\u003eA: drop the brown sachet in the plastic glass\n\u003c/PRED\u003e\u003ctask:affordance:discriminative:discrete:True\u003e\ndrop the brown sachet in the plastic glass Q: possible right now? \u003cPRED\u003eA: \u003cPRED:ANSWER\u003e\u003cPRED:DISCRETE\u003e\u003cPRED:BINARY\u003eyes\u003c/PRED:BINARY\u003e\u003c/PRED:DISCRETE\u003e\u003c/PRED:ANSWER\u003e\n\u003c/PRED\u003e\u003ctask:remaining5_planning_with_context20:freeform\u003e\ncurrent goal is: please place all the coffee condiments on the table back into the cup. last 20 steps: none Q: next 5 steps? \u003cPRED\u003eA: 1- drop the brown sachet in the plastic glass 2- pick a white packet from the table 3- drop the white packet in the plastic glass 4- pick the sachets from the table 5- drop the green packet in the plastic glass\n\u003c/PRED\u003e\u003ctask:planning:freeform\u003e\ncurrent goal is: please place all the coffee condiments on the table back into the cup. Q: immediate next step? \u003cPRED\u003eA: drop the brown sachet in the plastic glass\n\u003c/PRED\u003e\u003ctask:success:discrete:False\u003e\ndrop the brown sachet in the plastic glass Q: satisfied? \u003cPRED\u003eA: \u003cPRED:ANSWER\u003e\u003cPRED:DISCRETE\u003e\u003cPRED:BINARY\u003eno\u003c/PRED:BINARY\u003e\u003c/PRED:DISCRETE\u003e\u003c/PRED:ANSWER\u003e\n\u003c/PRED\u003e\u003ctask:immediate_planning_with_context20:freeform\u003e\ncurrent goal is: please place all the coffee condiments on the table back into the cup. last 20 steps: none Q: immediate next step? \u003cPRED\u003eA: drop the brown sachet in the plastic glass\n\u003c/PRED\u003e","video":"14989971876239671643.mp4"} +{"uid":"7845415266276637754","text":"\u003ctask:success:discrete:True\u003e\npick up green packet Q: satisfied? \u003cPRED\u003eA: \u003cPRED:ANSWER\u003e\u003cPRED:DISCRETE\u003e\u003cPRED:BINARY\u003eyes\u003c/PRED:BINARY\u003e\u003c/PRED:DISCRETE\u003e\u003c/PRED:ANSWER\u003e\n\u003c/PRED\u003e\u003ctask:success:discrete:True\u003e\npick up the white and green packet Q: satisfied? \u003cPRED\u003eA: \u003cPRED:ANSWER\u003e\u003cPRED:DISCRETE\u003e\u003cPRED:BINARY\u003eyes\u003c/PRED:BINARY\u003e\u003c/PRED:DISCRETE\u003e\u003c/PRED:ANSWER\u003e\n\u003c/PRED\u003e\u003ctask:past_description:freeform\u003e\nQ: what just happened? \u003cPRED\u003eA: pick up the white and green packet\n\u003c/PRED\u003e\u003ctask:past_description:freeform\u003e\nQ: what just happened? \u003cPRED\u003eA: pick up green packet\n\u003c/PRED\u003e","video":"7344627454494368553.mp4"} diff --git a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/10015443583468041553.mp4 b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/10015443583468041553.mp4 new file mode 100644 index 000000000..6cc850345 --- /dev/null +++ b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/10015443583468041553.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:3ca3182bc1d9a2a2d5fa3841391f25146ea76b1cbab5b03b798da74c881aa7af +size 577530 diff --git a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/14989971876239671643.mp4 b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/14989971876239671643.mp4 new file mode 100644 index 000000000..d5a5874c2 --- /dev/null +++ b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/14989971876239671643.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:642b7563e0873440f41c948e3d03c6a4f17dad9018c8b65940bfa65023a35b4d +size 145116 diff --git a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/215018504990484899.mp4 b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/215018504990484899.mp4 new file mode 100644 index 000000000..0d3f75f60 --- /dev/null +++ b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/215018504990484899.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:8fc70190471ea3aa8f23cbed37dfa7d86ce2ce56057a2527bf691b4b625f6056 +size 439805 diff --git a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/294863854345401650.mp4 b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/294863854345401650.mp4 new file mode 100644 index 000000000..e55dfe59d --- /dev/null +++ b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/294863854345401650.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:e37f54760a02dba33fd72689d2a15a29cdd61e5017fc25619b9a83050ea3b5c2 +size 191368 diff --git a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/7344627454494368553.mp4 b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/7344627454494368553.mp4 new file mode 100644 index 000000000..5b266c3f9 --- /dev/null +++ b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/7344627454494368553.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:ad81c70c5b243bc81c86e95909da1f70a55f5bb57d8b0fb7f750f6a521dfefab +size 205041 diff --git a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/8217664127105486720.mp4 b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/8217664127105486720.mp4 new file mode 100644 index 000000000..9850a3c39 --- /dev/null +++ b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/8217664127105486720.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:03d91fe4c54f2d01341c37797dcf4b65c4efbe43b1121175de4c9a1d831294db +size 205723 diff --git a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/eval/eval.config b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/eval/eval.config new file mode 100644 index 000000000..97294fc1c --- /dev/null +++ b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/eval/eval.config @@ -0,0 +1,50 @@ +{ + "evals": [ + { + "id": "smoke-rows-and-preflight", + "prompt": "On this machine's single GPU, prepare the Qwen3-VL-8B RoboVQA SFT training rows from the skill's bundled RoboVQA clips (assets/robovqa) and run the framework's SFT preflight (--validate-manifest and --validate-rows) on them with the skill's recipe. Save the preflight output as outputs/preflight.log and the frame check summary printed by the verify step as outputs/verify.json.", + "expected_outcome": "The bundled clips are split 4/0/2, the 4 training files pass the frame check, and the SFT preflight's train gate line in outputs/preflight.log reports rows 4 and failed 0.", + "assertions": [ + "The SFT environment was built with the skill's uv sync command (uv 0.11.29, Python 3.13, the sft and cu130-runtime groups) and apex imports in it", + "prepare_robovqa.py split, frames and verify ran on assets/robovqa, and outputs/verify.json reports rows 4 with all_ok true", + "outputs/preflight.log contains a 'train: rows 4' gate line ending in 'failed 0'", + "The skill's documented workflow executed without errors on the first attempt", + "result_summary.md reports the split counts, the gate line and the wall time of each step" + ], + "checks": [ + {"path": "outputs/preflight.log", "min_bytes": 100}, + {"path": "outputs/verify.json", "min_bytes": 40}, + {"path": "result_summary.md", "min_bytes": 150}, + {"path": "transcript.md", "min_bytes": 300} + ], + "stages": ["prerequisites", "environments", "model download", "training rows", "preflight", "verify + summarize"] + }, + { + "id": "smoke-zero-shot-score", + "prompt": "Score Qwen/Qwen3-VL-8B-Instruct (revision 0c351dd01ed87e9c1b53cbc748cba10e6187ff3b) zero-shot with the skill's RoboVQA evaluator on the held-out clips of the skill's bundled RoboVQA sample (assets/robovqa), and copy the evaluator's preds.jsonl and scores.json to outputs/.", + "expected_outcome": "outputs/preds.jsonl holds 2 greedy answers and outputs/scores.json the exact match, token F1 and yes/no accuracy over the 2 held-out clips, produced by eval_robovqa.py in the evaluation environment.", + "assertions": [ + "The evaluation environment was built as the skill documents (Python 3.12, torch 2.10.0+cu130, transformers 4.57.6, torchcodec 0.10.0) and the evaluator ran from it", + "The bundled clips were split with prepare_robovqa.py and the evaluator read the resulting hf/test.jsonl and samples_test.jsonl", + "outputs/preds.jsonl has 2 rows and outputs/scores.json reports n = 2 with no missing predictions", + "The skill's documented workflow executed without errors on the first attempt", + "result_summary.md reports the scores, the wall time and the GPU memory observed" + ], + "checks": [ + {"path": "outputs/preds.jsonl", "min_bytes": 50}, + {"path": "outputs/scores.json", "min_bytes": 50}, + {"path": "result_summary.md", "min_bytes": 150}, + {"path": "transcript.md", "min_bytes": 300} + ], + "stages": ["prerequisites", "evaluation environment", "model download", "split", "evaluation", "verify + summarize"] + } + ], + "credentials": ["HF_TOKEN"], + "gpus": 1, + "budget": {"minutes": 75, "tokens": 3000000}, + "setup": { + "repo": "https://github.com/NVIDIA/cosmos-framework.git", + "dest": "/app/repo", + "src_mount": "/opt/cosmos-src" + } +} diff --git a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/references/measured-run.md b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/references/measured-run.md new file mode 100644 index 000000000..4f333964c --- /dev/null +++ b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/references/measured-run.md @@ -0,0 +1,93 @@ +# Measured runs: Qwen3-VL-8B SFT on RoboVQA + +Two runs of this recipe at seed 0, each on one NVIDIA GB300 (256,703 MiB, aarch64, driver 595.91.07, CUDA 13.0), +directly on the host without a container, in October 2026, with the framework at commit +`9a70e64d2ad7ff1417484d12d10fb121b270baaf`. The commands were the ones in `SKILL.md` steps 1 to 8, with the training +config, data preparation and evaluator unchanged; the runs drove them from one shell script, and wrote the HF export +through the materializer with 5 GB shards (the change that ships with this skill). + +## Versions + +| Environment | Python | Key packages | +| --- | --- | --- | +| SFT (`docker/posttrain/environment`, groups `sft`, `cu130-runtime`, `cu130-aarch64`) | 3.13.15 | torch 2.11.0+cu130, transformers 5.8.1, torchcodec 0.11, apex `575968bc` built from source | +| Evaluation | 3.12 | torch 2.10.0+cu130, torchvision 0.25.0, torchcodec 0.10.0, transformers 4.57.6, nvidia-npp 13.0.0.50 | + +Pins: `Qwen/Qwen3-VL-8B-Instruct` at `0c351dd01ed87e9c1b53cbc748cba10e6187ff3b`; `Tianli/robovqa` at +`eb5819490f0d9ec18a79774233d4b1c9ebd52eea`. + +## Results on the 1,000 held-out clips + +| | exact match | token-F1 | yes/no accuracy (597 questions) | +| --- | --- | --- | --- | +| Qwen3-VL-8B-Instruct, zero-shot | 0.548 | 0.676 | 0.918 | +| run 1, after one epoch | 0.632 | 0.809 | 0.953 | +| run 2, after one epoch | 0.620 | 0.798 | 0.946 | + +- The zero-shot answers are deterministic: both runs (two independent setups) gave the same 1,000 answers. +- Training is not bit-reproducible: gradient norms differ from update 2 on, and the two fine-tuned models give the + same answer on 763 of the 1,000 clips. A Hugging Face + TRL full SFT of the same data and epoch on 8 H100s gave + 0.633 ± 0.007 over three seeds (outside this skill; a reference point only). +- Part of the gain is the model learning RoboVQA's answer style (short, lower-case, imperative), which exact match + rewards; yes/no accuracy, which reads only the first yes or no, rose from 0.918 to 0.946 and 0.953. + +## Time and memory + +| Step | Run 1 | Run 2 | Notes | +| --- | --- | --- | --- | +| 1 + model download | 704 s | 684 s | the apex build takes about 10 minutes of it | +| 2 + 3 (dataset download and training rows) | 1,367 s | 1,424 s | 48 processes; 7,998 frame files, 35,369,466,594 bytes, 119,827 frames | +| 4 preflight | 753 s | not run | CPU | +| 5 before | 802 s | 856 s | the evaluator alone 670 s and 761 s | +| 6 train | 7,512 s | 8,064 s | median 60.4 s and 63.1 s per update (updates 2 to 125); update 1 took 107 s and 134 s | +| 7 export | 121 s | 119 s | CPU; 16.3 GiB, 750 tensors in 4 shards | +| 8 after | 605 s | 657 s | the evaluator alone 526 s and 557 s | + +- Training memory: torch peak 138,981.7 MiB allocated and 143,566 to 143,716 MiB reserved; nvidia-smi peaked + 144,357 MiB above the idle baseline (run 1). +- GPU time: about 2.5 GPU-hours (steps 5, 6 and 8). +- Disk: about 110 GB in all; frames 33 GB, model 17 GB, checkpoint 17 GB, export 17 GB. + +## Training loss (run 2) + +| update | 1 | 2 | 3 | 4 | 5 | ... | 120 | 121 | 122 | 123 | 124 | 125 | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| `train/loss_avg` | 1.041 | 0.630 | 0.815 | 0.602 | 0.616 | ... | 0.698 | 0.695 | 0.543 | 0.544 | 0.499 | 0.482 | + +Run 1 had 1.041 at update 1 and 0.487 at update 125; the mean of updates 1 to 10 was 0.717 and of 116 to 125 0.591. + +## Example answers (run 1) + +The first held-out clip of each task family, fixed before any fine-tuned answer existed. ✓ / ✗ is the scorer's +normalised exact match. + +| task | question | reference | zero-shot | fine-tuned | +| --- | --- | --- | --- | --- | +| success | Put the orange pen in the penholder Q: satisfied? | yes | ✓ yes | ✓ yes | +| affordance | put the box on the table Q: possible right now? | no | ✗ yes | ✓ no | +| planning | current goal is: Please place the Peas and strawberries into the bowl Q: immediate next step? | place strawberry in the bowl | ✗ Put the strawberry in the paper bowl | ✗ put the strawberry in the bowl | +| past_description | Q: what just happened? | pick up the white and green packet | ✗ The robot arm picked up a green packet from the shelf. | ✗ place the packet in the shelf | +| future_prediction | Q: what is likely to happen next? | drop the brown sachet in the plastic glass | ✗ Place the brown sugar packet in the glass | ✗ put the brown packet in the glass | + +## Troubleshooting + +Each entry: symptom, cause, check, fix. + +- **`libnppicc.so.13: cannot open shared object file`** when TorchCodec loads. Cause: NVIDIA NPP is not on the + library path. Check: `ls "$SFT_LIBS" "$EVAL_LIBS" | grep nppicc`. Fix: keep the `nvidia-npp` installs of step 1 and + the `LD_LIBRARY_PATH=...` prefix of every decoding command; never clear `LD_LIBRARY_PATH`. +- **`ImportError: libc10.so` importing `amp_C`.** Cause: apex's extensions link against torch's libraries. Check: + `"$SFT_PY" -c "import torch, amp_C"`. Fix: import `torch` first. +- **apex missing after re-running step 1a.** Cause: an exact `uv sync` uninstalls packages the lock does not list. + Check: `"$SFT_PY" -c "import torch, amp_C"`. Fix: keep `--inexact`; re-run 1b. +- **The SFT environment was built on Python 3.12.** Cause: with `--project`, uv does not read the checkout's + `.python-version` and takes any interpreter in range. Check: `"$SFT_PY" --version`. Fix: delete + `docker/posttrain/environment/.venv-sft` and re-run 1a with `--python 3.13`. +- **`[materialize] FATAL ... model.safetensors.index.json missing`.** Cause: Transformers 5.8.1 writes a model under + 50 GB as one file without an index, and the materializer's verifier requires the index. Check: + `ls "$ART/export.tmp" "$ART/export" 2>/dev/null`. Fix: a materializer that writes 5 GB shards (this skill's + change), then re-run step 7. +- **The training stage prints nothing for about three minutes.** Cause: model load and the first update's warm-up. + Check: `nvidia-smi` shows the process growing toward 140 GB. Fix: none; wait for `step 1/125`. +- **`--validate-rows` prints `train: dp_world not checked`.** Expected outside torchrun; one GPU divides every row + count. diff --git a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh new file mode 100755 index 000000000..73a843ce7 --- /dev/null +++ b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 +# Paths and tool pins shared by every SKILL.md block. Source it from the repository root at the top of each block: +# source .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh +# It only sets variables (and creates $ART); it installs nothing. +if [ ! -f pyproject.toml ] || [ ! -d cosmos_framework ]; then + echo "env.sh: run from the cosmos-framework repository root (the directory with pyproject.toml)" >&2 + return 1 2>/dev/null || exit 1 +fi +export PATH="$HOME/.local/bin:$PATH" # where the uv installer puts uv / uvx +export SKILL_DIR="$PWD/.agents/skills/cosmos3-qwen3-vl-robovqa-sft" +export UV="uvx --from uv==0.11.29 uv" # the checkout pins uv 0.11.29 (tool.uv.required-version) +export ART="$PWD/artifacts/qwen3-vl-robovqa" # every artifact of this skill (about 110 GB for the full run) +export SFT_VENV="$PWD/docker/posttrain/environment/.venv-sft" +export SFT_PY="$SFT_VENV/bin/python" +export SFT_LIBS="$SFT_VENV/lib/python3.13/site-packages/nvidia/cu13/lib" # libnppicc.so.13 for TorchCodec 0.11 +export EVAL_VENV="$ART/venvs/eval" +export EVAL_PY="$EVAL_VENV/bin/python" +export EVAL_LIBS="$EVAL_VENV/lib/python3.12/site-packages/nvidia/cu13/lib" # libnppicc.so.13 for TorchCodec 0.10 +export SFT_MODEL_PATH="$ART/models/Qwen3-VL-8B-Instruct" # read by the recipe config +export CONFIG="cosmos_framework/posttrain/configs/sft_qwen3_vl_8b_robovqa.py" +mkdir -p "$ART/logs" diff --git a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/eval_robovqa.py b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/eval_robovqa.py new file mode 100755 index 000000000..eebc7bbb6 --- /dev/null +++ b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/eval_robovqa.py @@ -0,0 +1,239 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 +r"""Answer and score the RoboVQA held-out clips with a Hugging Face Qwen3-VL checkpoint. + +Run it on the untouched model (the "before" score) and on the SFT export (the "after" score) with the same +arguments; only ``--model`` and ``--out`` change. It runs on plain Transformers in its own environment (Transformers +4.57.6, TorchCodec 0.10, one GPU; the skill's ``SKILL.md`` step 1), the stack the recipe's numbers were measured with. + +- Frames: :func:`decode_video` keeps every round(fps / 2)-th frame from frame 0, at most 32 (evenly thinned when there + are more), and returns the effective frame rate. ``prepare_robovqa.py frames`` writes the training clips with this + same function, so training and evaluation see identical frames. +- Generation: greedy, at most ``--max-new-tokens`` (64) new tokens, batches of ``--batch`` (4), bfloat16, SDPA. +- Metrics, overall and per task, deterministic (no judge): ``exact`` = normalised exact match (lower case, punctuation + to spaces, articles dropped, spaces collapsed); ``f1`` = token F1 of the normalised strings; ``binary_acc`` = on the + yes/no tasks (tags containing ``:discrete:``) the first ``yes`` or ``no`` in the prediction against the reference, + none counting as wrong. + +Writes ``preds.jsonl``, ``scores.json`` and ``eval_meta.json`` under ``--out`` and ends with the overall line, for +example ``exact 0.548 token-F1 0.676 yes/no 0.918 (n=1000)``. ``--score-only`` re-scores an existing ``preds.jsonl``. + + python .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/eval_robovqa.py --model $MODEL \ + --data $DATA/hf/test.jsonl --ref $DATA/samples_test.jsonl --out $OUT +""" + +from __future__ import annotations + +import argparse +import collections +import json +import os +import re +import string +import sys +import time +from collections.abc import Sequence +from typing import Any + +MAX_FRAMES = 32 +TARGET_FPS = 2.0 +_ARTICLES = {"a", "an", "the"} + + +def decode_video(path: str) -> tuple[list[Any], float]: + """The clip's frames as PIL images and their effective frame rate: every round(fps / 2)-th frame from frame 0, at + most ``MAX_FRAMES`` (evenly thinned when there are more). ``prepare_robovqa.py frames`` writes the training clips + with this function, so the frames the model trains on are the frames it is scored on.""" + import numpy as np + from PIL import Image + from torchcodec.decoders import VideoDecoder + + decoder = VideoDecoder(path) + n = decoder.metadata.num_frames or 0 + fps = float(decoder.metadata.average_fps or 0.0) or 30.0 + stride = max(1, int(round(fps / TARGET_FPS))) + indices = list(range(0, n, stride)) + if len(indices) > MAX_FRAMES: + step = len(indices) / MAX_FRAMES + indices = [indices[int(i * step)] for i in range(MAX_FRAMES)] + frames = decoder.get_frames_at(indices=indices).data.permute(0, 2, 3, 1).contiguous().numpy().astype(np.uint8) + return [Image.fromarray(frame) for frame in frames], len(indices) / n * fps + + +def normalize(text: str) -> str: + """Lower case, punctuation to spaces, articles dropped, whitespace collapsed.""" + text = "".join(" " if ch in string.punctuation else ch for ch in text.lower()) + return " ".join(word for word in text.split() if word not in _ARTICLES) + + +def token_f1(prediction: str, reference: str) -> float: + """Token-level F1 between the normalised prediction and reference (SQuAD style).""" + pred_tokens, ref_tokens = normalize(prediction).split(), normalize(reference).split() + if not pred_tokens or not ref_tokens: + return float(pred_tokens == ref_tokens) + common = sum((collections.Counter(pred_tokens) & collections.Counter(ref_tokens)).values()) + if common == 0: + return 0.0 + precision, recall = common / len(pred_tokens), common / len(ref_tokens) + return 2 * precision * recall / (precision + recall) + + +def first_yes_no(text: str) -> str | None: + """The first standalone ``yes`` or ``no`` in the text, lower-cased; None when there is neither.""" + match = re.search(r"\b(yes|no)\b", text.lower()) + return match.group(1) if match else None + + +def score(refs: Sequence[dict[str, Any]], preds: dict[str, str]) -> dict[str, dict[str, float]]: + """Per task tag and overall (``__all__``): ``n``, then each metric's mean and its count (``_n``). A clip + without a prediction scores as an empty answer.""" + per_task: dict[str, list[dict[str, float]]] = collections.defaultdict(list) + for ref in refs: + prediction, task = preds.get(ref["id"], ""), ref["task"] + row = { + "exact": float(normalize(prediction) == normalize(ref["answer"])), + "f1": token_f1(prediction, ref["answer"]), + } + if ":discrete:" in task: + row["binary_acc"] = float(first_yes_no(prediction) == ref["answer"].lower()) + per_task[task].append(row) + per_task["__all__"].append(row) + out: dict[str, dict[str, float]] = {} + for task, rows in per_task.items(): + agg: dict[str, float] = {"n": len(rows)} + for key in ("exact", "f1", "binary_acc"): + values = [row[key] for row in rows if key in row] + if values: + agg[key] = sum(values) / len(values) + agg[key + "_n"] = len(values) + out[task] = agg + return out + + +def write_scores(ref_path: str, pred_path: str, out_path: str) -> dict[str, Any]: + """Score ``preds.jsonl`` against ``samples_test.jsonl`` and write ``scores.json``.""" + with open(ref_path) as handle: + refs = [json.loads(line) for line in handle if line.strip()] + preds: dict[str, str] = {} + with open(pred_path) as handle: + for line in handle: + if line.strip(): + row = json.loads(line) + preds[row["id"]] = row.get("prediction", "") + missing = [ref["id"] for ref in refs if ref["id"] not in preds] + result: dict[str, Any] = score(refs, preds) + result["__meta__"] = {"n_ref": len(refs), "n_pred": len(preds), "missing_predictions": len(missing)} + with open(out_path, "w", newline="\n") as handle: + json.dump(result, handle, indent=2) + print(json.dumps({"__all__": result["__all__"], "__meta__": result["__meta__"]})) + return result + + +def summary_line(result: dict[str, Any]) -> str: + overall = result["__all__"] + return "exact %.3f token-F1 %.3f yes/no %.3f (n=%d)" % ( + overall["exact"], + overall["f1"], + overall["binary_acc"], + overall["n"], + ) + + +def generate(model_path: str, data_path: str, out_dir: str, *, batch: int, max_new_tokens: int, limit: int) -> int: + """Greedy answers for every test row into ``/preds.jsonl``; returns the number of rows answered.""" + import torch + from transformers import AutoProcessor, Qwen3VLForConditionalGeneration + from transformers.video_utils import VideoMetadata + + processor = AutoProcessor.from_pretrained(model_path) + processor.tokenizer.padding_side = "left" + model = Qwen3VLForConditionalGeneration.from_pretrained( + model_path, dtype=torch.bfloat16, attn_implementation="sdpa" + ) + model = model.cuda().eval() + with open(data_path) as handle: + rows = [json.loads(line) for line in handle if line.strip()] + if limit: + rows = rows[:limit] + start = time.time() + with open(os.path.join(out_dir, "preds.jsonl"), "w", newline="\n") as out: + for i in range(0, len(rows), batch): + chunk = rows[i : i + batch] + texts, videos, metadata = [], [], [] + for row in chunk: + frames, fps = decode_video(row["video_path"]) + user = [message for message in row["messages"] if message["role"] == "user"][0] + content = [ + {"type": "video"} if item["type"] == "video" else {"type": "text", "text": item["text"]} + for item in user["content"] + ] + texts.append( + processor.apply_chat_template( + [{"role": "user", "content": content}], tokenize=False, add_generation_prompt=True + ) + ) + videos.append(frames) + metadata.append( + VideoMetadata( + total_num_frames=len(frames), + fps=fps, + duration=len(frames) / fps, + frames_indices=list(range(len(frames))), + video_backend="torchcodec", + ) + ) + inputs = processor( + text=texts, + videos=videos, + videos_kwargs={"do_sample_frames": False, "video_metadata": metadata}, + return_tensors="pt", + padding=True, + ).to("cuda") + with torch.inference_mode(): + generated = model.generate(**inputs, max_new_tokens=max_new_tokens, do_sample=False) + answers = processor.batch_decode(generated[:, inputs["input_ids"].shape[1] :], skip_special_tokens=True) + for row, answer in zip(chunk, answers): + reference = row["messages"][-1]["content"][0]["text"] + out.write( + json.dumps( + {"id": row["id"], "prediction": answer.strip(), "reference": reference, "task": row["task"]} + ) + + "\n" + ) + out.flush() + if (i // batch) % 20 == 0: + print(f"{i + len(chunk)}/{len(rows)} {time.time() - start:.0f}s", flush=True) + return len(rows) + + +def main(argv: Sequence[str] | None = None) -> int: + parser = argparse.ArgumentParser(description="Answer and score the RoboVQA held-out clips (see the module doc).") + parser.add_argument("--model", help="Hugging Face Qwen3-VL directory to evaluate (not needed with --score-only)") + parser.add_argument("--data", help="hf/test.jsonl written by prepare_robovqa.py split") + parser.add_argument("--ref", required=True, help="samples_test.jsonl written by prepare_robovqa.py split") + parser.add_argument("--out", required=True, help="output directory: preds.jsonl, scores.json, eval_meta.json") + parser.add_argument("--batch", type=int, default=4, help="clips per generate call (default 4)") + parser.add_argument("--max-new-tokens", type=int, default=64, help="answer length cap (default 64)") + parser.add_argument("--limit", type=int, default=0, help="score the first N clips only (0 = all)") + parser.add_argument("--score-only", action="store_true", help="re-score /preds.jsonl without generating") + args = parser.parse_args(argv) + os.makedirs(args.out, exist_ok=True) + if not args.score_only: + if not args.model or not args.data: + parser.error("--model and --data are required unless --score-only is given") + start = time.time() + n = generate( + args.model, args.data, args.out, batch=args.batch, max_new_tokens=args.max_new_tokens, limit=args.limit + ) + elapsed = time.time() - start + with open(os.path.join(args.out, "eval_meta.json"), "w", newline="\n") as handle: + json.dump({"eval_seconds": elapsed, "n": n, "model": args.model}, handle) + print(f"EVAL_DONE {n} samples in {elapsed:.0f}s") + result = write_scores(args.ref, os.path.join(args.out, "preds.jsonl"), os.path.join(args.out, "scores.json")) + print(summary_line(result)) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/prepare_robovqa.py b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/prepare_robovqa.py new file mode 100755 index 000000000..ca42c497a --- /dev/null +++ b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/prepare_robovqa.py @@ -0,0 +1,422 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 +r"""Write the RoboVQA split and the SFT training rows of ``cosmos_framework/posttrain/configs/sft_qwen3_vl_8b_robovqa.py``. + +Source: ``Tianli/robovqa`` (Apache-2.0) at revision ``eb5819490f0d9ec18a79774233d4b1c9ebd52eea``, downloaded with +``videos/*``, ``json/*`` and ``LICENSE.txt``. Only rows whose mp4 is present are usable (9,999 clips). Each row's text +holds one to nine task segments ``\n Q: ? A: \n``; discrete answers +are wrapped in ``yes|no``. + +Three steps, each in the environment named (the skill's ``SKILL.md`` has the commands): + +1. ``split`` (any Python 3.10+, CPU): shuffle the sorted video ids with ``random.Random(seed)`` and cut 80/10/10 into + train / val / test; move every member of a group of byte-identical clips to the split of the group's smallest id; + keep one question per clip (``random.Random(seed + int(md5(video id)[:8], 16))`` picks the segment). Writes + ``splits/``, ``samples_{train,val,test}.jsonl`` (``{id, video, task, question, answer, split, uid}``), + ``hf/{train,val,test}.jsonl`` (the source ``video_path`` and the conversation) and ``split_manifest.json`` + (counts, task histogram, moved duplicates, leakage check), and prints the md5 of the train and test samples. +2. ``frames`` (evaluation environment, CPU): for every training clip, decode the frames the evaluator decodes + (``eval_robovqa.decode_video``) and write them losslessly (PNG in MOV, rgb24) at the effective frame rate under + ``frames/``; then ``train.jsonl``, the SFT rows the recipe reads, each pointing at its file, and + ``train.jsonl.frames.jsonl`` (per clip: frame count, rate, size, sha256 of the raw frames). +3. ``verify`` (SFT environment, CPU): for every training file, check that the training stack decodes exactly the + recorded frames at the recorded frame rate, and that the Qwen3-VL processor called the way the SFT packer calls it + (the path, ``do_sample_frames=False``) gives the same ``input_ids``, video grid and pixel values as the evaluator's + call (the frames plus their ``VideoMetadata``). Writes ``train.jsonl.verify.jsonl``; exits 1 unless every row passes. + +Run it as a file (it imports ``eval_robovqa`` from its own directory): + + python .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/prepare_robovqa.py split --src $SRC --out $DATA +""" + +from __future__ import annotations + +import argparse +import collections +import glob +import hashlib +import json +import os +import random +import re +import sys +from collections.abc import Sequence +from fractions import Fraction +from pathlib import Path +from typing import Any + +SPLITS = ("train", "val", "test") +_SEGMENT = re.compile(r"]+)>\n(.*?)A: (.*?)\n", re.S) +_BINARY = re.compile(r"(yes|no)") +# md5 of the samples files that `split --seed 0` writes from the pinned revision (the recipe's measured split). +SEED0_MD5 = { + "samples_train.jsonl": "0b3739a50f5b715a6f4a5ab11081807a", + "samples_test.jsonl": "1a9b1924eddcf34cf26ce7ffbcb69184", +} + + +def _default_workers() -> int: + return min(os.cpu_count() or 1, 48) + + +def _md5_file(path: str | Path) -> str: + with open(path, "rb") as handle: + return hashlib.md5(handle.read()).hexdigest() + + +def parse_segments(text: str) -> list[dict[str, str]]: + """The task segments of one RoboVQA row: ``{task, question, answer}``, a binary answer reduced to yes/no.""" + out = [] + for tag, question, answer in _SEGMENT.findall(text): + binary = _BINARY.search(answer) + out.append({"task": tag, "question": question.strip(), "answer": binary.group(1) if binary else answer.strip()}) + return out + + +# ------------------------------------------------------------------------------------------------ split -------------- +def split(src: Path, out: Path, *, seed: int, fractions: Sequence[float]) -> dict[str, Any]: + """Write the split, the samples, the conversation rows and the manifest; return the manifest.""" + out.mkdir(parents=True, exist_ok=True) + present = {path.name for path in (src / "videos").glob("*.mp4")} + rows = [] + for json_path in sorted(glob.glob(str(src / "json" / "*" / "*.json"))): + with open(json_path) as handle: + for line in handle: + if line.strip(): + row = json.loads(line) + if row["video"] in present: + rows.append(row) + by_video: dict[str, list[dict[str, Any]]] = {} + for row in rows: + by_video.setdefault(row["video"], []).append(row) + if not all(len(group) == 1 for group in by_video.values()): + raise ValueError("expected one source row per video") + ids = sorted(by_video) + random.Random(seed).shuffle(ids) + n = len(ids) + n_train, n_val = int(n * fractions[0]), int(n * fractions[1]) + splits = {"train": ids[:n_train], "val": ids[n_train : n_train + n_val], "test": ids[n_train + n_val :]} + # Co-locate byte-identical clips: every member of an md5 group goes to the split of its lexicographically smallest + # id (deterministic), so identical videos never straddle splits. + md5_of = {video: _md5_file(src / "videos" / video) for video in ids} + groups: dict[str, list[str]] = collections.defaultdict(list) + for video in ids: + groups[md5_of[video]].append(video) + where = {video: name for name, videos in splits.items() for video in videos} + moved = [] + for group in groups.values(): + if len(group) > 1: + target = where[min(group)] + for video in group: + if where[video] != target: + moved.append((video, where[video], target)) + where[video] = target + splits = {name: [video for video in ids if where[video] == name] for name in splits} + + (out / "splits").mkdir(exist_ok=True) + samples: dict[str, list[dict[str, Any]]] = {} + histogram: collections.Counter[tuple[str, str]] = collections.Counter() + unparsed = 0 + for name, videos in splits.items(): + (out / "splits" / f"{name}_videos.txt").write_text("\n".join(videos) + "\n", newline="\n") + samples[name] = [] + for video in videos: + segments = parse_segments(by_video[video][0]["text"]) + if not segments: + unparsed += 1 + continue + pick = random.Random(seed + int(hashlib.md5(video.encode()).hexdigest()[:8], 16)).choice(segments) + samples[name].append( + { + "id": f"{name}_{video[:-4]}", + "video": video, + "task": pick["task"], + "question": pick["question"], + "answer": pick["answer"], + "split": name, + "uid": by_video[video][0]["uid"], + } + ) + histogram[(name, pick["task"])] += 1 + with open(out / f"samples_{name}.jsonl", "w", newline="\n") as handle: + for sample in samples[name]: + handle.write(json.dumps(sample) + "\n") + + # Leakage check: video ids, video bytes and source uids are disjoint across splits. + split_of_md5: dict[str, set[str]] = {} + for name, videos in splits.items(): + for video in videos: + split_of_md5.setdefault(md5_of[video], set()).add(name) + train, val, test = (set(splits[name]) for name in SPLITS) + uids = {name: {sample["uid"] for sample in samples[name]} for name in SPLITS} + + (out / "hf").mkdir(exist_ok=True) + for name in SPLITS: + with open(out / "hf" / f"{name}.jsonl", "w", newline="\n") as handle: + for sample in samples[name]: + conversation = [ + { + "role": "user", + "content": [ + {"type": "video", "video": "video_0"}, + {"type": "text", "text": sample["question"]}, + ], + }, + {"role": "assistant", "content": [{"type": "text", "text": sample["answer"]}]}, + ] + video_path = str((src / "videos" / sample["video"]).resolve()) + row = {"id": sample["id"], "video_path": video_path, "task": sample["task"], "messages": conversation} + handle.write(json.dumps(row) + "\n") + + manifest = { + "source": "Tianli/robovqa (Apache-2.0 LICENSE.txt)", + "seed": seed, + "fractions": list(fractions), + "rule": "shuffle sorted video ids with random.Random(seed); 80/10/10; one Q/A per video", + "usable_videos": n, + "unparsed_rows": unparsed, + "counts": {name: len(samples[name]) for name in SPLITS}, + "task_hist": {f"{name}|{task}": count for (name, task), count in sorted(histogram.items())}, + "duplicates_moved": moved, + "leakage_check": { + "video_id_overlap": len(train & val | train & test | val & test), + "identical_video_bytes_across_splits": sum(1 for names in split_of_md5.values() if len(names) > 1), + "uid_overlap_train_vs_val_test": len(uids["train"] & uids["test"]) + len(uids["train"] & uids["val"]), + }, + } + with open(out / "split_manifest.json", "w", newline="\n") as handle: + json.dump(manifest, handle, indent=2) + return manifest + + +# ------------------------------------------------------------------------------------------------ frames ------------- +def _write_clip_frames(job: tuple[dict[str, Any], str]) -> dict[str, Any]: + """Decode one clip the evaluator's way and write the frames losslessly at the effective frame rate.""" + import av + from eval_robovqa import decode_video # this script's directory: the evaluator's own decoder + + row, frames_dir = job + frames, fps_eff = decode_video(row["video_path"]) + rate = Fraction(fps_eff).limit_denominator(10**6) + clip = os.path.splitext(os.path.basename(row["video_path"]))[0] + path = os.path.join(frames_dir, f"{clip}.mov") + tmp = path + ".tmp.mov" + tick = Fraction(rate.denominator, rate.numerator) # one codec tick per frame + with av.open(tmp, "w", format="mov") as container: + stream = container.add_stream("png", rate=rate) + stream.width, stream.height = frames[0].size + stream.pix_fmt = "rgb24" + stream.codec_context.time_base = tick + for index, frame in enumerate(frames): + video_frame = av.VideoFrame.from_image(frame) + video_frame.pts = index + video_frame.time_base = tick + for packet in stream.encode(video_frame): + container.mux(packet) + for packet in stream.encode(): + container.mux(packet) + os.replace(tmp, path) + digest = hashlib.sha256() + for frame in frames: + digest.update(frame.tobytes()) + return { + "id": row["id"], + "src": row["video_path"], + "mov": path, + "T": len(frames), + "fps_eff": fps_eff, + "rate": f"{rate.numerator}/{rate.denominator}", + "w": frames[0].size[0], + "h": frames[0].size[1], + "bytes": os.path.getsize(path), + "frames_sha256": digest.hexdigest(), + } + + +def write_frames(out: Path, *, workers: int, limit: int) -> dict[str, Any]: + """Write ``frames/``, ``train.jsonl`` and ``train.jsonl.frames.jsonl`` from ``hf/train.jsonl``.""" + from multiprocessing import Pool + + source = out / "hf" / "train.jsonl" + frames_dir = out / "frames" + frames_dir.mkdir(exist_ok=True) + with open(source) as handle: + rows = [json.loads(line) for line in handle if line.strip()] + if limit: + rows = rows[:limit] + with Pool(workers) as pool: + meta = pool.map(_write_clip_frames, [(row, str(frames_dir)) for row in rows], chunksize=8) + by_id = {clip["id"]: clip for clip in meta} + rows_path = out / "train.jsonl" + with open(rows_path, "w", newline="\n") as handle: + for row in rows: + messages = [] + for message in row["messages"]: + parts = [ + {"type": "video", "video": by_id[row["id"]]["mov"]} + if item["type"] == "video" + else {"type": "text", "text": item["text"]} + for item in message["content"] + ] + messages.append({"role": message["role"], "content": parts}) + handle.write(json.dumps({"messages": messages, "meta": {"item": row["id"]}}, ensure_ascii=False) + "\n") + with open(f"{rows_path}.frames.jsonl", "w", newline="\n") as handle: + for clip in meta: + handle.write(json.dumps(clip) + "\n") + return { + "rows": len(rows), + "src_md5": _md5_file(source), + "rows_md5": _md5_file(rows_path), + "bytes_total": sum(clip["bytes"] for clip in meta), + "T_total": sum(clip["T"] for clip in meta), + } + + +# ------------------------------------------------------------------------------------------------ verify ------------- +_PROCESSOR: Any = None + + +def _verify_clip(job: tuple[dict[str, Any], dict[str, Any], str]) -> dict[str, Any]: + """One training file in the training stack: same frames, same rate, same processor output as the evaluator's.""" + import torch + from PIL import Image + from torchcodec.decoders import VideoDecoder + from transformers import AutoProcessor + from transformers.video_utils import VideoMetadata + + global _PROCESSOR + row, clip, model_path = job + if _PROCESSOR is None: + _PROCESSOR = AutoProcessor.from_pretrained(model_path) + decoder = VideoDecoder(clip["mov"]) + file_meta = decoder.metadata + data = ( + decoder.get_frames_at(indices=list(range(file_meta.num_frames))).data.permute(0, 2, 3, 1).contiguous().numpy() + ) + digest = hashlib.sha256() + for index in range(data.shape[0]): + digest.update(data[index].tobytes()) + frames = [Image.fromarray(data[index]) for index in range(data.shape[0])] + messages = [ + { + "role": message["role"], + "content": [ + {"type": "video"} if item["type"] == "video" else {"type": "text", "text": item["text"]} + for item in message["content"] + ], + } + for message in row["messages"] + ] + text = _PROCESSOR.apply_chat_template(messages, tokenize=False, add_generation_prompt=False) + count, fps = clip["T"], clip["fps_eff"] + evaluator_metadata = VideoMetadata( + total_num_frames=count, + fps=fps, + duration=count / fps, + frames_indices=list(range(count)), + video_backend="torchcodec", + ) + as_evaluated = _PROCESSOR( + text=[text], + videos=[frames], + return_tensors="pt", + videos_kwargs={"do_sample_frames": False, "video_metadata": [evaluator_metadata]}, + ) + as_trained = _PROCESSOR( + text=[text], videos=[clip["mov"]], return_tensors="pt", videos_kwargs={"do_sample_frames": False} + ) + return { + "id": clip["id"], + "T": count, + "n_file": file_meta.num_frames, + "sha_ok": digest.hexdigest() == clip["frames_sha256"], + "fps_rel_err": abs(float(file_meta.average_fps) - fps) / fps, + "ids_equal": torch.equal(as_evaluated["input_ids"], as_trained["input_ids"]), + "grid_equal": torch.equal(as_evaluated["video_grid_thw"], as_trained["video_grid_thw"]), + "pix_equal": torch.equal(as_evaluated["pixel_values_videos"], as_trained["pixel_values_videos"]), + "len": int(as_trained["input_ids"].shape[1]), + } + + +def verify_frames(out: Path, *, model_path: str, workers: int) -> dict[str, Any]: + """Check every training file; write ``train.jsonl.verify.jsonl``; return the summary.""" + from multiprocessing import Pool + + rows_path = out / "train.jsonl" + with open(rows_path) as handle: + rows = [json.loads(line) for line in handle if line.strip()] + with open(f"{rows_path}.frames.jsonl") as handle: + clips = {clip["id"]: clip for clip in (json.loads(line) for line in handle if line.strip())} + with Pool(workers) as pool: + results = pool.map(_verify_clip, [(row, clips[row["meta"]["item"]], model_path) for row in rows], chunksize=4) + bad = [ + result + for result in results + if not ( + result["sha_ok"] + and result["ids_equal"] + and result["grid_equal"] + and result["pix_equal"] + and result["T"] == result["n_file"] + and result["fps_rel_err"] < 1e-6 + ) + ] + with open(f"{rows_path}.verify.jsonl", "w", newline="\n") as handle: + for result in results: + handle.write(json.dumps(result) + "\n") + return { + "rows": len(results), + "all_ok": not bad, + "bad": len(bad), + "first_bad": bad[:3], + "max_fps_rel_err": max(result["fps_rel_err"] for result in results), + "len_max": max(result["len"] for result in results), + "len_sum": sum(result["len"] for result in results), + } + + +# ------------------------------------------------------------------------------------------------ CLI ---------------- +def main(argv: Sequence[str] | None = None) -> int: + parser = argparse.ArgumentParser( + description="RoboVQA split and SFT rows for sft_qwen3_vl_8b_robovqa (see the module doc)." + ) + steps = parser.add_subparsers(dest="step", required=True) + split_args = steps.add_parser("split", help="split RoboVQA and write samples, conversation rows and the manifest") + split_args.add_argument("--src", required=True, help="local Tianli/robovqa download (json/, videos/)") + split_args.add_argument("--out", required=True, help="data root (the recipe's SFT_DATA_ROOT)") + split_args.add_argument("--seed", type=int, default=0) + split_args.add_argument("--frac", default="0.8,0.1,0.1", help="train,val,test fractions") + frames_args = steps.add_parser("frames", help="write the training clips' evaluation frames and train.jsonl") + frames_args.add_argument("--out", required=True, help="data root written by split") + frames_args.add_argument("--workers", type=int, default=_default_workers()) + frames_args.add_argument("--limit", type=int, default=0, help="first N training clips only (0 = all)") + verify_args = steps.add_parser("verify", help="check the training files in the SFT environment") + verify_args.add_argument("--out", required=True, help="data root written by split and frames") + verify_args.add_argument( + "--model", default=os.environ.get("SFT_MODEL_PATH"), help="Qwen3-VL snapshot (default $SFT_MODEL_PATH)" + ) + verify_args.add_argument("--workers", type=int, default=_default_workers()) + args = parser.parse_args(argv) + out = Path(args.out).resolve() + + if args.step == "split": + fractions = [float(value) for value in args.frac.split(",")] + manifest = split(Path(args.src), out, seed=args.seed, fractions=fractions) + print(json.dumps({"counts": manifest["counts"], "leakage_check": manifest["leakage_check"]})) + for name, expected in SEED0_MD5.items(): + note = f" (seed-0 reference {expected})" if args.seed == 0 else "" + print(f"{_md5_file(out / name)} {name}{note}") + return 0 + if args.step == "frames": + print(json.dumps(write_frames(out, workers=args.workers, limit=args.limit))) + return 0 + if not args.model: + parser.error("verify needs --model or SFT_MODEL_PATH") + summary = verify_frames(out, model_path=args.model, workers=args.workers) + print(json.dumps(summary)) + return 0 if summary["all_ok"] else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/cosmos_framework/posttrain/configs/sft_qwen3_vl_8b_robovqa.py b/cosmos_framework/posttrain/configs/sft_qwen3_vl_8b_robovqa.py new file mode 100644 index 000000000..e65495a7e --- /dev/null +++ b/cosmos_framework/posttrain/configs/sft_qwen3_vl_8b_robovqa.py @@ -0,0 +1,116 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 +r"""Qwen3-VL-8B-Instruct full-parameter SFT on RoboVQA, on one GPU, through the i4-native policy builder. + +A worked example with a measured before and after: one epoch over the RoboVQA training clips written by the +``prepare_robovqa.py`` script of the ``cosmos3-qwen3-vl-robovqa-sft`` skill, scored by the skill's ``eval_robovqa.py`` on +the 1,000 held-out clips before and after training. Commands, expected output and timings: +``.agents/skills/cosmos3-qwen3-vl-robovqa-sft/SKILL.md``. + +Measured on one NVIDIA GB300 at seed 0, two independent runs (training is not bit-reproducible): + +| | exact match | token-F1 | yes/no accuracy | +|---|---|---|---| +| Qwen3-VL-8B-Instruct, zero-shot | 0.548 | 0.676 | 0.918 | +| after this recipe (run 1 / run 2) | 0.632 / 0.620 | 0.809 / 0.798 | 0.953 / 0.946 | + +Training stack, each field with its reason: +- ``model_backend='i4'`` with ``build_qwen3_vl_native_policy``: the seat the SFT guide gives Qwen3-VL + (``docs/posttrain/guides/sft.md`` section 1), with the precision stack of ``sft_qwen3_vl_moe_native.py``: bfloat16 + storage and compute, fp32 master weights and moments inside ``OptimizerConfig(name='fusedadam')`` (apex), SDPA, + full activation checkpointing. +- ``micro_batch_size=1``, ``grad_accum=64``, ``max_steps=125``: one epoch of the 7,998 training clips at 64 clips per + update. +- ``lr=2e-5``, betas (0.9, 0.999), no weight decay, ``grad_clip_norm=1.0``, no warm-up, cosine to ~0 at + ``max_steps``; every parameter trains (``train_keys`` empty). +- ``max_length=10240``: the longest training row renders 4,113 tokens with these frames, above the 4,096 default; + the packer refuses an over-long row instead of truncating it. +- ``checkpoint_format='consolidated'``, ``checkpoint_components=['model']``, ``save_every=max_steps``: one model-only + save after the last update, the input of the HF materializer. +- ``videos_kwargs.do_sample_frames=False``: every training row points at a lossless file that holds exactly the frames + the evaluator decodes (the skill's ``prepare_robovqa.py frames``, checked by its ``verify``), so the processor takes + every frame at the file's frame rate. +- Validation is off: the held-out measure is the evaluator's, run before and after training. + +Set ``SFT_MODEL_PATH`` to a complete local snapshot of ``Qwen/Qwen3-VL-8B-Instruct`` (tested at revision +``0c351dd01ed87e9c1b53cbc748cba10e6187ff3b``) and ``SFT_DATA_ROOT`` to the ``--out`` directory of +``prepare_robovqa.py``. Torchrun chooses the DP world; one GPU with about 150 GB is enough (torch peak 143,566 MiB +reserved on a GB300). Override after ``--``, for example ``-- rl.seed=1 job.name=seed1``; a shorter run that should +still save sets both ``rl.max_steps`` and ``rl.save_every``. +""" + +from __future__ import annotations + +import os + +from cosmos_framework.posttrain.config.config_sft import PosttrainSFTConfig +from cosmos_framework.posttrain.config.config_sft import make_config as make_root_config +from cosmos_framework.posttrain.config.model import ( + NativePolicyBuildSpec, + NativePolicyParallelismConfig, + Qwen3VLModelConfig, +) +from cosmos_framework.posttrain.config.rl import OptimizerConfig +from cosmos_framework.posttrain.config.sft import SFTDatasetConfig, SFTLRScheduleConfig, SFTValidationConfig + +NATIVE_POLICY_BUILDER = "cosmos_framework.posttrain.model.qwen3_vl.build_qwen3_vl_native_policy" +# One epoch of the 7,998 training clips of prepare_robovqa.py's seed-0 split at 64 clips per update. +EPOCH_STEPS = 125 + + +def build_config() -> PosttrainSFTConfig: + config = make_root_config() + config.job.project = "posttrain_sft" + config.job.group = "qwen3_vl_8b_robovqa" + config.job.name = "seed0" + config.job.wandb_mode = "disabled" + config.model_backend = "i4" + config.model = NativePolicyBuildSpec( + builder=NATIVE_POLICY_BUILDER, + model_id=os.environ.get("SFT_MODEL_PATH", "Qwen/Qwen3-VL-8B-Instruct"), + model_config=Qwen3VLModelConfig( + # NativePolicyBuildSpec.model_id is the single artifact identity; the builder injects it. + name_or_path="", + trust_remote_code=False, + checkpoint_path="", + credential_path="", + dtype="bfloat16", + attn_impl="sdpa", + activation_checkpointing="full", + fsdp_reduce_dtype="float32", + ), + parallelism=NativePolicyParallelismConfig(dp_replicate=1, dp_shard=-1), + requires_collated_policy_forward=True, + ) + config.rl.seed = 0 + config.rl.master_dtype = "bfloat16" + config.rl.param_dtype = "bfloat16" + config.rl.reduce_dtype = "float32" + config.rl.mixed_precision_mode = "fsdp" + config.rl.max_steps = EPOCH_STEPS + config.rl.micro_batch_size = 1 + config.rl.grad_accum = 64 + config.rl.max_length = 10240 + config.rl.checkpoint_format = "consolidated" + config.rl.checkpoint_components = ["model"] + config.rl.save_every = EPOCH_STEPS + config.rl.optimizer = OptimizerConfig( + name="fusedadam", + lr=2e-5, + betas=[0.9, 0.999], + eps=1e-8, + weight_decay=0.0, + grad_clip_norm=1.0, + warmup_steps=0, + ) + config.rl.lr_schedule = SFTLRScheduleConfig(kind="cosine", f_start=0.01, f_min=1e-9, cycle_steps=0) + data_root = os.environ.get("SFT_DATA_ROOT", "artifacts/qwen3-vl-robovqa/data") + config.rl.data = SFTDatasetConfig(path=os.path.join(data_root, "train.jsonl")) + config.rl.processor_kwargs = {"videos_kwargs": {"do_sample_frames": False}} + config.rl.validation = SFTValidationConfig(enabled=False) + return config + + +def make_config() -> PosttrainSFTConfig: + """The cosmos_framework.utils.config.load_config entry point.""" + return build_config() diff --git a/cosmos_framework/posttrain/tests/test_config_sft_qwen3_vl_8b_robovqa.py b/cosmos_framework/posttrain/tests/test_config_sft_qwen3_vl_8b_robovqa.py new file mode 100644 index 000000000..dd5b0c7cb --- /dev/null +++ b/cosmos_framework/posttrain/tests/test_config_sft_qwen3_vl_8b_robovqa.py @@ -0,0 +1,64 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 +"""The Qwen3-VL-8B RoboVQA SFT recipe: one epoch of 64-clip updates on the native seat, accepted by the SFT preflight.""" + +from __future__ import annotations + +import importlib.util +import json +import types +from pathlib import Path +from typing import Any + +import pytest + +pytestmark = [pytest.mark.L0, pytest.mark.CPU] + + +@pytest.fixture +def apex_present(monkeypatch: pytest.MonkeyPatch) -> None: + """Make ``importlib.util.find_spec("apex")`` succeed so fusedadam acceptance does not depend on the test runtime.""" + real = importlib.util.find_spec + + def find_spec(name: str, *args: Any, **kwargs: Any) -> Any: + return types.SimpleNamespace(name="apex") if name == "apex" else real(name, *args, **kwargs) + + monkeypatch.setattr(importlib.util, "find_spec", find_spec) + + +@pytest.fixture +def recipe(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Any: + from cosmos_framework.posttrain.configs.sft_qwen3_vl_8b_robovqa import make_config + + model_dir = tmp_path / "model" + model_dir.mkdir() + (model_dir / "config.json").write_text(json.dumps({"model_type": "qwen3_vl"})) + data_root = tmp_path / "data" + data_root.mkdir() + (data_root / "train.jsonl").write_text("{}\n") + monkeypatch.setenv("SFT_MODEL_PATH", str(model_dir)) + monkeypatch.setenv("SFT_DATA_ROOT", str(data_root)) + return make_config() + + +def test_recipe_is_one_epoch_of_64_clip_updates(recipe: Any, tmp_path: Path) -> None: + assert recipe.model_backend == "i4" + assert (recipe.rl.micro_batch_size, recipe.rl.grad_accum, recipe.rl.max_steps) == (1, 64, 125) + assert recipe.rl.save_every == recipe.rl.max_steps + assert recipe.rl.checkpoint_format == "consolidated" + assert recipe.rl.max_length == 10240 + assert recipe.rl.optimizer.name == "fusedadam" and recipe.rl.optimizer.lr == 2e-5 + assert recipe.rl.optimizer.train_keys == [] + assert recipe.rl.processor_kwargs == {"videos_kwargs": {"do_sample_frames": False}} + assert recipe.rl.validation.enabled is False + assert recipe.rl.data.path == str(tmp_path / "data" / "train.jsonl") + + +def test_recipe_passes_the_sft_preflight_on_the_native_qwen3_vl_seat( + recipe: Any, tmp_path: Path, apex_present: None +) -> None: + from cosmos_framework.posttrain.entrypoints.sft import sft_policy_source, validate_config + + rl = validate_config(recipe) + assert rl.data.path == str(tmp_path / "data" / "train.jsonl") + assert sft_policy_source(recipe) == (str(tmp_path / "model"), False) diff --git a/docs/posttrain/guides/sft.md b/docs/posttrain/guides/sft.md index 540b18fc3..aad4ef796 100644 --- a/docs/posttrain/guides/sft.md +++ b/docs/posttrain/guides/sft.md @@ -59,7 +59,9 @@ export SFT_DATA_ROOT="$PWD/.ene/scratch/sft-example" The generator creates 32 training and 8 validation conversations with local PNG/MP4 media; it refuses to overwrite a nonempty directory. Video decoding requires TorchCodec 0.11.0 -and compatible FFmpeg shared libraries. This is a functional example, not a quality benchmark. +and compatible FFmpeg shared libraries. This is a functional example, not a quality benchmark; for a +worked example on real data with a measured before and after on one GPU, see the +[`cosmos3-qwen3-vl-robovqa-sft` skill](../../../.agents/skills/cosmos3-qwen3-vl-robovqa-sft/SKILL.md). Preflight checks configuration, local file availability and role wiring, not GPU memory or remote access. `--validate-rows` then runs every row through the real packer on CPU and refuses over-long rows, masks that disagree with the assistant turns, and mislabelled tool turns before any From bba35bde88c0a0cd3dfce126e7d66c708d40e7bc Mon Sep 17 00:00:00 2001 From: sbabamohamad Date: Thu, 29 Oct 2026 10:55:04 -0700 Subject: [PATCH 02/14] posttrain: write HF exports in 5 GB shards so verify_index finds the index Transformers 5 defaults save_pretrained's max_shard_size to 50 GB, so a model under 50 GB (Qwen3-VL-8B, 16.3 GiB) is written as one model.safetensors without model.safetensors.index.json, and the materializer's own verify_index refuses the export ("[materialize] FATAL ... model.safetensors.index.json missing"). Writing 5 GB shards, the Transformers 4 default, restores the index; the Qwen3-VL-8B export then verifies with 750 tensors in 4 shards. Signed-off-by: sbabamohamad --- cosmos_framework/posttrain/checkpoint/conversion/hf.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/cosmos_framework/posttrain/checkpoint/conversion/hf.py b/cosmos_framework/posttrain/checkpoint/conversion/hf.py index b5edfd4fe..b262ec1a7 100644 --- a/cosmos_framework/posttrain/checkpoint/conversion/hf.py +++ b/cosmos_framework/posttrain/checkpoint/conversion/hf.py @@ -913,7 +913,9 @@ def overlay_consolidated_onto_hf( _log(f"writing HF-format directory: {staging_dir}") os.makedirs(staging_dir, exist_ok=True) - model.save_pretrained(staging_dir, safe_serialization=True) + # 5 GB shards (the Transformers 4 default): Transformers 5 defaults to 50 GB and writes a smaller model as one file + # without the INDEX_FILE that verify_index requires, so an 8B export failed its own verification. + model.save_pretrained(staging_dir, safe_serialization=True, max_shard_size="5GB") AutoProcessor.from_pretrained(model_id).save_pretrained(staging_dir) if expert_orig: rewritten = rewrite_expert_shards(staging_dir, expert_orig) From d4da5ad8bd68e6a0abc7197c6ba8f2456710b4e2 Mon Sep 17 00:00:00 2001 From: sbabamohamad Date: Thu, 29 Oct 2026 11:12:57 -0700 Subject: [PATCH 03/14] skills: one command per step in cosmos3-qwen3-vl-robovqa-sft scripts/run.sh runs each step of the example as one command (setup, download, data, preflight, before, train, export, after; --smoke for the six bundled clips and 2 updates), so SKILL.md reads as eight one-line steps with their expected output. scripts/env.sh is folded into run.sh. Signed-off-by: sbabamohamad --- .../cosmos3-qwen3-vl-robovqa-sft/SKILL.md | 374 ++++-------------- .../references/measured-run.md | 31 +- .../scripts/env.sh | 23 -- .../scripts/run.sh | 202 ++++++++++ 4 files changed, 304 insertions(+), 326 deletions(-) delete mode 100755 .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh create mode 100755 .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh diff --git a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/SKILL.md b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/SKILL.md index 045bc0fdd..08bdfd035 100644 --- a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/SKILL.md +++ b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/SKILL.md @@ -15,10 +15,10 @@ description: > # Qwen3-VL-8B SFT on RoboVQA (one GPU) -One job: full-parameter SFT of `Qwen/Qwen3-VL-8B-Instruct` on the RoboVQA training clips through -`cosmos_framework.posttrain.entrypoints.sft`, then the same scorer on 1,000 held-out clips before and after. -Example from the held-out set: the clip of a robot arm by a table, "put the box on the table Q: possible right -now?", reference `no`: the base model answers `yes`, the fine-tuned model `no`. +One job: full-parameter SFT of `Qwen/Qwen3-VL-8B-Instruct` on the RoboVQA training clips through the framework's +posttrain SFT path, then the same scorer on 1,000 held-out clips before and after. Example from the held-out set: +"put the box on the table Q: possible right now?", reference `no`: the base model answers `yes`, the fine-tuned +model `no`. Measured on one NVIDIA GB300 (aarch64, driver 595, CUDA 13.0), two runs at seed 0 (training is not bit-reproducible): exact match 0.548 before, 0.632 and 0.620 after. Details: @@ -26,366 +26,164 @@ bit-reproducible): exact match 0.548 before, 0.632 and 0.620 after. Details: ## Read this first -1. **Path convention.** Run every block from the repository root (the directory with `pyproject.toml`). Every - block starts with `source .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh`, which sets the paths, - the uv pin and the artifact directory `$ART`; no block relies on shell state from an earlier block. -2. **Pick the variant from the GPU** (table below): the full run needs one GPU with about 150 GB free; the smoke - variant (steps 0 to 5 on the bundled clips) runs on any GPU that holds the 8B model in bfloat16. -3. **Two Python environments, never mixed**: the SFT environment (framework, Transformers 5.8.1, TorchCodec 0.11) - prepares the training files, checks and trains; the evaluation environment (plain Transformers 4.57.6, - TorchCodec 0.10) decodes the evaluation frames and scores. Each command names its interpreter. -4. **Check each step's expected output** (given under the step and in "Verify the output") before the next step. -5. **After a failure**, read "Things not obvious from the docs" and - [`references/measured-run.md`](references/measured-run.md) § Troubleshooting before changing a command. +1. **One command per step.** Every step is + `bash .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh `, run from the repository root (the + directory with `pyproject.toml`). The script sets its own paths and pins, so each step can run in a fresh shell; + run the steps in order. [`scripts/run.sh`](scripts/run.sh) shows every command a step runs, with comments. +2. **Pick the variant from the GPU** (table below). The smoke variant adds `--smoke` to every step. +3. **Check each step's expected output** (under the step) before the next step. Each step also saves its output in + the `logs` folder of the artifacts directory, which the script creates as artifacts/qwen3-vl-robovqa under the + repository root. +4. **After a failure**, read "Things not obvious from the docs" and the troubleshooting section of + [`references/measured-run.md`](references/measured-run.md) before changing a command. ## When to use this skill - Fine-tune Qwen3-VL-8B on RoboVQA with Cosmos Framework, or reproduce the before-and-after numbers above - A worked, measured example of the posttrain SFT path for a Hugging Face VLM on video question answering -- Score any Hugging Face Qwen3-VL directory on the RoboVQA held-out clips (step 5 / step 8) +- Score a Hugging Face Qwen3-VL model on the RoboVQA held-out clips (steps `before` and `after`) - Not for Cosmos3 models: `../cosmos3-post-training/SKILL.md`. Not for LoRA: the SFT path does not support it (`docs/posttrain/guides/sft.md` section 1). Environment errors: `../cosmos3-env-troubleshoot/SKILL.md` ## Variants and hardware -| Variant | GPU | Steps | Time | Status | +| Variant | Data | GPU | Time | Status | | --- | --- | --- | --- | --- | -| Full run | one GPU with at least 150,000 MiB free; torch peaks at 143,566 MiB reserved, so 80 to 141 GB cards do not fit | 0 to 8 | about 3.5 h (training 2 h 05 to 2 h 14) | verified twice on one GB300, October 2026 | -| Smoke | any GPU for step 5 (the 8B model in bfloat16, 16.3 GiB of weights); steps 3 and 4 run on the CPU | 0 to 5 on `assets/robovqa` | environments plus a few minutes | the same commands on 6 bundled clips; measured by this skill's eval only | +| Full run | RoboVQA, 7,998 training clips, 125 updates | one GPU with at least 150,000 MiB free for `train` (torch peaks at 143,566 MiB reserved); every other step fits a smaller GPU | about 3.5 h, training 2 h 05 to 2 h 14 | verified twice on one GB300, October 2026 | +| Smoke (`--smoke`) | the six bundled clips, 2 updates | `setup` to `before` on any GPU that holds the 8B model in bfloat16 (16.3 GiB); `train`, `export`, `after` need the same 150 GB GPU | environments plus about 15 minutes | measured by this skill's eval | -Disk: about 110 GB for the full run (frames 33 GB, model, checkpoint and export 17 GB each), about 40 GB for the -smoke variant (environments and the model). The x86_64 branches below (`nvidia-npp` from PyPI) are unverified. -To score a model without training: steps 0 and 1c, the model download of step 2, the `split` line of step 3 and -step 5. +Disk: about 110 GB for the full run, about 75 GB for the smoke variant (environments, model, checkpoint, export). +x86_64 hosts take the `nvidia-npp` wheel from PyPI in `setup`; that route is unverified. ## Bundled assets | Asset | Path | Used by | | --- | --- | --- | -| Six RoboVQA clips with their source rows (Tianli/robovqa, Apache-2.0, revision `eb58194`) | `assets/robovqa/videos/`, `assets/robovqa/json/train/sample.json`, `assets/robovqa/LICENSE.txt` | smoke variant, steps 3 to 5 | +| Six RoboVQA clips with their source rows (Tianli/robovqa, Apache-2.0, revision `eb58194`) | `assets/robovqa/videos/`, `assets/robovqa/json/train/sample.json`, `assets/robovqa/LICENSE.txt` | every `--smoke` step | -The bundled directory has the layout of the full download, so the smoke commands differ from the full ones in -their `--src` / `--out` paths only. Six clips split 4 / 0 / 2 (train / val / test). +The bundled directory has the layout of the full download. Six clips split 4 / 0 / 2 (train / val / test). -## Step 0 — prerequisites +## Steps -Check the GPU, the driver's CUDA version (the wheels are CUDA 13.0), the system tools and uv: +### 1. setup — environments (about 12 minutes) ```bash -source .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh -nvidia-smi --query-gpu=name,memory.total,memory.free,driver_version,compute_cap --format=csv,noheader -CUDA_MAJOR=$(nvidia-smi | sed -n 's/.*CUDA Version: \([0-9]*\)\..*/\1/p' | head -1) -[ "${CUDA_MAJOR:-0}" -ge 13 ] || { echo "driver supports CUDA ${CUDA_MAJOR}; this recipe uses CUDA 13.0 wheels (a CUDA 12 route is unverified)"; exit 1; } -MISSING="" -for tool in git ffmpeg g++ curl; do command -v "$tool" >/dev/null || MISSING="$MISSING $tool"; done -if [ -n "$MISSING" ]; then - if [ "$(id -u)" = 0 ]; then apt-get update -q && apt-get install -y -q git ffmpeg g++ curl - else echo "missing:$MISSING (sudo apt-get install -y git ffmpeg g++ curl)"; exit 1; fi -fi -command -v uvx >/dev/null || curl -LsSf https://astral.sh/uv/install.sh | sh -$UV --version -ffmpeg -version | head -1 -df -h "$ART" | tail -1 +bash .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh setup ``` -Continue when the CUDA check passes, `uv 0.11.29` prints, `ffmpeg` is 4 to 8, and the free disk covers the -variant (above). +Checks the GPU, the driver (CUDA 13), `git`, `ffmpeg` and `g++`; builds the SFT environment from +`docs/posttrain/guides/sft.md` section 2; builds apex (the recipe's optimizer) unless it is already installed; +builds the evaluation environment (plain Transformers). Expected last lines: +`SFT environment: 2.11.0+cu130 5.8.1 apex ok` and `evaluation environment: 2.10.0+cu130 4.57.6`. -## Step 1 — environments - -**1a. SFT environment** — `docs/posttrain/guides/sft.md` section 2 plus `--python 3.13` (with `--project`, uv does -not read the checkout's `.python-version` and can pick a system 3.12), the aarch64 group that adds `nvidia-npp` -(TorchCodec needs `libnppicc.so.13`) and `--inexact` (an exact sync uninstalls apex, built in 1b): - -```bash -source .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh -ARCH_GROUP=""; [ "$(uname -m)" = aarch64 ] && ARCH_GROUP="--group cu130-aarch64" -UV_PROJECT_ENVIRONMENT="$SFT_VENV" $UV sync \ - --project docker/posttrain/environment \ - --frozen \ - --python 3.13 \ - --no-group transformers4 \ - --group sft \ - --group cu130-runtime \ - $ARCH_GROUP \ - --inexact -[ "$(uname -m)" = aarch64 ] || $UV pip install --python "$SFT_PY" "nvidia-npp==13.0.0.50" -LD_LIBRARY_PATH="$SFT_LIBS" "$SFT_PY" -c "import torch, transformers, torchcodec.decoders; print(torch.__version__, transformers.__version__)" -``` - -Expected: `2.11.0+cu130 5.8.1`. - -**1b. apex** — the native Qwen3-VL seat trains with apex `FusedAdam`, and the SFT preflight refuses `fusedadam` -without apex. NVIDIA training containers ship it; otherwise this builds it from source with the CUDA 13.0 compiler -and headers from PyPI (about 10 minutes on a GB300), for the GPU's own architecture, with one compile job per -4 GiB of free host memory: - -```bash -source .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh -if "$SFT_PY" -c "import torch, amp_C" 2>/dev/null; then - echo "apex already importable" -else - TC="$ART/venvs/cuda13-toolchain" - $UV venv --allow-existing --python 3.13 "$TC" - $UV pip install --python "$TC/bin/python" ninja \ - "nvidia-cuda-nvcc==13.0.88" "nvidia-cuda-cccl==13.0.85" "nvidia-cuda-crt==13.0.88" "nvidia-nvvm==13.0.88" \ - "nvidia-cuda-runtime==13.0.96" "nvidia-cuda-profiler-api==13.0.85" "nvidia-cublas==13.1.0.3" \ - "nvidia-cusparse==12.6.3.3" "nvidia-cusolver==12.0.4.66" "nvidia-curand==10.4.0.35" \ - "nvidia-cufft==12.0.0.61" "nvidia-nvjitlink==13.0.88" - export CUDA_HOME="$TC/lib/python3.13/site-packages/nvidia/cu13" - (cd "$CUDA_HOME/lib" && for f in libcudart libcublas libcublasLt libcusparse libcusolver libcurand libcufft libnvJitLink; do - ln -sfn "$(ls $f.so.* | head -1)" "$f.so"; done) - ln -sfn lib "$CUDA_HOME/lib64" - export PATH="$CUDA_HOME/bin:$TC/bin:$PATH" - [ -d "$ART/apex/.git" ] || git clone https://github.com/NVIDIA/apex.git "$ART/apex" - git -C "$ART/apex" checkout --detach 575968bc1f9127ccd61003a681472a83af4ff1a1 - JOBS=$(awk -v n="$(nproc)" '/MemAvailable/ {m = int($2 / 4194304); print (m < n ? m : n)}' /proc/meminfo) - TORCH_CUDA_ARCH_LIST="$(nvidia-smi --query-gpu=compute_cap --format=csv,noheader | head -1)" \ - APEX_CPP_EXT=1 APEX_CUDA_EXT=1 MAX_JOBS="$JOBS" \ - $UV pip install --python "$SFT_PY" --no-build-isolation "$ART/apex" -fi -LD_LIBRARY_PATH="$SFT_LIBS" "$SFT_PY" -c "import torch, transformers, torchcodec.decoders, amp_C; print('apex ok')" -``` - -Expected: `apex ok`. (The GB300 runs built with 64 jobs; the memory-based cap is a recommendation for smaller hosts.) - -**1c. Evaluation environment** — plain Transformers, the stack the scores were measured with: - -```bash -source .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh -$UV venv --allow-existing --python 3.12 "$EVAL_VENV" -$UV pip install --python "$EVAL_PY" "torch==2.10.0" "torchvision==0.25.0" "torchcodec==0.10.0" \ - --index-url https://download.pytorch.org/whl/cu130 -$UV pip install --python "$EVAL_PY" "transformers==4.57.6" "accelerate==1.15.0" "av==19.0.1" "numpy==2.2.6" \ - pillow safetensors "huggingface_hub==0.36.2" "nvidia-npp==13.0.0.50" -LD_LIBRARY_PATH="$EVAL_LIBS" "$EVAL_PY" -c "import torch, transformers, torchcodec.decoders; print(torch.__version__, transformers.__version__)" -``` - -Expected: `2.10.0+cu130 4.57.6`. - -## Step 2 — model and data - -Both pinned, downloaded with the evaluation environment's `hf` (huggingface_hub 0.36.2; newer `hf` releases read -the patterns after the first `--include` as file names and skip the videos folder). No token is needed. - -```bash -source .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh -"$EVAL_VENV/bin/hf" download Qwen/Qwen3-VL-8B-Instruct \ - --revision 0c351dd01ed87e9c1b53cbc748cba10e6187ff3b \ - --local-dir "$SFT_MODEL_PATH" -``` - -**Full run only** (3.5 GB; the smoke variant uses the bundled clips instead): - -```bash -source .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh -"$EVAL_VENV/bin/hf" download Tianli/robovqa --repo-type dataset \ - --revision eb5819490f0d9ec18a79774233d4b1c9ebd52eea \ - --include "videos/*" "json/*" "LICENSE.txt" \ - --local-dir "$ART/robovqa" -``` - -## Step 3 — training rows - -[`scripts/prepare_robovqa.py`](scripts/prepare_robovqa.py): `split` (80/10/10 by clip, one question per clip, -byte-identical clips kept in one split), `frames` (each training clip's evaluation frames, every round(fps/2)-th -frame up to 32, written losslessly so training sees exactly what the evaluator decodes; evaluation environment), -`verify` (the training stack decodes the same frames and the processor gives the same tokens and pixels; SFT -environment). Full run, about 25 minutes with 48 CPU processes: +### 2. download — model and data ```bash -source .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh -"$EVAL_PY" "$SKILL_DIR/scripts/prepare_robovqa.py" split --src "$ART/robovqa" --out "$ART/data" -CUDA_VISIBLE_DEVICES= LD_LIBRARY_PATH="$EVAL_LIBS" \ - "$EVAL_PY" "$SKILL_DIR/scripts/prepare_robovqa.py" frames --out "$ART/data" -CUDA_VISIBLE_DEVICES= HF_HUB_OFFLINE=1 LD_LIBRARY_PATH="$SFT_LIBS" \ - "$SFT_PY" "$SKILL_DIR/scripts/prepare_robovqa.py" verify --out "$ART/data" --model "$SFT_MODEL_PATH" +bash .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh download ``` -Expected: counts `train 7998, val 1001, test 1000`; the md5 lines equal their seed-0 references -(`0b3739a5…` train, `1a9b1924…` test); `frames` ends with `"rows": 7998` and `"T_total": 119827`; `verify` ends with -`"all_ok": true, "bad": 0` and `"len_max": 4113`. +Qwen3-VL-8B-Instruct (17 GB) and RoboVQA (3.5 GB, skipped with `--smoke`), both at fixed revisions. No token needed. -**Smoke variant** (the bundled clips; the md5 references apply to the full download only): +### 3. data — training rows (about 25 minutes) ```bash -source .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh -"$EVAL_PY" "$SKILL_DIR/scripts/prepare_robovqa.py" split --src "$SKILL_DIR/assets/robovqa" --out "$ART/smoke" -CUDA_VISIBLE_DEVICES= LD_LIBRARY_PATH="$EVAL_LIBS" \ - "$EVAL_PY" "$SKILL_DIR/scripts/prepare_robovqa.py" frames --out "$ART/smoke" -CUDA_VISIBLE_DEVICES= HF_HUB_OFFLINE=1 LD_LIBRARY_PATH="$SFT_LIBS" \ - "$SFT_PY" "$SKILL_DIR/scripts/prepare_robovqa.py" verify --out "$ART/smoke" --model "$SFT_MODEL_PATH" +bash .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh data ``` -Expected: counts `train 4, val 0, test 2`; `frames` `"rows": 4`; `verify` `"all_ok": true, "bad": 0`. +[`scripts/prepare_robovqa.py`](scripts/prepare_robovqa.py) splits the clips 80/10/10 with one question per clip and +byte-identical clips kept in one split, writes each training clip's evaluation frames losslessly so training sees +exactly what the evaluator scores, and checks every file in the training stack. Expected: counts +`train 7998, val 1001, test 1000`; md5 lines equal to their seed-0 references; `"rows": 7998`; `"all_ok": true`. +With `--smoke`: counts `train 4, val 0, test 2` (the md5 references apply to the full download only), +`"rows": 4`, `"all_ok": true`. -## Step 4 — preflight - -The two CPU checks of `docs/posttrain/guides/sft.md` section 2 with the recipe that trains, -`cosmos_framework/posttrain/configs/sft_qwen3_vl_8b_robovqa.py` (it reads `SFT_MODEL_PATH` and `SFT_DATA_ROOT`). -Full run, about 13 minutes: +### 4. preflight — the recipe's CPU checks (about 13 minutes) ```bash -source .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh -for check in --validate-manifest --validate-rows; do - CUDA_VISIBLE_DEVICES= PYTHONPATH=. SFT_DATA_ROOT="$ART/data" IMAGINAIRE_OUTPUT_ROOT="$ART/runs/preflight" \ - LD_LIBRARY_PATH="$SFT_LIBS" \ - "$SFT_PY" -m cosmos_framework.posttrain.entrypoints.sft --config "$CONFIG" "$check" || exit 1 -done 2>&1 | tee "$ART/logs/preflight.log" +bash .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh preflight ``` -Expected gate line: -`train: rows 7998 | loss_tokens == assistant_tokens on 7998/7998 | longest 4113 <= 10240 | failed 0`; the next -line, `train: dp_world not checked`, is expected outside torchrun. +`--validate-manifest` and `--validate-rows` of the SFT entrypoint with the recipe +`cosmos_framework/posttrain/configs/sft_qwen3_vl_8b_robovqa.py`. Expected: +`train: rows 7998 | loss_tokens == assistant_tokens on 7998/7998 | longest 4113 <= 10240 | failed 0` (`rows 4` with +`--smoke`). The line `train: dp_world not checked` that follows is expected outside torchrun. -**Smoke variant:** +### 5. before — score the untouched model (about 14 minutes) ```bash -source .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh -for check in --validate-manifest --validate-rows; do - CUDA_VISIBLE_DEVICES= PYTHONPATH=. SFT_DATA_ROOT="$ART/smoke" IMAGINAIRE_OUTPUT_ROOT="$ART/runs/preflight-smoke" \ - LD_LIBRARY_PATH="$SFT_LIBS" \ - "$SFT_PY" -m cosmos_framework.posttrain.entrypoints.sft --config "$CONFIG" "$check" || exit 1 -done 2>&1 | tee "$ART/logs/preflight-smoke.log" +bash .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh before ``` -Expected gate line: `train: rows 4 | loss_tokens == assistant_tokens on 4/4 | ... | failed 0`. - -## Step 5 — before: score the untouched model +[`scripts/eval_robovqa.py`](scripts/eval_robovqa.py) answers every held-out clip greedily and scores normalised exact +match, token F1 and yes/no accuracy. Expected last line: `exact 0.548 token-F1 0.676 yes/no 0.918 (n=1000)` +(`n=2` with `--smoke`). -[`scripts/eval_robovqa.py`](scripts/eval_robovqa.py) answers greedily (at most 64 new tokens, batches of 4, -bfloat16) and scores normalised exact match, token F1 and yes/no accuracy. Full run, about 14 minutes: +### 6. train — fine-tune (about 2 hours) ```bash -source .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh -LD_LIBRARY_PATH="$EVAL_LIBS" "$EVAL_PY" "$SKILL_DIR/scripts/eval_robovqa.py" \ - --model "$SFT_MODEL_PATH" \ - --data "$ART/data/hf/test.jsonl" \ - --ref "$ART/data/samples_test.jsonl" \ - --out "$ART/eval/zeroshot" +bash .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh train ``` -Expected last line: `exact 0.548 token-F1 0.676 yes/no 0.918 (n=1000)` (deterministic). +One GPU through torchrun, 125 updates of 64 clips (2 with `--smoke`). It stops before starting when the GPU has less +than 150,000 MiB free. The first update appears after about three minutes, then one `step N/125` line about every +minute; the loss starts near 1.0 and ends near 0.5. Expected end: `SFT DONE {'start': 0, 'final_step': 125, ...}`. +Agents: start it detached and keep the log, for example +`nohup bash .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh train > train.out 2>&1 &`. -**Smoke variant** (2 held-out clips; a pipeline check, not a quality measure): +### 7. export — Hugging Face format (about 2 minutes) ```bash -source .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh -LD_LIBRARY_PATH="$EVAL_LIBS" "$EVAL_PY" "$SKILL_DIR/scripts/eval_robovqa.py" \ - --model "$SFT_MODEL_PATH" \ - --data "$ART/smoke/hf/test.jsonl" \ - --ref "$ART/smoke/samples_test.jsonl" \ - --out "$ART/eval/zeroshot-smoke" +bash .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh export ``` -Expected: `$ART/eval/zeroshot-smoke/preds.jsonl` with 2 rows, `$ART/eval/zeroshot-smoke/scores.json`, and a last line -`exact … token-F1 … yes/no … (n=2)`. - -## Step 6 — train (full run only) +The framework's HF materializer on the CPU, in 5 GB shards. Expected: +`[materialize] OK -> .../export (16.3 GiB, 750 tensors in 4 shards, 8 sidecars identical to base, ...)`. -One GPU through torchrun, as `docs/posttrain/guides/sft.md` requires; started detached with its log kept. The block -stops before launching when the GPU has less than 150,000 MiB free: +### 8. after — score the fine-tuned model (about 11 minutes) ```bash -source .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh -FREE=$(nvidia-smi --query-gpu=memory.free --format=csv,noheader,nounits | head -1) -[ "$FREE" -ge 150000 ] || { echo "${FREE} MiB free; full SFT of the 8B model needs about 150 GB (measured peak 143,566 MiB reserved)"; exit 1; } -PYTHONPATH=. SFT_DATA_ROOT="$ART/data" IMAGINAIRE_OUTPUT_ROOT="$ART/runs" LD_LIBRARY_PATH="$SFT_LIBS" \ - setsid nohup "$SFT_PY" -m torch.distributed.run \ - --standalone \ - --nproc_per_node=1 \ - -m cosmos_framework.posttrain.entrypoints.sft \ - --config "$CONFIG" \ - > "$ART/logs/train.log" 2>&1 < /dev/null & -echo "training started; log: $ART/logs/train.log" +bash .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh after ``` -Wait for it (about three minutes to update 1, then one `step N/125` line about every minute; deadline 3 h): - -```bash -source .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh -for minute in $(seq 1 180); do - grep -q "SFT DONE" "$ART/logs/train.log" && break - pgrep -f "entrypoints.sft --config $CONFIG" >/dev/null || { echo "training exited"; tail -n 40 "$ART/logs/train.log"; exit 1; } - sleep 60 -done -grep -E "step +(1|125)/125 |SFT DONE" "$ART/logs/train.log" | cut -c1-120 -``` - -Expected: the logged training loss (`loss_avg`) near 1.0 at update 1 and near 0.5 at update 125 (single updates -move with each batch of 64 clips), then `SFT DONE {'start': 0, 'final_step': 125, ...}`. A second seed: append `-- rl.seed=1 job.name=seed1` -to the launch command. - -## Step 7 — export to Hugging Face format - -The framework's HF materializer, on the CPU (about 2 minutes); it writes 5 GB shards and verifies all 750 tensors: - -```bash -source .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh -CKPT=$(ls -d "$ART"/runs/posttrain_sft/qwen3_vl_8b_robovqa/seed0/*/checkpoint_artifacts/iter_* | sort | tail -1) -CUDA_VISIBLE_DEVICES= PYTHONPATH=. LD_LIBRARY_PATH="$SFT_LIBS" \ - "$SFT_PY" -m cosmos_framework.posttrain.checkpoint.tools.materialize_hf \ - --ckpt-dir "$CKPT" \ - --base "$SFT_MODEL_PATH" \ - --out "$ART/export" \ - --expect-tensors 750 \ - --min-tensors 750 -``` - -Expected: `[materialize] OK -> .../export (16.3 GiB, 750 tensors in 4 shards, 8 sidecars identical to base, ...)`. - -## Step 8 — after: score the fine-tuned model - -```bash -source .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh -LD_LIBRARY_PATH="$EVAL_LIBS" "$EVAL_PY" "$SKILL_DIR/scripts/eval_robovqa.py" \ - --model "$ART/export" \ - --data "$ART/data/hf/test.jsonl" \ - --ref "$ART/data/samples_test.jsonl" \ - --out "$ART/eval/sft" -``` - -Expected: exact match near 0.62 to 0.63 (0.632 and 0.620 in the two measured runs). A score near 0.55 means the -fine-tuned weights were not loaded. +Expected: exact match near 0.62 to 0.63 (0.632 and 0.620 in the two measured runs). Near 0.55 means the fine-tuned +weights were not loaded. With `--smoke` the score is only a check that the export loads and answers. ## Verify the output -| Step | File | Correct when | -| --- | --- | --- | -| 3 | `$ART/data/split_manifest.json` | counts 7998 / 1001 / 1000; every `leakage_check` value is 0 | -| 3 | `$ART/data/train.jsonl.verify.jsonl` | 7,998 lines, all `sha_ok`, `ids_equal`, `grid_equal`, `pix_equal` true | -| 4 | `$ART/logs/preflight.log` | the gate line ends in `failed 0` | -| 5, 8 | `$ART/eval//scores.json` | `__all__.n` is 1000 and `__meta__.missing_predictions` is 0 | -| 6 | `$ART/logs/train.log` | 125 `step N/125` lines and `SFT DONE` | -| 7 | `$ART/export/model.safetensors.index.json` | 750 entries in `weight_map` | +Every step saves its console output in the artifacts directory's `logs` folder, named after the step (with a +`-smoke` suffix for the smoke variant). A run is correct when: -Smoke variant: the same files under `$ART/smoke`, `$ART/logs/preflight-smoke.log` and `$ART/eval/zeroshot-smoke`, -with 4 training rows and `n` = 2. +| Step | Correct when | +| --- | --- | +| data | the split manifest's `leakage_check` values are all 0 and verify prints `"all_ok": true, "bad": 0` | +| preflight | the train gate line ends in `failed 0` | +| before, after | the overall line reports `n=1000` (`n=2` with `--smoke`) and the scores file shows no missing predictions | +| train | the log has a `step N/125` line per update and `SFT DONE` | +| export | `[materialize] OK` with 750 tensors | ## Reference files | File | What it holds | | --- | --- | +| [`scripts/run.sh`](scripts/run.sh) | every step's commands, with comments | | [`references/measured-run.md`](references/measured-run.md) | the two measured runs: versions, timings, memory, loss, example answers, troubleshooting | | `cosmos_framework/posttrain/configs/sft_qwen3_vl_8b_robovqa.py` | the recipe, each field with its reason | | [`scripts/prepare_robovqa.py`](scripts/prepare_robovqa.py) | split, frames, verify | | [`scripts/eval_robovqa.py`](scripts/eval_robovqa.py) | evaluator, scorer and the shared frame decoder | -| [`scripts/env.sh`](scripts/env.sh) | paths and pins every block sources | ## Things not obvious from the docs -- **`libnppicc.so.13: cannot open shared object file`**: TorchCodec needs NVIDIA NPP. Every command that decodes - video sets `LD_LIBRARY_PATH` to its environment's NPP library directory (`$SFT_LIBS` or `$EVAL_LIBS`). Do not clear - `LD_LIBRARY_PATH`. +- **`libnppicc.so.13: cannot open shared object file`**: TorchCodec needs NVIDIA NPP; `setup` installs it and every + step that decodes video points `LD_LIBRARY_PATH` at it. Do not clear `LD_LIBRARY_PATH`. - **`ImportError: libc10.so` importing `amp_C`**: import `torch` first; apex's extensions link against torch. -- **apex gone after a second `uv sync`**: an exact sync uninstalls packages the lock does not list; keep `--inexact`. -- **SFT environment on Python 3.12**: `$SFT_LIBS` then points at a missing `python3.13` path; keep `--python 3.13`. -- **`[materialize] FATAL ... model.safetensors.index.json missing`**: a checkout whose materializer still writes one - 50 GB shard; Transformers 5 writes a model under 50 GB as one file without the index the verifier requires. The - materializer now writes 5 GB shards. -- **`hf download` fetched no videos**: a newer `hf`; use `$EVAL_VENV/bin/hf` (huggingface_hub 0.36.2). -- **Training needs one GPU with about 150 GB**: 80 GB and 141 GB cards run every step except step 6. -- **The first update takes about two minutes and prints nothing before it**: model load and warm-up, not a hang. +- **apex gone after re-running `uv sync` by hand**: an exact sync uninstalls packages the lock does not list; + `setup` syncs with `--inexact`. +- **SFT environment on Python 3.12**: TorchCodec then cannot find NPP; `setup` pins `--python 3.13`. +- **`[materialize] FATAL ... model.safetensors.index.json missing`**: a framework checkout older than the 5 GB shard + change; Transformers 5 writes a model under 50 GB as one file without the index the verifier requires. +- **`hf download` fetched no videos**: a newer `hf`; `download` uses the evaluation environment's (huggingface_hub + 0.36.2). +- **The first training update takes about two minutes and nothing prints before it**: model load and warm-up. ## Related skills diff --git a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/references/measured-run.md b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/references/measured-run.md index 4f333964c..26ff350dc 100644 --- a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/references/measured-run.md +++ b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/references/measured-run.md @@ -2,9 +2,9 @@ Two runs of this recipe at seed 0, each on one NVIDIA GB300 (256,703 MiB, aarch64, driver 595.91.07, CUDA 13.0), directly on the host without a container, in October 2026, with the framework at commit -`9a70e64d2ad7ff1417484d12d10fb121b270baaf`. The commands were the ones in `SKILL.md` steps 1 to 8, with the training -config, data preparation and evaluator unchanged; the runs drove them from one shell script, and wrote the HF export -through the materializer with 5 GB shards (the change that ships with this skill). +`9a70e64d2ad7ff1417484d12d10fb121b270baaf`. The runs used the commands of [`../scripts/run.sh`](../scripts/run.sh) +from an earlier version of that script, with the same training config, data preparation and evaluator, and wrote the +HF export through the materializer with 5 GB shards (the change that ships with this skill). ## Versions @@ -71,23 +71,24 @@ normalised exact match. ## Troubleshooting -Each entry: symptom, cause, check, fix. +Each entry: symptom, cause, check, fix. Paths are relative to the repository root. - **`libnppicc.so.13: cannot open shared object file`** when TorchCodec loads. Cause: NVIDIA NPP is not on the - library path. Check: `ls "$SFT_LIBS" "$EVAL_LIBS" | grep nppicc`. Fix: keep the `nvidia-npp` installs of step 1 and - the `LD_LIBRARY_PATH=...` prefix of every decoding command; never clear `LD_LIBRARY_PATH`. + library path. Check: `ls docker/posttrain/environment/.venv-sft/lib/python3.13/site-packages/nvidia/cu13/lib | grep nppicc`. + Fix: re-run `run.sh setup`, which installs `nvidia-npp` in both environments; `run.sh` puts it on + `LD_LIBRARY_PATH` for every command that decodes video. Never clear `LD_LIBRARY_PATH`. - **`ImportError: libc10.so` importing `amp_C`.** Cause: apex's extensions link against torch's libraries. Check: - `"$SFT_PY" -c "import torch, amp_C"`. Fix: import `torch` first. -- **apex missing after re-running step 1a.** Cause: an exact `uv sync` uninstalls packages the lock does not list. - Check: `"$SFT_PY" -c "import torch, amp_C"`. Fix: keep `--inexact`; re-run 1b. + `docker/posttrain/environment/.venv-sft/bin/python -c "import torch, amp_C"`. Fix: import `torch` first. +- **apex missing after running `uv sync` by hand.** Cause: an exact sync uninstalls packages the lock does not list. + Check: the same import. Fix: re-run `run.sh setup` (it syncs with `--inexact` and rebuilds apex only when missing). - **The SFT environment was built on Python 3.12.** Cause: with `--project`, uv does not read the checkout's - `.python-version` and takes any interpreter in range. Check: `"$SFT_PY" --version`. Fix: delete - `docker/posttrain/environment/.venv-sft` and re-run 1a with `--python 3.13`. + `.python-version` and takes any interpreter in range. Check: `docker/posttrain/environment/.venv-sft/bin/python + --version`. Fix: delete `docker/posttrain/environment/.venv-sft` and re-run `run.sh setup`, which pins + `--python 3.13`. - **`[materialize] FATAL ... model.safetensors.index.json missing`.** Cause: Transformers 5.8.1 writes a model under - 50 GB as one file without an index, and the materializer's verifier requires the index. Check: - `ls "$ART/export.tmp" "$ART/export" 2>/dev/null`. Fix: a materializer that writes 5 GB shards (this skill's - change), then re-run step 7. -- **The training stage prints nothing for about three minutes.** Cause: model load and the first update's warm-up. + 50 GB as one file without an index, and the materializer's verifier requires the index. Fix: a framework checkout + that writes 5 GB shards (the change that ships with this skill), then re-run `run.sh export`. +- **The training step prints nothing for about three minutes.** Cause: model load and the first update's warm-up. Check: `nvidia-smi` shows the process growing toward 140 GB. Fix: none; wait for `step 1/125`. - **`--validate-rows` prints `train: dp_world not checked`.** Expected outside torchrun; one GPU divides every row count. diff --git a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh deleted file mode 100755 index 73a843ce7..000000000 --- a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh +++ /dev/null @@ -1,23 +0,0 @@ -#!/usr/bin/env bash -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: OpenMDW-1.1 -# Paths and tool pins shared by every SKILL.md block. Source it from the repository root at the top of each block: -# source .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/env.sh -# It only sets variables (and creates $ART); it installs nothing. -if [ ! -f pyproject.toml ] || [ ! -d cosmos_framework ]; then - echo "env.sh: run from the cosmos-framework repository root (the directory with pyproject.toml)" >&2 - return 1 2>/dev/null || exit 1 -fi -export PATH="$HOME/.local/bin:$PATH" # where the uv installer puts uv / uvx -export SKILL_DIR="$PWD/.agents/skills/cosmos3-qwen3-vl-robovqa-sft" -export UV="uvx --from uv==0.11.29 uv" # the checkout pins uv 0.11.29 (tool.uv.required-version) -export ART="$PWD/artifacts/qwen3-vl-robovqa" # every artifact of this skill (about 110 GB for the full run) -export SFT_VENV="$PWD/docker/posttrain/environment/.venv-sft" -export SFT_PY="$SFT_VENV/bin/python" -export SFT_LIBS="$SFT_VENV/lib/python3.13/site-packages/nvidia/cu13/lib" # libnppicc.so.13 for TorchCodec 0.11 -export EVAL_VENV="$ART/venvs/eval" -export EVAL_PY="$EVAL_VENV/bin/python" -export EVAL_LIBS="$EVAL_VENV/lib/python3.12/site-packages/nvidia/cu13/lib" # libnppicc.so.13 for TorchCodec 0.10 -export SFT_MODEL_PATH="$ART/models/Qwen3-VL-8B-Instruct" # read by the recipe config -export CONFIG="cosmos_framework/posttrain/configs/sft_qwen3_vl_8b_robovqa.py" -mkdir -p "$ART/logs" diff --git a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh new file mode 100755 index 000000000..4b4556797 --- /dev/null +++ b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh @@ -0,0 +1,202 @@ +#!/usr/bin/env bash +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 +# +# Fine-tune Qwen3-VL-8B-Instruct on RoboVQA with Cosmos Framework, one step per command. +# Run from the cosmos-framework repository root: +# +# bash .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh [--smoke] +# +# Steps, in order: +# setup the SFT environment, apex and the evaluation environment +# download Qwen3-VL-8B-Instruct and RoboVQA, at fixed revisions +# data split RoboVQA, write the training frames, check them +# preflight the SFT path's CPU checks of the recipe +# before score the untouched model on the held-out clips +# train fine-tune: one epoch, 125 updates of 64 clips, one GPU with about 150 GB +# export write the trained weights as a Hugging Face model +# after score the fine-tuned model the same way +# +# --smoke uses the skill's six bundled clips and 2 training updates: a quick check of every step. +# Every artifact goes under artifacts/qwen3-vl-robovqa/; each step's output is also saved under its logs/. +set -euo pipefail + +STEP=${1:-} +SMOKE=0 +[ "${2:-}" = "--smoke" ] && SMOKE=1 +if [ ! -f pyproject.toml ] || [ ! -d cosmos_framework ]; then + echo "run.sh: run it from the cosmos-framework repository root" >&2 + exit 1 +fi + +export PATH="$HOME/.local/bin:$PATH" +SKILL_DIR="$PWD/.agents/skills/cosmos3-qwen3-vl-robovqa-sft" +UV="uvx --from uv==0.11.29 uv" # the checkout pins uv 0.11.29 +ART="$PWD/artifacts/qwen3-vl-robovqa" +SFT_VENV="$PWD/docker/posttrain/environment/.venv-sft" +SFT_PY="$SFT_VENV/bin/python" +SFT_LIBS="$SFT_VENV/lib/python3.13/site-packages/nvidia/cu13/lib" # NVIDIA NPP, needed by TorchCodec +EVAL_VENV="$ART/venvs/eval" +EVAL_PY="$EVAL_VENV/bin/python" +EVAL_LIBS="$EVAL_VENV/lib/python3.12/site-packages/nvidia/cu13/lib" +CONFIG="cosmos_framework/posttrain/configs/sft_qwen3_vl_8b_robovqa.py" +export SFT_MODEL_PATH="$ART/models/Qwen3-VL-8B-Instruct" # read by the recipe +if [ "$SMOKE" = 1 ]; then + SRC="$SKILL_DIR/assets/robovqa" DATA="$ART/smoke" RUN="$ART/runs-smoke" EVAL="$ART/eval-smoke" + EXPORT="$ART/export-smoke" STEPS=2 TAG="-smoke" +else + SRC="$ART/robovqa" DATA="$ART/data" RUN="$ART/runs" EVAL="$ART/eval" EXPORT="$ART/export" STEPS=125 TAG="" +fi +export SFT_DATA_ROOT="$DATA" # read by the recipe +mkdir -p "$ART/logs" + +say() { printf '\n==> %s\n' "$*"; } + +setup() { + say "GPU and driver" + nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv,noheader + local cuda + cuda=$(nvidia-smi | sed -n 's/.*CUDA Version: \([0-9]*\)\..*/\1/p' | head -1) + if [ "${cuda:-0}" -lt 13 ]; then + echo "the driver supports CUDA ${cuda}; this example uses CUDA 13.0 wheels" >&2 + exit 1 + fi + local missing="" + for tool in git ffmpeg g++ curl; do command -v "$tool" >/dev/null || missing="$missing $tool"; done + if [ -n "$missing" ]; then + if [ "$(id -u)" = 0 ]; then apt-get update -q && apt-get install -y -q git ffmpeg g++ curl; else + echo "install$missing first: sudo apt-get install -y git ffmpeg g++ curl" >&2 + exit 1 + fi + fi + command -v uvx >/dev/null || curl -LsSf https://astral.sh/uv/install.sh | sh + + say "SFT environment (docs/posttrain/guides/sft.md section 2)" + # --python 3.13: with --project, uv does not read the checkout's .python-version. + # --group cu130-aarch64 adds NVIDIA NPP on aarch64. --inexact keeps apex when this is re-run. + local arch=() + [ "$(uname -m)" = aarch64 ] && arch=(--group cu130-aarch64) + UV_PROJECT_ENVIRONMENT="$SFT_VENV" $UV sync --project docker/posttrain/environment --frozen --python 3.13 \ + --no-group transformers4 --group sft --group cu130-runtime "${arch[@]}" --inexact + [ "$(uname -m)" = aarch64 ] || $UV pip install --python "$SFT_PY" "nvidia-npp==13.0.0.50" # x86_64: unverified + + say "apex (the recipe's FusedAdam optimizer)" + if "$SFT_PY" -c "import torch, amp_C" 2>/dev/null; then echo "already installed"; else build_apex; fi + + say "evaluation environment (plain Transformers, the stack the scores were measured with)" + $UV venv --allow-existing --python 3.12 "$EVAL_VENV" + $UV pip install --python "$EVAL_PY" torch==2.10.0 torchvision==0.25.0 torchcodec==0.10.0 \ + --index-url https://download.pytorch.org/whl/cu130 + $UV pip install --python "$EVAL_PY" transformers==4.57.6 accelerate==1.15.0 av==19.0.1 numpy==2.2.6 \ + pillow safetensors huggingface_hub==0.36.2 nvidia-npp==13.0.0.50 + + say "check" + LD_LIBRARY_PATH="$SFT_LIBS" "$SFT_PY" -c \ + "import torch, transformers, torchcodec.decoders, amp_C; print('SFT environment:', torch.__version__, transformers.__version__, 'apex ok')" + LD_LIBRARY_PATH="$EVAL_LIBS" "$EVAL_PY" -c \ + "import torch, transformers, torchcodec.decoders; print('evaluation environment:', torch.__version__, transformers.__version__)" +} + +build_apex() { + # The CUDA 13.0 compiler and headers come from PyPI, so the host needs no CUDA toolkit; about 10 minutes. + local tc="$ART/venvs/cuda13-toolchain" + $UV venv --allow-existing --python 3.13 "$tc" + $UV pip install --python "$tc/bin/python" ninja \ + nvidia-cuda-nvcc==13.0.88 nvidia-cuda-cccl==13.0.85 nvidia-cuda-crt==13.0.88 nvidia-nvvm==13.0.88 \ + nvidia-cuda-runtime==13.0.96 nvidia-cuda-profiler-api==13.0.85 nvidia-cublas==13.1.0.3 \ + nvidia-cusparse==12.6.3.3 nvidia-cusolver==12.0.4.66 nvidia-curand==10.4.0.35 nvidia-cufft==12.0.0.61 \ + nvidia-nvjitlink==13.0.88 + export CUDA_HOME="$tc/lib/python3.13/site-packages/nvidia/cu13" + (cd "$CUDA_HOME/lib" && for f in libcudart libcublas libcublasLt libcusparse libcusolver libcurand libcufft libnvJitLink; do + ln -sfn "$(ls $f.so.* | head -1)" "$f.so" + done) + ln -sfn lib "$CUDA_HOME/lib64" + export PATH="$CUDA_HOME/bin:$tc/bin:$PATH" + [ -d "$ART/apex/.git" ] || git clone https://github.com/NVIDIA/apex.git "$ART/apex" + git -C "$ART/apex" checkout --detach 575968bc1f9127ccd61003a681472a83af4ff1a1 + # One compile job per 4 GiB of free host memory, at most one per CPU; built for this GPU's architecture. + local jobs + jobs=$(awk -v n="$(nproc)" '/MemAvailable/ {m = int($2 / 4194304); print (m < n ? m : n)}' /proc/meminfo) + TORCH_CUDA_ARCH_LIST="$(nvidia-smi --query-gpu=compute_cap --format=csv,noheader | head -1)" \ + APEX_CPP_EXT=1 APEX_CUDA_EXT=1 MAX_JOBS="$jobs" \ + $UV pip install --python "$SFT_PY" --no-build-isolation "$ART/apex" +} + +download() { + say "Qwen3-VL-8B-Instruct" + "$EVAL_VENV/bin/hf" download Qwen/Qwen3-VL-8B-Instruct --revision 0c351dd01ed87e9c1b53cbc748cba10e6187ff3b \ + --local-dir "$SFT_MODEL_PATH" + if [ "$SMOKE" = 1 ]; then + echo "smoke: RoboVQA comes from the skill's bundled clips" + return + fi + say "RoboVQA (3.5 GB)" + # The evaluation environment's hf (huggingface_hub 0.36.2) reads every pattern after --include. + "$EVAL_VENV/bin/hf" download Tianli/robovqa --repo-type dataset --revision eb5819490f0d9ec18a79774233d4b1c9ebd52eea \ + --include "videos/*" "json/*" "LICENSE.txt" --local-dir "$SRC" +} + +data() { + say "split: 80/10/10 by clip, one question per clip" + "$EVAL_PY" "$SKILL_DIR/scripts/prepare_robovqa.py" split --src "$SRC" --out "$DATA" + say "frames: each training clip's evaluation frames, written losslessly" + CUDA_VISIBLE_DEVICES= LD_LIBRARY_PATH="$EVAL_LIBS" \ + "$EVAL_PY" "$SKILL_DIR/scripts/prepare_robovqa.py" frames --out "$DATA" + say "verify: the training stack sees exactly the evaluator's frames" + CUDA_VISIBLE_DEVICES= HF_HUB_OFFLINE=1 LD_LIBRARY_PATH="$SFT_LIBS" \ + "$SFT_PY" "$SKILL_DIR/scripts/prepare_robovqa.py" verify --out "$DATA" --model "$SFT_MODEL_PATH" +} + +preflight() { + for check in --validate-manifest --validate-rows; do + say "sft $check" + CUDA_VISIBLE_DEVICES= PYTHONPATH=. IMAGINAIRE_OUTPUT_ROOT="$RUN/preflight" LD_LIBRARY_PATH="$SFT_LIBS" \ + "$SFT_PY" -m cosmos_framework.posttrain.entrypoints.sft --config "$CONFIG" "$check" + done +} + +score() { # score + LD_LIBRARY_PATH="$EVAL_LIBS" "$EVAL_PY" "$SKILL_DIR/scripts/eval_robovqa.py" --model "$1" \ + --data "$DATA/hf/test.jsonl" --ref "$DATA/samples_test.jsonl" --out "$EVAL/$2" +} + +before() { + say "score the untouched model on the held-out clips" + score "$SFT_MODEL_PATH" before +} + +train() { + local free + free=$(nvidia-smi --query-gpu=memory.free --format=csv,noheader,nounits | head -1) + if [ "$free" -lt 150000 ]; then + echo "${free} MiB free on the GPU; training the 8B model needs about 150 GB (measured peak 143,566 MiB)" >&2 + exit 1 + fi + say "train: $STEPS updates of 64 clips on one GPU" + PYTHONPATH=. IMAGINAIRE_OUTPUT_ROOT="$RUN" LD_LIBRARY_PATH="$SFT_LIBS" \ + "$SFT_PY" -m torch.distributed.run --standalone --nproc_per_node=1 \ + -m cosmos_framework.posttrain.entrypoints.sft --config "$CONFIG" -- rl.max_steps="$STEPS" rl.save_every="$STEPS" +} + +export_hf() { + local ckpt + ckpt=$(ls -d "$RUN"/posttrain_sft/qwen3_vl_8b_robovqa/seed0/*/checkpoint_artifacts/iter_* | sort | tail -1) + say "export $ckpt" + CUDA_VISIBLE_DEVICES= PYTHONPATH=. LD_LIBRARY_PATH="$SFT_LIBS" \ + "$SFT_PY" -m cosmos_framework.posttrain.checkpoint.tools.materialize_hf \ + --ckpt-dir "$ckpt" --base "$SFT_MODEL_PATH" --out "$EXPORT" --expect-tensors 750 --min-tensors 750 +} + +after() { + say "score the fine-tuned model the same way" + score "$EXPORT" after +} + +case "$STEP" in + setup | download | data | preflight | before | train | after) "$STEP" 2>&1 | tee "$ART/logs/$STEP$TAG.log" ;; + export) export_hf 2>&1 | tee "$ART/logs/export$TAG.log" ;; + *) + echo "usage: run.sh [--smoke]" >&2 + exit 2 + ;; +esac From 105cadde480d1fd1ef3ab70786cb5da47992e297 Mon Sep 17 00:00:00 2001 From: sbabamohamad Date: Thu, 29 Oct 2026 11:32:51 -0700 Subject: [PATCH 04/14] skills: say what each long step is doing while it runs The apex build printed nothing for about ten minutes; setup now says how long it takes and streams one line per compiled source file, and fails with a clear message when the build does not produce amp_C. The model download and the training launch also announce what to expect before their quiet stretches. Signed-off-by: sbabamohamad --- .../scripts/run.sh | 23 +++++++++++++------ 1 file changed, 16 insertions(+), 7 deletions(-) diff --git a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh index 4b4556797..92188e9d4 100755 --- a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh +++ b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh @@ -71,7 +71,7 @@ setup() { fi command -v uvx >/dev/null || curl -LsSf https://astral.sh/uv/install.sh | sh - say "SFT environment (docs/posttrain/guides/sft.md section 2)" + say "SFT environment (docs/posttrain/guides/sft.md section 2): about 270 packages, 1 to 2 minutes" # --python 3.13: with --project, uv does not read the checkout's .python-version. # --group cu130-aarch64 adds NVIDIA NPP on aarch64. --inexact keeps apex when this is re-run. local arch=() @@ -83,7 +83,7 @@ setup() { say "apex (the recipe's FusedAdam optimizer)" if "$SFT_PY" -c "import torch, amp_C" 2>/dev/null; then echo "already installed"; else build_apex; fi - say "evaluation environment (plain Transformers, the stack the scores were measured with)" + say "evaluation environment (plain Transformers, the stack the scores were measured with): 1 to 2 minutes" $UV venv --allow-existing --python 3.12 "$EVAL_VENV" $UV pip install --python "$EVAL_PY" torch==2.10.0 torchvision==0.25.0 torchcodec==0.10.0 \ --index-url https://download.pytorch.org/whl/cu130 @@ -115,17 +115,24 @@ build_apex() { [ -d "$ART/apex/.git" ] || git clone https://github.com/NVIDIA/apex.git "$ART/apex" git -C "$ART/apex" checkout --detach 575968bc1f9127ccd61003a681472a83af4ff1a1 # One compile job per 4 GiB of free host memory, at most one per CPU; built for this GPU's architecture. - local jobs + local jobs arch jobs=$(awk -v n="$(nproc)" '/MemAvailable/ {m = int($2 / 4194304); print (m < n ? m : n)}' /proc/meminfo) - TORCH_CUDA_ARCH_LIST="$(nvidia-smi --query-gpu=compute_cap --format=csv,noheader | head -1)" \ - APEX_CPP_EXT=1 APEX_CUDA_EXT=1 MAX_JOBS="$jobs" \ - $UV pip install --python "$SFT_PY" --no-build-isolation "$ART/apex" + arch=$(nvidia-smi --query-gpu=compute_cap --format=csv,noheader | head -1) + echo "compiling apex for compute capability $arch with $jobs parallel jobs: about 10 minutes on a GB300." + echo "Each line below is one source file being compiled; the last step links the extensions." + # -v shows the compiler's progress line by line instead of a silent "Building apex" for ten minutes. + TORCH_CUDA_ARCH_LIST="$arch" APEX_CPP_EXT=1 APEX_CUDA_EXT=1 MAX_JOBS="$jobs" \ + $UV pip install -v --python "$SFT_PY" --no-build-isolation "$ART/apex" 2>&1 | + grep --line-buffered -E '^\[[0-9]+/[0-9]+\]|Successfully built|Installed|error|Error' | + sed -u 's/^\[\([0-9]*\)\/\([0-9]*\)\].*\/\([^/ ]*\.\(cu\|cpp\)\).*/[\1\/\2] \3/' + "$SFT_PY" -c "import torch, amp_C" 2>/dev/null || { echo "apex did not build; the full log is above" >&2; exit 1; } } download() { - say "Qwen3-VL-8B-Instruct" + say "Qwen3-VL-8B-Instruct (17 GB, 16 files; one progress bar per file)" "$EVAL_VENV/bin/hf" download Qwen/Qwen3-VL-8B-Instruct --revision 0c351dd01ed87e9c1b53cbc748cba10e6187ff3b \ --local-dir "$SFT_MODEL_PATH" + du -sh "$SFT_MODEL_PATH" | sed 's/^/model on disk: /' if [ "$SMOKE" = 1 ]; then echo "smoke: RoboVQA comes from the skill's bundled clips" return @@ -173,6 +180,8 @@ train() { exit 1 fi say "train: $STEPS updates of 64 clips on one GPU" + echo "loading the model and warming up: the first 'step 1/$STEPS' line appears after about 3 minutes," + echo "then one line per update, about a minute each. Watch train/loss_avg fall from about 1.0." PYTHONPATH=. IMAGINAIRE_OUTPUT_ROOT="$RUN" LD_LIBRARY_PATH="$SFT_LIBS" \ "$SFT_PY" -m torch.distributed.run --standalone --nproc_per_node=1 \ -m cosmos_framework.posttrain.entrypoints.sft --config "$CONFIG" -- rl.max_steps="$STEPS" rl.save_every="$STEPS" From f3bbc0033eabc12f67fa5cd7fdb26dc90635c116 Mon Sep 17 00:00:00 2001 From: sbabamohamad Date: Thu, 29 Oct 2026 12:52:13 -0700 Subject: [PATCH 05/14] skills: show the apex compiler output during setup Building apex through pip hid the compiler for the whole ten minutes (the progress filter matched nothing). setup now builds the wheel with setup.py directly, streams the compiler output and saves it to logs/apex-build.log, caps the parallel jobs at 32, and fails with the log's path when no wheel comes out. Signed-off-by: sbabamohamad --- .../scripts/run.sh | 24 ++++++++++++------- 1 file changed, 15 insertions(+), 9 deletions(-) diff --git a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh index 92188e9d4..04f210852 100755 --- a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh +++ b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh @@ -114,18 +114,24 @@ build_apex() { export PATH="$CUDA_HOME/bin:$tc/bin:$PATH" [ -d "$ART/apex/.git" ] || git clone https://github.com/NVIDIA/apex.git "$ART/apex" git -C "$ART/apex" checkout --detach 575968bc1f9127ccd61003a681472a83af4ff1a1 - # One compile job per 4 GiB of free host memory, at most one per CPU; built for this GPU's architecture. + # One compile job per 4 GiB of free host memory, at most 32 and at most one per CPU; built for this GPU. local jobs arch - jobs=$(awk -v n="$(nproc)" '/MemAvailable/ {m = int($2 / 4194304); print (m < n ? m : n)}' /proc/meminfo) + jobs=$(awk -v n="$(nproc)" '/MemAvailable/ {m = int($2 / 4194304); if (m > n) m = n; if (m > 32) m = 32; print m}' /proc/meminfo) arch=$(nvidia-smi --query-gpu=compute_cap --format=csv,noheader | head -1) echo "compiling apex for compute capability $arch with $jobs parallel jobs: about 10 minutes on a GB300." - echo "Each line below is one source file being compiled; the last step links the extensions." - # -v shows the compiler's progress line by line instead of a silent "Building apex" for ten minutes. - TORCH_CUDA_ARCH_LIST="$arch" APEX_CPP_EXT=1 APEX_CUDA_EXT=1 MAX_JOBS="$jobs" \ - $UV pip install -v --python "$SFT_PY" --no-build-isolation "$ART/apex" 2>&1 | - grep --line-buffered -E '^\[[0-9]+/[0-9]+\]|Successfully built|Installed|error|Error' | - sed -u 's/^\[\([0-9]*\)\/\([0-9]*\)\].*\/\([^/ ]*\.\(cu\|cpp\)\).*/[\1\/\2] \3/' - "$SFT_PY" -c "import torch, amp_C" 2>/dev/null || { echo "apex did not build; the full log is above" >&2; exit 1; } + echo "The compiler's own output follows (one block per source file); it is also saved to $ART/logs/apex-build.log." + # pip's own build output stays hidden; running setup.py directly shows the compiler as it works. + $UV pip install --python "$SFT_PY" wheel # bdist_wheel needs it; not in the SFT lock + (cd "$ART/apex" && TORCH_CUDA_ARCH_LIST="$arch" APEX_CPP_EXT=1 APEX_CUDA_EXT=1 MAX_JOBS="$jobs" \ + "$SFT_PY" setup.py bdist_wheel --dist-dir "$ART/apex-wheel" 2>&1 | tee "$ART/logs/apex-build.log") || true + local wheel + wheel=$(ls -t "$ART"/apex-wheel/apex-*.whl 2>/dev/null | head -1) + if [ -z "$wheel" ]; then + echo "apex did not build; see $ART/logs/apex-build.log (the last lines are above)" >&2 + exit 1 + fi + $UV pip install --python "$SFT_PY" "$wheel" + "$SFT_PY" -c "import torch, amp_C" 2>/dev/null || { echo "apex installed but does not import; see $ART/logs/apex-build.log" >&2; exit 1; } } download() { From 259a4f88bba81424910d9f8949d9633c3fee73af Mon Sep 17 00:00:00 2001 From: sbabamohamad Date: Thu, 29 Oct 2026 13:25:39 -0700 Subject: [PATCH 06/14] skills: one line per stage on screen, full output in logs Every step of run.sh now prints one line per stage (the label, then "done" with the elapsed time) and a one-line result, and writes each stage's full output to artifacts/qwen3-vl-robovqa/logs/.log. A failing stage prints FAILED, the last 30 lines of its log and the log's path, and the step stops. Training shows only the step lines and the end marker; the framework's full log goes to the file. Signed-off-by: sbabamohamad --- .../cosmos3-qwen3-vl-robovqa-sft/SKILL.md | 43 +++-- .../scripts/run.sh | 165 ++++++++++++------ 2 files changed, 137 insertions(+), 71 deletions(-) diff --git a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/SKILL.md b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/SKILL.md index 08bdfd035..be485b736 100644 --- a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/SKILL.md +++ b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/SKILL.md @@ -31,9 +31,11 @@ bit-reproducible): exact match 0.548 before, 0.632 and 0.620 after. Details: directory with `pyproject.toml`). The script sets its own paths and pins, so each step can run in a fresh shell; run the steps in order. [`scripts/run.sh`](scripts/run.sh) shows every command a step runs, with comments. 2. **Pick the variant from the GPU** (table below). The smoke variant adds `--smoke` to every step. -3. **Check each step's expected output** (under the step) before the next step. Each step also saves its output in - the `logs` folder of the artifacts directory, which the script creates as artifacts/qwen3-vl-robovqa under the - repository root. +3. **Check each step's expected output** (under the step) before the next step. The screen shows one line per + stage (`done` with its time) and a one-line result; each stage's full output goes to a file in the `logs` folder + of the artifacts directory, which the script creates as artifacts/qwen3-vl-robovqa under the repository root + (`-smoke` suffix for the smoke variant). A failing stage prints `FAILED`, the last 30 lines of its log and the + log's path, and the step stops there. 4. **After a failure**, read "Things not obvious from the docs" and the troubleshooting section of [`references/measured-run.md`](references/measured-run.md) before changing a command. @@ -71,10 +73,11 @@ The bundled directory has the layout of the full download. Six clips split 4 / 0 bash .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh setup ``` -Checks the GPU, the driver (CUDA 13), `git`, `ffmpeg` and `g++`; builds the SFT environment from -`docs/posttrain/guides/sft.md` section 2; builds apex (the recipe's optimizer) unless it is already installed; -builds the evaluation environment (plain Transformers). Expected last lines: -`SFT environment: 2.11.0+cu130 5.8.1 apex ok` and `evaluation environment: 2.10.0+cu130 4.57.6`. +Four stages: the host check (CUDA 13 driver, `git`, `ffmpeg`, `g++`, uv), the SFT environment from +`docs/posttrain/guides/sft.md` section 2, apex (the recipe's optimizer, compiled from source unless already +installed; the long stage, about 10 minutes), and the evaluation environment (plain Transformers). Expected last +lines: `SFT environment: 2.11.0+cu130 5.8.1 apex ok` and `evaluation environment: 2.10.0+cu130 4.57.6`. The +compiler's output is in the logs folder as apex-build.log. ### 2. download — model and data @@ -83,6 +86,7 @@ bash .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh download ``` Qwen3-VL-8B-Instruct (17 GB) and RoboVQA (3.5 GB, skipped with `--smoke`), both at fixed revisions. No token needed. +Expected last line: `model on disk: 17G dataset: 9999 clips` (`6 clips` with `--smoke`). ### 3. data — training rows (about 25 minutes) @@ -92,10 +96,10 @@ bash .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh data [`scripts/prepare_robovqa.py`](scripts/prepare_robovqa.py) splits the clips 80/10/10 with one question per clip and byte-identical clips kept in one split, writes each training clip's evaluation frames losslessly so training sees -exactly what the evaluator scores, and checks every file in the training stack. Expected: counts -`train 7998, val 1001, test 1000`; md5 lines equal to their seed-0 references; `"rows": 7998`; `"all_ok": true`. -With `--smoke`: counts `train 4, val 0, test 2` (the md5 references apply to the full download only), -`"rows": 4`, `"all_ok": true`. +exactly what the evaluator scores, and checks every file in the training stack. Expected last lines: the counts +`"train": 7998, "val": 1001, "test": 1000` with every `leakage_check` value 0, and `frame check: "all_ok": true, +"bad": 0`. With `--smoke`: `"train": 4, "val": 0, "test": 2`. The split log also prints the md5 of the samples +files next to their seed-0 references; they match on the full download only. ### 4. preflight — the recipe's CPU checks (about 13 minutes) @@ -104,9 +108,9 @@ bash .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh preflight ``` `--validate-manifest` and `--validate-rows` of the SFT entrypoint with the recipe -`cosmos_framework/posttrain/configs/sft_qwen3_vl_8b_robovqa.py`. Expected: +`cosmos_framework/posttrain/configs/sft_qwen3_vl_8b_robovqa.py`. Expected last line: `train: rows 7998 | loss_tokens == assistant_tokens on 7998/7998 | longest 4113 <= 10240 | failed 0` (`rows 4` with -`--smoke`). The line `train: dp_world not checked` that follows is expected outside torchrun. +`--smoke`). The log's `train: dp_world not checked` line is expected outside torchrun. ### 5. before — score the untouched model (about 14 minutes) @@ -125,10 +129,10 @@ bash .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh train ``` One GPU through torchrun, 125 updates of 64 clips (2 with `--smoke`). It stops before starting when the GPU has less -than 150,000 MiB free. The first update appears after about three minutes, then one `step N/125` line about every -minute; the loss starts near 1.0 and ends near 0.5. Expected end: `SFT DONE {'start': 0, 'final_step': 125, ...}`. -Agents: start it detached and keep the log, for example -`nohup bash .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh train > train.out 2>&1 &`. +than 150,000 MiB free. The screen shows one line per update, `step N/125 train/loss_avg … … s`, the first after +about three minutes and then about one a minute; the loss starts near 1.0 and ends near 0.5. Expected end: +`SFT DONE {'start': 0, 'final_step': 125, ...}`. The framework's full log is train.log in the logs folder. Agents: start it +detached, for example `nohup bash .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh train > train.out 2>&1 &`. ### 7. export — Hugging Face format (about 2 minutes) @@ -136,7 +140,7 @@ Agents: start it detached and keep the log, for example bash .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh export ``` -The framework's HF materializer on the CPU, in 5 GB shards. Expected: +The framework's HF materializer on the CPU, in 5 GB shards. Expected last line: `[materialize] OK -> .../export (16.3 GiB, 750 tensors in 4 shards, 8 sidecars identical to base, ...)`. ### 8. after — score the fine-tuned model (about 11 minutes) @@ -150,7 +154,8 @@ weights were not loaded. With `--smoke` the score is only a check that the expor ## Verify the output -Every step saves its console output in the artifacts directory's `logs` folder, named after the step (with a +Every stage's full output is in the artifacts directory's `logs` folder, named after the stage (`setup-sft-env`, +`apex-build`, `data-split`, `data-frames`, `data-verify`, `preflight-rows`, `before`, `train`, `export`, `after`, …; `-smoke` suffix for the smoke variant). A run is correct when: | Step | Correct when | diff --git a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh index 04f210852..2e61747e7 100755 --- a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh +++ b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh @@ -52,26 +52,42 @@ mkdir -p "$ART/logs" say() { printf '\n==> %s\n' "$*"; } -setup() { - say "GPU and driver" - nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv,noheader - local cuda +# stage