diff --git a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/SKILL.md b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/SKILL.md new file mode 100644 index 00000000..61125c34 --- /dev/null +++ b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/SKILL.md @@ -0,0 +1,247 @@ +--- +name: cosmos3-qwen3-vl-robovqa-sft +description: > + Fine-tune Qwen3-VL-8B-Instruct, a vision-language model from outside the + Cosmos family, with the Cosmos Framework posttrain SFT path on robot-video + question answering (RoboVQA), on one GPU, and score it before and after with + the same evaluator. Use when the user asks to "fine-tune Qwen3-VL with Cosmos + Framework", "train a non-Cosmos model or a VLM in the framework", "run SFT on + robot videos / RoboVQA", "reproduce the Qwen3-VL RoboVQA result", or wants a + worked post-training example with a measured before and after (environments, + data preparation, preflight, training, Hugging Face export, evaluation). Not + for Cosmos3 checkpoints (use cosmos3-post-training) and not for LoRA (the SFT + path trains every parameter). +--- + +# Qwen3-VL-8B SFT on RoboVQA (one GPU) + +One job: full-parameter SFT of `Qwen/Qwen3-VL-8B-Instruct` on the RoboVQA training clips through the framework's +posttrain SFT path, then the same scorer on 1,000 held-out clips before and after. Example from the held-out set: +"put the box on the table Q: possible right now?", reference `no`: the base model answers `yes`, the fine-tuned +model `no`. + +Measured on one NVIDIA GB300 (aarch64, driver 595, CUDA 13.0), two runs at seed 0 (training is not +bit-reproducible): exact match 0.548 before, 0.632 and 0.620 after. Details: +[`references/measured-run.md`](references/measured-run.md). + +## For an agent + +The whole job is the eight steps below, run in order, each one command. A request that fits in one line: + +> Using the `cosmos3-qwen3-vl-robovqa-sft` skill in this Cosmos Framework checkout: set up the environment, +> download the model and RoboVQA, prepare the training rows, run the preflight, score Qwen3-VL-8B before +> training, fine-tune for one epoch, export, and score after. Report both scores. + +Rules: run the steps in the order listed; after each step compare the last lines with the step's "Expected"; +stop at the first `FAILED` and report the step, the stage and the log path it printed; do not change the +commands or the recipe. With `--smoke` on every step the same request is a 25-minute check on six bundled clips +and 2 updates (the scores then mean nothing). + +## Read this first + +1. **One command per step.** Every step is + `bash .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh `, run from the repository root (the + directory with `pyproject.toml`). The script sets its own paths and pins, so each step can run in a fresh shell; + run the steps in order. [`scripts/run.sh`](scripts/run.sh) shows every command a step runs, with comments. +2. **Pick the variant from the GPU** (table below). The smoke variant adds `--smoke` to every step. +3. **Check each step's expected output** (under the step) before the next step. The screen shows one line per + stage (`done` with its time) and a one-line result; each stage's full output goes to a file in the `logs` folder + of the artifacts directory, which the script creates as artifacts/qwen3-vl-robovqa under the repository root + (`-smoke` suffix for the smoke variant). A failing stage prints `FAILED`, the last 30 lines of its log and the + log's path, and the step stops there. +4. **After a failure**, read "Things not obvious from the docs" and the troubleshooting section of + [`references/measured-run.md`](references/measured-run.md) before changing a command. + +## Your own data or another Qwen3-VL size + +- **Another dataset.** The recipe reads `SFT_DATA_ROOT` (the `data` step writes it as `artifacts/qwen3-vl-robovqa/data`). + Point it at your own folder and skip the `data` step. The folder needs `train.jsonl` in the framework's SFT + conversation format (`docs/posttrain/guides/sft.md` section 3: one conversation per line, a `video` or `image` + part plus the question in the user turn, the answer in the assistant turn, media paths relative to the file), and + for the `before` and `after` steps `hf/test.jsonl` and `samples_test.jsonl` as `scripts/prepare_robovqa.py` writes + them. The scorer's metrics (normalised exact match, token F1, yes/no accuracy) suit short-answer VQA; other tasks + need their own scorer. Run `preflight` first: it refuses rows the recipe cannot train on. +- **Another Qwen3-VL size.** The model is one line of the recipe, `cosmos_framework/posttrain/configs/sft_qwen3_vl_8b_robovqa.py` + (`SFT_MODEL_PATH`, which the `download` step sets to the 8B snapshot). Copy the config, change the model, and + pass the copy as `CONFIG` to `run.sh`; the framework's SFT path supports the Qwen3-VL and Qwen3-VL-MoE + families (`docs/posttrain/guides/sft.md` section 1). Memory scales with size: 8B peaks at 143,566 MiB on one + GPU. Only the 8B on RoboVQA is measured. +- **Another model family** (Llama, Gemma, InternVL, ...): not through this path; the SFT builder accepts the Qwen + families only. + +## When to use this skill + +- Fine-tune Qwen3-VL-8B on RoboVQA with Cosmos Framework, or reproduce the before-and-after numbers above +- A worked, measured example of the posttrain SFT path for a Hugging Face VLM on video question answering +- Score a Hugging Face Qwen3-VL model on the RoboVQA held-out clips (steps `before` and `after`) +- Not for Cosmos3 models: `../cosmos3-post-training/SKILL.md`. Not for LoRA: the SFT path does not support it + (`docs/posttrain/guides/sft.md` section 1). Environment errors: `../cosmos3-env-troubleshoot/SKILL.md` + +## Variants and hardware + +| Variant | Data | GPU | Time | Status | +| ----------------- | ------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------- | ----------------------------------------- | +| Full run | RoboVQA, 7,998 training clips, 125 updates | one GPU with at least 150,000 MiB free for `train` (torch peaks at 143,566 MiB reserved); every other step fits a smaller GPU | about 3.5 h, training 2 h 05 to 2 h 14 | verified twice on one GB300, October 2026 | +| Smoke (`--smoke`) | the six bundled clips, 2 updates | `setup` to `before` on any GPU that holds the 8B model in bfloat16 (16.3 GiB); `train`, `export`, `after` need the same 150 GB GPU | environments plus about 15 minutes | measured by this skill's eval | + +Disk: about 110 GB for the full run, about 75 GB for the smoke variant (environments, model, checkpoint, export). +x86_64 hosts take the `nvidia-npp` wheel from PyPI in `setup`; that route is unverified. On a host with several +GPUs the script uses the one with the most memory for every GPU step (it sets `CUDA_VISIBLE_DEVICES` unless the +caller already did). The bundled clips are Git LFS objects: `download --smoke` fetches them when the clone holds +pointer files, and asks for `git-lfs` to be installed when it is missing. + +## Bundled assets + +| Asset | Path | Used by | +| ----------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | -------------------- | +| Six RoboVQA clips with their source rows (Tianli/robovqa, Apache-2.0, revision `eb58194`) | `assets/robovqa/videos/`, `assets/robovqa/json/train/sample.json`, `assets/robovqa/LICENSE.txt` | every `--smoke` step | + +The bundled directory has the layout of the full download. Six clips split 4 / 0 / 2 (train / val / test). + +## Steps + +### 1. setup — environments (about 12 minutes) + +```bash +bash .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh setup +``` + +Four stages: the host check (CUDA 13 driver, `git`, `ffmpeg`, `g++`, uv; run as root it installs the missing tools +with apt-get itself, otherwise it prints the apt-get command and stops), the SFT environment from +`docs/posttrain/guides/sft.md` section 2, apex (the recipe's optimizer, compiled from source unless already +installed; the long stage, about 10 minutes), and the evaluation environment (plain Transformers). Expected last +lines: `SFT environment: 2.11.0+cu130 5.8.1 apex ok` and `evaluation environment: 2.10.0+cu130 4.57.6`. The +compiler's output is in the logs folder as apex-build.log (apex-build-smoke.log with `--smoke`). + +Agents: this is one command that runs for about 12 minutes. Run it as one shell call with a long timeout, or start it +detached the way step 6 shows for `train` (`nohup bash .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh setup +> setup.out 2>&1 &`) and wait for its last lines before the next step. If your shell tool stops it early, run the same +command again: the finished environments and the apex build are kept, so the re-run continues from where it stopped. + +### 2. download — model and data + +```bash +bash .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh download +``` + +Qwen3-VL-8B-Instruct (17 GB) and RoboVQA (3.5 GB), both at fixed revisions; no token needed. With `--smoke` the +dataset stage instead makes sure the six bundled clips are real videos, not Git LFS pointers. Expected last line: +`model on disk: 17G dataset: 9999 clips` (`6 clips` with `--smoke`). + +### 3. data — training rows (about 25 minutes) + +```bash +bash .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh data +``` + +[`scripts/prepare_robovqa.py`](scripts/prepare_robovqa.py) splits the clips 80/10/10 with one question per clip and +byte-identical clips kept in one split, writes each training clip's evaluation frames losslessly so training sees +exactly what the evaluator scores, and checks every file in the training stack. Expected last lines: the counts +`"train": 7998, "val": 1001, "test": 1000` with every `leakage_check` value 0, and `frame check: "all_ok": true, +"bad": 0`. With `--smoke`: `"train": 4, "val": 0, "test": 2`. The split log also prints the md5 of the samples +files next to their seed-0 references; they match on the full download only. + +### 4. preflight — the recipe's CPU checks (about 13 minutes) + +```bash +bash .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh preflight +``` + +`--validate-manifest` and `--validate-rows` of the SFT entrypoint with the recipe +`cosmos_framework/posttrain/configs/sft_qwen3_vl_8b_robovqa.py`. Expected last line: +`train: rows 7998 | loss_tokens == assistant_tokens on 7998/7998 | longest 4113 <= 10240 | failed 0` (`rows 4` with +`--smoke`). The log's `train: dp_world not checked` line is expected outside torchrun. + +### 5. before — score the untouched model (about 14 minutes) + +```bash +bash .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh before +``` + +[`scripts/eval_robovqa.py`](scripts/eval_robovqa.py) answers every held-out clip greedily and scores normalised exact +match, token F1 and yes/no accuracy. Expected last line: `exact 0.548 token-F1 0.676 yes/no 0.918 (n=1000)` +(`n=2` with `--smoke`). + +### 6. train — fine-tune (about 2 hours) + +```bash +bash .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh train +``` + +One GPU through torchrun, 125 updates of 64 clips (2 with `--smoke`). It stops before starting when the GPU has less +than 150,000 MiB free. The screen shows one line per update, `step N/125 train/loss_avg … … s`, the first after +about three minutes and then about one a minute; the loss starts near 1.0 and ends near 0.5. Expected end: +`SFT DONE {'start': 0, 'final_step': 125, ...}`. The framework's full log is train.log in the logs folder. Agents: start it +detached, for example `nohup bash .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh train > train.out 2>&1 &`. + +### 7. export — Hugging Face format (about 2 minutes) + +```bash +bash .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh export +``` + +The framework's HF materializer on the CPU, in 5 GB shards. Expected last line: +`[materialize] OK -> .../export (16.3 GiB, 750 tensors in 4 shards, 8 sidecars identical to base, ...)`. + +### 8. after — score the fine-tuned model (about 11 minutes) + +```bash +bash .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh after +``` + +Expected: exact match near 0.62 to 0.63 (0.632 and 0.620 in the two measured runs). Near 0.55 means the fine-tuned +weights were not loaded. With `--smoke` the score is only a check that the export loads and answers. + +## Verify the output + +Every stage's full output is in the artifacts directory's `logs` folder, named after the stage (`setup-sft-env`, +`apex-build`, `data-split`, `data-frames`, `data-verify`, `preflight-rows`, `before`, `train`, `export`, `after`, …; +`-smoke` suffix for the smoke variant; a re-run of a stage keeps the previous attempt's log as `.log.prev`). A run +is correct when: + +| Step | Correct when | +| ------------- | --------------------------------------------------------------------------------------------------------- | +| data | the split manifest's `leakage_check` values are all 0 and verify prints `"all_ok": true, "bad": 0` | +| preflight | the train gate line ends in `failed 0` | +| before, after | the overall line reports `n=1000` (`n=2` with `--smoke`) and the scores file shows no missing predictions | +| train | the log has a `step N/125` line per update and `SFT DONE` | +| export | `[materialize] OK` with 750 tensors | + +## Reference files + +| File | What it holds | +| --------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | +| [`scripts/run.sh`](scripts/run.sh) | every step's commands, with comments | +| [`references/measured-run.md`](references/measured-run.md) | the two measured runs: versions, timings, memory, loss, example answers, troubleshooting | +| `cosmos_framework/posttrain/configs/sft_qwen3_vl_8b_robovqa.py` | the recipe, each field with its reason | +| [`scripts/prepare_robovqa.py`](scripts/prepare_robovqa.py) | split, frames, verify | +| [`scripts/eval_robovqa.py`](scripts/eval_robovqa.py) | evaluator, scorer and the shared frame decoder | + +## Things not obvious from the docs + +- **`could not read the CUDA version from the nvidia-smi banner`**: the host check reads the banner's `CUDA Version:` + label (drivers up to R595) or `CUDA UMD Version:` (R615 and later, where the old label is deprecated). Another label + is a format this skill has not seen: check that `nvidia-smi -q | grep -i 'cuda.*version'` reports 13 or newer, then + open an issue with the banner line. +- **`libnppicc.so.13: cannot open shared object file`**: TorchCodec needs NVIDIA NPP; `setup` installs it and every + step that decodes video points `LD_LIBRARY_PATH` at it. Do not clear `LD_LIBRARY_PATH`. +- **`ImportError: libc10.so` importing `amp_C`**: import `torch` first; apex's extensions link against torch. +- **apex gone after re-running `uv sync` by hand**: an exact sync uninstalls packages the lock does not list; + `setup` syncs with `--inexact`. +- **SFT environment on Python 3.12**: TorchCodec then cannot find NPP; `setup` pins `--python 3.13`. +- **`[materialize] FATAL ... model.safetensors.index.json missing`**: a framework checkout older than the 5 GB shard + change; Transformers 5 writes a model under 50 GB as one file without the index the verifier requires. +- **`hf download` fetched no videos**: a newer `hf`; `download` uses the evaluation environment's (huggingface_hub + 0.36.2). +- **The first training update takes about two minutes and nothing prints before it**: model load and warm-up. +- **Running `train` or `export` a second time in the same checkout**: the framework refuses to start at step 0 in a + run directory that already holds a checkpoint, and the materializer refuses an existing output. Both steps move + the earlier result aside under a timestamped name and say so; nothing is deleted. + +## Related skills + +| Skill | When to use | +| -------------------------------------- | ---------------------------------------------------------- | +| `../cosmos3-setup/SKILL.md` | framework installation and checkpoint downloads in general | +| `../cosmos3-post-training/SKILL.md` | post-training Cosmos3 models | +| `../cosmos3-env-troubleshoot/SKILL.md` | import, CUDA, shared-library and download errors | diff --git a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/LICENSE.txt b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/LICENSE.txt new file mode 100644 index 00000000..4bbb8433 --- /dev/null +++ b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/LICENSE.txt @@ -0,0 +1,202 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright 2023 Google LLC + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/json/train/sample.json b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/json/train/sample.json new file mode 100644 index 00000000..60d2da9e --- /dev/null +++ b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/json/train/sample.json @@ -0,0 +1,6 @@ +{"uid":"12991928966725870442","text":"\u003ctask:affordance:discriminative:discrete:False\u003e\nput the box on the table Q: possible right now? \u003cPRED\u003eA: \u003cPRED:ANSWER\u003e\u003cPRED:DISCRETE\u003e\u003cPRED:BINARY\u003eno\u003c/PRED:BINARY\u003e\u003c/PRED:DISCRETE\u003e\u003c/PRED:ANSWER\u003e\n\u003c/PRED\u003e","video":"10015443583468041553.mp4"} +{"uid":"12974193841542607726","text":"\u003ctask:remaining5_planning_with_context20:freeform\u003e\ncurrent goal is: Please place the Peas and strawberries into the bowl. last 20 steps: 1- place peas in the bowl 2- place pea in the bowl 3- place pea in the bowl 4- place pea in the bowl 5- place pea in the bowl 6- place pea in the bowl 7- place pea in the bowl 8- place peas in the bowl 9- place pea in the bowl 10- place pea in the bowl 11- place peas in the bowl 12- place pea in the bowl 13- place pea in the bowl 14- place strawberry in the bowl Q: next 5 steps? \u003cPRED\u003eA: 1- place strawberry in the bowl 2- place strawberry in the bowl 3- place strawberry in the bowl 4- place strawberry in the bowl 5- place strawberry in the bowl\n\u003c/PRED\u003e\u003ctask:affordance:generative:positive:freeform\u003e\nQ: what action is possible right now? \u003cPRED\u003eA: place strawberry in the bowl\n\u003c/PRED\u003e\u003ctask:planning:freeform\u003e\ncurrent goal is: Please place the Peas and strawberries into the bowl Q: immediate next step? \u003cPRED\u003eA: place strawberry in the bowl\n\u003c/PRED\u003e\u003ctask:immediate_planning_with_context20:freeform\u003e\ncurrent goal is: Please place the Peas and strawberries into the bowl. last 20 steps: 1- place peas in the bowl 2- place pea in the bowl 3- place pea in the bowl 4- place pea in the bowl 5- place pea in the bowl 6- place pea in the bowl 7- place pea in the bowl 8- place peas in the bowl 9- place pea in the bowl 10- place pea in the bowl 11- place peas in the bowl 12- place pea in the bowl 13- place pea in the bowl 14- place strawberry in the bowl Q: immediate next step? \u003cPRED\u003eA: place strawberry in the bowl\n\u003c/PRED\u003e\u003ctask:future_prediction:freeform\u003e\nQ: what is likely to happen next? \u003cPRED\u003eA: place strawberry in the bowl\n\u003c/PRED\u003e\u003ctask:affordance:discriminative:discrete:True\u003e\nplace strawberry in the bowl Q: possible right now? \u003cPRED\u003eA: \u003cPRED:ANSWER\u003e\u003cPRED:DISCRETE\u003e\u003cPRED:BINARY\u003eyes\u003c/PRED:BINARY\u003e\u003c/PRED:DISCRETE\u003e\u003c/PRED:ANSWER\u003e\n\u003c/PRED\u003e","video":"215018504990484899.mp4"} +{"uid":"17235415102297408195","text":"\u003ctask:immediate_planning_with_context20:freeform\u003e\ncurrent goal is: please use the brush to sweep everything off the desks. last 20 steps: 1- pick the brush from the table 2- sweep the table 3- move towards the popcorn 4- move all the objects to left 5- sweep the blue chip packet 6- move your arm upwards 7- drop sweep the table 8- sweep the candy's on the floor Q: immediate next step? \u003cPRED\u003eA: done\n\u003c/PRED\u003e\u003ctask:planning:freeform\u003e\ncurrent goal is: please use the brush to sweep everything off the desks. Q: immediate next step? \u003cPRED\u003eA: done\n\u003c/PRED\u003e\u003ctask:success:discrete:True\u003e\nsweep the candy's on the floor Q: satisfied? \u003cPRED\u003eA: \u003cPRED:ANSWER\u003e\u003cPRED:DISCRETE\u003e\u003cPRED:BINARY\u003eyes\u003c/PRED:BINARY\u003e\u003c/PRED:DISCRETE\u003e\u003c/PRED:ANSWER\u003e\n\u003c/PRED\u003e\u003ctask:remaining5_planning_with_context20:freeform\u003e\ncurrent goal is: please use the brush to sweep everything off the desks. last 20 steps: 1- pick the brush from the table 2- sweep the table 3- move towards the popcorn 4- move all the objects to left 5- sweep the blue chip packet 6- move your arm upwards 7- drop sweep the table 8- sweep the candy's on the floor Q: next 5 steps? \u003cPRED\u003eA: 1- done\n\u003c/PRED\u003e\u003ctask:past_description:freeform\u003e\nQ: what just happened? \u003cPRED\u003eA: sweep the candy's on the floor\n\u003c/PRED\u003e","video":"294863854345401650.mp4"} +{"uid":"5447814328030553394","text":"\u003ctask:past_description:freeform\u003e\nQ: what just happened? \u003cPRED\u003eA: Put the orange pen in the penholder\n\u003c/PRED\u003e\u003ctask:success:discrete:True\u003e\nPut the orange pen in the penholder Q: satisfied? \u003cPRED\u003eA: \u003cPRED:ANSWER\u003e\u003cPRED:DISCRETE\u003e\u003cPRED:BINARY\u003eyes\u003c/PRED:BINARY\u003e\u003c/PRED:DISCRETE\u003e\u003c/PRED:ANSWER\u003e\n\u003c/PRED\u003e","video":"8217664127105486720.mp4"} +{"uid":"8365152957462210433","text":"\u003ctask:affordance:generative:positive:freeform\u003e\nQ: what action is possible right now? \u003cPRED\u003eA: drop the brown sachet in the plastic glass\n\u003c/PRED\u003e\u003ctask:future_prediction:freeform\u003e\nQ: what is likely to happen next? \u003cPRED\u003eA: drop the brown sachet in the plastic glass\n\u003c/PRED\u003e\u003ctask:affordance:discriminative:discrete:True\u003e\ndrop the brown sachet in the plastic glass Q: possible right now? \u003cPRED\u003eA: \u003cPRED:ANSWER\u003e\u003cPRED:DISCRETE\u003e\u003cPRED:BINARY\u003eyes\u003c/PRED:BINARY\u003e\u003c/PRED:DISCRETE\u003e\u003c/PRED:ANSWER\u003e\n\u003c/PRED\u003e\u003ctask:remaining5_planning_with_context20:freeform\u003e\ncurrent goal is: please place all the coffee condiments on the table back into the cup. last 20 steps: none Q: next 5 steps? \u003cPRED\u003eA: 1- drop the brown sachet in the plastic glass 2- pick a white packet from the table 3- drop the white packet in the plastic glass 4- pick the sachets from the table 5- drop the green packet in the plastic glass\n\u003c/PRED\u003e\u003ctask:planning:freeform\u003e\ncurrent goal is: please place all the coffee condiments on the table back into the cup. Q: immediate next step? \u003cPRED\u003eA: drop the brown sachet in the plastic glass\n\u003c/PRED\u003e\u003ctask:success:discrete:False\u003e\ndrop the brown sachet in the plastic glass Q: satisfied? \u003cPRED\u003eA: \u003cPRED:ANSWER\u003e\u003cPRED:DISCRETE\u003e\u003cPRED:BINARY\u003eno\u003c/PRED:BINARY\u003e\u003c/PRED:DISCRETE\u003e\u003c/PRED:ANSWER\u003e\n\u003c/PRED\u003e\u003ctask:immediate_planning_with_context20:freeform\u003e\ncurrent goal is: please place all the coffee condiments on the table back into the cup. last 20 steps: none Q: immediate next step? \u003cPRED\u003eA: drop the brown sachet in the plastic glass\n\u003c/PRED\u003e","video":"14989971876239671643.mp4"} +{"uid":"7845415266276637754","text":"\u003ctask:success:discrete:True\u003e\npick up green packet Q: satisfied? \u003cPRED\u003eA: \u003cPRED:ANSWER\u003e\u003cPRED:DISCRETE\u003e\u003cPRED:BINARY\u003eyes\u003c/PRED:BINARY\u003e\u003c/PRED:DISCRETE\u003e\u003c/PRED:ANSWER\u003e\n\u003c/PRED\u003e\u003ctask:success:discrete:True\u003e\npick up the white and green packet Q: satisfied? \u003cPRED\u003eA: \u003cPRED:ANSWER\u003e\u003cPRED:DISCRETE\u003e\u003cPRED:BINARY\u003eyes\u003c/PRED:BINARY\u003e\u003c/PRED:DISCRETE\u003e\u003c/PRED:ANSWER\u003e\n\u003c/PRED\u003e\u003ctask:past_description:freeform\u003e\nQ: what just happened? \u003cPRED\u003eA: pick up the white and green packet\n\u003c/PRED\u003e\u003ctask:past_description:freeform\u003e\nQ: what just happened? \u003cPRED\u003eA: pick up green packet\n\u003c/PRED\u003e","video":"7344627454494368553.mp4"} diff --git a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/10015443583468041553.mp4 b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/10015443583468041553.mp4 new file mode 100644 index 00000000..6cc85034 --- /dev/null +++ b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/10015443583468041553.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:3ca3182bc1d9a2a2d5fa3841391f25146ea76b1cbab5b03b798da74c881aa7af +size 577530 diff --git a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/14989971876239671643.mp4 b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/14989971876239671643.mp4 new file mode 100644 index 00000000..d5a5874c --- /dev/null +++ b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/14989971876239671643.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:642b7563e0873440f41c948e3d03c6a4f17dad9018c8b65940bfa65023a35b4d +size 145116 diff --git a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/215018504990484899.mp4 b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/215018504990484899.mp4 new file mode 100644 index 00000000..0d3f75f6 --- /dev/null +++ b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/215018504990484899.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:8fc70190471ea3aa8f23cbed37dfa7d86ce2ce56057a2527bf691b4b625f6056 +size 439805 diff --git a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/294863854345401650.mp4 b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/294863854345401650.mp4 new file mode 100644 index 00000000..e55dfe59 --- /dev/null +++ b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/294863854345401650.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:e37f54760a02dba33fd72689d2a15a29cdd61e5017fc25619b9a83050ea3b5c2 +size 191368 diff --git a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/7344627454494368553.mp4 b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/7344627454494368553.mp4 new file mode 100644 index 00000000..5b266c3f --- /dev/null +++ b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/7344627454494368553.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:ad81c70c5b243bc81c86e95909da1f70a55f5bb57d8b0fb7f750f6a521dfefab +size 205041 diff --git a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/8217664127105486720.mp4 b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/8217664127105486720.mp4 new file mode 100644 index 00000000..9850a3c3 --- /dev/null +++ b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/assets/robovqa/videos/8217664127105486720.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:03d91fe4c54f2d01341c37797dcf4b65c4efbe43b1121175de4c9a1d831294db +size 205723 diff --git a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/eval/eval.config b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/eval/eval.config new file mode 100644 index 00000000..97294fc1 --- /dev/null +++ b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/eval/eval.config @@ -0,0 +1,50 @@ +{ + "evals": [ + { + "id": "smoke-rows-and-preflight", + "prompt": "On this machine's single GPU, prepare the Qwen3-VL-8B RoboVQA SFT training rows from the skill's bundled RoboVQA clips (assets/robovqa) and run the framework's SFT preflight (--validate-manifest and --validate-rows) on them with the skill's recipe. Save the preflight output as outputs/preflight.log and the frame check summary printed by the verify step as outputs/verify.json.", + "expected_outcome": "The bundled clips are split 4/0/2, the 4 training files pass the frame check, and the SFT preflight's train gate line in outputs/preflight.log reports rows 4 and failed 0.", + "assertions": [ + "The SFT environment was built with the skill's uv sync command (uv 0.11.29, Python 3.13, the sft and cu130-runtime groups) and apex imports in it", + "prepare_robovqa.py split, frames and verify ran on assets/robovqa, and outputs/verify.json reports rows 4 with all_ok true", + "outputs/preflight.log contains a 'train: rows 4' gate line ending in 'failed 0'", + "The skill's documented workflow executed without errors on the first attempt", + "result_summary.md reports the split counts, the gate line and the wall time of each step" + ], + "checks": [ + {"path": "outputs/preflight.log", "min_bytes": 100}, + {"path": "outputs/verify.json", "min_bytes": 40}, + {"path": "result_summary.md", "min_bytes": 150}, + {"path": "transcript.md", "min_bytes": 300} + ], + "stages": ["prerequisites", "environments", "model download", "training rows", "preflight", "verify + summarize"] + }, + { + "id": "smoke-zero-shot-score", + "prompt": "Score Qwen/Qwen3-VL-8B-Instruct (revision 0c351dd01ed87e9c1b53cbc748cba10e6187ff3b) zero-shot with the skill's RoboVQA evaluator on the held-out clips of the skill's bundled RoboVQA sample (assets/robovqa), and copy the evaluator's preds.jsonl and scores.json to outputs/.", + "expected_outcome": "outputs/preds.jsonl holds 2 greedy answers and outputs/scores.json the exact match, token F1 and yes/no accuracy over the 2 held-out clips, produced by eval_robovqa.py in the evaluation environment.", + "assertions": [ + "The evaluation environment was built as the skill documents (Python 3.12, torch 2.10.0+cu130, transformers 4.57.6, torchcodec 0.10.0) and the evaluator ran from it", + "The bundled clips were split with prepare_robovqa.py and the evaluator read the resulting hf/test.jsonl and samples_test.jsonl", + "outputs/preds.jsonl has 2 rows and outputs/scores.json reports n = 2 with no missing predictions", + "The skill's documented workflow executed without errors on the first attempt", + "result_summary.md reports the scores, the wall time and the GPU memory observed" + ], + "checks": [ + {"path": "outputs/preds.jsonl", "min_bytes": 50}, + {"path": "outputs/scores.json", "min_bytes": 50}, + {"path": "result_summary.md", "min_bytes": 150}, + {"path": "transcript.md", "min_bytes": 300} + ], + "stages": ["prerequisites", "evaluation environment", "model download", "split", "evaluation", "verify + summarize"] + } + ], + "credentials": ["HF_TOKEN"], + "gpus": 1, + "budget": {"minutes": 75, "tokens": 3000000}, + "setup": { + "repo": "https://github.com/NVIDIA/cosmos-framework.git", + "dest": "/app/repo", + "src_mount": "/opt/cosmos-src" + } +} diff --git a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/references/measured-run.md b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/references/measured-run.md new file mode 100644 index 00000000..8878b328 --- /dev/null +++ b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/references/measured-run.md @@ -0,0 +1,94 @@ +# Measured runs: Qwen3-VL-8B SFT on RoboVQA + +Two runs of this recipe at seed 0, each on one NVIDIA GB300 (256,703 MiB, aarch64, driver 595.91.07, CUDA 13.0), +directly on the host without a container, in October 2026, with the framework at commit +`9a70e64d2ad7ff1417484d12d10fb121b270baaf`. The runs used the commands of [`../scripts/run.sh`](../scripts/run.sh) +from an earlier version of that script, with the same training config, data preparation and evaluator, and wrote the +HF export through the materializer with 5 GB shards (the change that ships with this skill). + +## Versions + +| Environment | Python | Key packages | +| ------------------------------------------------------------------------------------ | ------- | ---------------------------------------------------------------------------------------------------- | +| SFT (`docker/posttrain/environment`, groups `sft`, `cu130-runtime`, `cu130-aarch64`) | 3.13.15 | torch 2.11.0+cu130, transformers 5.8.1, torchcodec 0.11, apex `575968bc` built from source | +| Evaluation | 3.12 | torch 2.10.0+cu130, torchvision 0.25.0, torchcodec 0.10.0, transformers 4.57.6, nvidia-npp 13.0.0.50 | + +Pins: `Qwen/Qwen3-VL-8B-Instruct` at `0c351dd01ed87e9c1b53cbc748cba10e6187ff3b`; `Tianli/robovqa` at +`eb5819490f0d9ec18a79774233d4b1c9ebd52eea`. + +## Results on the 1,000 held-out clips + +| | exact match | token-F1 | yes/no accuracy (597 questions) | +| ------------------------------- | ----------- | -------- | ------------------------------- | +| Qwen3-VL-8B-Instruct, zero-shot | 0.548 | 0.676 | 0.918 | +| run 1, after one epoch | 0.632 | 0.809 | 0.953 | +| run 2, after one epoch | 0.620 | 0.798 | 0.946 | + +- The zero-shot answers are deterministic: both runs (two independent setups) gave the same 1,000 answers. +- Training is not bit-reproducible: gradient norms differ from update 2 on, and the two fine-tuned models give the + same answer on 763 of the 1,000 clips. A Hugging Face + TRL full SFT of the same data and epoch on 8 H100s gave + 0.633 ± 0.007 over three seeds (outside this skill; a reference point only). +- Part of the gain is the model learning RoboVQA's answer style (short, lower-case, imperative), which exact match + rewards; yes/no accuracy, which reads only the first yes or no, rose from 0.918 to 0.946 and 0.953. + +## Time and memory + +| Step | Run 1 | Run 2 | Notes | +| ------------------------------------------ | ------- | ------- | ------------------------------------------------------------------------------------- | +| 1 + model download | 704 s | 684 s | the apex build takes about 10 minutes of it | +| 2 + 3 (dataset download and training rows) | 1,367 s | 1,424 s | 48 processes; 7,998 frame files, 35,369,466,594 bytes, 119,827 frames | +| 4 preflight | 753 s | not run | CPU | +| 5 before | 802 s | 856 s | the evaluator alone 670 s and 761 s | +| 6 train | 7,512 s | 8,064 s | median 60.4 s and 63.1 s per update (updates 2 to 125); update 1 took 107 s and 134 s | +| 7 export | 121 s | 119 s | CPU; 16.3 GiB, 750 tensors in 4 shards | +| 8 after | 605 s | 657 s | the evaluator alone 526 s and 557 s | + +- Training memory: torch peak 138,981.7 MiB allocated and 143,566 to 143,716 MiB reserved; nvidia-smi peaked + 144,357 MiB above the idle baseline (run 1). +- GPU time: about 2.5 GPU-hours (steps 5, 6 and 8). +- Disk: about 110 GB in all; frames 33 GB, model 17 GB, checkpoint 17 GB, export 17 GB. + +## Training loss (run 2) + +| update | 1 | 2 | 3 | 4 | 5 | ... | 120 | 121 | 122 | 123 | 124 | 125 | +| ---------------- | ----- | ----- | ----- | ----- | ----- | --- | ----- | ----- | ----- | ----- | ----- | ----- | +| `train/loss_avg` | 1.041 | 0.630 | 0.815 | 0.602 | 0.616 | ... | 0.698 | 0.695 | 0.543 | 0.544 | 0.499 | 0.482 | + +Run 1 had 1.041 at update 1 and 0.487 at update 125; the mean of updates 1 to 10 was 0.717 and of 116 to 125 0.591. + +## Example answers (run 1) + +The first held-out clip of each task family, fixed before any fine-tuned answer existed. ✓ / ✗ is the scorer's +normalised exact match. + +| task | question | reference | zero-shot | fine-tuned | +| ----------------- | --------------------------------------------------------------------------------------------- | ------------------------------------------ | -------------------------------------------------------- | ----------------------------------- | +| success | Put the orange pen in the penholder Q: satisfied? | yes | ✓ yes | ✓ yes | +| affordance | put the box on the table Q: possible right now? | no | ✗ yes | ✓ no | +| planning | current goal is: Please place the Peas and strawberries into the bowl Q: immediate next step? | place strawberry in the bowl | ✗ Put the strawberry in the paper bowl | ✗ put the strawberry in the bowl | +| past_description | Q: what just happened? | pick up the white and green packet | ✗ The robot arm picked up a green packet from the shelf. | ✗ place the packet in the shelf | +| future_prediction | Q: what is likely to happen next? | drop the brown sachet in the plastic glass | ✗ Place the brown sugar packet in the glass | ✗ put the brown packet in the glass | + +## Troubleshooting + +Each entry: symptom, cause, check, fix. Paths are relative to the repository root. + +- **`libnppicc.so.13: cannot open shared object file`** when TorchCodec loads. Cause: NVIDIA NPP is not on the + library path. Check: `ls docker/posttrain/environment/.venv-sft/lib/python3.13/site-packages/nvidia/cu13/lib | grep nppicc`. + Fix: re-run `run.sh setup`, which installs `nvidia-npp` in both environments; `run.sh` puts it on + `LD_LIBRARY_PATH` for every command that decodes video. Never clear `LD_LIBRARY_PATH`. +- **`ImportError: libc10.so` importing `amp_C`.** Cause: apex's extensions link against torch's libraries. Check: + `docker/posttrain/environment/.venv-sft/bin/python -c "import torch, amp_C"`. Fix: import `torch` first. +- **apex missing after running `uv sync` by hand.** Cause: an exact sync uninstalls packages the lock does not list. + Check: the same import. Fix: re-run `run.sh setup` (it syncs with `--inexact` and rebuilds apex only when missing). +- **The SFT environment was built on Python 3.12.** Cause: with `--project`, uv does not read the checkout's + `.python-version` and takes any interpreter in range. Check: `docker/posttrain/environment/.venv-sft/bin/python + --version`. Fix: delete `docker/posttrain/environment/.venv-sft` and re-run `run.sh setup`, which pins + `--python 3.13`. +- **`[materialize] FATAL ... model.safetensors.index.json missing`.** Cause: Transformers 5.8.1 writes a model under + 50 GB as one file without an index, and the materializer's verifier requires the index. Fix: a framework checkout + that writes 5 GB shards (the change that ships with this skill), then re-run `run.sh export`. +- **The training step prints nothing for about three minutes.** Cause: model load and the first update's warm-up. + Check: `nvidia-smi` shows the process growing toward 140 GB. Fix: none; wait for `step 1/125`. +- **`--validate-rows` prints `train: dp_world not checked`.** Expected outside torchrun; one GPU divides every row + count. diff --git a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/eval_robovqa.py b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/eval_robovqa.py new file mode 100755 index 00000000..eebc7bbb --- /dev/null +++ b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/eval_robovqa.py @@ -0,0 +1,239 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 +r"""Answer and score the RoboVQA held-out clips with a Hugging Face Qwen3-VL checkpoint. + +Run it on the untouched model (the "before" score) and on the SFT export (the "after" score) with the same +arguments; only ``--model`` and ``--out`` change. It runs on plain Transformers in its own environment (Transformers +4.57.6, TorchCodec 0.10, one GPU; the skill's ``SKILL.md`` step 1), the stack the recipe's numbers were measured with. + +- Frames: :func:`decode_video` keeps every round(fps / 2)-th frame from frame 0, at most 32 (evenly thinned when there + are more), and returns the effective frame rate. ``prepare_robovqa.py frames`` writes the training clips with this + same function, so training and evaluation see identical frames. +- Generation: greedy, at most ``--max-new-tokens`` (64) new tokens, batches of ``--batch`` (4), bfloat16, SDPA. +- Metrics, overall and per task, deterministic (no judge): ``exact`` = normalised exact match (lower case, punctuation + to spaces, articles dropped, spaces collapsed); ``f1`` = token F1 of the normalised strings; ``binary_acc`` = on the + yes/no tasks (tags containing ``:discrete:``) the first ``yes`` or ``no`` in the prediction against the reference, + none counting as wrong. + +Writes ``preds.jsonl``, ``scores.json`` and ``eval_meta.json`` under ``--out`` and ends with the overall line, for +example ``exact 0.548 token-F1 0.676 yes/no 0.918 (n=1000)``. ``--score-only`` re-scores an existing ``preds.jsonl``. + + python .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/eval_robovqa.py --model $MODEL \ + --data $DATA/hf/test.jsonl --ref $DATA/samples_test.jsonl --out $OUT +""" + +from __future__ import annotations + +import argparse +import collections +import json +import os +import re +import string +import sys +import time +from collections.abc import Sequence +from typing import Any + +MAX_FRAMES = 32 +TARGET_FPS = 2.0 +_ARTICLES = {"a", "an", "the"} + + +def decode_video(path: str) -> tuple[list[Any], float]: + """The clip's frames as PIL images and their effective frame rate: every round(fps / 2)-th frame from frame 0, at + most ``MAX_FRAMES`` (evenly thinned when there are more). ``prepare_robovqa.py frames`` writes the training clips + with this function, so the frames the model trains on are the frames it is scored on.""" + import numpy as np + from PIL import Image + from torchcodec.decoders import VideoDecoder + + decoder = VideoDecoder(path) + n = decoder.metadata.num_frames or 0 + fps = float(decoder.metadata.average_fps or 0.0) or 30.0 + stride = max(1, int(round(fps / TARGET_FPS))) + indices = list(range(0, n, stride)) + if len(indices) > MAX_FRAMES: + step = len(indices) / MAX_FRAMES + indices = [indices[int(i * step)] for i in range(MAX_FRAMES)] + frames = decoder.get_frames_at(indices=indices).data.permute(0, 2, 3, 1).contiguous().numpy().astype(np.uint8) + return [Image.fromarray(frame) for frame in frames], len(indices) / n * fps + + +def normalize(text: str) -> str: + """Lower case, punctuation to spaces, articles dropped, whitespace collapsed.""" + text = "".join(" " if ch in string.punctuation else ch for ch in text.lower()) + return " ".join(word for word in text.split() if word not in _ARTICLES) + + +def token_f1(prediction: str, reference: str) -> float: + """Token-level F1 between the normalised prediction and reference (SQuAD style).""" + pred_tokens, ref_tokens = normalize(prediction).split(), normalize(reference).split() + if not pred_tokens or not ref_tokens: + return float(pred_tokens == ref_tokens) + common = sum((collections.Counter(pred_tokens) & collections.Counter(ref_tokens)).values()) + if common == 0: + return 0.0 + precision, recall = common / len(pred_tokens), common / len(ref_tokens) + return 2 * precision * recall / (precision + recall) + + +def first_yes_no(text: str) -> str | None: + """The first standalone ``yes`` or ``no`` in the text, lower-cased; None when there is neither.""" + match = re.search(r"\b(yes|no)\b", text.lower()) + return match.group(1) if match else None + + +def score(refs: Sequence[dict[str, Any]], preds: dict[str, str]) -> dict[str, dict[str, float]]: + """Per task tag and overall (``__all__``): ``n``, then each metric's mean and its count (``_n``). A clip + without a prediction scores as an empty answer.""" + per_task: dict[str, list[dict[str, float]]] = collections.defaultdict(list) + for ref in refs: + prediction, task = preds.get(ref["id"], ""), ref["task"] + row = { + "exact": float(normalize(prediction) == normalize(ref["answer"])), + "f1": token_f1(prediction, ref["answer"]), + } + if ":discrete:" in task: + row["binary_acc"] = float(first_yes_no(prediction) == ref["answer"].lower()) + per_task[task].append(row) + per_task["__all__"].append(row) + out: dict[str, dict[str, float]] = {} + for task, rows in per_task.items(): + agg: dict[str, float] = {"n": len(rows)} + for key in ("exact", "f1", "binary_acc"): + values = [row[key] for row in rows if key in row] + if values: + agg[key] = sum(values) / len(values) + agg[key + "_n"] = len(values) + out[task] = agg + return out + + +def write_scores(ref_path: str, pred_path: str, out_path: str) -> dict[str, Any]: + """Score ``preds.jsonl`` against ``samples_test.jsonl`` and write ``scores.json``.""" + with open(ref_path) as handle: + refs = [json.loads(line) for line in handle if line.strip()] + preds: dict[str, str] = {} + with open(pred_path) as handle: + for line in handle: + if line.strip(): + row = json.loads(line) + preds[row["id"]] = row.get("prediction", "") + missing = [ref["id"] for ref in refs if ref["id"] not in preds] + result: dict[str, Any] = score(refs, preds) + result["__meta__"] = {"n_ref": len(refs), "n_pred": len(preds), "missing_predictions": len(missing)} + with open(out_path, "w", newline="\n") as handle: + json.dump(result, handle, indent=2) + print(json.dumps({"__all__": result["__all__"], "__meta__": result["__meta__"]})) + return result + + +def summary_line(result: dict[str, Any]) -> str: + overall = result["__all__"] + return "exact %.3f token-F1 %.3f yes/no %.3f (n=%d)" % ( + overall["exact"], + overall["f1"], + overall["binary_acc"], + overall["n"], + ) + + +def generate(model_path: str, data_path: str, out_dir: str, *, batch: int, max_new_tokens: int, limit: int) -> int: + """Greedy answers for every test row into ``/preds.jsonl``; returns the number of rows answered.""" + import torch + from transformers import AutoProcessor, Qwen3VLForConditionalGeneration + from transformers.video_utils import VideoMetadata + + processor = AutoProcessor.from_pretrained(model_path) + processor.tokenizer.padding_side = "left" + model = Qwen3VLForConditionalGeneration.from_pretrained( + model_path, dtype=torch.bfloat16, attn_implementation="sdpa" + ) + model = model.cuda().eval() + with open(data_path) as handle: + rows = [json.loads(line) for line in handle if line.strip()] + if limit: + rows = rows[:limit] + start = time.time() + with open(os.path.join(out_dir, "preds.jsonl"), "w", newline="\n") as out: + for i in range(0, len(rows), batch): + chunk = rows[i : i + batch] + texts, videos, metadata = [], [], [] + for row in chunk: + frames, fps = decode_video(row["video_path"]) + user = [message for message in row["messages"] if message["role"] == "user"][0] + content = [ + {"type": "video"} if item["type"] == "video" else {"type": "text", "text": item["text"]} + for item in user["content"] + ] + texts.append( + processor.apply_chat_template( + [{"role": "user", "content": content}], tokenize=False, add_generation_prompt=True + ) + ) + videos.append(frames) + metadata.append( + VideoMetadata( + total_num_frames=len(frames), + fps=fps, + duration=len(frames) / fps, + frames_indices=list(range(len(frames))), + video_backend="torchcodec", + ) + ) + inputs = processor( + text=texts, + videos=videos, + videos_kwargs={"do_sample_frames": False, "video_metadata": metadata}, + return_tensors="pt", + padding=True, + ).to("cuda") + with torch.inference_mode(): + generated = model.generate(**inputs, max_new_tokens=max_new_tokens, do_sample=False) + answers = processor.batch_decode(generated[:, inputs["input_ids"].shape[1] :], skip_special_tokens=True) + for row, answer in zip(chunk, answers): + reference = row["messages"][-1]["content"][0]["text"] + out.write( + json.dumps( + {"id": row["id"], "prediction": answer.strip(), "reference": reference, "task": row["task"]} + ) + + "\n" + ) + out.flush() + if (i // batch) % 20 == 0: + print(f"{i + len(chunk)}/{len(rows)} {time.time() - start:.0f}s", flush=True) + return len(rows) + + +def main(argv: Sequence[str] | None = None) -> int: + parser = argparse.ArgumentParser(description="Answer and score the RoboVQA held-out clips (see the module doc).") + parser.add_argument("--model", help="Hugging Face Qwen3-VL directory to evaluate (not needed with --score-only)") + parser.add_argument("--data", help="hf/test.jsonl written by prepare_robovqa.py split") + parser.add_argument("--ref", required=True, help="samples_test.jsonl written by prepare_robovqa.py split") + parser.add_argument("--out", required=True, help="output directory: preds.jsonl, scores.json, eval_meta.json") + parser.add_argument("--batch", type=int, default=4, help="clips per generate call (default 4)") + parser.add_argument("--max-new-tokens", type=int, default=64, help="answer length cap (default 64)") + parser.add_argument("--limit", type=int, default=0, help="score the first N clips only (0 = all)") + parser.add_argument("--score-only", action="store_true", help="re-score /preds.jsonl without generating") + args = parser.parse_args(argv) + os.makedirs(args.out, exist_ok=True) + if not args.score_only: + if not args.model or not args.data: + parser.error("--model and --data are required unless --score-only is given") + start = time.time() + n = generate( + args.model, args.data, args.out, batch=args.batch, max_new_tokens=args.max_new_tokens, limit=args.limit + ) + elapsed = time.time() - start + with open(os.path.join(args.out, "eval_meta.json"), "w", newline="\n") as handle: + json.dump({"eval_seconds": elapsed, "n": n, "model": args.model}, handle) + print(f"EVAL_DONE {n} samples in {elapsed:.0f}s") + result = write_scores(args.ref, os.path.join(args.out, "preds.jsonl"), os.path.join(args.out, "scores.json")) + print(summary_line(result)) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/prepare_robovqa.py b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/prepare_robovqa.py new file mode 100755 index 00000000..ca42c497 --- /dev/null +++ b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/prepare_robovqa.py @@ -0,0 +1,422 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 +r"""Write the RoboVQA split and the SFT training rows of ``cosmos_framework/posttrain/configs/sft_qwen3_vl_8b_robovqa.py``. + +Source: ``Tianli/robovqa`` (Apache-2.0) at revision ``eb5819490f0d9ec18a79774233d4b1c9ebd52eea``, downloaded with +``videos/*``, ``json/*`` and ``LICENSE.txt``. Only rows whose mp4 is present are usable (9,999 clips). Each row's text +holds one to nine task segments ``\n Q: ? A: \n``; discrete answers +are wrapped in ``yes|no``. + +Three steps, each in the environment named (the skill's ``SKILL.md`` has the commands): + +1. ``split`` (any Python 3.10+, CPU): shuffle the sorted video ids with ``random.Random(seed)`` and cut 80/10/10 into + train / val / test; move every member of a group of byte-identical clips to the split of the group's smallest id; + keep one question per clip (``random.Random(seed + int(md5(video id)[:8], 16))`` picks the segment). Writes + ``splits/``, ``samples_{train,val,test}.jsonl`` (``{id, video, task, question, answer, split, uid}``), + ``hf/{train,val,test}.jsonl`` (the source ``video_path`` and the conversation) and ``split_manifest.json`` + (counts, task histogram, moved duplicates, leakage check), and prints the md5 of the train and test samples. +2. ``frames`` (evaluation environment, CPU): for every training clip, decode the frames the evaluator decodes + (``eval_robovqa.decode_video``) and write them losslessly (PNG in MOV, rgb24) at the effective frame rate under + ``frames/``; then ``train.jsonl``, the SFT rows the recipe reads, each pointing at its file, and + ``train.jsonl.frames.jsonl`` (per clip: frame count, rate, size, sha256 of the raw frames). +3. ``verify`` (SFT environment, CPU): for every training file, check that the training stack decodes exactly the + recorded frames at the recorded frame rate, and that the Qwen3-VL processor called the way the SFT packer calls it + (the path, ``do_sample_frames=False``) gives the same ``input_ids``, video grid and pixel values as the evaluator's + call (the frames plus their ``VideoMetadata``). Writes ``train.jsonl.verify.jsonl``; exits 1 unless every row passes. + +Run it as a file (it imports ``eval_robovqa`` from its own directory): + + python .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/prepare_robovqa.py split --src $SRC --out $DATA +""" + +from __future__ import annotations + +import argparse +import collections +import glob +import hashlib +import json +import os +import random +import re +import sys +from collections.abc import Sequence +from fractions import Fraction +from pathlib import Path +from typing import Any + +SPLITS = ("train", "val", "test") +_SEGMENT = re.compile(r"]+)>\n(.*?)A: (.*?)\n", re.S) +_BINARY = re.compile(r"(yes|no)") +# md5 of the samples files that `split --seed 0` writes from the pinned revision (the recipe's measured split). +SEED0_MD5 = { + "samples_train.jsonl": "0b3739a50f5b715a6f4a5ab11081807a", + "samples_test.jsonl": "1a9b1924eddcf34cf26ce7ffbcb69184", +} + + +def _default_workers() -> int: + return min(os.cpu_count() or 1, 48) + + +def _md5_file(path: str | Path) -> str: + with open(path, "rb") as handle: + return hashlib.md5(handle.read()).hexdigest() + + +def parse_segments(text: str) -> list[dict[str, str]]: + """The task segments of one RoboVQA row: ``{task, question, answer}``, a binary answer reduced to yes/no.""" + out = [] + for tag, question, answer in _SEGMENT.findall(text): + binary = _BINARY.search(answer) + out.append({"task": tag, "question": question.strip(), "answer": binary.group(1) if binary else answer.strip()}) + return out + + +# ------------------------------------------------------------------------------------------------ split -------------- +def split(src: Path, out: Path, *, seed: int, fractions: Sequence[float]) -> dict[str, Any]: + """Write the split, the samples, the conversation rows and the manifest; return the manifest.""" + out.mkdir(parents=True, exist_ok=True) + present = {path.name for path in (src / "videos").glob("*.mp4")} + rows = [] + for json_path in sorted(glob.glob(str(src / "json" / "*" / "*.json"))): + with open(json_path) as handle: + for line in handle: + if line.strip(): + row = json.loads(line) + if row["video"] in present: + rows.append(row) + by_video: dict[str, list[dict[str, Any]]] = {} + for row in rows: + by_video.setdefault(row["video"], []).append(row) + if not all(len(group) == 1 for group in by_video.values()): + raise ValueError("expected one source row per video") + ids = sorted(by_video) + random.Random(seed).shuffle(ids) + n = len(ids) + n_train, n_val = int(n * fractions[0]), int(n * fractions[1]) + splits = {"train": ids[:n_train], "val": ids[n_train : n_train + n_val], "test": ids[n_train + n_val :]} + # Co-locate byte-identical clips: every member of an md5 group goes to the split of its lexicographically smallest + # id (deterministic), so identical videos never straddle splits. + md5_of = {video: _md5_file(src / "videos" / video) for video in ids} + groups: dict[str, list[str]] = collections.defaultdict(list) + for video in ids: + groups[md5_of[video]].append(video) + where = {video: name for name, videos in splits.items() for video in videos} + moved = [] + for group in groups.values(): + if len(group) > 1: + target = where[min(group)] + for video in group: + if where[video] != target: + moved.append((video, where[video], target)) + where[video] = target + splits = {name: [video for video in ids if where[video] == name] for name in splits} + + (out / "splits").mkdir(exist_ok=True) + samples: dict[str, list[dict[str, Any]]] = {} + histogram: collections.Counter[tuple[str, str]] = collections.Counter() + unparsed = 0 + for name, videos in splits.items(): + (out / "splits" / f"{name}_videos.txt").write_text("\n".join(videos) + "\n", newline="\n") + samples[name] = [] + for video in videos: + segments = parse_segments(by_video[video][0]["text"]) + if not segments: + unparsed += 1 + continue + pick = random.Random(seed + int(hashlib.md5(video.encode()).hexdigest()[:8], 16)).choice(segments) + samples[name].append( + { + "id": f"{name}_{video[:-4]}", + "video": video, + "task": pick["task"], + "question": pick["question"], + "answer": pick["answer"], + "split": name, + "uid": by_video[video][0]["uid"], + } + ) + histogram[(name, pick["task"])] += 1 + with open(out / f"samples_{name}.jsonl", "w", newline="\n") as handle: + for sample in samples[name]: + handle.write(json.dumps(sample) + "\n") + + # Leakage check: video ids, video bytes and source uids are disjoint across splits. + split_of_md5: dict[str, set[str]] = {} + for name, videos in splits.items(): + for video in videos: + split_of_md5.setdefault(md5_of[video], set()).add(name) + train, val, test = (set(splits[name]) for name in SPLITS) + uids = {name: {sample["uid"] for sample in samples[name]} for name in SPLITS} + + (out / "hf").mkdir(exist_ok=True) + for name in SPLITS: + with open(out / "hf" / f"{name}.jsonl", "w", newline="\n") as handle: + for sample in samples[name]: + conversation = [ + { + "role": "user", + "content": [ + {"type": "video", "video": "video_0"}, + {"type": "text", "text": sample["question"]}, + ], + }, + {"role": "assistant", "content": [{"type": "text", "text": sample["answer"]}]}, + ] + video_path = str((src / "videos" / sample["video"]).resolve()) + row = {"id": sample["id"], "video_path": video_path, "task": sample["task"], "messages": conversation} + handle.write(json.dumps(row) + "\n") + + manifest = { + "source": "Tianli/robovqa (Apache-2.0 LICENSE.txt)", + "seed": seed, + "fractions": list(fractions), + "rule": "shuffle sorted video ids with random.Random(seed); 80/10/10; one Q/A per video", + "usable_videos": n, + "unparsed_rows": unparsed, + "counts": {name: len(samples[name]) for name in SPLITS}, + "task_hist": {f"{name}|{task}": count for (name, task), count in sorted(histogram.items())}, + "duplicates_moved": moved, + "leakage_check": { + "video_id_overlap": len(train & val | train & test | val & test), + "identical_video_bytes_across_splits": sum(1 for names in split_of_md5.values() if len(names) > 1), + "uid_overlap_train_vs_val_test": len(uids["train"] & uids["test"]) + len(uids["train"] & uids["val"]), + }, + } + with open(out / "split_manifest.json", "w", newline="\n") as handle: + json.dump(manifest, handle, indent=2) + return manifest + + +# ------------------------------------------------------------------------------------------------ frames ------------- +def _write_clip_frames(job: tuple[dict[str, Any], str]) -> dict[str, Any]: + """Decode one clip the evaluator's way and write the frames losslessly at the effective frame rate.""" + import av + from eval_robovqa import decode_video # this script's directory: the evaluator's own decoder + + row, frames_dir = job + frames, fps_eff = decode_video(row["video_path"]) + rate = Fraction(fps_eff).limit_denominator(10**6) + clip = os.path.splitext(os.path.basename(row["video_path"]))[0] + path = os.path.join(frames_dir, f"{clip}.mov") + tmp = path + ".tmp.mov" + tick = Fraction(rate.denominator, rate.numerator) # one codec tick per frame + with av.open(tmp, "w", format="mov") as container: + stream = container.add_stream("png", rate=rate) + stream.width, stream.height = frames[0].size + stream.pix_fmt = "rgb24" + stream.codec_context.time_base = tick + for index, frame in enumerate(frames): + video_frame = av.VideoFrame.from_image(frame) + video_frame.pts = index + video_frame.time_base = tick + for packet in stream.encode(video_frame): + container.mux(packet) + for packet in stream.encode(): + container.mux(packet) + os.replace(tmp, path) + digest = hashlib.sha256() + for frame in frames: + digest.update(frame.tobytes()) + return { + "id": row["id"], + "src": row["video_path"], + "mov": path, + "T": len(frames), + "fps_eff": fps_eff, + "rate": f"{rate.numerator}/{rate.denominator}", + "w": frames[0].size[0], + "h": frames[0].size[1], + "bytes": os.path.getsize(path), + "frames_sha256": digest.hexdigest(), + } + + +def write_frames(out: Path, *, workers: int, limit: int) -> dict[str, Any]: + """Write ``frames/``, ``train.jsonl`` and ``train.jsonl.frames.jsonl`` from ``hf/train.jsonl``.""" + from multiprocessing import Pool + + source = out / "hf" / "train.jsonl" + frames_dir = out / "frames" + frames_dir.mkdir(exist_ok=True) + with open(source) as handle: + rows = [json.loads(line) for line in handle if line.strip()] + if limit: + rows = rows[:limit] + with Pool(workers) as pool: + meta = pool.map(_write_clip_frames, [(row, str(frames_dir)) for row in rows], chunksize=8) + by_id = {clip["id"]: clip for clip in meta} + rows_path = out / "train.jsonl" + with open(rows_path, "w", newline="\n") as handle: + for row in rows: + messages = [] + for message in row["messages"]: + parts = [ + {"type": "video", "video": by_id[row["id"]]["mov"]} + if item["type"] == "video" + else {"type": "text", "text": item["text"]} + for item in message["content"] + ] + messages.append({"role": message["role"], "content": parts}) + handle.write(json.dumps({"messages": messages, "meta": {"item": row["id"]}}, ensure_ascii=False) + "\n") + with open(f"{rows_path}.frames.jsonl", "w", newline="\n") as handle: + for clip in meta: + handle.write(json.dumps(clip) + "\n") + return { + "rows": len(rows), + "src_md5": _md5_file(source), + "rows_md5": _md5_file(rows_path), + "bytes_total": sum(clip["bytes"] for clip in meta), + "T_total": sum(clip["T"] for clip in meta), + } + + +# ------------------------------------------------------------------------------------------------ verify ------------- +_PROCESSOR: Any = None + + +def _verify_clip(job: tuple[dict[str, Any], dict[str, Any], str]) -> dict[str, Any]: + """One training file in the training stack: same frames, same rate, same processor output as the evaluator's.""" + import torch + from PIL import Image + from torchcodec.decoders import VideoDecoder + from transformers import AutoProcessor + from transformers.video_utils import VideoMetadata + + global _PROCESSOR + row, clip, model_path = job + if _PROCESSOR is None: + _PROCESSOR = AutoProcessor.from_pretrained(model_path) + decoder = VideoDecoder(clip["mov"]) + file_meta = decoder.metadata + data = ( + decoder.get_frames_at(indices=list(range(file_meta.num_frames))).data.permute(0, 2, 3, 1).contiguous().numpy() + ) + digest = hashlib.sha256() + for index in range(data.shape[0]): + digest.update(data[index].tobytes()) + frames = [Image.fromarray(data[index]) for index in range(data.shape[0])] + messages = [ + { + "role": message["role"], + "content": [ + {"type": "video"} if item["type"] == "video" else {"type": "text", "text": item["text"]} + for item in message["content"] + ], + } + for message in row["messages"] + ] + text = _PROCESSOR.apply_chat_template(messages, tokenize=False, add_generation_prompt=False) + count, fps = clip["T"], clip["fps_eff"] + evaluator_metadata = VideoMetadata( + total_num_frames=count, + fps=fps, + duration=count / fps, + frames_indices=list(range(count)), + video_backend="torchcodec", + ) + as_evaluated = _PROCESSOR( + text=[text], + videos=[frames], + return_tensors="pt", + videos_kwargs={"do_sample_frames": False, "video_metadata": [evaluator_metadata]}, + ) + as_trained = _PROCESSOR( + text=[text], videos=[clip["mov"]], return_tensors="pt", videos_kwargs={"do_sample_frames": False} + ) + return { + "id": clip["id"], + "T": count, + "n_file": file_meta.num_frames, + "sha_ok": digest.hexdigest() == clip["frames_sha256"], + "fps_rel_err": abs(float(file_meta.average_fps) - fps) / fps, + "ids_equal": torch.equal(as_evaluated["input_ids"], as_trained["input_ids"]), + "grid_equal": torch.equal(as_evaluated["video_grid_thw"], as_trained["video_grid_thw"]), + "pix_equal": torch.equal(as_evaluated["pixel_values_videos"], as_trained["pixel_values_videos"]), + "len": int(as_trained["input_ids"].shape[1]), + } + + +def verify_frames(out: Path, *, model_path: str, workers: int) -> dict[str, Any]: + """Check every training file; write ``train.jsonl.verify.jsonl``; return the summary.""" + from multiprocessing import Pool + + rows_path = out / "train.jsonl" + with open(rows_path) as handle: + rows = [json.loads(line) for line in handle if line.strip()] + with open(f"{rows_path}.frames.jsonl") as handle: + clips = {clip["id"]: clip for clip in (json.loads(line) for line in handle if line.strip())} + with Pool(workers) as pool: + results = pool.map(_verify_clip, [(row, clips[row["meta"]["item"]], model_path) for row in rows], chunksize=4) + bad = [ + result + for result in results + if not ( + result["sha_ok"] + and result["ids_equal"] + and result["grid_equal"] + and result["pix_equal"] + and result["T"] == result["n_file"] + and result["fps_rel_err"] < 1e-6 + ) + ] + with open(f"{rows_path}.verify.jsonl", "w", newline="\n") as handle: + for result in results: + handle.write(json.dumps(result) + "\n") + return { + "rows": len(results), + "all_ok": not bad, + "bad": len(bad), + "first_bad": bad[:3], + "max_fps_rel_err": max(result["fps_rel_err"] for result in results), + "len_max": max(result["len"] for result in results), + "len_sum": sum(result["len"] for result in results), + } + + +# ------------------------------------------------------------------------------------------------ CLI ---------------- +def main(argv: Sequence[str] | None = None) -> int: + parser = argparse.ArgumentParser( + description="RoboVQA split and SFT rows for sft_qwen3_vl_8b_robovqa (see the module doc)." + ) + steps = parser.add_subparsers(dest="step", required=True) + split_args = steps.add_parser("split", help="split RoboVQA and write samples, conversation rows and the manifest") + split_args.add_argument("--src", required=True, help="local Tianli/robovqa download (json/, videos/)") + split_args.add_argument("--out", required=True, help="data root (the recipe's SFT_DATA_ROOT)") + split_args.add_argument("--seed", type=int, default=0) + split_args.add_argument("--frac", default="0.8,0.1,0.1", help="train,val,test fractions") + frames_args = steps.add_parser("frames", help="write the training clips' evaluation frames and train.jsonl") + frames_args.add_argument("--out", required=True, help="data root written by split") + frames_args.add_argument("--workers", type=int, default=_default_workers()) + frames_args.add_argument("--limit", type=int, default=0, help="first N training clips only (0 = all)") + verify_args = steps.add_parser("verify", help="check the training files in the SFT environment") + verify_args.add_argument("--out", required=True, help="data root written by split and frames") + verify_args.add_argument( + "--model", default=os.environ.get("SFT_MODEL_PATH"), help="Qwen3-VL snapshot (default $SFT_MODEL_PATH)" + ) + verify_args.add_argument("--workers", type=int, default=_default_workers()) + args = parser.parse_args(argv) + out = Path(args.out).resolve() + + if args.step == "split": + fractions = [float(value) for value in args.frac.split(",")] + manifest = split(Path(args.src), out, seed=args.seed, fractions=fractions) + print(json.dumps({"counts": manifest["counts"], "leakage_check": manifest["leakage_check"]})) + for name, expected in SEED0_MD5.items(): + note = f" (seed-0 reference {expected})" if args.seed == 0 else "" + print(f"{_md5_file(out / name)} {name}{note}") + return 0 + if args.step == "frames": + print(json.dumps(write_frames(out, workers=args.workers, limit=args.limit))) + return 0 + if not args.model: + parser.error("verify needs --model or SFT_MODEL_PATH") + summary = verify_frames(out, model_path=args.model, workers=args.workers) + print(json.dumps(summary)) + return 0 if summary["all_ok"] else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh new file mode 100755 index 00000000..6ba7121f --- /dev/null +++ b/.agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh @@ -0,0 +1,357 @@ +#!/usr/bin/env bash +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: OpenMDW-1.1 +# +# Fine-tune Qwen3-VL-8B-Instruct on RoboVQA with Cosmos Framework, one step per command. +# Run from the cosmos-framework repository root: +# +# bash .agents/skills/cosmos3-qwen3-vl-robovqa-sft/scripts/run.sh [--smoke] +# +# Steps, in order: +# setup the SFT environment, apex and the evaluation environment +# download Qwen3-VL-8B-Instruct and RoboVQA, at fixed revisions +# data split RoboVQA, write the training frames, check them +# preflight the SFT path's CPU checks of the recipe +# before score the untouched model on the held-out clips +# train fine-tune: one epoch, 125 updates of 64 clips, one GPU with about 150 GB +# export write the trained weights as a Hugging Face model +# after score the fine-tuned model the same way +# +# --smoke uses the skill's six bundled clips and 2 training updates: a quick check of every step. +# Every artifact goes under artifacts/qwen3-vl-robovqa/; each step's output is also saved under its logs/. +set -euo pipefail + +STEP=${1:-} +SMOKE=0 +[ "${2:-}" = "--smoke" ] && SMOKE=1 +if [ ! -f pyproject.toml ] || [ ! -d cosmos_framework ]; then + echo "run.sh: run it from the cosmos-framework repository root" >&2 + exit 1 +fi + +export PATH="$HOME/.local/bin:$PATH" +# On a host with several GPUs, use the one with the most memory for every GPU step (training needs about 150 GB). +if [ -z "${CUDA_VISIBLE_DEVICES:-}" ] && command -v nvidia-smi >/dev/null; then + GPU_INDEX=$(nvidia-smi --query-gpu=index,memory.total --format=csv,noheader,nounits | sort -t, -k2 -n -r | head -1 | cut -d, -f1) + export CUDA_VISIBLE_DEVICES=$GPU_INDEX +fi +gpu() { nvidia-smi -i "${CUDA_VISIBLE_DEVICES:-0}" "$@"; } # the selected GPU +SKILL_DIR="$PWD/.agents/skills/cosmos3-qwen3-vl-robovqa-sft" +UV="uvx --from uv==0.11.29 uv" # the checkout pins uv 0.11.29 +ART="$PWD/artifacts/qwen3-vl-robovqa" +SFT_VENV="$PWD/docker/posttrain/environment/.venv-sft" +SFT_PY="$SFT_VENV/bin/python" +SFT_LIBS="$SFT_VENV/lib/python3.13/site-packages/nvidia/cu13/lib" # NVIDIA NPP, needed by TorchCodec +EVAL_VENV="$ART/venvs/eval" +EVAL_PY="$EVAL_VENV/bin/python" +EVAL_LIBS="$EVAL_VENV/lib/python3.12/site-packages/nvidia/cu13/lib" +# Three knobs, read from the environment when set (SKILL.md "Your own data or another Qwen3-VL size"): +# CONFIG the recipe file; a copy with another Qwen3-VL model trains that model +# SFT_MODEL_PATH the model snapshot the recipe and the scorer load +# SFT_DATA_ROOT the data folder (train.jsonl, hf/test.jsonl, samples_test.jsonl); the data step writes it +CONFIG=${CONFIG:-cosmos_framework/posttrain/configs/sft_qwen3_vl_8b_robovqa.py} +export SFT_MODEL_PATH=${SFT_MODEL_PATH:-"$ART/models/Qwen3-VL-8B-Instruct"} +if [ "$SMOKE" = 1 ]; then + SRC="$SKILL_DIR/assets/robovqa" DATA=${SFT_DATA_ROOT:-"$ART/smoke"} RUN="$ART/runs-smoke" EVAL="$ART/eval-smoke" + EXPORT="$ART/export-smoke" STEPS=2 TAG="-smoke" +else + SRC="$ART/robovqa" DATA=${SFT_DATA_ROOT:-"$ART/data"} RUN="$ART/runs" EVAL="$ART/eval" EXPORT="$ART/export" STEPS=125 TAG="" +fi +export SFT_DATA_ROOT="$DATA" +mkdir -p "$ART/logs" + +say() { printf '\n==> %s\n' "$*"; } + +# stage