Add Nano Brev launchable for native MVBench evaluation

.agents/skills/evaluation/cosmos3/reasoner/vlmevalkit/.dockerignore

@@ -0,0 +1,26 @@
+.git
+.venv
+__pycache__
+*.pyc
+*.egg-info
+**/__pycache__
+**/.ipynb_checkpoints
+launchable/state
+**/.env*
+**/.aws
+**/.ssh
+**/*credentials*
+**/*.pem
+**/*.key
+**/scoring-assets.json
+**/scoring-assets
+**/.mvbench
+**/mvbench-data
+**/*.tsv
+**/*.xlsx
+**/*.pkl
+**/results
+**/env.sh
+cosmos_eval_out
+out
+outputs

.agents/skills/evaluation/cosmos3/reasoner/vlmevalkit/EXTERNAL_CONFIGURATION.md

@@ -40,3 +40,55 @@ constructor or dataset configuration. Original private dataset YAML defaults
 remain available. Missing or nonexistent grounding annotation files produce a
 clear `ValueError`. Omni3D retains its existing private configuration and can be
 pointed at another authorized dataset through `raw_image_path` and `json_path`.
+
+## Brev Nano and public MVBench evaluation
+
+The [Brev launchable](launchable/README.md) uses the Framework-native kit,
+the pinned Nano revision, and `OpenGVLab/MVBench` at
+`230a2d4fac8900333c61754641c7a13e069ac9c6`. It provides public download,
+coverage validation, native inference, and local exact-match scoring for 20
+tasks and 4,000 annotated multiple-choice questions. No private annotation
+bundle, private S3, GPU, or model endpoint is required for score-only execution.
+
+Set `LAUNCHABLE_STATE_DIR` to an absolute writable host directory outside the
+checkout. `MVBENCH_DATA_DIR` defaults to its `mvbench` child and is mounted
+read-write at `/mvbench`; cache and results use separate mounts. The notebook
+downloads the selected public scope itself. Approved media terms restrict use
+to academic research; public access is not commercial-use or redistribution
+permission. Full coverage requires manually acquired authorized NTU RGB+D
+media under `video/nturgbd`; removed videos are never fetched from historical
+revisions or mirrors. Missing media produces a coverage error, not an incomplete
+full score. Default smoke is five fixed Egocentric Navigation rows.
+
+Native `run.py` configuration uses the local dataset class directly:
+
+```json
+{
+  "data": {
+    "MVBench": {
+      "class": "MVBenchLocal",
+      "dataset": "MVBench",
+      "data_root": "/mvbench",
+      "nframe": 16,
+      "smoke": false
+    }
+  }
+}
+```
+
+The controller composes this with `CosmosReason2` and the existing generation
+recipe, defaulting to one API worker. Temporal sampling and prompts are native
+MVBench, with sampled-image input and isolated clip/frame caches. The starting
+16-frame target and scale/center-crop-224 frame preprocessing follow the original
+recipe; Cosmos image encoding/normalization and answer parsing still differ from VideoChat2.
+Native count-based accuracy (fraction), task macro, coverage and provenance
+remain explicit. No LLM judge or published-score claim is involved.
+
+Use `launchable.py score --scope smoke|full` with either an existing controller
+`--work-dir` or native `--predictions` plus a fresh `--score-dir`. Exact selected
+identities and complete coverage are required. Trusted answers come from pinned
+annotations, never prediction-supplied fields. Score-only skips endpoint checks
+and model construction. BYOM uses `COSMOS_API_BASE`, `COSMOS_MODEL`, optional
+environment-only `COSMOS_API_KEY`, and an explicit `COSMOS_MODEL_REVISION`;
+setup then skips local serving. This launchable does not implement a
+customer-data evaluation agent or alter other model deployment tracks.

.agents/skills/evaluation/cosmos3/reasoner/vlmevalkit/README.md

@@ -3,16 +3,28 @@ SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All
 SPDX-License-Identifier: Apache-2.0
 -->
 
-# TAR and VANTAGE Benchmarks for Cosmos3 Model
+# Cosmos3 evaluation: MVBench launchable and native benchmark recipes
 
-This repository is an **NVIDIA-customized [VLMEvalKit](https://github.com/open-compass/VLMEvalKit)** that
-releases the NVIDIA Metropolis "smart infra" benchmarks and makes the cosmos-reasoner scores on them
-**publicly reproducible**. The reproduction kit lives in [`cosmos_eval/`](cosmos_eval/); the stock
-VLMEvalKit inference + scoring engine it drives is the rest of the tree (upstream toolkit docs:
-[`README-vlmevalkit.md`](README-vlmevalkit.md)).
+This directory is an **NVIDIA-customized [VLMEvalKit](https://github.com/open-compass/VLMEvalKit)**
+maintained in Cosmos Framework. It contains reference settings for NVIDIA
+Metropolis "smart infra" evaluation; [`cosmos_eval/`](cosmos_eval/) composes the
+native inference and scoring engine. Toolkit docs live in
+[`README-vlmevalkit.md`](README-vlmevalkit.md). These settings do not make private
+datasets publicly reproducible or establish official score parity.
 
 For user-hosted endpoints and local dataset inputs, see
 [Deployment and dataset configuration](EXTERNAL_CONFIGURATION.md).
+For the Nano-only Brev notebook, use the
+[MVBench launchable](launchable/README.md): 20 tasks, 4,000 public annotated
+multiple-choice questions, pinned public downloads, native inference, and local
+exact-match scoring without an API judge. Its default smoke is five Egocentric
+Navigation questions, not a full score; full evaluation is explicit opt-in.
+The approved dataset revision requires manual authorized acquisition of 320 NTU
+RGB+D videos and restricts video use to academic research. Public access does
+not establish commercial rights, redistribution permission, or published-score
+parity. Data and state stay outside the checkout. This is not a customer-data
+evaluation-agent implementation. Existing native recipes below are independent
+of this launchable and remain unchanged.
 
 The kit ships, for each benchmark, reference `run.py --config` settings —
 split into a shared dataset layer (`cosmos_eval/data/`) and a per-family model layer

.agents/skills/evaluation/cosmos3/reasoner/vlmevalkit/launchable/.gitignore

@@ -0,0 +1,3 @@
+state/
+.env
+.ipynb_checkpoints/

.agents/skills/evaluation/cosmos3/reasoner/vlmevalkit/launchable/Dockerfile

@@ -0,0 +1,21 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+FROM python:3.11.14-slim-bookworm@sha256:65a93d69fa75478d554f4ad27c85c1e69fa184956261b4301ebaf6dbb0a3543d
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    build-essential ffmpeg git libgl1 libglib2.0-0 libcairo2 \
+    && rm -rf /var/lib/apt/lists/*
+WORKDIR /opt/vlmevalkit
+COPY requirements.txt setup.py setup.cfg README-vlmevalkit.md ./
+# Evaluation is API-based; keep its PyTorch stack off the serving GPU.
+RUN pip install --no-cache-dir torch==2.8.0 torchvision==0.23.0 --index-url https://download.pytorch.org/whl/cpu \
+    && pip install --no-cache-dir -r requirements.txt einops accelerate 'setuptools<81' evaluate jupyterlab==4.4.10
+COPY . .
+RUN pip install --no-cache-dir --no-deps -e . \
+    && python -c 'import vlmeval' \
+    && pip freeze > /opt/evaluation-packages.txt
+ARG COSMOS_COMMIT=unknown
+ENV COSMOS_COMMIT=${COSMOS_COMMIT} PYTHONUNBUFFERED=1 HF_HOME=/cache/huggingface \
+    AWS_CONFIG_FILE=/dev/null AWS_SHARED_CREDENTIALS_FILE=/dev/null \
+    AWS_EC2_METADATA_DISABLED=true MVBENCH_DATA_DIR=/mvbench LMUData=/cache/lmudata
+WORKDIR /opt/vlmevalkit/launchable
+CMD ["python", "jupyter_config.py"]

.agents/skills/evaluation/cosmos3/reasoner/vlmevalkit/launchable/Evaluate_Cosmos3.ipynb

@@ -0,0 +1,215 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Evaluate Cosmos3-Nano on MVBench\n",
+    "\n",
+    "**Download public inputs → validate coverage → infer → score locally.** The default is the fixed five-question Egocentric Navigation smoke, not a full score. Set `RUN_FULL = True` to select full evaluation directly; it does not rerun smoke first.\n",
+    "\n",
+    "[MVBench](https://arxiv.org/abs/2311.17005) contains 4,000 annotated multiple-choice questions across 20 video-understanding tasks. This notebook uses the native VLMEvalKit MVBench task definitions, temporal bounds, prompts, and exact-match evaluator. There is no LLM judge or automatic upload. The Nano serving revision is `e59a53c25979a090fa8706c9acc0c254a6e89b92`.\n",
+    "\n",
+    "**Research-only media terms:** public access does not establish commercial rights or redistribution permission. The card mentions 320 NTU videos, but this pinned revision's manual list contains 200 RGB AVI files for its 200 Fine-grained Pose questions. Obtain authorized access from https://rose1.ntu.edu.sg/dataset/actionRecognition/ and use the pinned video/MVBench_videos_ntu.txt list. Removed media is never downloaded from historical revisions or mirrors. No published-score reproduction or commercial-rights claim is made. This is not a customer-data evaluation-agent implementation.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1 · Choose one execution scope\n",
+    "\n",
+    "The external host `MVBENCH_DATA_DIR` is writable at `/mvbench`; caches and results are separate external mounts. No AWS credentials, Docker socket, labels, media, or model weights are embedded in this image.\n",
+    "\n",
+    "For **score-only**, choose `SCORE_SCOPE` explicitly (`smoke` or `full`) and set exactly one of `PREDICTIONS_FILE` (native TSV/XLSX/JSON) or `EXISTING_WORK` (a completed controller inference directory). No endpoint check, GPU, model construction, or private S3 access is used. The selected scope must match the predictions exactly; arbitrary partial scores are not accepted.\n",
+    "\n",
+    "For full runs, stage authorized NTU files under `/mvbench/video/nturgbd` using Jupyter's authenticated file browser/terminal before preparation. The download cell retrieves only currently approved public files. Preparation reports missing media and refuses to run an incomplete full benchmark.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "import os\n",
+    "from pathlib import Path\n",
+    "import subprocess\n",
+    "import sys\n",
+    "from datetime import datetime, timezone\n",
+    "from IPython.display import FileLink, display\n",
+    "\n",
+    "SCORE_ONLY = os.environ.get('EVAL_SCORE_ONLY') == '1'\n",
+    "PREDICTIONS_FILE = None\n",
+    "EXISTING_WORK = None\n",
+    "SCORE_SCOPE = 'smoke'  # Explicitly change to 'full' for 4,000 existing predictions.\n",
+    "RUN_FULL = False      # Full inference is opt-in; selecting it skips the smoke.\n",
+    "DOWNLOAD_PUBLIC = True\n",
+    "NFRAME = 16           # Reference frame count; not a model-score parity claim.\n",
+    "API_NPROC = 1\n",
+    "DATA = Path(os.environ.get('MVBENCH_DATA_DIR', '/mvbench'))\n",
+    "SCOPE = SCORE_SCOPE if SCORE_ONLY else ('full' if RUN_FULL else 'smoke')\n",
+    "STAMP = datetime.now(timezone.utc).strftime('%Y%m%dT%H%M%S%fZ')\n",
+    "WORK = Path('/results') / f'mvbench-{SCOPE}-{STAMP}'\n",
+    "SCORE_DIR = Path('/results') / f'mvbench-score-{SCOPE}-{STAMP}'\n",
+    "CLI = [sys.executable, '-u', str(Path('/opt/vlmevalkit/launchable/launchable.py'))]\n",
+    "\n",
+    "def invoke(command):\n",
+    "    # Stream instead of buffering an entire long-running inference job.\n",
+    "    with subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,\n",
+    "                          text=True, bufsize=1) as process:\n",
+    "        for line in process.stdout:\n",
+    "            print(line, end='', flush=True)\n",
+    "        if process.wait():\n",
+    "            raise subprocess.CalledProcessError(process.returncode, command)\n",
+    "\n",
+    "if SCOPE not in {'smoke', 'full'}:\n",
+    "    raise ValueError('Choose explicit smoke or full scope.')\n",
+    "if SCORE_ONLY and (PREDICTIONS_FILE is None) == (EXISTING_WORK is None):\n",
+    "    raise ValueError('Set exactly one of PREDICTIONS_FILE or EXISTING_WORK for score-only execution.')\n",
+    "print('Score existing predictions' if SCORE_ONLY else 'Generate and score predictions', SCOPE)\n",
+    "print('Data:', DATA, '| frames:', NFRAME, '| concurrent API workers:', API_NPROC)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2 · Download the chosen public scope\n",
+    "\n",
+    "The downloader pins `OpenGVLab/MVBench` to `230a2d4fac8900333c61754641c7a13e069ac9c6`, fetches all 20 official annotation JSONs and licensing/NTU notes, and safely extracts selected current public archives, including the June-2024 `data0613` overlay where applicable. Smoke needs only the approximately 40 MB `vlnqa.zip` archive. Full download is substantially larger and **cannot supply the manual NTU videos**. Existing approved files are reused. Set `DOWNLOAD_PUBLIC = False` for offline scoring with already staged inputs.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "if DOWNLOAD_PUBLIC:\n",
+    "    invoke(CLI + ['download', '--data-root', str(DATA), '--scope', SCOPE])\n",
+    "else:\n",
+    "    print('Using already staged public annotations and media.')\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3 · Prepare trusted annotations and require complete selected coverage\n",
+    "\n",
+    "Smoke selects exactly stable indices 3400–3404: five of 4,000 rows, one of 20 tasks. Full requires all 4,000 rows and all 20 tasks; no missing clips or questions are silently dropped. Missing full NTU media is an actionable prerequisite failure, not a partial full score. A structured coverage report is retained under `/mvbench/.mvbench/`.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "scope_flag = ['--smoke'] if SCOPE == 'smoke' else []\n",
+    "invoke(CLI + ['prepare', '--data-root', str(DATA)] + scope_flag)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 4 · Check the endpoint only for inference\n",
+    "\n",
+    "BYOM can be selected before host setup with `COSMOS_API_BASE` (complete `/v1/chat/completions` URL), exact `COSMOS_MODEL`, optional `COSMOS_API_KEY`, and `COSMOS_MODEL_REVISION`. Keep keys in the environment, not this notebook. The endpoint must accept sampled-image messages; fixed frame count is meaningful because no re-encoded video is sent. Generation uses the existing CosmosReason2 recipe: deterministic temperature 0, seed 1, no thinking, maximum 8,192 output tokens.\n",
+    "\n",
+    "The default 16 frames and bicubic scale/center-crop to 224 follow the reference recipe. Frames are sent as sampled images; Cosmos owns downstream image encoding/normalization, so VideoChat2 tensor normalization is not applied and model/parser differences remain. There is no paper-score parity claim.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "if not SCORE_ONLY:\n",
+    "    invoke(CLI + ['check', '--wait', '1800'])\n",
+    "else:\n",
+    "    print('Score-only: endpoint check and model inference skipped.')\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 5 · Native inference and local scoring, or score existing outputs\n",
+    "\n",
+    "Progress streams here and into the private inference `run.log`. Fresh directories prevent stale prediction reuse. Inference and scoring reject wrong/missing/duplicate identities and blank or transport-failure outputs; the scorer reconstructs trusted identities and labels from pinned annotations. Malformed but nonempty model answers receive native wrong-answer semantics, not fabricated successes. Native exact matching runs locally without an API judge.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "if SCORE_ONLY:\n",
+    "    source = (['--work-dir', str(EXISTING_WORK)] if EXISTING_WORK is not None\n",
+    "              else ['--predictions', str(PREDICTIONS_FILE)])\n",
+    "    invoke(CLI + ['score', '--data-root', str(DATA), '--scope', SCOPE,\n",
+    "                  '--score-dir', str(SCORE_DIR), '--nframe', str(NFRAME)] + source)\n",
+    "    SUMMARY = SCORE_DIR / 'summary.json'\n",
+    "else:\n",
+    "    invoke(CLI + ['run', '--data-root', str(DATA), '--mode', 'all',\n",
+    "                  '--nframe', str(NFRAME), '--api-nproc', str(API_NPROC),\n",
+    "                  '--work-dir', str(WORK)] + scope_flag)\n",
+    "    SUMMARY = WORK / 'scores' / 'summary.json'\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 6 · Inspect metrics, coverage, and provenance\n",
+    "\n",
+    "The primary metric is count-based multiple-choice accuracy as a **fraction**, with per-task metrics and the task-macro average reported separately. Smoke is prominently marked as a subset. Native scorer version/source, dataset revision and manifest, prediction hashes, selected IDs, frame protocol, generation parameters, and model/source revisions are recorded in private artifacts. A BYOM display name alone does not establish its checkpoint revision. No reference/published-score parity is claimed until independently established with an agreed protocol.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "summary = json.loads(SUMMARY.read_text())\n",
+    "print(json.dumps({key: summary[key] for key in\n",
+    "                  ('status', 'coverage', 'primary_metric', 'per_task', 'provenance')}, indent=2))\n",
+    "display(FileLink(str(SUMMARY), url_prefix='/files'))\n",
+    "if not SCORE_ONLY:\n",
+    "    invoke(CLI + ['results', '--work-dir', str(WORK)])\n",
+    "    display(FileLink(str(WORK / 'run.json'), url_prefix='/files'))\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Stop and retain private results\n",
+    "\n",
+    "Container lifecycle commands belong in the **host** terminal, not this notebook: `docker compose --profile local stop` preserves state. `bash setup.sh --score-only` starts only authenticated Jupyter for existing-prediction scoring. There is no Docker socket in the evaluator and the local model API has no host port. Stopping containers does not stop underlying Brev VM billing; stop that instance separately after retaining authorized results.\n",
+    "\n",
+    "Clear outputs before sharing. Do not commit restricted media, labels, credentials, or answer-bearing artifacts. Native Linux x86-64 is the supported runtime; code changes alone do not demonstrate notebook/hardware execution or full benchmark reproduction.\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "name": "python",
+   "version": "3.11.14"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}