Add notebook-driven Brev launchable for Cosmos3 VANTAGE evaluation

evaluation/cosmos3/reasoner/vlmevalkit/.dockerignore

@@ -0,0 +1,15 @@
+.git
+.venv
+__pycache__
+*.pyc
+*.egg-info
+**/__pycache__
+**/.ipynb_checkpoints
+launchable/state
+launchable/.env
+.env
+**/.env
+**/env.sh
+cosmos_eval_out
+out
+outputs

evaluation/cosmos3/reasoner/vlmevalkit/README.md

@@ -45,6 +45,13 @@ the inputs and reads the outputs.
 
 ## 1. Environment setup
 
+For a Brev VM with a notebook-driven Docker deployment, use the
+[Cosmos3 VANTAGE launchable](launchable/README.md). Its default **public** path
+uses the dataset-linked VANTAGE harness for public data preparation and official
+submission generation. Its separate **internal** path retains this NVIDIA fork
+and the configs below; the two paths do not share prepared dataset roots.
+
+
 Use a clean virtual environment with **Python 3.10+**, run from the **repo root** (the directory that
 holds this README):
 

evaluation/cosmos3/reasoner/vlmevalkit/launchable/.gitignore

@@ -0,0 +1,3 @@
+state/
+.env
+.ipynb_checkpoints/

evaluation/cosmos3/reasoner/vlmevalkit/launchable/Dockerfile

@@ -0,0 +1,35 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+FROM python:3.11.14-slim-bookworm@sha256:65a93d69fa75478d554f4ad27c85c1e69fa184956261b4301ebaf6dbb0a3543d
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    build-essential ffmpeg git libgl1 libglib2.0-0 libcairo2 \
+    && rm -rf /var/lib/apt/lists/*
+WORKDIR /opt/vlmevalkit
+COPY requirements.txt setup.py setup.cfg README-vlmevalkit.md ./
+# Evaluation is API-based; keep its PyTorch stack off the serving GPU.
+RUN pip install --no-cache-dir torch==2.8.0 torchvision==0.23.0 --index-url https://download.pytorch.org/whl/cpu \
+    && pip install --no-cache-dir -r requirements.txt einops accelerate 'setuptools<81' evaluate jupyterlab==4.4.10
+
+COPY launchable/public-harness.patch /tmp/public-harness.patch
+# Keep the public release and internal NVIDIA fork in separate Python environments.
+RUN git init /opt/vantage-public \
+    && git -C /opt/vantage-public remote add origin https://github.com/Clemson-Capstone/VANTAGE-Bench.git \
+    && git -C /opt/vantage-public fetch --depth 1 origin ec1ebdebb955332a9d7fdd9e98f39627b2822ff5 \
+    && git -C /opt/vantage-public checkout --detach FETCH_HEAD \
+    && git -C /opt/vantage-public apply /tmp/public-harness.patch \
+    && python -m venv /opt/vantage-public/.venv \
+    && /opt/vantage-public/.venv/bin/pip install --no-cache-dir torch==2.8.0 torchvision==0.23.0 --index-url https://download.pytorch.org/whl/cpu \
+    && /opt/vantage-public/.venv/bin/pip install --no-cache-dir -r /opt/vantage-public/requirements.txt \
+        'transformers==4.57.1' 'qwen-vl-utils==0.0.14' 'setuptools<81' evaluate gdown \
+    && cd /opt/vantage-public \
+    && .venv/bin/pip install --no-cache-dir --no-deps -e . \
+    && .venv/bin/python -c 'import vlmeval' \
+    && .venv/bin/pip freeze > /opt/public-evaluation-packages.txt
+COPY . .
+RUN pip install --no-cache-dir --no-deps -e . \
+    && python -c 'import vlmeval' \
+    && pip freeze > /opt/evaluation-packages.txt
+ARG COSMOS_COMMIT=unknown
+ENV COSMOS_COMMIT=${COSMOS_COMMIT} PYTHONUNBUFFERED=1 HF_HOME=/cache/huggingface
+WORKDIR /opt/vlmevalkit/launchable
+CMD ["python", "jupyter_config.py"]

evaluation/cosmos3/reasoner/vlmevalkit/launchable/Evaluate_Cosmos3.ipynb

@@ -0,0 +1,293 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "welcome",
+   "metadata": {},
+   "source": [
+    "# Cosmos3 · VANTAGE evaluation\n",
+    "\n",
+    "**Prepare → connect → five real predictions → full submission**\n",
+    "\n",
+    "This notebook is a thin front end to `launchable.py`, using the official public VANTAGE harness by default. The full eight-task run is **opt-in**: **Run All runs the five-question smoke only** while `RUN_FULL = False`.\n",
+    "\n",
+    "**Before opening:** run `bash setup.sh --brev` on the Brev VM from the launchable directory. It binds Jupyter to the private mesh address for the authenticated Brev link on port 8888. Docker setup/lifecycle commands run on the VM host, not here; this evaluator has no Docker socket. See [README](README.md) for launch fields and credentials.\n",
+    "\n",
+    "**Validation boundary:** the public five-question VQA notebook passed on a Brev 96 GB RTX PRO 6000 Blackwell GPU at commit `23cfa62968f0a395481b52c4fd91bae9ce1106dd`, using a resumed VM and retained cache. An uninterrupted fresh-VM cold launch, full evaluation, portal acceptance, and reference-score parity remain unverified. Five predictions are not a leaderboard score."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "configure-help",
+   "metadata": {},
+   "source": [
+    "## 1 · Configure a fresh run\n",
+    "\n",
+    "The pinned Nano model and public VQA data support anonymous access; no HF token or hidden prompt is needed for this smoke. An inherited secret `HF_TOKEN` remains available for gated assets. Full SOT preparation requires a read token with accepted [PhysicalAI-SmartSpaces access terms](https://huggingface.co/datasets/nvidia/PhysicalAI-SmartSpaces). Public mode does **not** need internal AWS/NGC/dataset-service or API judge keys. Existing endpoint/model environment settings are preserved.\n",
+    "\n",
+    "Use a new timestamped directory for each run. Re-running this cell creates new output paths; do not mix smoke and full outputs. Change `SOURCE` to `internal` only after configuring the host AWS mount described at the end."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "configure",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "import subprocess\n",
+    "import sys\n",
+    "from datetime import datetime, timezone\n",
+    "from getpass import getpass\n",
+    "from pathlib import Path\n",
+    "from IPython.display import FileLink, display\n",
+    "\n",
+    "SOURCE = 'public'\n",
+    "RUN_FULL = False  # Explicitly opt in before running section 6.\n",
+    "LAUNCHABLE = Path('/opt/vlmevalkit/launchable/launchable.py')\n",
+    "CLI = [sys.executable, str(LAUNCHABLE)]\n",
+    "STAMP = datetime.now(timezone.utc).strftime('%Y%m%dT%H%M%S%fZ')\n",
+    "SMOKE_WORK = Path('/results') / f'{SOURCE}-smoke-{STAMP}'\n",
+    "FULL_WORK = Path('/results') / f'public-full-{STAMP}'\n",
+    "\n",
+    "os.environ.setdefault('COSMOS_API_BASE', 'http://cosmos-endpoint:8080/v1/chat/completions')\n",
+    "# Do not overwrite COSMOS_MODEL: deployment/BYOM configuration selects it.\n",
+    "print(f'Source: {SOURCE}; smoke output: {SMOKE_WORK}; full run enabled: {RUN_FULL}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "byom-help",
+   "metadata": {},
+   "source": [
+    "### Optional · Bring your own model\n",
+    "\n",
+    "Prefer setting `COSMOS_API_BASE`, `COSMOS_MODEL`, and an optional secret `COSMOS_API_KEY` at **initial deployment**: setup then skips the local GPU endpoint. For a session-only override, set `USE_BYOM = True` below **before** the health check. The URL must end in `/v1/chat/completions` and be reachable from the evaluator container. The service must accept the harness's image/video requests. Do not put credentials in the URL.\n",
+    "\n",
+    "Changing these settings here does not stop an already-running local GPU container. Leave this cell disabled to preserve all inherited settings. A blank hidden key prompt preserves the inherited key."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "byom",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "USE_BYOM = False\n",
+    "if USE_BYOM:\n",
+    "    os.environ['COSMOS_API_BASE'] = input('Full chat/completions URL (no embedded credentials): ').strip()\n",
+    "    os.environ['COSMOS_MODEL'] = input('Exact served model ID: ').strip()\n",
+    "    byom_key = getpass('Endpoint key (hidden; blank preserves inherited key): ')\n",
+    "    if byom_key:\n",
+    "        os.environ['COSMOS_API_KEY'] = byom_key\n",
+    "    del byom_key"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "prepare-help",
+   "metadata": {},
+   "source": [
+    "## 2 · Prepare VQA with the selected harness\n",
+    "\n",
+    "Public mode uses the official public preparation path and pinned HF revision. VQA-only preparation downloads the VQA subtree rather than the entire benchmark; it still needs real media and can take time. Internal mode instead uses the authorized internal S3/cache constructor. Public and internal data are isolated under `/data/public/LMUData` and `/data/internal/LMUData`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "prepare-vqa",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "subprocess.run(CLI + ['prepare', '--source', SOURCE, '--benchmarks', 'VANTAGE_VQA'], check=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "endpoint-help",
+   "metadata": {},
+   "source": [
+    "## 3 · Wait for the endpoint\n",
+    "\n",
+    "Wait up to 30 minutes for `/v1/models` and verify the requested model is served. This checks readiness, not GPU compatibility or benchmark quality. Failures propagate instead of being suppressed. The local model API is available only on the Compose network; do not expose it publicly."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "endpoint-check",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "subprocess.run(CLI + ['check', '--wait', '1800'], check=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "smoke-help",
+   "metadata": {},
+   "source": [
+    "## 4 · Infer five real VQA questions\n",
+    "\n",
+    "The helper applies a deterministic five-row VQA selection (`limit=5`, `random_state=0`) and runs inference only. It checks actual prediction artifacts because the native harness can catch exceptions and exit zero without producing usable results. This is an integration smoke, **not** a full benchmark, local score, or valid submission."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "smoke-run",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "subprocess.run(CLI + ['run', '--source', SOURCE, '--benchmarks', 'VANTAGE_VQA',\n",
+    "                      '--mode', 'infer', '--smoke', '--work-dir', str(SMOKE_WORK)], check=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "results-help",
+   "metadata": {},
+   "source": [
+    "## 5 · Inspect smoke results\n",
+    "\n",
+    "Inspect prediction paths and run evidence. Native outputs can be TSV rather than XLSX. Public labels are evaluated by the official server; no local public leaderboard score is claimed. Use Jupyter's `/results` browser for logs, generated configs, and provenance; logs may contain dataset/model content, so share carefully."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "smoke-results",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "subprocess.run(CLI + ['results', '--source', SOURCE, '--work-dir', str(SMOKE_WORK)], check=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "full-help",
+   "metadata": {},
+   "source": [
+    "## 6 · Opt in to the full eight-task public run\n",
+    "\n",
+    "**Disabled by default.** Set `RUN_FULL = True` in the configure cell, then run this section. Use `SOURCE = 'public'`. The following cells otherwise do no preparation, inference, or export. Full preparation needs substantial disk/network time: roughly 42 GB public data plus roughly 16 GB SOT source data, in addition to weights, containers, and extraction/cache headroom. SOT requires the official SmartSpaces license/access prerequisites.\n",
+    "\n",
+    "All eight tasks run: VQA, EventVerification, 2DGrounding, 2DPointing, Astro2D, Temporal, DVC, and SOT. No runtime or score-parity promise is made. The public harness, dataset classes, prompts, and ID emission differ from the internal fork even where sampling/model parameters are preserved.\n",
+    "\n",
+    "This run requests **inference**, then the official submission packager. Public `--mode all` also uses inference because public ground truth is held out; it does not produce a local leaderboard score. Never reuse the five-question directory for this run."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "full-prepare",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "if RUN_FULL:\n",
+    "    if SOURCE != 'public':\n",
+    "        raise ValueError('This full-submission section requires SOURCE = public.')\n",
+    "    if not os.environ.get('HF_TOKEN'):\n",
+    "        token = getpass('HF read token with accepted SmartSpaces access (hidden): ').strip()\n",
+    "        if not token:\n",
+    "            raise ValueError('Full SOT preparation requires authorized SmartSpaces access and HF_TOKEN.')\n",
+    "        os.environ['HF_TOKEN'] = token\n",
+    "        del token\n",
+    "    subprocess.run(CLI + ['prepare', '--source', 'public', '--benchmarks', 'all'], check=True)\n",
+    "else:\n",
+    "    print('Full public run disabled; set RUN_FULL = True to opt in.')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "full-run",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "if RUN_FULL:\n",
+    "    if SOURCE != 'public':\n",
+    "        raise ValueError('This full-submission section requires SOURCE = public.')\n",
+    "    subprocess.run(CLI + ['run', '--source', 'public', '--benchmarks', 'all',\n",
+    "                          '--mode', 'infer', '--work-dir', str(FULL_WORK)], check=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "full-results",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "if RUN_FULL:\n",
+    "    subprocess.run(CLI + ['results', '--source', 'public', '--work-dir', str(FULL_WORK)], check=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "export-help",
+   "metadata": {},
+   "source": [
+    "## 7 · Export and download the official submission\n",
+    "\n",
+    "The public helper validates complete task coverage and uses the official packager, not an invented archive schema. Pillars require complete task sets: semantic = VQA + EventVerification; spatial = Grounding + Pointing + Astro2D; temporal = Temporal + DVC; tracking = SOT. This notebook's submission path runs **all eight**, not a partial pillar or smoke.\n",
+    "\n",
+    "Download the generated artifact using the authenticated Jupyter link below, then follow [official submission instructions](https://github.com/Clemson-Capstone/VANTAGE-Bench/blob/ec1ebdebb955332a9d7fdd9e98f39627b2822ff5/SUBMISSION.md) and submit at [vantage-bench.org/submit](https://vantage-bench.org/submit). Server scoring may take up to 24 hours. Nothing is uploaded automatically. Clear notebook outputs before sharing; do not redistribute restricted dataset media or credentials."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "full-export",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "if RUN_FULL:\n",
+    "    subprocess.run(CLI + ['export', '--source', 'public', '--work-dir', str(FULL_WORK)], check=True)\n",
+    "    archive = FULL_WORK.parent / f'{FULL_WORK.name}-submission.tar.gz'\n",
+    "    display(FileLink(str(archive), url_prefix='/files'))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "internal-help",
+   "metadata": {},
+   "source": [
+    "## Optional · Authorized internal evaluation\n",
+    "\n",
+    "Public VQA smoke supports anonymous HF access. For internal mode, configure **at initial deployment** an existing host `AWS_CREDENTIALS_DIR` containing AWS `config` and `credentials` with the authorized `team-cosmos` profile (or securely populate the default `state/credentials/aws` before setup). Compose mounts it read-only under `/credentials/aws`, using `AWS_CONFIG_FILE=/credentials/aws/config` and `AWS_SHARED_CREDENTIALS_FILE=/credentials/aws/credentials`. Credentials must remain valid during preparation. A notebook variable cannot create or change a host bind mount. Never paste AWS values into this notebook.\n",
+    "\n",
+    "Then set `SOURCE = 'internal'` in section 1, keep `RUN_FULL = False`, and rerun configuration, VQA preparation, health check, smoke, and results. Those same commands pass `--source internal` to the NVIDIA harness. Internal data stays under `/data/internal/LMUData`; public data stays under `/data/public/LMUData`, with separate Python/import roots as well. Internal `export --source internal` creates an evidence bundle, **not** a public leaderboard submission. See the README for CLI examples.\n",
+    "\n",
+    "These eight VANTAGE tasks need no additional dataset-service, NGC, or API judge keys. TAR and other benchmark families are outside scope.\n",
+    "\n",
+    "### Stop/restart safely\n",
+    "Use the **VM terminal**, in the launchable directory: `docker compose --profile local stop` preserves results/caches. For existing local services, `docker compose --profile local start cosmos-endpoint vantage-eval`; for BYOM, `docker compose start vantage-eval`. For changed configuration rerun setup with the intended deployment environment. See README for log commands. Stopping containers does not stop VM billing."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.11"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}