diff --git a/.agents/skills/evaluation/cosmos3/reasoner/vlmevalkit/.dockerignore b/.agents/skills/evaluation/cosmos3/reasoner/vlmevalkit/.dockerignore new file mode 100644 index 000000000..8153879df --- /dev/null +++ b/.agents/skills/evaluation/cosmos3/reasoner/vlmevalkit/.dockerignore @@ -0,0 +1,31 @@ +.git +.venv +__pycache__ +*.pyc +*.egg-info +**/__pycache__ +**/.ipynb_checkpoints +launchable/state +**/.env* +**/.aws +**/.ssh +**/*credentials* +**/*.pem +**/*.key +**/scoring-assets.json +**/scoring-assets +**/.perception-test +**/perception-test-data +**/*.tsv +**/*.xlsx +**/*.pkl +**/results +**/env.sh +cosmos_eval_out +out +outputs +**/cache +**/*.mp4 +**/*.zip +**/mc_question_valid.json +**/cut_frame_mapping_valid.json diff --git a/.agents/skills/evaluation/cosmos3/reasoner/vlmevalkit/EXTERNAL_CONFIGURATION.md b/.agents/skills/evaluation/cosmos3/reasoner/vlmevalkit/EXTERNAL_CONFIGURATION.md index 24023fdb7..a67f6826f 100644 --- a/.agents/skills/evaluation/cosmos3/reasoner/vlmevalkit/EXTERNAL_CONFIGURATION.md +++ b/.agents/skills/evaluation/cosmos3/reasoner/vlmevalkit/EXTERNAL_CONFIGURATION.md @@ -40,3 +40,80 @@ constructor or dataset configuration. Original private dataset YAML defaults remain available. Missing or nonexistent grounding annotation files produce a clear `ValueError`. Omni3D retains its existing private configuration and can be pointed at another authorized dataset through `raw_image_path` and `json_path`. + +## Brev Nano and public Perception Test validation + +The [Brev launchable](launchable/README.md) uses the Framework-native kit, +the pinned public Nano checkpoint, and Google DeepMind's public Perception Test +validation MCQ data: 19,140 questions over 5,260 videos. Default smoke selects +`video_8241`, official question IDs 0–4 (stable indices 13404–13408). +Full validation is explicit opt-in, never the held-out test split. +Dataset source revision is `3938d2f1ba3a6b502025741cea4cd73c7b3bdfaf`; +official annotations, cut mapping, README and public archives are SHA-256 pinned +in `vlmeval/perception_test_local.py`. No private dataset or dataset credentials +are required. The Perception Test authors' non-code materials are +[CC BY 4.0](https://creativecommons.org/licenses/by/4.0/); retain attribution to +the [authors/source](https://github.com/google-deepmind/perception_test). +Code is Apache 2.0. Public access is not an additional clearance determination. + +Set `LAUNCHABLE_STATE_DIR` to an absolute writable host directory outside the +checkout. `PERCEPTION_TEST_DATA_DIR` defaults to its `perception-test` child, +mounted read-write at `/perception-test`; cache and results use separate mounts. +Compose project is `cosmos3-perception-test`, evaluator service `perception-eval`. +Authenticated Jupyter binds only to a private or loopback address. The local +model has no host port; no Docker socket or cloud credentials are mounted. + +Native `run.py` configuration uses the local dataset class directly: + +```json +{ + "data": { + "PerceptionTest": { + "class": "PerceptionTestLocal", + "dataset": "PerceptionTest", + "data_root": "/perception-test", + "nframe": 16, + "smoke": false + } + } +} +``` + +The controller composes this with `CosmosReason2` and the existing generation +recipe, defaulting to one API worker. The launchable prototype samples 16 frames +uniformly inside official source cutpoints: `-1` means all frames, positive cuts +are exclusive stops and invalid cuts fail. Aspect-preserving downscale only to +maximum side 768, without cropping, precedes direct sampled-image input. This +protocol and text-to-option parser are adaptations, not official parser parity +or a published-model/reference-score reproduction. + +`launchable.py download --scope metadata|smoke|full` stages public inputs. +`prepare --smoke` and `prepare` require complete smoke/full media respectively. +`run --mode infer|all` generates predictions and optionally scores them; the +notebook defaults to `all`, not a manual Docker benchmark command. +`score --scope smoke|full` accepts either an existing successful inference +`--work-dir` or native `--predictions` plus a fresh `--score-dir`. Score-only +downloads metadata only and prepares with `require_media=False`: no video files, +endpoint, GPU, or model construction. Canonical annotation/cut/scope fingerprints +are independent of media presence; media hashes are recorded separately. + +Exact selected identities are mandatory. Missing/blank/transport-failure rows +reject incomplete runs; nonempty malformed/ambiguous answers count wrong and +invalid. Trusted labels come only from pinned annotations. Primary metric is +local top1 (`correct / total selected questions`), with per-area, per-reasoning +and per-tag diagnostics, never an API judge. Fresh outputs, failed-run artifacts, +streamed progress, source/model revisions and UTC elapsed timings are retained. +The notebook shows concise metrics/coverage and artifact links. + +BYOM uses complete `COSMOS_API_BASE`, exact `COSMOS_MODEL`, optional +environment-only `COSMOS_API_KEY`, and explicit `COSMOS_MODEL_REVISION`; +setup then skips local serving. Score-only setup uses `bash setup.sh --score-only`. +For your own labeled video MCQ data, follow the +[customer-data recipe](launchable/CUSTOM_DATA_RECIPE.md) and +[Customer_Data.ipynb](launchable/Customer_Data.ipynb). This is a separate +canonical JSONL path: official benchmark identity/checksum gates remain intact. +Questions, answer options and actual ground-truth labels are required before +GPU use; footage alone cannot produce an accuracy score. Existing agents can +perform the one-off conversion described by the recipe. No customer-ingestion +agent, benchmark builder, automatic converter or other model deployment track +is added. diff --git a/.agents/skills/evaluation/cosmos3/reasoner/vlmevalkit/README.md b/.agents/skills/evaluation/cosmos3/reasoner/vlmevalkit/README.md index a01f18d98..9a05f4816 100644 --- a/.agents/skills/evaluation/cosmos3/reasoner/vlmevalkit/README.md +++ b/.agents/skills/evaluation/cosmos3/reasoner/vlmevalkit/README.md @@ -3,16 +3,33 @@ SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All SPDX-License-Identifier: Apache-2.0 --> -# TAR and VANTAGE Benchmarks for Cosmos3 Model +# Cosmos3 evaluation: public benchmark launchable and native recipes -This repository is an **NVIDIA-customized [VLMEvalKit](https://github.com/open-compass/VLMEvalKit)** that -releases the NVIDIA Metropolis "smart infra" benchmarks and makes the cosmos-reasoner scores on them -**publicly reproducible**. The reproduction kit lives in [`cosmos_eval/`](cosmos_eval/); the stock -VLMEvalKit inference + scoring engine it drives is the rest of the tree (upstream toolkit docs: -[`README-vlmevalkit.md`](README-vlmevalkit.md)). +This directory is an **NVIDIA-customized [VLMEvalKit](https://github.com/open-compass/VLMEvalKit)** +maintained in Cosmos Framework. It contains reference settings for NVIDIA +Metropolis "smart infra" evaluation; [`cosmos_eval/`](cosmos_eval/) composes the +native inference and scoring engine. Toolkit docs live in +[`README-vlmevalkit.md`](README-vlmevalkit.md). These settings do not make private +datasets publicly reproducible or establish official score parity. For user-hosted endpoints and local dataset inputs, see [Deployment and dataset configuration](EXTERNAL_CONFIGURATION.md). +For the Brev notebook workflow, use the +[public benchmark launchable](launchable/README.md): MMSI-Video-Bench is the +default full reference run, with Perception Test and VideoPhy2 also +available. Public inputs are downloaded by the customer; local scoring does +not use an external judge or private dataset credentials. Expected pinned +measurements and their comparison limits are included. + +For your own labeled video multiple-choice data, follow the +[standalone customer-data recipe](launchable/CUSTOM_DATA_RECIPE.md), then use +[Customer_Data.ipynb](launchable/Customer_Data.ipynb). Videos alone do not +provide an accuracy score: actual questions, options and ground-truth labels +are required. Existing agents perform one-off schema conversion; no benchmark +builder, packaging workflow or automatic ingestion agent is included. +Data and state stay outside the checkout. The launchable does not claim +identical published-reference protocols. Existing native recipes below remain +independent and unchanged. The kit ships, for each benchmark, reference `run.py --config` settings — split into a shared dataset layer (`cosmos_eval/data/`) and a per-family model layer @@ -85,7 +102,7 @@ Serve the model behind an OpenAI-compatible `/v1/chat/completions` endpoint (see **[Reasoner with vLLM](../../../../cookbooks/cosmos3/reasoner/README.md#run-with-vllm)**): ```bash -vllm serve nvidia/Cosmos3-Nano \ +VLLM_USE_FLASHINFER_SAMPLER=0 vllm serve nvidia/Cosmos3-Nano \ --async-scheduling \ --allowed-local-media-path / \ --port 8080 \ @@ -104,6 +121,41 @@ configs request, and `--allowed-local-media-path /` lets it read local media. (R hosted NIM, …) as long as it answers `POST /v1/chat/completions` and `GET /v1/models`; the wrapper health-checks the endpoint and auto-detects the served model id from `/v1/models`. +The launchable pins `vllm/vllm-openai:cosmos3@sha256:db0bb920b0b54e82ea96a98659bbd21921f87d0dcfc86feffdafa2db3f08be55` +and model revision `e59a53c25979a090fa8706c9acc0c254a6e89b92`. Its inspected +vLLM version is `0.23.1rc1.dev1306+gbfee6a802`. This build does **not** expose +`--trust-request-mm-kwargs` in `vllm serve --help=all`: the supported equivalent +is its unconditional request override path. `renderers/online_renderer.py` +passes request `mm_processor_kwargs` into `RenderParams`; `renderers/hf.py` +forwards those settings into the multimodal prompt/processor. Do not add the +unsupported flag or replace this digest merely to obtain it. Other runtime +versions must be checked for their own trust/override policy. The sampler +environment override above also applies to the launchable's serving container. + +Public benchmark runs save redacted requested/observed execution evidence per +response and an `execution-summary.json` aggregate, referenced by their +summary/provenance. Native `CosmosReason2` collection is opt-in via +`execution_evidence_dir` and `execution_processor_profile`; disabled collection +does not change its payload or return API. Evidence requests raw token IDs but +retains only visual groups, actual response model, usage and finish reasons—not +prompts, text token IDs, labels, media, endpoint URLs or credentials. + +Nano's explicit processor profile comes from the pinned public +[`config.json`](https://huggingface.co/nvidia/Cosmos3-Nano/resolve/e59a53c25979a090fa8706c9acc0c254a6e89b92/config.json) +and [`preprocessor_config.json`](https://huggingface.co/nvidia/Cosmos3-Nano/resolve/e59a53c25979a090fa8706c9acc0c254a6e89b92/preprocessor_config.json): +patch 16, spatial merge 2, temporal patch 2. Each bounded consecutive visual +token group yields spatial grid area `tokens × (16 × 2)²` **pixels**; video +groups yield **frame slots** `groups × 2`, not decoded or unique frame counts. +These derivations assume that processor profile and unpruned placeholder +expansion, not a server-attested revision. Per-group geometry is checked, not +just total tokens. Width/height, decoder FPS, decoded frame counts and effective +generation settings remain explicitly unknown when the response cannot expose +them. Non-32-aligned image inputs can be resized by the processor, so original +input dimensions are not relabeled as effective pixel geometry. Requested +generation/revision and historical scientific provenance remain requests and +provenance; they are not newly verified by a successful completion. + + --- ## 3. Environment variables diff --git a/.agents/skills/evaluation/cosmos3/reasoner/vlmevalkit/launchable/.gitignore b/.agents/skills/evaluation/cosmos3/reasoner/vlmevalkit/launchable/.gitignore new file mode 100644 index 000000000..dd26ad7de --- /dev/null +++ b/.agents/skills/evaluation/cosmos3/reasoner/vlmevalkit/launchable/.gitignore @@ -0,0 +1,3 @@ +state/ +.env +.ipynb_checkpoints/ diff --git a/.agents/skills/evaluation/cosmos3/reasoner/vlmevalkit/launchable/CUSTOM_DATA_RECIPE.md b/.agents/skills/evaluation/cosmos3/reasoner/vlmevalkit/launchable/CUSTOM_DATA_RECIPE.md new file mode 100644 index 000000000..ee143aa16 --- /dev/null +++ b/.agents/skills/evaluation/cosmos3/reasoner/vlmevalkit/launchable/CUSTOM_DATA_RECIPE.md @@ -0,0 +1,177 @@ +# Evaluate your labeled video multiple-choice data + +This is a local **customer-video MCQ** evaluation, not an official benchmark protocol, published Cosmos score, or leaderboard submission. It does not convert arbitrary datasets automatically, build benchmarks, use an external judge, or obtain private labels. Each question has 2–26 options, one known correct answer, and optional category and temporal bounds. There is no requirement to use MVBench's tasks or three options. + +## 1. Labels and permissions first — before provisioning a GPU + +Obtain the selected videos and their actual ground-truth answers, confirm permission to process them, and confirm that every selected question has exactly one labeled answer. An unlabeled test split, a caption-only collection, or a dataset whose answers require inaccessible credentials is **not ready**. Stop before GPU provisioning if labels or videos are missing. Never infer, invent, or ask the evaluated model to generate missing ground truth. Record the split, source revision, license, inclusion rules, and any deliberate subset. A small subset is a customer subset, not the full source benchmark. + +Choose an external persistent directory, for example `/mnt/customer-eval/inspection-v1/`. Keep inputs and outputs outside the source checkout and outside source control. Do not paste private data or credentials into shared notebooks, commits, or conversion-agent prompts without authorization. Conversion agents should work with authorized local files only. + +Use exactly this layout (video filenames are examples, not supplied assets): + +```text +/mnt/customer-eval/inspection-v1/ + rows.jsonl + config-nano.json + config-qwen.json + media/ + invented_clip_001.mp4 + invented_clip_002.mp4 +``` + +No annotations or videos are included in this recipe. The examples below are invented metadata to explain the schema, not evaluation data or labels for any real collection. + +## 2. Map your source columns into canonical `rows.jsonl` + +Have an existing Codex or Claude coding agent inspect the source schema, source documentation, and a few authorized rows. Give it the schema and mapping instructions below. Ask it to write canonical JSONL directly into the external directory; do not create a reusable converter product or alter the public benchmark loaders. + +Before conversion, require the agent to state a concrete mapping table, including these decisions: + +| Canonical field | Mapping decision to document | +| --- | --- | +| `id` | Stable source question ID, including split/video/question identity if source IDs are not globally unique. Never a model-dependent ID. | +| `video` | Existing local video path relative to `media_root`; map source video IDs/filenames explicitly. Do not guess by caption similarity. | +| `question` | Actual question text, not an answer-bearing explanation, caption, rationale, or task label. | +| `candidates` | Ordered source options as an array of full strings. Preserve order and exact text; 2–26 unique nonempty entries. | +| `answer` | Full text of the one correct candidate. Resolve a source letter/index/option-column through the documented option order and index base. Never treat `0` and `1` index conventions interchangeably. | +| `category` | Optional existing source task/category or a documented, fixed grouping. Do not invent a new taxonomy per model. | +| `start`, `end` | Optional paired numeric seconds into the source video, with source units converted explicitly. If the source already provides trimmed clips, use the clip's own timeline, not original-video timestamps. | + +Suitable conversion-agent instruction: + +> Read CUSTOM_DATA_RECIPE.md and inspect the authorized local dataset. Report the source-to-canonical mapping, answer index base, split, selected count, video mapping, timestamp units and category meanings first. Produce only rows.jsonl and a config JSON in my external data directory using the exact schema below. Preserve source question IDs, option order, actual labels and video content. Resolve each source label to exactly one complete candidate string. Do not generate missing labels, download untrusted replacements, change the public benchmark loaders, add a converter tool, or request a GPU. Validate every selected row and list missing/ambiguous inputs rather than silently skipping them. Run custom_eval.py validate and resolve all validation failures before inference. Record the inclusion rules and source revision in dataset.source/revision. Do not include credentials in any artifact. + +Canonical `rows.jsonl` is UTF-8, one complete JSON object per line, no blank lines. These two invented rows illustrate different option counts and category/temporal settings: + +```jsonl +{"id":"inspection-v1:invented-001:q1","video":"invented_clip_001.mp4","question":"Which direction does the box move?","candidates":["Left","Right","It stays still","It leaves the frame"],"answer":"Right","category":"object-motion","start":0.0,"end":2.0} +{"id":"inspection-v1:invented-002:q1","video":"invented_clip_002.mp4","question":"Does the person place the cup on the table?","candidates":["Yes","No"],"answer":"Yes","category":"action-completion"} +``` + +### Content and temporal semantics + +- All rows are selected; there is no hidden sampling or automatic row dropping. Make a separate fixed manifest and dataset revision for an intentional subset. +- IDs are nonempty strings, globally unique within the manifest. Multiple questions can refer to one video but must have different IDs. +- `video` uses a safe relative POSIX path within `media_root`; absolute paths, URLs, `..`, backslashes, colon-containing paths and symlink escapes are rejected. Copy authorized media into the root or use paths that resolve within it. Missing or empty files fail validation. +- `question`, candidate strings, ID and category cannot be blank or have surrounding whitespace. Unknown fields are rejected so source labels/rationales are not accidentally sent to inference. Candidate duplicates are rejected. Keep semantically duplicate options out of the source selection too; validation cannot determine semantic equivalence. +- `answer` must match exactly one candidate string, including case and punctuation. The runner sends *all candidate texts* but never sends which one is correct, the answer field, IDs, categories or source labels. The source text itself must not disclose the answer in a rationale or annotation; inspect this during mapping. +- `start` and `end` must both be omitted for a full clip, or both supplied as finite numeric seconds with `0 <= start < end <= duration`. Booleans, NaN and infinity are invalid. Questions must be answerable from the selected segment; validation checks bounds, not semantic relevance. Native decoding performs temporal selection before sampling. +- Categories describe the question, not the predicted answer. Missing category is reported as `uncategorized`. Category accuracy is a question-weighted percentage within that category; overall accuracy is question-weighted across every selected row, not an average of category percentages. If a category is literally named `uncategorized`, it shares the missing-category bucket. + +## 3. Write `config-nano.json` + +Paths resolve relative to the **config file**, never the current working directory. Absolute external input paths also work. Adapt the invented dataset identity, source, model revision and pixel/frame settings to your data and deployed model: + +```json +{ + "dataset": { + "name": "customer-inspection-mcq", + "revision": "inspection-v1-labeled-fixed-selection", + "source": "Authorized local inspection export; source revision example-r1; fixed selected split; mapping documented by data owner" + }, + "manifest": "rows.jsonl", + "media_root": "media", + "nframes": 32, + "min_pixels": 50176, + "max_pixels": 50176, + "image_patch_size": 16, + "model": "nvidia/Cosmos3-Nano", + "model_revision": "unknown", + "generation": { + "temperature": 0, + "seed": 42, + "max_tokens": 32, + "enable_thinking": false + } +} +``` + +Customization points: + +- `dataset.name/revision/source` are required nonempty strings. Set revision to the real fixed export/selection identity; record transformations and licensing/source context in `source`. Do not put secrets or credential-bearing URLs here. +- `nframes` is an even integer at least 4, requested of the existing native video utility. Choose it to fit the segment durations and model capacity. Native decoding can reject clips that cannot satisfy the request; the runner pre-decodes every selected clip before any endpoint request. +- `min_pixels` and `max_pixels` are positive per-frame spatial pixel budgets (`min <= max`), not a total video token limit. `image_patch_size` must match the deployed processor; the example is not a universal model profile. Native shape rounding may change actual dimensions. Keep budgets fixed for like-for-like comparisons unless explicitly studying budgets. +- `model` and `model_revision` identify the deployed model. If omitted, use `COSMOS_MODEL` and `COSMOS_MODEL_REVISION`; model must be supplied in config or environment, and absent revision is honestly `unknown`. A config value takes precedence. The runner does not select/deploy a different endpoint based on the model field; point the endpoint to that actual deployment. +- Optional `processor_profile` is an evidence object supplied only from verified deployment/processor provenance, with `patch_size`, `spatial_merge_size`, `temporal_patch_size`, `image_token_id`, `video_token_id`, `provenance`, and the bounded-group markers `vision_start_token_id`/`vision_end_token_id` as supported by `vlmeval.execution_evidence`. When omitted for the exact `nvidia/Cosmos3-Nano` model name, the shared helper uses its pinned, provenance-recorded Nano processor profile; other models do not inherit that profile. Do not copy Nano values to another model or invent a profile to obtain a verified status. Missing runtime observability remains unknown; requested budgets alone are not proof of effective execution. Even verified bounded token-group counts do not attest decoded frame count, decoder FPS, generation settings or effective model revision. +- `generation` supports only `temperature`, nonnegative integer `seed`, positive integer `max_tokens`, and boolean `enable_thinking`. Defaults are the example values. The prompt requires one uppercase letter; a very small token cap or thinking mode can cause truncation, which counts wrong. Do not retry malformed answers selectively. +- No judge, scoring URL, label URL, automatic converter, output overwrite, or resume settings exist. Scoring is always local strict MCQ accuracy. Output location is passed separately with a fresh `--work-dir`. + +## 4. Validate all inputs before using an endpoint + +The toolkit is `.agents/skills/evaluation/cosmos3/reasoner/vlmevalkit` in this repository. In a launchable installation, use its toolkit root containing `vlmeval/` and `launchable/`. Use the installed launchable Python environment (including decord, ffmpeg and the native Cosmos video dependencies). From that root: + +```bash +python launchable/custom_eval.py validate --config /mnt/customer-eval/inspection-v1/config-nano.json +``` + +Validation is CPU-side and sends no endpoint requests. It validates all labels, IDs, candidate counts, safe media paths, media duration and temporal ranges, probes media frames, and hashes every selected media file. Resolve every failure; never remove inconvenient rows just to obtain a completed score. An intentional subset needs a new documented identity before model comparison. Keep the returned `dataset_identity`, expected count and `manifest_sha256`. The semantic data identity includes dataset metadata, canonical rows and video hashes/metadata; reformatting JSON does not change it, but changing labels, options, selection, categories, temporal bounds or media bytes does. The raw manifest hash can change on harmless JSON reformatting. + +## 5. Run from notebook or CLI + +Open `launchable/Customer_Data.ipynb` from the `launchable/Evaluate_Cosmos3.ipynb` overview. Set its visible `CONFIG_FILE` to your absolute external config path; the notebook invokes the same validate/run CLI and renders `summary.json`. Use the overview/launchable endpoint workflow for deployment. The same Python API can also be called from a notebook cell after making the toolkit root and `launchable/` importable: + +```python +from pathlib import Path +import sys +sys.path.insert(0, str(Path("launchable").resolve())) +import custom_eval + +config = Path("/mnt/customer-eval/inspection-v1/config-nano.json") +validated = custom_eval.validate(config) +summary = custom_eval.run(config, Path("/mnt/customer-eval/results/nano-inspection-v1-run01"), api_nproc=1) +summary +``` + +This cell assumes the notebook working directory is the toolkit root; adjust the import path to the actual toolkit installation if not. Endpoint startup and model deployment remain the notebook/launchable's responsibility. Do not provision an endpoint until validation succeeds. + +For CLI, export the actual complete chat-completions URL; keep a real API key only in the environment: + +```bash +export COSMOS_API_BASE=http://cosmos-endpoint:8080/v1/chat/completions +export COSMOS_MODEL=nvidia/Cosmos3-Nano +export COSMOS_MODEL_REVISION=unknown +export COSMOS_API_KEY=EMPTY +python launchable/custom_eval.py run \ + --config /mnt/customer-eval/inspection-v1/config-nano.json \ + --work-dir /mnt/customer-eval/results/nano-inspection-v1-run01 \ + --api-nproc 1 +``` + +`EMPTY` is only for a local endpoint that does not require authentication. Use your authorized secret environment for authenticated endpoints; never store keys in config. `COSMOS_TIMEOUT` optionally sets positive finite request timeout seconds (default 300). Concurrency is 1–4; use 1 first. No endpoint URL or key is deliberately persisted. Dataset/source strings are user metadata, so keep them credential-free. + +Choose an external output parent that the evaluation process owns or can write to. Inputs may remain read-only. If an operator-provided output directory has the wrong owner, correct that directory's ownership or choose a writable output location before running; do not broaden raw-data permissions or alter the mapping to bypass a filesystem error. + +A result directory **must not already exist**, even if empty. Each run validates inputs again, pre-decodes every selected clip with the existing native video processor, and only then issues requests. Temporary transport files can contain encoded customer video while running and are removed when the run exits normally or through handled exceptions; keep work storage private and remove abandoned temporary files after an unclean process kill. + +## 6. Inspect local scoring and compare models + +Outputs under the fresh work directory: + +- `validated.json`: fixed dataset identity, source metadata, media SHA256/durations, selected count, requested sampling/model/generation. +- `requested-execution.json`: requested model/generation/processor budget/profile, not proof of runtime execution. +- `responses/000000.json`, etc.: per-request HTTP status, shared requested/observed execution evidence, strict parsed answer outcome, or redacted exception type. No inference prompt or video base64 is retained in response evidence. Arbitrary malformed model text is not retained. +- `execution-summary.json`: shared aggregate of recorded requested settings, observed bounded visual-group shapes, verification counts and unknown fields. Also embedded in `summary.json` as `execution_evidence`; incomplete evidence coverage or matching token geometry does not establish endpoint/model attestation or all effective media settings. +- `results.jsonl`: one result per processed row, stable ID, category, parsed single-letter prediction (or null), validity, correctness, finish reason and response-evidence pointer; transport errors have an `error` instead of a correctness value. +- `summary.json`: `benchmark=customer-video-mcq`, `scope=custom`, dataset/model identity, `status`, expected/scored coverage, elapsed seconds, execution pointers, overall accuracy and category breakdown. + +A valid answer is exactly one uppercase candidate letter after surrounding whitespace is removed. Full option text, lowercase letters, multiple choices, explanations, blanks and truncated (`finish_reason=length`) answers are **wrong** and remain in the denominator. Ground-truth labels are stored locally in your input manifest, never sent as an answer field. HTTP/connection/JSON transport failures are **not** wrong answers: the run fails, exits nonzero, and does not report a completed accuracy. Successful HTTP JSON responses with missing/malformed answer choices count as invalid wrong answers, while their execution evidence remains unverified. Inspect `status=completed`, `coverage.expected == coverage.scored`, and evidence availability before quoting any score. A preflight failure also fails the run without issuing requests. Never compare a failed/partial result as if complete. + +For Qwen or another compatible deployment, copy `config-nano.json` to `config-qwen.json` and change only the actual model/revision and provenance-grounded processor settings needed for that deployment. Point `COSMOS_API_BASE` to its endpoint. Do not regenerate the manifest, relabel options, alter selection, or replace media between runs. Use a different fresh work directory, for example `/mnt/customer-eval/results/qwen-inspection-v1-run01`. Use the exact model-supported patch size/profile; document any unavoidable processor budget differences. + +Before reporting comparison, require both runs completed, equal expected/scored counts, and identical `dataset_identity`; verify manifest/media identities if transferring between machines. Compare `primary_metric.value` (percent) with `correct/total`, per-category `accuracy_percent`, elapsed seconds and requested/observed execution evidence. Identify model revision, any budget differences, unknown runtime facts and transport/deployment environment. Preserve first-run results rather than overwriting or selectively regenerating wrong answers. The result measures this fixed customer selection under these settings, not official benchmark parity or leaderboard rank. + +### Fields reported as unknown + +Every run lists the same five fields in `execution_evidence.unknown_fields`. A standard chat-completions response does not report these values, so they are left unknown rather than filled in from the requested settings. + +| Field | Meaning | +| --- | --- | +| `decoded_video_frames` | Frames actually decoded and the source frame indices used. Frame slots are reported separately and can include repeated or padded frames. | +| `processor_width_height` | Width, height and crop used by the processor. Grid area is reported, but area alone does not determine dimensions. | +| `effective_decoder_fps` | Sampling FPS the decoder actually used, as opposed to the requested or source FPS. | +| `effective_generation_settings` | Generation parameters the server actually applied, as opposed to the requested configuration. | +| `effective_model_revision` | Checkpoint revision the server actually loaded, as opposed to the configured revision or returned model name. | + +For image inputs, the three video-decoder fields do not apply. What is reported: requested settings, the model name returned by the server, token usage, and visual-token counts. Frame slots and grid areas are derived from those token counts using the processor profile. + diff --git a/.agents/skills/evaluation/cosmos3/reasoner/vlmevalkit/launchable/Customer_Data.ipynb b/.agents/skills/evaluation/cosmos3/reasoner/vlmevalkit/launchable/Customer_Data.ipynb new file mode 100644 index 000000000..a94812d4f --- /dev/null +++ b/.agents/skills/evaluation/cosmos3/reasoner/vlmevalkit/launchable/Customer_Data.ipynb @@ -0,0 +1,172 @@ +{ + "cells": [ + { + "id": "welcome", + "cell_type": "markdown", + "metadata": {}, + "source": [ + "[← All benchmarks](Evaluate_Cosmos3.ipynb)\n", + "\n", + "# Your labeled video data\n", + "\n", + "**Labels first:** footage alone cannot produce an accuracy score. You need questions, answer options, and actual ground-truth answers. Confirm data permissions before using a GPU.\n", + "\n", + "Follow the [standalone customer-data recipe](CUSTOM_DATA_RECIPE.md) to map your source files into canonical JSONL and a config JSON. This is a customer-defined evaluation, not an official benchmark or leaderboard submission.\n" + ] + }, + { + "id": "helpers", + "cell_type": "code", + "metadata": { + "jupyter": { + "source_hidden": true + }, + "scrolled": true + }, + "execution_count": null, + "outputs": [], + "source": [ + "import json\n", + "import os\n", + "from pathlib import Path\n", + "import sys\n", + "from datetime import datetime, timezone\n", + "from notebook_ui import invoke, show_customer_results\n", + "\n", + "CLI = [sys.executable, \"-u\", str(Path.cwd() / \"custom_eval.py\")]\n" + ] + }, + { + "id": "configure", + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 01 / Point to your configuration\n", + "\n", + "Set the absolute path to your external config file. It names your manifest, video directory, model, frame/pixel budget, and generation settings. Paths inside that file resolve relative to the config—not this notebook. Keep inputs fixed when comparing models.\n" + ] + }, + { + "id": "settings", + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "CONFIG_FILE = Path(\"/datasets/customer-eval/config-nano.json\")\n", + "API_NPROC = 1 # 1–4; run one benchmark notebook at a time.\n", + "\n", + "RESULTS_ROOT = Path(os.environ.get(\"EVAL_RESULTS_DIR\", \"/results\"))\n", + "STAMP = datetime.now(timezone.utc).strftime(\"%Y%m%dT%H%M%S%fZ\")\n", + "WORK = RESULTS_ROOT / \"customer-video-mcq\" / f\"custom-{STAMP}\"\n", + "LOG = WORK.with_suffix(\".log\")\n", + "if not CONFIG_FILE.is_absolute():\n", + " raise ValueError(\"Set CONFIG_FILE to your absolute external configuration path.\")\n", + "print(f\"Configuration: {CONFIG_FILE}\\nNew result directory: {WORK}\")\n" + ] + }, + { + "id": "validate", + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 02 / Validate labels and media\n", + "\n", + "This CPU-only step checks every selected question, label, ID, option list, video, and temporal bound before any model request. Missing or ambiguous inputs stop the run; they are not silently skipped. Review the reported count and dataset identity.\n" + ] + }, + { + "id": "validate-inputs", + "cell_type": "code", + "metadata": { + "jupyter": { + "source_hidden": true + }, + "scrolled": true + }, + "execution_count": null, + "outputs": [], + "source": [ + "invoke(CLI + [\"validate\", \"--config\", str(CONFIG_FILE)], WORK.with_suffix(\".validation.log\"))\n" + ] + }, + { + "id": "run", + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 03 / Run and score locally\n", + "\n", + "The endpoint must already serve the model named in your configuration. Only the question and all options are sent—not the correct-answer field. Invalid or truncated answers count wrong; transport failures cannot become a completed score. No external judge or manual benchmark command.\n" + ] + }, + { + "id": "infer-and-score", + "cell_type": "code", + "metadata": { + "jupyter": { + "source_hidden": true + }, + "scrolled": true + }, + "execution_count": null, + "outputs": [], + "source": [ + "invoke(CLI + [\"run\", \"--config\", str(CONFIG_FILE), \"--work-dir\", str(WORK),\n", + " \"--api-nproc\", str(API_NPROC)], LOG)\n" + ] + }, + { + "id": "review", + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 04 / Inspect the score and execution evidence\n", + "\n", + "Check complete coverage, category scores, dataset identity, and requested versus observable effective input settings. An unknown effective setting is not a verified setting. Retain the model revision and any budget differences.\n" + ] + }, + { + "id": "results", + "cell_type": "code", + "metadata": { + "jupyter": { + "source_hidden": true + }, + "scrolled": false + }, + "execution_count": null, + "outputs": [], + "source": [ + "summary = json.loads((WORK / \"summary.json\").read_text())\n", + "show_customer_results(summary, WORK, LOG)\n" + ] + }, + { + "id": "compare", + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Compare another model\n", + "\n", + "Deploy or select the second endpoint using the [recipe](CUSTOM_DATA_RECIPE.md#6-inspect-local-scoring-and-compare-models). Change only the model/revision and any required processor profile in a copy of your config. Keep the manifest and videos unchanged; rerun this notebook for a fresh result directory. Require equal dataset identities and coverage before comparing scores. Closing a notebook tab does not stop a running kernel or Brev VM billing.\n", + "\n", + "Share reviewed aggregate scores and code—not labels, videos, raw predictions, credentials, or executed notebooks.\n" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.11" + }, + "title": "Your labeled video data" + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/.agents/skills/evaluation/cosmos3/reasoner/vlmevalkit/launchable/Dockerfile b/.agents/skills/evaluation/cosmos3/reasoner/vlmevalkit/launchable/Dockerfile new file mode 100644 index 000000000..6d8912f72 --- /dev/null +++ b/.agents/skills/evaluation/cosmos3/reasoner/vlmevalkit/launchable/Dockerfile @@ -0,0 +1,21 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +FROM python:3.11.14-slim-bookworm@sha256:65a93d69fa75478d554f4ad27c85c1e69fa184956261b4301ebaf6dbb0a3543d +RUN apt-get update && apt-get install -y --no-install-recommends \ + build-essential ffmpeg git libgl1 libglib2.0-0 libcairo2 \ + && rm -rf /var/lib/apt/lists/* +WORKDIR /opt/vlmevalkit +COPY requirements.txt setup.py setup.cfg README-vlmevalkit.md ./ +# Evaluation is API-based; keep its PyTorch stack off the serving GPU. +RUN pip install --no-cache-dir torch==2.8.0 torchvision==0.23.0 --index-url https://download.pytorch.org/whl/cpu \ + && pip install --no-cache-dir -r requirements.txt einops accelerate 'setuptools<81' evaluate jupyterlab==4.4.10 +COPY . . +RUN pip install --no-cache-dir --no-deps -e . \ + && python -c 'import vlmeval' \ + && pip freeze > /opt/evaluation-packages.txt +ARG COSMOS_COMMIT=unknown +ENV COSMOS_COMMIT=${COSMOS_COMMIT} PYTHONUNBUFFERED=1 HF_HOME=/cache/huggingface \ + AWS_CONFIG_FILE=/dev/null AWS_SHARED_CREDENTIALS_FILE=/dev/null \ + AWS_EC2_METADATA_DISABLED=true PERCEPTION_TEST_DATA_DIR=/perception-test BENCHMARK_DATA_DIR=/datasets LMUData=/cache/lmudata +WORKDIR /opt/vlmevalkit/launchable +CMD ["python", "jupyter_config.py"] diff --git a/.agents/skills/evaluation/cosmos3/reasoner/vlmevalkit/launchable/Evaluate_Cosmos3.ipynb b/.agents/skills/evaluation/cosmos3/reasoner/vlmevalkit/launchable/Evaluate_Cosmos3.ipynb new file mode 100644 index 000000000..fb6bc877b --- /dev/null +++ b/.agents/skills/evaluation/cosmos3/reasoner/vlmevalkit/launchable/Evaluate_Cosmos3.ipynb @@ -0,0 +1,52 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "overview", + "metadata": {}, + "source": [ + "
NVIDIA Cosmos · Evaluation workspace
\n", + "Run the default reference benchmark, compare another public benchmark, or evaluate your own labeled videos. All scoring is local.
\n", + "Perception Test
What happened in the video?
Multiple-choice accuracy
Open notebook →\n", + "VideoPhy2
Does the video match its caption and obey physics?
Mean SA / PC Pearson correlation ×100
Open notebook →\n", + "MMSI-Video-Bench · Default reference run
Can the model reason about space across video views?
1,106 questions · Official-extractor answer accuracy
Open notebook →\n", + "{escape(item["scope"] if scope == "full" else item["smoke"])}
' + '{escape(item["protocol"])}
{escape(str(data))}{escape(str(work))}{escape(item["license"])}
Publisher source' + 'Public hosting does not clear all underlying redistribution rights. ' + 'Downloads stay on your instance; no raw data is bundled or uploaded by this notebook.
Grid areas and frame slots are conditional on the explicit processor profile. ' + 'They do not attest decoded/unique frames, width/height, or generation settings. ' + 'See the detailed record for unknowns and geometry differences.
') + + +def _show_results(summary, benchmark, scope, work, item, log_path=None): + if summary.get("benchmark") != benchmark or summary.get("scope") != scope: + raise ValueError("Result identity does not match this notebook's selected benchmark/scope") + if summary.get("status") not in {"completed", "completed_with_generation_issues"}: + raise ValueError("This run is not complete; inspect retained diagnostics before reporting a score") + metric = summary["primary_metric"] + value, unit = metric["value"], metric["unit"] + if value is not None and not math.isfinite(value): + raise ValueError("Metric must be finite or explicitly undefined") + headline = "Not defined" if value is None else f"{value:.2f}{'%' if unit == 'percent' else ''}" + coverage = summary["coverage"] + if coverage["scored"] != coverage["expected"]: + raise ValueError("Refusing to present incomplete coverage as a completed benchmark") + warnings = list(summary.get("warnings", [])) + issues = summary["status"] == "completed_with_generation_issues" + notes = "".join(f'Benchmark result
' + f'{escape(scope.title())} scope · {escape(benchmark)}
' + f'{notice}{escape(detail)}
Published-score equivalence has not been established.
' + f'{escape(json.dumps({"primary_metric": metric, "protocol": summary["protocol"]}, indent=2))}'
+ '{escape(json.dumps(execution, indent=2))}'
+ 'Null means not recorded in this artifact, not verified. ' + 'Effective input geometry is derived from observed processor tokens when supported; ' + 'requested generation parameters are not a server attestation.