[freshyjmp]

Summary

Notebook-driven Brev launchable for Cosmos3-Nano and VANTAGE evaluation, plus a repeatable setup for an existing SSH-accessible LaunchPad host.

• Pinned Nano checkpoint and vLLM image; model API has no published host port. setup.sh --brev binds authenticated Jupyter to Brev's private mesh interface, preserving provider login and Jupyter-token authentication. Non-Brev setup defaults to loopback.
• The notebook prepares public data, checks the endpoint, runs five real VQA questions, verifies predictions/canonical IDs, and keeps the full eight-task run opt-in.
• The public Nano/VQA smoke needs no Hugging Face token. Gated assets, including full SOT/SmartSpaces preparation, still require the appropriate token and accepted access terms.
• Public and internal harnesses, Python environments, and dataset roots remain separate. Public labels are withheld; public predictions are not locally scored as ground truth. Cross-harness/reference-score parity is not claimed.
• LaunchPad setup reuses the original controller and native dataset/scorer. Preparation is separate from credential-free, network-isolated inference. A resumable, data-only workstation-staging route supports hosts that cannot reach PDX.
• Reject empty/API/video-load failures and incorrect smoke row counts. Public subprocesses use a headless plotting backend rather than inheriting Jupyter's inline backend.

Verification exercised

• Brev, commit 23cfa62968f0a395481b52c4fd91bae9ce1106dd: actual authenticated Jupyter UI Run All on one 96 GB RTX PRO 6000 Blackwell Server Edition; prepared 1,195 public VQA questions/282 videos; produced exactly five valid native predictions and five unique canonical IDs. Full-eight execution remained disabled. Screenshots, executed notebook, and provenance retained privately. Test VM confirmed stopped.
• LaunchPad, commit 0297efb4ee50584055f5ba4f95293d433f71c0d0: actual single-H200 lifecycle and endpoint checks; staged 1,195 native questions/282 videos using the original converter/preparer; verified all referenced video paths. Cached preparation, five-question inference, and native local scoring passed without any remote AWS profile. Final source digest and acceptance/lifecycle evidence retained privately.
• Exercised repeated up, status/check, stop/resume, and refusal to overwrite existing results. Final project endpoint stopped; all eight GPUs free; original Jupyter/Traefik containers unchanged. The existing LaunchPad instance remains allocated.
• Seven artifact-safety regression tests passed on native Linux. Actual invalid public prediction artifacts were accepted before the verifier fix and rejected afterward. Actual CLI source isolation rejected .envrc and escaping symlinks before state creation.
• Real notebook execution reproduced and then verified fixes for private-proxy connection refusal and inherited inline-backend failure; these were not CLI-only simulations.

Run / review entry points

bash evaluation/cosmos3/reasoner/vlmevalkit/launchable/setup.sh --brev

• Brev setup and notebook workflow
• LaunchPad lifecycle, staging, smoke, and scoring

Tokens, private dataset media/labels, predictions, local state, and operational evidence are not included in this PR. Temporary copied AWS credentials were removed; the user's existing local profile was not modified by cleanup.

Draft / remaining release gates

• Brev validation used the same stopped/resumed VM and retained caches. An uninterrupted fresh-VM cold deployment has not been rerun.
• Full eight-task inference, official submission acceptance, and published-reference/leaderboard parity remain unverified. A five-question native score is only an integration smoke, not a benchmark result.
• Repository placement still needs Dan/maintainer confirmation: this draft targets NVIDIA/cosmos, while new-code guidance points to Cosmos Framework. Do not merge before that decision and applicable data/licensing/scoring approvals.
• No public launchable release or PR merge performed. Observed hardware is a tested smoke configuration, not a minimum-sizing or full-workload compatibility guarantee.

[d2b4]

can we make use a proper package manager like uv for managing the runtime path? Doing it this way will fail at runtime if we add dependencies in KIT that aren't included in root

[d2b4]

can we use a cred other than team-cosmos, or move the data we need into a public HF bucket asap?

[d2b4]

would it be appropriate to use the OSS version of vantage bench or code based on vlmevalkit for this, it seems we might be reimplementing more than we should

[d2b4]

We want to have a very high standard of code quality for OSS releases and I feel like this is far short of where we should be aiming, as a draft it has the right shape but can we give this another few passes before shipping?

[d2b4]

Let's make sure to focus on evaluation that requires only publicly available datasets; we want to be able to compute a score end to end using data that's already public or that we can release along with this code.

[d2b4]

For this project I believe we're only interested in benchmarks that can be automatically scored. We should remove anything about preparing a submission for external scoring and use only datasets that we can fully evaluate with public data.

[freshyjmp]

The latest Nano/Brev implementation is now in the maintained Framework repository: NVIDIA/cosmos-framework#321 (draft). Please use that PR for the current code review. It contains source, documentation, an unexecuted notebook and synthetic fixtures only; sensitive datasets, annotations, benchmark outputs and credentials are not included. The runtime code matches the privately validated candidate, while publication removes private operational notes and uses a clean DCO-signed commit. This older mixed-scope draft is superseded for the Nano/Brev track; no LaunchPad deployment is included in the Framework draft.

[yaoxu-crypto]

Closing in favor of NVIDIA/cosmos-framework#321, as @freshyjmp noted above. The evaluation suites moved to .agents/skills/evaluation/ in Cosmos Framework and were removed from this repo in #384, so review continues there. @d2b4's points (public-data-only evaluation, code quality before an OSS release) carry over to that PR.