Skip to content

Docs: refresh Generator benchmarks and replace old NIM guides #370

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
rahul-steiger-nv wants to merge 1 commit into from
+278 −1,951
Closed

Docs: refresh Generator benchmarks and replace old NIM guides #370

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ This repository is the home of the models: everything for exploring, running, an
<tr><td><b>See it work</b> — zero install</td><td><a href="https://build.nvidia.com/nvidia/cosmos3-nano">Video generation</a>, <a href="https://build.nvidia.com/nvidia/cosmos3-nano-reasoner">visual reasoning</a></td><td>1 min</td></tr>
<tr><td><b>Generate my first video</b></td><td><a href="#generate-your-first-video">Quickstart ↓</a></td><td>10 min</td></tr>
<tr><td><b>Reason over images &amp; video</b></td><td><a href="cookbooks/cosmos3/reasoner/run_with_vllm.ipynb">Reasoner notebook</a></td><td>10 min</td></tr>
<tr><td><b>Serve an OpenAI-compatible API</b></td><td><a href="cookbooks/cosmos3/README.md">Serving setup guide</a> — vLLM, vLLM-Omni, or NIM</td><td>30 min</td></tr>
<tr><td><b>Serve models through an API</b></td><td><a href="cookbooks/cosmos3/README.md">Serving setup guide</a> — vLLM, vLLM-Omni, or the <a href="cookbooks/cosmos3/nim/README.md">Cosmos3 Certified NIM</a></td><td>30 min</td></tr>
<tr><td><b>Post-train on my own data</b> — SFT, distillation, RL</td><td><a href="https://github.com/NVIDIA/cosmos-framework">Cosmos Framework</a>, then <a href="evaluation/">evaluate here</a></td><td>hours</td></tr>
<tr><td><b>Explore runnable notebooks</b></td><td><a href="cookbooks/cosmos3/README.md">Cookbooks</a></td><td>browse</td></tr>
<tr><td><b>Evaluate a model</b></td><td><a href="evaluation/">Evaluation suites</a> — PAIBench, Physics-IQ, VLMEvalKit</td><td>hours</td></tr>
Expand Down Expand Up @@ -164,7 +164,7 @@ Training, optimization, and deployment tooling lives in [Cosmos Framework](https
| [Cosmos Curator](https://github.com/NVIDIA/cosmos-curator) | Distributed data curation: processing, annotation, filtering, dedup |
| [Cosmos Evaluator](https://github.com/NVIDIA/cosmos-evaluator) | Automated evaluation system for world generation & reasoning outputs |

Cosmos 3 runs on Diffusers, Transformers, vLLM, vLLM-Omni, SGLang, TensorRT-LLM, and NIM — pick a backend in the [environment setup guide](cookbooks/cosmos3/README.md).
Cosmos 3 runs on Diffusers, Transformers, vLLM, vLLM-Omni, SGLang, TensorRT-LLM, and [NIM](cookbooks/cosmos3/nim/README.md) — pick a backend in the [environment setup guide](cookbooks/cosmos3/README.md).

## Community & contributing

Expand Down
96 changes: 6 additions & 90 deletions cookbooks/cosmos3/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,6 @@ backend you want to run and follow that one section.
| [vLLM](#vllm) | OpenAI-compatible reasoning server (image/video understanding) | Reasoner |
| [vLLM-Omni](#vllm-omni) | OpenAI-compatible generation server (image/video/audio/action/transfer) | Generator (Audiovisual, Action, **Transfer**) |
| [Cosmos3 Certified NIM](#cosmos3-certified-nim) | Prebuilt NGC container serving either the Generator (image/video/action/transfer) or the Reasoner from one image | Reasoner, Generator (Audiovisual, Action, Transfer) |
| [Reasoner NIM](#reasoner-nim) | Prebuilt OpenAI-compatible reasoning server (image/video understanding); no venv | Reasoner |
| [Generator NIM](#generator-nim) | Prebuilt NGC container serving the Cosmos3 Generator for Text-to-Video and Image-to-Video inference | Generator (Audiovisual) |

## Prerequisites

Expand All @@ -35,9 +33,9 @@ backend you want to run and follow that one section.
To disable the guardrail, set `enable_safety_checker=False` (Diffusers),
`TRTLLM_DISABLE_COSMOS3_GUARDRAILS=1` or `use_guardrails: false` through
`extra_params` (TensorRT-LLM), `guardrails: false` (vLLM-Omni
`extra_params`/`extra_args`), or `--no-guardrails` (Cosmos Framework). For Generator NIM set environment variables `NIM_ENABLE_TEXT_GUARDRAILS=0 NIM_ENABLE_VIDEO_GUARDRAILS=0`.
- NIMs don't need Hugging Face access; instead, an NGC API key is required
(used as `NGC_API_KEY`). You can generate one on [build.nvidia.com](https://build.nvidia.com/) or [NGC](https://catalog.ngc.nvidia.com/), then run `docker login nvcr.io` once (username `$oauthtoken`, password = your key). This repository uses the Reasoner NIM image `nvcr.io/nim/nvidia/cosmos3-reasoner` and the Generator NIM image `nvcr.io/nim/nvidia/cosmos3-generator`.
`extra_params`/`extra_args`), or `--no-guardrails` (Cosmos Framework). For the NIM Generator runtime, see [Guardrails](nim/operations.md#guardrails).
- NIM doesn't need Hugging Face access; instead, an NGC API key is required
(used as `NGC_API_KEY`). You can generate one on [build.nvidia.com](https://build.nvidia.com/) or [NGC](https://catalog.ngc.nvidia.com/), then run `docker login nvcr.io` once (username `$oauthtoken`, password = your key). Use [NIM](nim/README.md), image `nvcr.io/nim/nvidia/cosmos3:2.0.0`, and select the Generator or Reasoner runtime at launch.
- For the Cosmos Framework backend: access to `git@github.com:NVIDIA/cosmos-framework.git`.
- Enough local disk for the venv/image, the uv cache, and the model cache. Nano
downloads plus CUDA dependencies can take tens of GiB.
Expand Down Expand Up @@ -779,7 +777,7 @@ Ensure the server has enough GPUs for the product of enabled degrees

## NIM

Prebuilt NGC containers for Cosmos3. Like vLLM-Omni, NIM runs from Docker, so
A prebuilt NGC container image for Cosmos3. Like vLLM-Omni, NIM runs from Docker, so
there is no venv or `--torch-backend` to manage. Unlike the Hugging Face based
backends, NIM authenticates with an NGC API key instead of a Hugging Face token
(see [Prerequisites](#prerequisites)).
Expand All @@ -791,7 +789,7 @@ export NGC_API_KEY=<your_key>
echo "$NGC_API_KEY" | docker login nvcr.io --username '$oauthtoken' --password-stdin
```

Each NIM below exposes readiness at `GET /v1/health/ready` after model
NIM exposes readiness at `GET /v1/health/ready` after model
download, engine initialization, and warmup complete.

### Cosmos3 Certified NIM
Expand Down Expand Up @@ -833,95 +831,13 @@ selects the checkpoint: `nano` or `super` for either runtime, plus
`super-i2v-4step` for the Generator. On DGX Spark/GB10 and Jetson AGX Thor, the
Reasoner also needs `NIM_GPU_MEMORY_UTILIZATION` set explicitly.

The [Cosmos3 Certified NIM cookbook](nim/README.md) is the full guide —
The [NIM cookbook](nim/README.md) is the full guide —
[deployment](nim/deployment.md), [support matrix](nim/support-matrix.md),
[configuration](nim/configuration.md), and [API reference](nim/api-reference.md),
with worked examples for [generation](nim/generation.md),
[action](nim/action.md), [transfer](nim/transfer.md), and
[reasoning](nim/reasoning.md).

### Reasoner NIM

A prebuilt container that serves the Reasoner over an OpenAI-compatible API for
image and video understanding.

Start a Nano Reasoner server (publishes the API on port 8000; the first run
downloads the model into `~/.cache/nim`):

```bash
export NGC_API_KEY=<your_key>

docker run --runtime=nvidia --gpus all \
--shm-size=32GB \
-e NGC_API_KEY="$NGC_API_KEY" \
-e NIM_MODEL_SIZE=nano \
-v ~/.cache/nim:/opt/nim/.cache \
-u $(id -u) \
-p 8000:8000 \
nvcr.io/nim/nvidia/cosmos3-reasoner:1.7.0
```

For **Cosmos3-Super-Reasoner** (the larger model), set `-e NIM_MODEL_SIZE=super`.
The container serves `nvidia/cosmos3-nano-reasoner` (or
`nvidia/cosmos3-super-reasoner`); pass that exact name as the request `model`, or
resolve it dynamically with `client.models.list()`.

### Generator NIM

A prebuilt container that serves **Cosmos3-Generator Text-to-Video and Image-to-Video
only** through `POST /v1/infer`. The NIM infers mode from the request fields:
non-empty `prompt` with no `image` means Text-to-Video; `image` provided means Image-to-Video. The
response is JSON with a base64-encoded MP4 in `b64_video`.

It does **not** expose text-to-image, video-to-video, sound/audio generation,
action modes, or transfer controls. Use vLLM-Omni or Cosmos Framework for those
broader Generator workflows.

Start a Nano Generator server (default `NIM_MODEL_SIZE=nano`, `NIM_PRECISION=fp8`,
`NIM_PERF_PROFILE=latency`):

```bash
export NGC_API_KEY=<your_key>
export LOCAL_NIM_CACHE="${LOCAL_NIM_CACHE:-$HOME/.cache/nim}"
mkdir -p "$LOCAL_NIM_CACHE"
chmod -R 777 "$LOCAL_NIM_CACHE" 2>/dev/null || true

docker run --runtime=nvidia --gpus all \
--shm-size=32GB \
--ulimit nofile=65536:65536 \
-e NGC_API_KEY="$NGC_API_KEY" \
-v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
-p 8000:8000 \
nvcr.io/nim/nvidia/cosmos3-generator:1.0.0
```

For **Cosmos3-Super Generator**, add `-e NIM_MODEL_SIZE=super`. Other selection
knobs:

| Env var | Values | Default | Use |
| --- | --- | --- | --- |
| `NIM_MODEL_SIZE` | `nano`, `super` | `nano` | Selects 8B Nano or 32B Super |
| `NIM_PRECISION` | `bf16`, `fp8`, `nvfp4` | `fp8` | Selects precision; `nvfp4` requires Blackwell |
| `NIM_PERF_PROFILE` | `latency`, `throughput` | `latency` | Optimizes profile selection objective |
| `NIM_TAGS_SELECTOR` | comma-separated `key=value` filters | unset | Advanced profile pinning, e.g. `model_size=super,nim_tp=2` |

A quick T2V smoke test:

```bash
curl -sS -X POST http://127.0.0.1:8000/v1/infer \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d '{
"prompt": "A humanoid robot walks through a futuristic warehouse, inspecting shelves of mechanical components.",
"seed": 42,
"guidance_scale": 6.0,
"steps": 35,
"resolution": "256",
"num_output_frames": 25,
"fps": 24.0
}' | jq -r '.b64_video' | base64 -d > /tmp/cosmos3_generator_nim_t2v.mp4
```

## Verify the environment

For the Cosmos Framework / Diffusers / vLLM venvs, check that PyTorch sees the GPU:
Expand Down
146 changes: 17 additions & 129 deletions cookbooks/cosmos3/generator/audiovisual/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,26 +5,27 @@ Generate images and video (with optional audio) from text, image or video prompt
distilled Cosmos3-Super students across Cosmos Framework, Diffusers, vLLM-Omni, TRT-LLM
and NIM backends. Sample prompts live under [`assets/`](./assets).

> **NIM scope:** `Cosmos3-Generator` NIM currently exposes Text2Video and
> Image2Video only. It does not expose text-to-image, video-to-video,
> sound/audio generation, action modes, or transfer controls.
> **NIM scope:** [Cosmos3 Certified NIM](../../nim/README.md) supports
> text-to-image, text-to-video, image-to-video, and video-to-video,
> plus [Action](../../nim/action.md) and [Transfer](../../nim/transfer.md).
> Sound/audio generation is not exposed by the public Generator API.

Environment setup for every backend is centralized in the shared
[Cosmos3 cookbooks environment setup](../../README.md) guide; each backend below
links to the section you need. The quickstarts are minimal text-to-video examples
to get one generation running per backend — run them from this folder.

Generator requires the Guardrail. Request access to the gated
For backends other than NIM, request access to the gated
[nvidia/Cosmos-1.0-Guardrail](https://huggingface.co/nvidia/Cosmos-1.0-Guardrail)
HF repository before running these examples. To disable the guardrail, set
`enable_safety_checker=False` (Diffusers), `TRTLLM_DISABLE_COSMOS3_GUARDRAILS=1`
or `use_guardrails: false` through `extra_params` (TensorRT-LLM),
`guardrails: false` (vLLM-Omni `extra_params`/`extra_args`), or
`--no-guardrails` (Cosmos Framework). For
Generator NIM set `NIM_ENABLE_TEXT_GUARDRAILS=0 NIM_ENABLE_VIDEO_GUARDRAILS=0`.
`--no-guardrails` (Cosmos Framework). For NIM, see
[Guardrails](../../nim/operations.md#guardrails).

NIM backends use NGC authentication instead of Hugging Face login; see the
[Generator NIM setup](../../README.md#generator-nim) for details.
NIM uses NGC authentication instead of Hugging Face login; see the
[NIM setup](../../nim/deployment.md) for details.

## Run with Cosmos Framework

Expand Down Expand Up @@ -423,125 +424,12 @@ live in the

## Run with NIM

### Quickstart

Set up the environment: [Generator NIM setup](../../README.md#generator-nim).
`Cosmos3-Generator` NIM is a prebuilt NGC container that serves Text2Video and
Image2Video through `POST /v1/infer`. It returns JSON with a base64-encoded MP4
in `b64_video`; unlike vLLM-Omni, it does not use `/v1/videos/sync` and does not
return MP4 bytes directly.

Authenticate Docker to NGC once:

```bash
export NGC_API_KEY=<your_key>
echo "$NGC_API_KEY" | docker login nvcr.io --username '$oauthtoken' --password-stdin
```

Launch **Cosmos3-Nano** (default model size, FP8, latency profile):

```bash
export LOCAL_NIM_CACHE="${LOCAL_NIM_CACHE:-$HOME/.cache/nim}"
mkdir -p "$LOCAL_NIM_CACHE"
chmod -R 777 "$LOCAL_NIM_CACHE" 2>/dev/null || true

docker run -it --rm --name cosmos3-generator \
--runtime=nvidia \
--gpus all \
--shm-size=32GB \
--ulimit nofile=65536:65536 \
-e NGC_API_KEY="$NGC_API_KEY" \
-v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
-p 8000:8000 \
nvcr.io/nim/nvidia/cosmos3-generator:1.0.0
```

Launch **Cosmos3-Super** by adding `NIM_MODEL_SIZE=super`:

```bash
docker run -it --rm --name cosmos3-generator \
--runtime=nvidia \
--gpus all \
--shm-size=32GB \
--ulimit nofile=65536:65536 \
-e NGC_API_KEY="$NGC_API_KEY" \
-e NIM_MODEL_SIZE=super \
-e NIM_PRECISION=fp8 \
-e NIM_PERF_PROFILE=latency \
-v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
-p 8000:8000 \
nvcr.io/nim/nvidia/cosmos3-generator:1.0.0
```

Wait until the readiness endpoint returns success:

```bash
curl -fsS http://127.0.0.1:8000/v1/health/ready
```

Send a Text2Video request. The NIM infers T2V from a non-empty `prompt` with no
`image` field:

```bash
curl -sS -X POST http://127.0.0.1:8000/v1/infer \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d '{
"prompt": "A humanoid robot walks through a futuristic warehouse, inspecting shelves of mechanical components. Photorealistic, cinematic lighting.",
"seed": 42,
"guidance_scale": 6.0,
"steps": 35,
"resolution": "256",
"num_output_frames": 25,
"fps": 24.0
}' | jq -r '.b64_video' | base64 -d > /tmp/cosmos3_nim_t2v.mp4
```

Send an Image2Video request with a local image encoded as a data URI:

```bash
IMG_B64=$(base64 -w 0 assets/images/image2video/humanoid_robot.jpg)
cat > /tmp/cosmos3_nim_i2v.json <<EOF
{
"prompt": "The humanoid robot performs a controlled standing backflip in a modern living room, then lands steadily on both feet.",
"image": "data:image/jpeg;base64,${IMG_B64}",
"seed": 123,
"guidance_scale": 6.0,
"steps": 35,
"resolution": "256",
"num_output_frames": 25,
"fps": 24.0
}
EOF

curl -sS -X POST http://127.0.0.1:8000/v1/infer \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d @/tmp/cosmos3_nim_i2v.json | jq -r '.b64_video' | base64 -d > /tmp/cosmos3_nim_i2v.mp4
```

Key request fields and constraints:

| Field | Constraint / default |
| --- | --- |
| `prompt` | Required for T2V; optional when `image` is provided; max 20000 chars |
| `negative_prompt` | Optional; omitted means the server uses the Cosmos3 default |
| `image` | I2V conditioning image; raw base64, `data:image/...;base64,...`, or public URL when URL inputs are enabled |
| `guidance_scale` | `1.0` to `7.0`, default `6.0` |
| `steps` | `1` to `100`, default `35` |
| `resolution` | `256`, `480`, `720`, optionally with `_16_9`, `_1_1`, `_9_16`, `_4_3`, or `_3_4` |
| `num_output_frames` | Must follow the 4k+1 cadence (`25, 29, 33, ...`); caps: `256 <= 397`, `480 <= 297`, `720 <= 197` |
| `fps` | `1.0` to `60.0`, recommended `10` to `30`, default `24.0` |

### Notebook walkthrough

[`run_with_nim.ipynb`](./run_with_nim.ipynb) launches the NIM container,
waits for readiness, inspects the service endpoints, sends T2V and I2V requests,
decodes `b64_video`, and previews the generated MP4 files inline.

### Limitations
Use [NIM](../../nim/README.md) with `NIM_MODEL_TYPE=generator`. Follow the
[deployment guide](../../nim/deployment.md)
to select a compatible model and latency or throughput profile, then use the
[generation guide](../../nim/generation.md) for text-to-image, text-to-video,
image-to-video, and video-to-video requests through `POST /v1/infer`.

`Cosmos3-Generator` NIM currently exposes **Text2Video** and **Image2Video** only.
It does **not** expose text-to-image, video-to-video, sound/audio generation,
action modes, or transfer controls. For those broader Generator API workflows,
use vLLM-Omni or Cosmos Framework as appropriate.
Requests select the task explicitly with `model_mode`. The guide includes
[Python examples](../../nim/examples/) for general-purpose and four-step
models, response decoding, and saving generated media.
Loading
Loading