Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 7 additions & 47 deletions .agents/skills/cookbooks/cosmos3/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -191,7 +191,7 @@ git lfs install

git clone https://github.com/NVIDIA/TensorRT-LLM.git
cd TensorRT-LLM
git checkout bca6761ab84fbcd58fc7f914eade7de48b32e35e
git checkout cc63e66daf52644f1441e117bb7ba67d20566016
git submodule update --init --recursive
git lfs pull

Expand Down Expand Up @@ -219,58 +219,18 @@ Then install the Cosmos3 guardrail package in the same environment unless you
explicitly disable guardrails before starting the server:

```bash
pip install cosmos_guardrail==0.3.0
pip install cosmos_guardrail==0.3.2
# On headless servers without libGL.so.1, replace the OpenCV wheel pulled in by
# cosmos_guardrail with the matching headless build:
pip uninstall -y opencv-python
pip install opencv-python-headless==5.0.0.93
```

#### Server-side NLTK data setup

With `cosmos_guardrail==0.3.0`, NLTK's path checks can reject tokenizer or
dictionary files that are symlinks from a Hugging Face snapshot into its
`blobs/` directory. Prepare a separate copy of the NLTK data as regular files
**on the server, in the same container and shell used to launch TensorRT-LLM**:

```bash
export NLTK_DATA="$(mktemp -d "${TMPDIR:-/tmp}/cosmos3-nltk.XXXXXX")"
python3 - <<'PY'
import os
import shutil
from pathlib import Path

import nltk
from huggingface_hub import snapshot_download

snapshot = snapshot_download(
"nvidia/Cosmos-1.0-Guardrail",
revision="cf03c0395fac8c4de386c0bdab12cc4fc8d66362",
allow_patterns=["blocklist/**"],
)
source = Path(snapshot) / "blocklist" / "nltk_data"
destination = Path(os.environ["NLTK_DATA"]).resolve()
shutil.copytree(source, destination, symlinks=False, dirs_exist_ok=True)
assert not any(path.is_symlink() for path in destination.rglob("*"))

# Check both resource lookups used by the text blocklist before starting a GPU server.
nltk.data.path[:] = [str(destination)]
tokens = nltk.word_tokenize("You are an autonomous vehicle planning system.")
assert nltk.WordNetLemmatizer().lemmatize("vehicles") == "vehicle"
print("Guardrail NLTK data ready:", destination, tokens)
PY
```

Keep `NLTK_DATA` exported when starting the server below. Repeat this setup
after recreating the container or removing the temporary directory. Running it
only in the client notebook's environment does not configure a remote server.
This workaround does not rewrite cached files or symlinks, and keeps NLTK
path security and `use_guardrails=True` enabled.

The separate `No safety models found, returning safe` warning in guardrail
0.3.0 refers to its intentionally empty video-content classifier list. Text
checks and face blurring remain configured; the NLTK workaround does not enable
video-content classification or suppress that warning.
`cosmos_guardrail` 0.3.2 copies its NLTK data out of the Hugging Face cache
on first use, so the server needs no separate NLTK data setup. Its
`No safety models found, returning safe` warning refers to the intentionally
empty video-content classifier list; text checks and face blurring remain
configured.

Set the TensorRT-LLM source root for the shared VisualGen config YAMLs. Run this
from inside the TensorRT-LLM checkout — the directory the `git clone` above
Expand Down
6 changes: 2 additions & 4 deletions .agents/skills/cookbooks/cosmos3/generator/action/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -176,16 +176,14 @@ have available.
### Quickstart

Set up and launch the VisualGen server with the one-GPU Nano config:
[TensorRT-LLM setup](../../README.md#tensorrt-llm-generator), including the
[server-side NLTK data setup](../../README.md#server-side-nltk-data-setup).
Keep `NLTK_DATA` exported in the server's shell. Action requests
[TensorRT-LLM setup](../../README.md#tensorrt-llm-generator). Action requests
upload a conditioning image as multipart `image_reference` or a video as
`video_reference`, and put the mode-specific fields under `extra_params`.
Use the same trained AV prompt as the Cosmos Framework reference; the legacy
`view_point` field is ignored by current TensorRT-LLM.

Both notebooks' current multipart fields were exercised against TensorRT-LLM
source revision `bca6761ab84fbcd58fc7f914eade7de48b32e35e`. Follow the source
source revision `cc63e66daf52644f1441e117bb7ba67d20566016`. Follow the source
pin in the shared setup.

```python
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,6 @@
"\n",
"Follow the shared [TensorRT-LLM setup](../../README.md#tensorrt-llm-generator), then run this\n",
"from the TensorRT-LLM checkout. This notebook is a client; keep the server in a separate terminal.\n",
"Complete the [server-side NLTK data setup](../../README.md#server-side-nltk-data-setup) there first, and keep `NLTK_DATA` exported in that server terminal.\n",
"\n",
"```bash\n",
"export TRTLLM_ROOT=\"${TRTLLM_ROOT:-$PWD}\"\n",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,6 @@
"\n",
"Follow the shared [TensorRT-LLM setup](../../README.md#tensorrt-llm-generator), then run this\n",
"from the TensorRT-LLM checkout. This notebook is a client; keep the server in a separate terminal.\n",
"Complete the [server-side NLTK data setup](../../README.md#server-side-nltk-data-setup) there first, and keep `NLTK_DATA` exported in that server terminal.\n",
"\n",
"```bash\n",
"export TRTLLM_ROOT=\"${TRTLLM_ROOT:-$PWD}\"\n",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -199,9 +199,7 @@ reuses the previews from [`preview_helpers.py`](./preview_helpers.py), writing o
### Quickstart

Set up and launch Nano or Super as described in the shared
[TensorRT-LLM setup](../../README.md#tensorrt-llm-generator), including the
[server-side NLTK data setup](../../README.md#server-side-nltk-data-setup).
Keep `NLTK_DATA` exported in the server's shell. TensorRT-LLM
[TensorRT-LLM setup](../../README.md#tensorrt-llm-generator). TensorRT-LLM
accepts a raw source video as multipart `video_reference` when edge or blur will
be computed on the server. The checked-in assets are already precomputed
controls, so this example base64-encodes the control inside its hint and does
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,6 @@
"\n",
"Follow the shared [TensorRT-LLM setup](../../README.md#tensorrt-llm-generator), then launch one\n",
"server from the TensorRT-LLM checkout.\n",
"Complete the [server-side NLTK data setup](../../README.md#server-side-nltk-data-setup) there first, and keep `NLTK_DATA` exported in that server terminal.\n",
"\n",
"```bash\n",
"export TRTLLM_ROOT=\"${TRTLLM_ROOT:-$PWD}\"\n",
Expand Down
Loading