diff --git a/README.md b/README.md index 343c5ed7..ffde5976 100644 --- a/README.md +++ b/README.md @@ -71,7 +71,7 @@ One model, two surfaces: | | Inputs | Outputs | Use it for | |---|---|---|---| | **Reasoner** | text, vision | text | world understanding, grounding, task planning, embodied reasoning | -| **Generator** | text, vision, sound, action | vision, sound, action | world simulation, future prediction, synthetic data, policy learning | +| **Generator** | text, vision, audio, action | vision, audio, action | world simulation, future prediction, synthetic data, policy learning | This repository is the home of the models: everything for exploring, running, and evaluating Cosmos. For model training (SFT, LoRA, RL, distillation etc.), go to [Cosmos Framework](https://github.com/NVIDIA/cosmos-framework). Use [Cosmos Curator](https://github.com/NVIDIA/cosmos-curator) for data curation, and [Cosmos Evaluator](https://github.com/NVIDIA/cosmos-evaluator) for model output evaluation. @@ -114,7 +114,7 @@ video = pipe(prompt="A mobile robot navigates a warehouse aisle and stops at a s export_to_video(video, "first_video.mp4", fps=24) ``` -First run downloads the 16B checkpoint; diffusion steps are compute-heavy, so long step times are normal. Full options, image/sound modes, and every other backend: [audiovisual cookbooks](cookbooks/cosmos3/generator/audiovisual/) · setup issues: [environment setup guide](cookbooks/cosmos3/README.md). +First run downloads the 16B checkpoint; diffusion steps are compute-heavy, so long step times are normal. Full options, image/audio modes, and every other backend: [audiovisual cookbooks](cookbooks/cosmos3/generator/audiovisual/) · setup issues: [environment setup guide](cookbooks/cosmos3/README.md). ## Models diff --git a/docs/reference/models.md b/docs/reference/models.md index ba08cf7f..f4a3c3fd 100644 --- a/docs/reference/models.md +++ b/docs/reference/models.md @@ -41,9 +41,9 @@ Cosmos 3 is built on a unified Mixture-of-Transformers (MoT) architecture combin t2vTextVideo
(256p, 480p, 720p)Video
(256p, 480p, 720p)Video
(256p, 480p) - t2svTextVideo with sound
(256p, 480p, 720p)Video with sound
(256p, 480p, 720p)Not supported + t2avTextVideo with audio
(256p, 480p, 720p)Video with audio
(256p, 480p, 720p)Not supported i2v — 1st frameText + image
(JPG/PNG/JPEG/WEBP)Video
(256p, 480p, 720p)Video
(256p, 480p, 720p)Video
(256p, 480p) - i2svText + image
(JPG/PNG/JPEG/WEBP)Video with sound
(256p, 480p, 720p)Video with sound
(256p, 480p, 720p)Not supported + i2avText + image
(JPG/PNG/JPEG/WEBP)Video with audio
(256p, 480p, 720p)Video with audio
(256p, 480p, 720p)Not supported v2v — transferText + MP4 video
(edge, blur, depth, segmentation map)Video
(256p, 480p, 720p)Video
(256p, 480p, 720p)Not supported v2v — predictText + MP4 video
(first 5 frames used, up to ~3 s of conditioning)Video
(256p, 480p, 720p)Video
(256p, 480p, 720p)Not supported t2iText stringImage
(256p, 480p, 720p)Image
(256p, 480p, 720p)Not supported @@ -91,4 +91,4 @@ Every workflow has a runnable notebook — see [cookbooks](../../cookbooks/cosmo ## Limitations -Cosmos 3 can produce artifacts in long, high-resolution, or physically complex outputs: temporal inconsistency, unstable camera/object motion, sound-video misalignment, action-state inconsistency, object morphing, inaccurate 3D structure, implausible dynamics. Physically grounded simulation, safety-critical control, and complex multi-agent behavior require additional validation, guardrails, and system-level safety analysis before deployment. +Cosmos 3 can produce artifacts in long, high-resolution, or physically complex outputs: temporal inconsistency, unstable camera/object motion, audio-video misalignment, action-state inconsistency, object morphing, inaccurate 3D structure, implausible dynamics. Physically grounded simulation, safety-critical control, and complex multi-agent behavior require additional validation, guardrails, and system-level safety analysis before deployment.