Repository navigation
docs(auto): clarify recommended inference configuration for 10-FPS inputs - #316
Merged
Merged
Conversation
Signed-off-by: Qianli Ma <qianlim@nvidia.com>
lfengad
approved these changes
Oct 27, 2026
lfengad
added a commit
that referenced
this pull request
Oct 28, 2026
## Summary Follow-up to merged #316: make the documented 10-FPS RGB-only Auto configuration a runtime option rather than a manual FFmpeg postprocessing step. Users keep synchronized 10-FPS RGB/WSM input frames unchanged and add: ```bash --output-fps 10 ``` Auto model conditioning defaults to 30 FPS. An explicit different `fps`, including one in input JSON or a custom defaults file, is rejected with guidance to use `output_fps`. Existing requests that explicitly use `fps=30` continue working. ## Implementation and scope - Identify Auto through existing exported task metadata, with exact official HF repository IDs as a fallback. No released checkpoint/config files are modified, and local directory names, view counts and `base_fps` are not used to guess model identity. - Attach the inference-only policy to the runtime model config. Keep the shared argument-builder signature and other models' existing FPS defaults/validation unchanged. Revalidate already-built samples at the shared generation entry point. - Add optional top-level `output_fps` / CLI `--output-fps`. Only the bidirectional and causal Auto video savers use the resolved playback rate, consistently for generated RGB and WSM controls. No frame dropping/duplication/interpolation; model FPS tensors, frame bounds, prompts and sampling are unchanged. - Record both FPS values in saved arguments and log the two clocks explicitly. - Initially support different playback FPS only for RGB-only Auto. Joint RGB/LiDAR requires output FPS to match its 30-FPS camera clock; other models and LiDAR-only requests reject the new option instead of silently ignoring it. - Update the shared inference documentation and Transfer-Auto/Sim-Auto guides. Preserve the timestamped-caption/model-clock and joint-sensor alignment caveats. Both current published Auto configs were checked read-only and already carry `per_view_caption_task=av_joint_camera_lidar_transfer`: - Transfer-Auto: `ac871d0981046cfd3e7bf99e57c449520b1f28c3` - Sim-Auto: `6f537e5f07306be5ff3f776f285d69ffe7bc4e2f` This covers the native `cosmos_framework.scripts.inference` pipeline; it does not add a new option to independent Diffusers/vLLM entry points. ## Validation - **414 tests passed**, covering the new Auto FPS tests, caption metadata, shared arguments, Sim-Auto/Sim-Transfer arguments, inference construction, causal multiview saving/dispatch and LiDAR transfer. - New tests exercise CLI/JSON/custom-defaults precedence, 1-camera Auto recognition, unsupported modes, and unchanged actual model batch tensors/caption text when only `output_fps` changes. - Real RGB and WSM MP4 encoding plus ffprobe: **300 frames at 10 FPS = 30 seconds**, with no lost frames. Existing causal saver tests also cover 10-FPS RGB/WSM output and unchanged joint output. - Ruff check/format, Pyrefly 0.55.0 on affected Python files (zero errors), pre-commit hooks and diff whitespace checks passed. - Updated one existing stale FPS-error assertion to match the current runtime wording (`trained source-frame clock`). Two unrelated tests were deselected from the final CPU regression command: the remote checkpoint-registry download test encountered a local uv cache installation error; the reasoner-output test requires CUDA and also fails on the unchanged base in this CPU environment. No GPU model generation or new 10-FPS visual-quality benchmark was run. Regression command: ```bash python -m pytest \ cosmos_framework/inference/auto_fps_test.py \ cosmos_framework/inference/common/multiview_captions_test.py \ cosmos_framework/inference/args_test.py \ cosmos_framework/inference/sim_auto_args_test.py \ cosmos_framework/inference/sim_transfer_args_test.py \ cosmos_framework/inference/inference_test.py \ cosmos_framework/inference/multiview_causal_test.py \ cosmos_framework/inference/sim_auto_dispatch_test.py \ cosmos_framework/inference/lidar_transfer_test.py \ -k 'not test_checkpoints and not test_generate_reasoner_batch_writes_outputs' \ --num-gpus=0 -q ``` Signed-off-by: Qianli Ma <qianlim@nvidia.com> Co-authored-by: lfengad <liangf@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Make the recommended Transfer-Auto configuration for 10-FPS RGB-only inputs explicit:
fps: 30/--fps 30), matching training.Replace the previous recommendation to match model FPS to source FPS and the "diagnostic only" framing. Explain that changing the model's FPS conditioning to 10 has been reported to reduce quality; this PR does not claim a new model-quality evaluation.
Include a frame-preserving FFmpeg playback conversion, distinguish caption frame bounds from textual timestamps, and retain the separate joint RGB/LiDAR clock/alignment requirement. Update the inference skill's short pointer so it no longer contradicts the reference document. Preserve the existing FPS section anchor. No runtime changes.
Validation
git diff --checkpassed.