Skip to content

docs(reasoner): clarify raw vLLM video sampling - #337

Closed
sylvesterkaczmarek wants to merge 1 commit into
NVIDIA:mainfrom
sylvesterkaczmarek:docs/fix-vllm-video-sampling-190
Closed

sylvesterkaczmarek wants to merge 1 commit into
NVIDIA:mainfrom
sylvesterkaczmarek:docs/fix-vllm-video-sampling-190

Conversation

@sylvesterkaczmarek

Copy link
Copy Markdown

Summary

  • document the correct vLLM frame-sampling request shape for video inputs
  • show both the OpenAI client extra_body form and equivalent raw /v1/chat/completions JSON
  • clarify that fps and do_sample_frames belong inside mm_processor_kwargs, not as standalone top-level chat-completion fields

Why

Issue #190 reports weak physical-plausibility classification while sending fps and do_sample_frames as top-level fields in a raw requests payload. The current Reasoner prompt guide documents video sampling through extra_body={"mm_processor_kwargs": ...} for the OpenAI client, but the vLLM quickstart does not show the equivalent raw JSON shape.

That distinction matters for motion-dependent tasks such as physical plausibility analysis because frame sampling determines which temporal events are presented to the Reasoner. This change makes the raw HTTP configuration explicit next to the vLLM quickstart.

Validation

Documentation-only change. Cross-checked against the current Reasoner prompt guide's mm_processor_kwargs sampling configuration and the request shown in #190.

Addresses #190

@sylvesterkaczmarek
sylvesterkaczmarek force-pushed the docs/fix-vllm-video-sampling-190 branch from 14418a1 to eb7efb8 Compare September 30, 2026 17:26
@sylvesterkaczmarek

Copy link
Copy Markdown
Author

Refreshed this docs PR onto current main today. It is now 0 commits behind and mergeable, and the diff remains limited to the intended documentation file. This has been quiet for over two weeks. Could a maintainer please review/merge it, or let me know if any wording change is needed?

Refreshed onto current upstream.

Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
@sylvesterkaczmarek
sylvesterkaczmarek force-pushed the docs/fix-vllm-video-sampling-190 branch from eb7efb8 to 5e00193 Compare September 30, 2026 17:41
@sylvesterkaczmarek

Copy link
Copy Markdown
Author

Following up on the raw-vLLM sampling documentation. Could a maintainer review the request example and approve the waiting workflow?

@yaoxu-crypto

Copy link
Copy Markdown
Collaborator

Thanks for this, @sylvesterkaczmarek. The Cosmos cookbooks and evaluation suites have moved to NVIDIA/cosmos-framework under .agents/skills/ and were removed from this repo in #384, so this PR can no longer merge here.

The raw /v1/chat/completions example (#190) isn't in Cosmos Framework yet. Could you re-submit it against .agents/skills/cookbooks/cosmos3/reasoner/README.md?

The relative paths are unchanged, just under .agents/skills/ (for example cookbooks/cosmos3/... is now .agents/skills/cookbooks/cosmos3/...). Please note that Cosmos Framework requires DCO sign-off on each commit (git commit -s, see CONTRIBUTING.md). I'm closing this PR because it can't merge here, but the work is welcome there. Please link this PR from the new one so the earlier discussion and reviews carry over. Thanks for your patience, and sorry it sat here while the move happened.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants