Repository navigation
Fix physics_iq scoring result reuse across tasks and validate staged artifacts - #379
Closed
JIN-strong wants to merge 1 commit into
Closed
JIN-strong wants to merge 1 commit into
JIN-strong wants to merge 1 commit into
Conversation
… artifacts The scoring cell defaulted RESULTS_DIR to one shared directory for every task, and the official scorer keys its cached CSV by the input folder's basename — which is "staged" for both the I2V and V2V sweeps. Scoring V2V after I2V therefore found a complete cached CSV and republished the I2V metrics as the V2V score. Results are now namespaced by the staged folder's parent (i2v_full, v2v_full, or the demo case id), so each task owns its own cache. The staging helpers also write through a temp file that is renamed into place only after the frame count probes back as expected, and an existing artifact is re-staged instead of trusted when it no longer probes as exactly 120 frames (staged clips) or 73 frames (padded conditioning) — a run killed mid-encode previously left a truncated clip that resume would reuse for conditioning or scoring. test_run_with_cosmos_framework.py extracts the real staging functions from the notebook and drives them with fake ffmpeg/ffprobe pairs: it covers the temp-file write path, valid reuse without a second encode, re-staging of corrupted artifacts for all three producers, and rejection of an unexpected source frame count. Signed-off-by: JIN-strong <1228766517@qq.com>
Collaborator
|
@JIN-strong , we have migrated the eval folder to cosmos-framework, please submit your PR here: https://github.com/NVIDIA/cosmos-framework/tree/main/.agents/skills/evaluation/cosmos3/generator/physics_iq |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The scoring cell defaulted
RESULTS_DIRto one shared directory for every task, and the official scorer keys its cached CSV by the input folder's basename — which isstagedfor both the I2V and V2V sweeps. Scoring V2V after I2V therefore found a complete cached CSV and republished the I2V metrics as the V2V score.This PR isolates the results per task and hardens the staged artifacts:
RESULTS_DIRnow defaults to$PHYSICSIQ_OUTPUT_ROOT/scoring/$(basename "$(dirname "$STAGED_DIR")"), soi2v_full,v2v_full, and demo runs each own their scoring cache.stage_i2v_last120,stage_v2v_omit73,prepare_v2v_conditioning) write through a temp file that is renamed into place only after the frame count probes back as expected, so an interrupted encode can never leave a truncated clip at the final path.Why
A truncated clip from a killed run could silently enter V2V conditioning or scoring, and the second task's scoring could silently republish the first task's metrics.
Validation
python3 test_run_with_cosmos_framework.py→ 8 tests pass. The test extracts the real staging function definitions from the notebook and drives them with fake ffmpeg/ffprobe pairs (frame counts embedded in the file bytes), covering the temp-file write path, valid reuse without a second encode, re-staging of corrupted artifacts for all three producers, and rejection of an unexpected source frame count. No GPU, network, ffmpeg, or venv is needed.compile(), and the bash cells passbash -n.