Repository navigation
Conversation
…n eval Compute prediction/ground-truth IoUs once per image instead of once per IoU threshold (10x fewer compute_2d_iou calls). IoUs are cached in a nested list so the exact values returned by compute_2d_iou are compared, leaving TP/FP assignment at threshold boundaries unchanged. Signed-off-by: Raushan Kumar <raus7n@gmail.com>
sylvesterkaczmarek
left a comment
There was a problem hiding this comment.
Thanks for carrying this over to cosmos-framework and addressing the precision concern from #313. I checked the new diff: the nested Python-list cache preserves the scalar values returned by compute_2d_iou without float32 conversion, and the confidence-order and per-threshold greedy matching remain unchanged. That resolves my earlier review concern. A focused near-threshold regression would be useful, but I don't see a blocker on this point.
lfengad
approved these changes
Oct 29, 2026
Xuanmeng-Zhang
approved these changes
Oct 29, 2026
pengcuo
approved these changes
Oct 29, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Speeds up the Metropolis2D detection eval by computing the IoU between each prediction and GT box once per image, instead of once per threshold.
This was originally NVIDIA/cosmos#313. I moved it here as asked.
Motivation
The eval checks matches at 10 IoU thresholds (0.50 to 0.95), but it recalculated the same prediction/GT IoUs from scratch for every one of them. So with
Npredictions,MGT boxes andTthresholds, it was doing aboutT × N × MIoU calculations whenN × Mis enough.Changes
With 10 thresholds, that's roughly 10x fewer IoU calculations.
Review feedback
@sylvesterkaczmarek pointed out on the old PR that storing the IoUs as float32 could round a value sitting right at a threshold to the other side. I've switched the cache to a plain Python list, so it compares exactly what
compute_2d_ioureturns and nothing gets rounded.Validation
python -m py_compile .agents/skills/evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/dataset/metropolis2d/detection_2d_dataset.pygit diff --checkTesting
The matching logic itself hasn't changed. It just reads IoUs from the cache instead of recomputing them. Syntax check passes.
cc @yaoxu-crypto