Skip to content

Reuse per-image IoU matrix across thresholds in Metropolis2D detectio… - #325

Open
raus7n wants to merge 2 commits into
NVIDIA:mainfrom
raus7n:optimize-metropolis2d-iou
Open

raus7n wants to merge 2 commits into
NVIDIA:mainfrom
raus7n:optimize-metropolis2d-iou

Conversation

@raus7n

@raus7n raus7n commented Oct 28, 2026

Copy link
Copy Markdown

Summary

Speeds up the Metropolis2D detection eval by computing the IoU between each prediction and GT box once per image, instead of once per threshold.

This was originally NVIDIA/cosmos#313. I moved it here as asked.

Motivation

The eval checks matches at 10 IoU thresholds (0.50 to 0.95), but it recalculated the same prediction/GT IoUs from scratch for every one of them. So with N predictions, M GT boxes and T thresholds, it was doing about T × N × M IoU calculations when N × M is enough.

Changes

  • Work out all the prediction/GT IoUs once per image.
  • Reuse those values when matching at each threshold.
  • Everything else stays the same: sorting by confidence, greedy matching, how GT boxes get marked as matched, and the AP/mAP calculation.

With 10 thresholds, that's roughly 10x fewer IoU calculations.

Review feedback

@sylvesterkaczmarek pointed out on the old PR that storing the IoUs as float32 could round a value sitting right at a threshold to the other side. I've switched the cache to a plain Python list, so it compares exactly what compute_2d_iou returns and nothing gets rounded.

Validation

  • python -m py_compile .agents/skills/evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/dataset/metropolis2d/detection_2d_dataset.py
  • git diff --check

Testing

The matching logic itself hasn't changed. It just reads IoUs from the cache instead of recomputing them. Syntax check passes.
cc @yaoxu-crypto

…n eval

Compute prediction/ground-truth IoUs once per image instead of once per IoU threshold (10x fewer compute_2d_iou calls). IoUs are cached in a nested list so the exact values returned by compute_2d_iou are compared, leaving TP/FP assignment at threshold boundaries unchanged.

Signed-off-by: Raushan Kumar <raus7n@gmail.com>

@sylvesterkaczmarek sylvesterkaczmarek left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for carrying this over to cosmos-framework and addressing the precision concern from #313. I checked the new diff: the nested Python-list cache preserves the scalar values returned by compute_2d_iou without float32 conversion, and the confidence-order and per-threshold greedy matching remain unchanged. That resolves my earlier review concern. A focused near-threshold regression would be useful, but I don't see a blocker on this point.

@yaoxu-crypto yaoxu-crypto added the Eval Evaluation harness or benchmark changes label Oct 30, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Eval Evaluation harness or benchmark changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants