Skip to content

feat(icons): align the ICONS selector with the reference implementation - #66

Merged
Walnutes merged 1 commit into
OpenDCAI:mainfrom
tongruiliu:pr/icons-align-official
Oct 14, 2026
Merged

Walnutes merged 1 commit into
OpenDCAI:mainfrom
tongruiliu:pr/icons-align-official

Conversation

@tongruiliu

Copy link
Copy Markdown
Contributor

The official code (github.com/princetonvisualai/icons) is now public, so close the gaps that keep this selector from reproducing it:

  • pool_groups: the reference gives the 13 MME subtask validation sets a single vote by max-pooling their per-slice influence vectors (icons/generalist.py). Slices can now be grouped the same way instead of each voting separately.
  • warn when only one task votes: the consensus then degenerates to plain top-p influence selection, which is easy to miss with task_boundaries unset.
  • vote_percentile: pin the paper's selection ratio p for the per-task threshold instead of always deriving it from the schedule (num_samples / len(dataset)).
  • clean stale gradient chunks before a collection and fail when the merged chunks do not cover every sample, rather than silently scoring zero rows for missing ones; the old "resuming" path never skipped a sample.
  • components.yaml: proj_dim 5120 / seed 0, matching the reference dims.

Voting was checked vote-for-vote against a transcription of icons/generalist.py (percentile threshold, >= comparison, vote sum, top-k) including the pooled and single-task cases.

The official code (github.com/princetonvisualai/icons) is now public, so close
the gaps that keep this selector from reproducing it:

- pool_groups: the reference gives the 13 MME subtask validation sets a single
  vote by max-pooling their per-slice influence vectors (icons/generalist.py).
  Slices can now be grouped the same way instead of each voting separately.
- warn when only one task votes: the consensus then degenerates to plain top-p
  influence selection, which is easy to miss with task_boundaries unset.
- vote_percentile: pin the paper's selection ratio p for the per-task threshold
  instead of always deriving it from the schedule (num_samples / len(dataset)).
- clean stale gradient chunks before a collection and fail when the merged chunks
  do not cover every sample, rather than silently scoring zero rows for missing
  ones; the old "resuming" path never skipped a sample.
- components.yaml: proj_dim 5120 / seed 0, matching the reference dims.

Voting was checked vote-for-vote against a transcription of icons/generalist.py
(percentile threshold, >= comparison, vote sum, top-k) including the pooled and
single-task cases.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@tongruiliu
tongruiliu requested a review from Walnutes October 14, 2026 10:00
@Walnutes
Walnutes merged commit 0e2eb4b into OpenDCAI:main Oct 14, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants