Skip to content

perf(eval): select percentile and median ranks instead of sorting the window - #619

Merged
mostafa merged 5 commits into
mainfrom
fix/percentile-selection
Oct 18, 2026
Merged

mostafa merged 5 commits into
mainfrom
fix/percentile-selection

Conversation

@mostafa

@mostafa mostafa commented Oct 18, 2026

Copy link
Copy Markdown
Member

Fixes #594.

value_percentile and value_median correlations sorted every finite value in the group's window on each event fed into the group, and chain bookkeeping deep-cloned every fired correlation result only to read its source index and group key.

Changes

  • Percentile and median select the needed ranks with select_nth_unstable_by(f64::total_cmp) instead of sorting. An interpolated percentile takes its upper neighbour as the minimum of the right-hand partition, and an even-count median takes its lower middle value as the maximum of the left-hand partition, so results are bit-identical to the sorted computation. check_condition and current_value now share window_percentile and window_median. Window state and snapshots are unchanged.
  • chain_correlations carries (source_idx, group_key) in pending instead of full EvaluationResults, which removes the per-fire clone of every result (including retained events in full event mode) and the header clone for each chained parent.

Benchmark

New correlation_numeric_window group in benches/correlation.rs: one group held at a steady window size, 1,000 events per iteration, each checked against the full window.

Case Before After
value_percentile, 1,000 values 10.15 ms 3.00 ms
value_percentile, 5,000 values 42.75 ms 10.15 ms
value_median, 1,000 values 9.79 ms 2.68 ms
value_median, 5,000 values 42.55 ms 10.00 ms

Tests

  • test_percentile_selection_matches_sorted_reference compares against a full-sort reference bit for bit, over window sizes 1 to 64 plus 99, 100, 1,000, and 5,001, at every integer rank and a few fractional ones. The samples include repeats and signed zeros.
  • test_window_median_matches_sorted_reference does the same for odd and even medians through both check_condition and current_value.
  • test_chaining_in_full_event_mode checks that a child carrying retained events still drives its parent, which fires with the child's group key and no events.
  • The Sigma corpus regression is identical to main.

@mostafa
mostafa added this pull request to stack #622 October 18, 2026 19:44
@mostafa
mostafa merged commit 27f2b2a into main Oct 18, 2026
25 checks passed
@mostafa
mostafa deleted the fix/percentile-selection branch October 18, 2026 20:07
@mostafa mostafa mentioned this pull request Oct 19, 2026
12 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Percentile and median correlations sort the whole window on every hit

1 participant