Skip to content

refactor(eval): make HIR retention opt-in and share the source-reference grammar - #620

Merged
mostafa merged 6 commits into
fix/percentile-selectionfrom
fix/eval-lean-up
Oct 18, 2026
Merged

mostafa merged 6 commits into
fix/percentile-selectionfrom
fix/eval-lean-up

Conversation

@mostafa

@mostafa mostafa commented Oct 18, 2026

Copy link
Copy Markdown
Member

Stacked on #619; merge that first.

Fixes #595.

HIR retention is opt-in

Engine kept every loaded rule's HIR for save_hir, which nothing in the CLI, daemon, MCP server, or LSP calls. New Engine::set_retain_hir(bool) and retain_hir_enabled(); retention is off by default and has to be on before rules load. save_hir returns EvalError::HirCache when retention is off, or when rules were loaded while it was off, instead of writing a cache that silently misses rules. Disabling retention drops the HIR already kept. Loading the 3,132 SigmaHQ rules with engine eval peaks at 222 to 229 MB without retention versus 238 to 243 MB with it (three interleaved runs each).

One evaluation loop per path

evaluate_{no_bloom,with_bloom}_path become one evaluate_candidates<B: BloomLookup>, and evaluate_with_logsource_{no_bloom,with_bloom} become one evaluate_matching_logsource<B>. The NoBloom instantiation monomorphizes to the same code as before. The include-event attachment the loops and the full-scan oracle all repeated is now evaluate_one. evaluate_with_logsource stays separate, since its subset matching differs from evaluate_pruned's conflict-based pruning. eval_single_event, eval_throughput, and eval_bloom_rejection move within about 2% in both directions, and the signs flip between runs.

One source-reference grammar

rsigma_eval::pipeline::sources::scan_source_templates is now the only ${source.*} scanner. It returns each template's id, sub-path, and byte offset, and both pipeline parsing and the runtime's TemplateExpander use it. A reference id is one or more ASCII letters, digits, _, or -, so ${source.1st_feed} (previously expanded by the runtime but not recorded as a reference) and ${source.threat-intel} (previously matched by neither side) both work. parse_dynamic_source rejects ids outside that set through is_valid_source_id. I widened the grammar rather than rejecting digit-leading and hyphenated ids, because source ids are also used outside templates (--disposition-source files and the /api/v1/sources/.../{source_id} routes), so rejecting them would break working configs.

Dead items

Removed: the LSP's uncalled diagnose(), the _use_err placeholder in the enrichment template module, the stale allow(dead_code) on exit_code::SUCCESS, rstix's unused PaginatedFilter::added_after() and next() getters, and the unread TaxiiResponse.clock_skew field. None is public API.

Tests

  • hir_cache.rs enables retention for the round trips and adds save_requires_retention, save_refuses_rules_loaded_before_retention, and disabling_retention_drops_retained_hir; hir_is_retained_only_when_enabled checks the default directly.
  • test_digit_leading_and_hyphenated_source_refs_are_dynamic and test_source_ids_templates_cannot_reference_are_rejected both fail on the previous grammar and parser.
  • Scanner unit tests check ids, sub-paths, and offsets, and expand_digit_leading_and_hyphenated_ids covers whole-value and inline expansion, including an unresolved template left in place.
  • The Sigma corpus regression is identical to main, and the wasm32 check for rsigma-eval and the no-default-features checks for the CLI and rstix pass.

@mostafa mostafa changed the title Make HIR retention opt-in, merge duplicated eval loops, and share the source-reference grammar refactor(eval): make HIR retention opt-in and share the source-reference grammar Oct 18, 2026
@mostafa
mostafa added this pull request to stack #622 October 18, 2026 19:44
@mostafa
mostafa merged commit 27f2b2a into main Oct 18, 2026
26 checks passed
@mostafa
mostafa deleted the fix/eval-lean-up branch October 18, 2026 20:07
@mostafa mostafa mentioned this pull request Oct 19, 2026
12 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Opt-in HIR retention, merged eval loops, one source-reference grammar, dead code removal

1 participant