Repository navigation
refactor(eval): make HIR retention opt-in and share the source-reference grammar - #620
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #619; merge that first.
Fixes #595.
HIR retention is opt-in
Enginekept every loaded rule's HIR forsave_hir, which nothing in the CLI, daemon, MCP server, or LSP calls. NewEngine::set_retain_hir(bool)andretain_hir_enabled(); retention is off by default and has to be on before rules load.save_hirreturnsEvalError::HirCachewhen retention is off, or when rules were loaded while it was off, instead of writing a cache that silently misses rules. Disabling retention drops the HIR already kept. Loading the 3,132 SigmaHQ rules withengine evalpeaks at 222 to 229 MB without retention versus 238 to 243 MB with it (three interleaved runs each).One evaluation loop per path
evaluate_{no_bloom,with_bloom}_pathbecome oneevaluate_candidates<B: BloomLookup>, andevaluate_with_logsource_{no_bloom,with_bloom}become oneevaluate_matching_logsource<B>. TheNoBloominstantiation monomorphizes to the same code as before. The include-event attachment the loops and the full-scan oracle all repeated is nowevaluate_one.evaluate_with_logsourcestays separate, since its subset matching differs fromevaluate_pruned's conflict-based pruning.eval_single_event,eval_throughput, andeval_bloom_rejectionmove within about 2% in both directions, and the signs flip between runs.One source-reference grammar
rsigma_eval::pipeline::sources::scan_source_templatesis now the only${source.*}scanner. It returns each template's id, sub-path, and byte offset, and both pipeline parsing and the runtime'sTemplateExpanderuse it. A reference id is one or more ASCII letters, digits,_, or-, so${source.1st_feed}(previously expanded by the runtime but not recorded as a reference) and${source.threat-intel}(previously matched by neither side) both work.parse_dynamic_sourcerejects ids outside that set throughis_valid_source_id. I widened the grammar rather than rejecting digit-leading and hyphenated ids, because source ids are also used outside templates (--disposition-sourcefiles and the/api/v1/sources/.../{source_id}routes), so rejecting them would break working configs.Dead items
Removed: the LSP's uncalled
diagnose(), the_use_errplaceholder in the enrichment template module, the staleallow(dead_code)onexit_code::SUCCESS, rstix's unusedPaginatedFilter::added_after()andnext()getters, and the unreadTaxiiResponse.clock_skewfield. None is public API.Tests
hir_cache.rsenables retention for the round trips and addssave_requires_retention,save_refuses_rules_loaded_before_retention, anddisabling_retention_drops_retained_hir;hir_is_retained_only_when_enabledchecks the default directly.test_digit_leading_and_hyphenated_source_refs_are_dynamicandtest_source_ids_templates_cannot_reference_are_rejectedboth fail on the previous grammar and parser.expand_digit_leading_and_hyphenated_idscovers whole-value and inline expansion, including an unresolved template left in place.main, and thewasm32check forrsigma-evaland the no-default-features checks for the CLI and rstix pass.