refactor(eval): make HIR retention opt-in and share the source-reference grammar

CHANGELOG.md

@@ -4,6 +4,12 @@ All notable changes to RSigma are documented in this file. Each entry correspond
 
 ## [Unreleased]
 
+### Engines no longer retain rule HIR by default, and source references share one grammar (#620)
+
+`Engine` kept every loaded rule's lowered HIR beside its compiled form, only so `save_hir` could write a restart cache, which nothing in the CLI, daemon, MCP server, or LSP calls. Retention is now opt-in through `Engine::set_retain_hir(true)`, which must be on before rules are loaded; loading the 3,132 SigmaHQ rules with `engine eval` peaks about 12 to 16 MB (roughly 6%) lower without it. This changes library behavior: `save_hir` now returns `EvalError::HirCache` unless retention was on for every rule loaded from parsed Sigma, rather than writing a cache that silently misses rules.
+
+The `${source.*}` reference grammar existed twice, and the copies disagreed. Pipeline parsing ignored ids starting with a digit, so a pipeline whose only reference was `${source.1st_feed}` was treated as static and kept the literal template, and ids containing `-`, such as `threat-intel`, matched neither copy. Both sides now use `rsigma_eval::pipeline::sources::scan_source_templates`, and a reference id is one or more ASCII letters, digits, `_`, or `-`. A source declaration whose id uses any other character, such as a dot or a space, now fails to load with an error naming the id, since no template could reference it.
+
 ### Percentile and median checks no longer sort the window (#619)
 
 `value_percentile` and `value_median` correlations sorted every finite value in the group's window on each event fed into the group. They now select the needed ranks with `select_nth_unstable_by`, which is O(n) on average and returns exactly the same values, including interpolated percentiles and even-count medians. In the new `correlation_numeric_window` benchmark, a group checked against a 5,000-value window processes events about 4x faster. Chained correlations also stop cloning every fired result, including its retained events in full event mode, just to read which correlation fired and for which group.

crates/rsigma-cli/src/exit_code.rs

@@ -8,7 +8,6 @@
 /// For `eval`: events were processed (detections may or may not have fired).
 /// For `lint`: no findings at the configured severity threshold.
 /// For `validate`: all rules parsed and compiled.
-#[allow(dead_code)]
 pub const SUCCESS: i32 = 0;
 
 /// Findings were produced.

crates/rsigma-eval/README.md

@@ -36,6 +36,9 @@ The free function `apply_filters(collection: &SigmaCollection) -> Vec<SigmaRule>
 | `bloom_max_bytes()` | Configured bloom memory budget, or `None` if using the default |
 | `set_cross_rule_ac(enabled: bool)` | Enable opt-in cross-rule Aho-Corasick pre-filter (requires `daachorse-index` feature; see [Cross-Rule AC Index](#cross-rule-aho-corasick-index-opt-in-feature-gated)) |
 | `cross_rule_ac_enabled()` | Whether the cross-rule AC pre-filter is currently enabled (`daachorse-index` only) |
+| `set_retain_hir(enabled: bool)` | Keep each loaded rule's HIR for `save_hir` (off by default; enable before loading rules) |
+| `retain_hir_enabled()` | Whether loaded rules keep their HIR |
+| `save_hir()` / `load_hir(bytes)` | Write the retained HIR as a versioned restart cache, and load rules from one; `save_hir` errors unless retention was on for every rule loaded from parsed Sigma |
 | `rule_count()` | Number of loaded rules |
 | `rules()` | Access the compiled rules slice |
 | `transform_rule(rule: &SigmaRule)` | Apply the engine's pipelines and return the rewritten rule as a `TransformedRule`, without compiling or loading (inspection only) |
@@ -94,6 +97,8 @@ A non-short-circuiting, bloom-free recording evaluator that backs `rsigma engine
 | `parse_sources_file(path: &Path)` | Parse a standalone source YAML file (top-level `sources:` list) |
 | `parse_sources_dir(dir: &Path)` | Parse all `*.yml`/`*.yaml` source files in a directory |
 | `validate_source_refs(pipeline, external_ids)` | Validate that `${source.*}` references resolve against declared and/or external source IDs |
+| `sources::scan_source_templates(text)` | Find every `${source.<id>}` or `${source.<id>.<path>}` template in a string, with its id, path, and byte offset; the one reference grammar used by pipeline parsing and runtime expansion |
+| `sources::is_valid_source_id(id)` | Whether an id is declarable: ASCII letters, digits, `_`, and `-` |
 | `apply_pipelines(pipelines, rule)` | Apply all pipelines to a rule, in slice order |
 | `apply_pipelines_with_state(pipelines, rule)` | Same ordering, and returns the merged `PipelineState` (for backends) |
 | `merge_pipelines(pipelines)` | Sort a pipeline slice by `priority` in place, lower first. Despite the name it does not combine or apply them; call it before the `apply_*` / `transform_*` functions, which walk the slice as given |

crates/rsigma-eval/src/engine/mod.rs

@@ -32,7 +32,7 @@ use crate::pipeline::{Pipeline, TransformedRule, apply_pipelines, transform_coll
 use crate::result::{EvaluationResult, MatchDetailLevel};
 use crate::rule_metadata::{RuleBundleMetadata, RuleMetadataLookup};
 
-use bloom_index::{BloomCache, FieldBloomIndex};
+use bloom_index::{BloomCache, BloomLookup, FieldBloomIndex};
 
 pub use filters::apply_filters;
 use filters::{
@@ -90,10 +90,16 @@ pub(crate) struct IdentifiedEvaluation {
 pub struct Engine<L = LogSourceExtractor> {
     rules: Vec<CompiledRule>,
     /// Post-pipeline, pre-filter HIR for rules added via the parsed-rule paths,
-    /// retained so [`Engine::save_hir`] can serialize a restart cache. Kept in
-    /// step with `rules` for those paths; rules added via `add_compiled_rule`
-    /// have no HIR and are not represented here.
+    /// retained so [`Engine::save_hir`] can serialize a restart cache. Empty
+    /// unless [`Engine::set_retain_hir`] enabled retention; rules added via
+    /// `add_compiled_rule` have no HIR and are not represented here.
     ir_rules: Vec<IrRule>,
+    /// Whether the parsed-rule paths keep each rule's HIR in `ir_rules`.
+    retain_hir: bool,
+    /// Rules added via the parsed-rule paths while retention was off. A
+    /// non-zero count makes `save_hir` fail instead of writing a cache that
+    /// silently misses them.
+    unretained_hir: usize,
     pipelines: Vec<Pipeline>,
     /// Global override: include the full event JSON in all match results.
     /// When `true`, overrides per-rule `rsigma.include_event` custom attributes.
@@ -186,6 +192,8 @@ impl<L> Engine<L> {
         Engine {
             rules: Vec::new(),
             ir_rules: Vec::new(),
+            retain_hir: false,
+            unretained_hir: 0,
             pipelines,
             include_event: false,
             match_detail: MatchDetailLevel::Off,
@@ -341,6 +349,26 @@ impl<L> Engine<L> {
         self.cross_rule_ac_enabled
     }
 
+    /// Enable or disable retention of each loaded rule's HIR for
+    /// [`Engine::save_hir`].
+    ///
+    /// Off by default: nothing on the evaluation path reads the HIR, so
+    /// keeping it only costs memory. Enable it before loading rules when the
+    /// engine's rules will be saved as a restart cache. Disabling it drops
+    /// any HIR already retained.
+    pub fn set_retain_hir(&mut self, enabled: bool) {
+        if !enabled {
+            self.unretained_hir += self.ir_rules.len();
+            self.ir_rules = Vec::new();
+        }
+        self.retain_hir = enabled;
+    }
+
+    /// Returns whether loaded rules keep their HIR for [`Engine::save_hir`].
+    pub fn retain_hir_enabled(&self) -> bool {
+        self.retain_hir
+    }
+
     /// Set global `include_event` — when `true`, all match results include
     /// the full event JSON regardless of per-rule custom attributes.
     pub fn set_include_event(&mut self, include: bool) {
@@ -431,20 +459,29 @@ impl<L> Engine<L> {
     }
 
     /// Lower a rule to HIR (applying any configured pipelines first), compile
-    /// it, and store both the HIR and the compiled rule. Shared by the single-
-    /// and batched-add paths so they stay behaviourally identical, and so the
-    /// retained HIR (`ir_rules`) tracks `rules` for [`Engine::save_hir`].
+    /// it, and store the compiled rule, plus its HIR when retention is on.
+    /// Shared by the single- and batched-add paths so they stay behaviourally
+    /// identical, and so the retained HIR (`ir_rules`) tracks `rules` for
+    /// [`Engine::save_hir`].
     ///
     /// Both pushes happen only after lowering and compilation succeed, so a
     /// failing rule leaves neither vector mutated.
     fn compile_and_store(&mut self, rule: &SigmaRule) -> Result<()> {
         let ir = self.lower_with_pipelines(rule)?;
         let compiled = compile_to_compiled(&ir)?;
-        self.ir_rules.push(ir);
-        self.rules.push(compiled);
+        self.store(ir, compiled);
         Ok(())
     }
 
+    fn store(&mut self, ir: IrRule, compiled: CompiledRule) {
+        if self.retain_hir {
+            self.ir_rules.push(ir);
+        } else {
+            self.unretained_hir += 1;
+        }
+        self.rules.push(compiled);
+    }
+
     /// Lower a rule to HIR, applying any configured pipelines first.
     fn lower_with_pipelines(&self, rule: &SigmaRule) -> Result<IrRule> {
         if self.pipelines.is_empty() {
@@ -465,7 +502,23 @@ impl<L> Engine<L> {
     /// injections (applied to the compiled rules) or rules added via
     /// [`Engine::add_compiled_rule`] / [`Engine::extend_compiled_rules`].
     /// Re-apply any filters after [`Engine::load_hir`] if the engine used them.
+    ///
+    /// Requires HIR retention ([`Engine::set_retain_hir`]) to have been on
+    /// for every rule loaded through those paths; otherwise this returns
+    /// [`EvalError::HirCache`] rather than a cache that misses rules.
     pub fn save_hir(&self) -> Result<Vec<u8>> {
+        if !self.retain_hir {
+            return Err(EvalError::HirCache(
+                "HIR retention is off; call Engine::set_retain_hir(true) before loading rules"
+                    .to_string(),
+            ));
+        }
+        if self.unretained_hir > 0 {
+            return Err(EvalError::HirCache(format!(
+                "{} rules were loaded while HIR retention was off and cannot be saved",
+                self.unretained_hir
+            )));
+        }
         Ok(rsigma_ir::encode_rules(&self.ir_rules)?)
     }
 
@@ -474,13 +527,13 @@ impl<L> Engine<L> {
     ///
     /// Rules are appended to any already loaded, so a warm start loads into a
     /// fresh [`Engine::new`]. A blob whose schema version differs from this
-    /// build's is rejected (see [`rsigma_ir::decode_rules`]).
+    /// build's is rejected (see [`rsigma_ir::decode_rules`]). The loaded HIR
+    /// is retained only when [`Engine::set_retain_hir`] is on.
     pub fn load_hir(&mut self, bytes: &[u8]) -> Result<()> {
         let rules = rsigma_ir::decode_rules(bytes)?;
         for ir in rules {
             let compiled = compile_to_compiled(&ir)?;
-            self.ir_rules.push(ir);
-            self.rules.push(compiled);
+            self.store(ir, compiled);
         }
         self.rebuild_index();
         Ok(())
@@ -748,9 +801,12 @@ impl<L> Engine<L> {
         event_logsource: Option<&LogSource>,
     ) -> Vec<IdentifiedEvaluation> {
         if self.bloom_prefilter {
-            self.evaluate_with_bloom_path(event, event_logsource)
+            let bloom = BloomCache::new(&self.bloom_index, event);
+            self.evaluate_candidates(event, event_logsource, &bloom)
         } else {
-            self.evaluate_no_bloom_path(event, event_logsource)
+            // `NoBloom` is zero-sized, so this instantiation compiles to the
+            // same straight-line code as an engine without a bloom filter.
+            self.evaluate_candidates(event, event_logsource, &bloom_index::NoBloom)
         }
     }
 
@@ -818,14 +874,12 @@ impl<L> Engine<L> {
         }
     }
 
-    fn evaluate_no_bloom_path<E: Event>(
+    fn evaluate_candidates<E: Event, B: BloomLookup>(
         &self,
         event: &E,
         event_logsource: Option<&LogSource>,
+        bloom: &B,
     ) -> Vec<IdentifiedEvaluation> {
-        // Pass the zero-sized `NoBloom` lookup so this monomorphizes to the
-        // same straight-line code as the pre-bloom engine while still
-        // threading the configured match-detail level.
         let keep = self.cross_rule_ac_keep_mask(event);
         // `event_logsource` is `None` (the default) unless pruning is enabled,
         // leaving the loop's behaviour unchanged.
@@ -845,15 +899,7 @@ impl<L> Engine<L> {
                 pruned += 1;
                 continue;
             }
-            if let Some(mut m) =
-                evaluate_rule_with_bloom(rule, event, &bloom_index::NoBloom, self.match_detail)
-            {
-                if self.include_event
-                    && let Some(d) = m.as_detection_mut()
-                    && d.event.is_none()
-                {
-                    d.event = Some(event.to_json());
-                }
+            if let Some(m) = self.evaluate_one(rule, event, bloom) {
                 results.push(IdentifiedEvaluation {
                     result: m,
                     rule_id: rule.id.clone(),
@@ -865,47 +911,21 @@ impl<L> Engine<L> {
         results
     }
 
-    fn evaluate_with_bloom_path<E: Event>(
+    /// Evaluate one rule, attaching the event when `include_event` is set.
+    fn evaluate_one<E: Event, B: BloomLookup>(
         &self,
+        rule: &CompiledRule,
         event: &E,
-        event_logsource: Option<&LogSource>,
-    ) -> Vec<IdentifiedEvaluation> {
-        let bloom = BloomCache::new(&self.bloom_index, event);
-        let keep = self.cross_rule_ac_keep_mask(event);
-        // `event_logsource` is `None` (the default) unless pruning is enabled,
-        // leaving the loop's behaviour unchanged.
-        let candidates = self.logsource_candidates(event, event_logsource);
-        let mut results = Vec::new();
-        let mut pruned = 0u64;
-        for idx in candidates {
-            if let Some(ref mask) = keep
-                && !mask[idx]
-            {
-                continue;
-            }
-            let rule = &self.rules[idx];
-            if let Some(event_ls) = event_logsource
-                && !logsource_compatible(&rule.logsource, event_ls)
-            {
-                pruned += 1;
-                continue;
-            }
-            if let Some(mut m) = evaluate_rule_with_bloom(rule, event, &bloom, self.match_detail) {
-                if self.include_event
-                    && let Some(d) = m.as_detection_mut()
-                    && d.event.is_none()
-                {
-                    d.event = Some(event.to_json());
-                }
-                results.push(IdentifiedEvaluation {
-                    result: m,
-                    rule_id: rule.id.clone(),
-                    rule_name: rule.name.clone(),
-                });
-            }
+        bloom: &B,
+    ) -> Option<EvaluationResult> {
+        let mut m = evaluate_rule_with_bloom(rule, event, bloom, self.match_detail)?;
+        if self.include_event
+            && let Some(d) = m.as_detection_mut()
+            && d.event.is_none()
+        {
+            d.event = Some(event.to_json());
         }
-        self.record_logsource_pruned(pruned);
-        results
+        Some(m)
     }
 
     /// Fold one event's residual logsource-conflict tally into the counter.
@@ -929,16 +949,18 @@ impl<L> Engine<L> {
         event_logsource: &LogSource,
     ) -> Vec<EvaluationResult> {
         if self.bloom_prefilter {
-            self.evaluate_with_logsource_with_bloom(event, event_logsource)
+            let bloom = BloomCache::new(&self.bloom_index, event);
+            self.evaluate_matching_logsource(event, event_logsource, &bloom)
         } else {
-            self.evaluate_with_logsource_no_bloom(event, event_logsource)
+            self.evaluate_matching_logsource(event, event_logsource, &bloom_index::NoBloom)
         }
     }
 
-    fn evaluate_with_logsource_no_bloom<E: Event>(
+    fn evaluate_matching_logsource<E: Event, B: BloomLookup>(
         &self,
         event: &E,
         event_logsource: &LogSource,
+        bloom: &B,
     ) -> Vec<EvaluationResult> {
         let keep = self.cross_rule_ac_keep_mask(event);
         let mut results = Vec::new();
@@ -950,46 +972,8 @@ impl<L> Engine<L> {
             }
             let rule = &self.rules[idx];
             if logsource_matches(&rule.logsource, event_logsource)
-                && let Some(mut m) =
-                    evaluate_rule_with_bloom(rule, event, &bloom_index::NoBloom, self.match_detail)
+                && let Some(m) = self.evaluate_one(rule, event, bloom)
             {
-                if self.include_event
-                    && let Some(d) = m.as_detection_mut()
-                    && d.event.is_none()
-                {
-                    d.event = Some(event.to_json());
-                }
-                results.push(m);
-            }
-        }
-        results
-    }
-
-    fn evaluate_with_logsource_with_bloom<E: Event>(
-        &self,
-        event: &E,
-        event_logsource: &LogSource,
-    ) -> Vec<EvaluationResult> {
-        let bloom = BloomCache::new(&self.bloom_index, event);
-        let keep = self.cross_rule_ac_keep_mask(event);
-        let mut results = Vec::new();
-        for idx in self.rule_index.candidates(event) {
-            if let Some(ref mask) = keep
-                && !mask[idx]
-            {
-                continue;
-            }
-            let rule = &self.rules[idx];
-            if logsource_matches(&rule.logsource, event_logsource)
-                && let Some(mut m) =
-                    evaluate_rule_with_bloom(rule, event, &bloom, self.match_detail)
-            {
-                if self.include_event
-                    && let Some(d) = m.as_detection_mut()
-                    && d.event.is_none()
-                {
-                    d.event = Some(event.to_json());
-                }
                 results.push(m);
             }
         }
@@ -1062,15 +1046,7 @@ impl<L> Engine<L> {
             if !keep(rule) {
                 continue;
             }
-            if let Some(mut m) =
-                evaluate_rule_with_bloom(rule, event, &bloom_index::NoBloom, self.match_detail)
-            {
-                if self.include_event
-                    && let Some(d) = m.as_detection_mut()
-                    && d.event.is_none()
-                {
-                    d.event = Some(event.to_json());
-                }
+            if let Some(m) = self.evaluate_one(rule, event, &bloom_index::NoBloom) {
                 results.push(m);
             }
         }