diff --git a/CHANGELOG.md b/CHANGELOG.md index 3eaa75cd..fea18850 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,17 @@ All notable changes to RSigma are documented in this file. Each entry correspond ## [Unreleased] +## [0.25.0] - 2026-10-19 + +**TL;DR** +RSigma v0.25.0 is the "secure defaults and trustworthy results" release: three security advisories are fixed, the daemon and MCP server refuse unsafe exposure unless told otherwise, rules match exactly what they say, and correlation state, alert pipelines, and sinks hold up under reloads, restarts, caps, and delivery failures. +* Security: OTLP/HTTP gzip bodies are capped at 4 MiB after decompression ([GHSA-vwg6-4q3h-8pvq](https://github.com/timescale/rsigma/security/advisories/GHSA-vwg6-4q3h-8pvq), #597), the MCP server refuses unconfined writes and unconfined HTTP serving ([GHSA-33jj-q567-34gj](https://github.com/timescale/rsigma/security/advisories/GHSA-33jj-q567-34gj), #598), and event values are escaped in enricher and webhook templates ([GHSA-3fh4-8r8j-8xw3](https://github.com/timescale/rsigma/security/advisories/GHSA-3fh4-8r8j-8xw3), #599). Every build now refuses to serve an unauthenticated daemon API on a non-loopback address (#601), `rustls` picks up four RUSTSEC fixes (#577), and OTLP log decoding is fuzzed (#617). +* Matching correctness: integer literals above 2^53 match exactly (#610), boolean keywords match the word (#609), partly case-insensitive regex pairs match the original text again (#608), time-part modifiers accept space-separated fractional timestamps (#611), match detail reports only the selections that decide the match (#612), `0 of` selectors are rejected (#562), and `value_percentile` thresholds apply (#621). +* State and delivery: hot-reload no longer rolls back correlation state (#604), state saves are one consistent, atomic cut (#606), restore and alert-pipeline reload enforce the dedup and incident caps (#605), entity-graph grouping no longer leaks join keys or drops live incidents (#603), the inhibit store is capped (#616), alerts dropped without a DLQ sink are logged (#615), OTLP sink exports time out (#600), and file and Unix socket sink retries no longer repeat lines (#621). +* Diagnostics, API, and performance: parse warnings are reported apart from parse errors (#566), lint and the parser agree on modifiers and conditions (#565), pipeline errors get their own variant (#561), embedders can plug in custom logsource extractors (#578, based on work by John Sonnenschein in #500), `engine eval -o` writes results to a file (#576), match detail costs one evaluation pass (#613), percentile and median checks no longer sort the window (#619), and engines no longer retain rule HIR by default (#620). +* Breaking: a non-loopback daemon API needs authentication or `--allow-unauthenticated`; `mcp serve --http` needs `--rules-dir` or `--allow-unconfined-paths`; event values in enricher and webhook URLs are encoded as one component each; `HIR_SCHEMA_VERSION` is 4 and `save_hir` requires HIR retention; `SigmaCollection::diagnostics` replaces `errors`; `EvalError` gains `Pipeline` and is `#[non_exhaustive]`; `EventFilter` takes the parsed line by value; numeric matchers and backend methods use `NumericLiteral`; `incompatible_modifiers` is a lint error; and `cel` 0.15 changes how some extract expressions evaluate. See the migration notes in the entries below. +* Platform and docs: `--no-default-features` builds compile again (#560), the docs are checked against the code on every pull request (#558, #559), the PostgreSQL backend's JSONB and correlation code moves into dedicated modules with byte-identical SQL (#567, #568), the `test` backend is compared against pySigma 2.0.1 (#564), and dependencies are updated, including the docs site's `sharp` security fix (#556, #563, #577). + ### `value_percentile` thresholds apply, sink retries stop repeating lines, and smaller eval and runtime fixes (#621) `value_percentile` correlations ignored their condition: the percentile was computed and the correlation fired whenever the window held a finite value, so a rule with `lte: 50` fired on a window whose percentile was 50.5. The predicates now apply to the percentile as they do for the other numeric aggregates. @@ -200,6 +211,8 @@ The docs site publishes the config JSON Schema at `/rsigma.schema.json`, the URL Rolls up the compatible Dependabot PRs that opened after #550, with `Cargo.lock` regenerated against current `main`. Rust: `rmcp` 3.3.0 to 3.5.0 (#555), which renames the deprecated `ServerInfo` alias to `ServerConfig` in the MCP server handler, and `tokio-rustls` 0.26.5 to 0.26.6 (#554). CI: `taiki-e/install-action` v2.87.21 to v2.87.22, repinned by commit SHA (#553). VS Code: `@types/vscode` 1.138.0 to 1.140.0 (#552). Held back: `yamlpath` 1.30.1 (#495) and `yamlpatch` 1.30.1 (#494) require rustc 1.97 through `tree-sitter-iter`, above the 1.96.0 MSRV; `tikv-jemallocator` 0.7.0 (#425) remains excluded after the measured musl daemon throughput regression. +[v0.24.0...v0.25.0](https://github.com/timescale/rsigma/compare/v0.24.0...v0.25.0) + ## [0.24.0] - 2026-10-05 **TL;DR** @@ -3406,6 +3419,7 @@ First release of rsigma -- a Sigma detection toolkit in Rust. Ships a parser, ev Initial crates.io publish. Reserved the `rsigma` crate name with a minimal CLI binary (parser + evaluator only, no linter/LSP/pipelines/correlation). Superseded the same day by v0.2.0, which is the first feature-complete release. +[0.25.0]: https://github.com/timescale/rsigma/releases/tag/v0.25.0 [0.24.0]: https://github.com/timescale/rsigma/releases/tag/v0.24.0 [0.23.0]: https://github.com/timescale/rsigma/releases/tag/v0.23.0 [0.22.0]: https://github.com/timescale/rsigma/releases/tag/v0.22.0 diff --git a/Cargo.lock b/Cargo.lock index 8538cc1d..9eb5953b 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4566,7 +4566,7 @@ dependencies = [ [[package]] name = "rsigma" -version = "0.24.0" +version = "0.25.0" dependencies = [ "anyhow", "arc-swap", @@ -4639,7 +4639,7 @@ dependencies = [ [[package]] name = "rsigma-convert" -version = "0.24.0" +version = "0.25.0" dependencies = [ "insta", "log", @@ -4658,7 +4658,7 @@ dependencies = [ [[package]] name = "rsigma-eval" -version = "0.24.0" +version = "0.25.0" dependencies = [ "ahash", "aho-corasick", @@ -4685,7 +4685,7 @@ dependencies = [ [[package]] name = "rsigma-ir" -version = "0.24.0" +version = "0.25.0" dependencies = [ "base64 0.23.1", "ciborium", @@ -4699,7 +4699,7 @@ dependencies = [ [[package]] name = "rsigma-lsp" -version = "0.24.0" +version = "0.25.0" dependencies = [ "env_logger", "insta", @@ -4712,7 +4712,7 @@ dependencies = [ [[package]] name = "rsigma-mcp" -version = "0.24.0" +version = "0.25.0" dependencies = [ "anyhow", "axum", @@ -4737,7 +4737,7 @@ dependencies = [ [[package]] name = "rsigma-parser" -version = "0.24.0" +version = "0.25.0" dependencies = [ "criterion", "fancy-regex", @@ -4759,7 +4759,7 @@ dependencies = [ [[package]] name = "rsigma-runtime" -version = "0.24.0" +version = "0.25.0" dependencies = [ "arc-swap", "async-nats", @@ -4827,7 +4827,7 @@ dependencies = [ [[package]] name = "rstix" -version = "0.24.0" +version = "0.25.0" dependencies = [ "base64 0.23.1", "email_address", diff --git a/Cargo.toml b/Cargo.toml index ec6ee50d..7ed22fbc 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -14,7 +14,7 @@ members = [ exclude = ["fuzz", "ci/wasm-smoke"] [workspace.package] -version = "0.24.0" +version = "0.25.0" edition = "2024" rust-version = "1.96.0" license = "MIT" diff --git a/ci/wasm-smoke/Cargo.lock b/ci/wasm-smoke/Cargo.lock index 6cd19bd3..16b2ff86 100644 --- a/ci/wasm-smoke/Cargo.lock +++ b/ci/wasm-smoke/Cargo.lock @@ -501,7 +501,7 @@ checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4" [[package]] name = "rsigma-eval" -version = "0.24.0" +version = "0.25.0" dependencies = [ "ahash", "aho-corasick", @@ -522,7 +522,7 @@ dependencies = [ [[package]] name = "rsigma-ir" -version = "0.24.0" +version = "0.25.0" dependencies = [ "base64", "ciborium", @@ -535,7 +535,7 @@ dependencies = [ [[package]] name = "rsigma-parser" -version = "0.24.0" +version = "0.25.0" dependencies = [ "fancy-regex", "globset", diff --git a/crates/rsigma-cli/Cargo.toml b/crates/rsigma-cli/Cargo.toml index 093ba2ce..c2d823ff 100644 --- a/crates/rsigma-cli/Cargo.toml +++ b/crates/rsigma-cli/Cargo.toml @@ -64,12 +64,12 @@ stix-enrich = [ ] [dependencies] -rsigma-parser = { path = "../rsigma-parser", version = "0.24.0" } -rsigma-eval = { path = "../rsigma-eval", version = "0.24.0", features = ["parallel"] } -rsigma-convert = { path = "../rsigma-convert", version = "0.24.0", features = ["sigma-cli"] } -rsigma-runtime = { path = "../rsigma-runtime", version = "0.24.0", optional = true } -rsigma-mcp = { path = "../rsigma-mcp", version = "0.24.0", optional = true } -rstix = { path = "../rstix", version = "0.24.0", optional = true } +rsigma-parser = { path = "../rsigma-parser", version = "0.25.0" } +rsigma-eval = { path = "../rsigma-eval", version = "0.25.0", features = ["parallel"] } +rsigma-convert = { path = "../rsigma-convert", version = "0.25.0", features = ["sigma-cli"] } +rsigma-runtime = { path = "../rsigma-runtime", version = "0.25.0", optional = true } +rsigma-mcp = { path = "../rsigma-mcp", version = "0.25.0", optional = true } +rstix = { path = "../rstix", version = "0.25.0", optional = true } clap = { version = "4", features = ["derive", "env"] } serde = { version = "1", features = ["derive"] } serde_json = "1" diff --git a/crates/rsigma-convert/Cargo.toml b/crates/rsigma-convert/Cargo.toml index f5988ef1..7d3970fd 100644 --- a/crates/rsigma-convert/Cargo.toml +++ b/crates/rsigma-convert/Cargo.toml @@ -22,9 +22,9 @@ default = [] sigma-cli = [] [dependencies] -rsigma-parser = { path = "../rsigma-parser", version = "0.24.0" } -rsigma-eval = { path = "../rsigma-eval", version = "0.24.0" } -rsigma-ir = { path = "../rsigma-ir", version = "0.24.0", default-features = false } +rsigma-parser = { path = "../rsigma-parser", version = "0.25.0" } +rsigma-eval = { path = "../rsigma-eval", version = "0.25.0" } +rsigma-ir = { path = "../rsigma-ir", version = "0.25.0", default-features = false } thiserror = "2" serde_json = "1" yaml_serde = "0.10" diff --git a/crates/rsigma-eval/Cargo.toml b/crates/rsigma-eval/Cargo.toml index 016fba44..18d2bf58 100644 --- a/crates/rsigma-eval/Cargo.toml +++ b/crates/rsigma-eval/Cargo.toml @@ -14,8 +14,8 @@ parallel = ["rayon"] daachorse-index = ["dep:daachorse"] [dependencies] -rsigma-parser = { path = "../rsigma-parser", version = "0.24.0", default-features = false } -rsigma-ir = { path = "../rsigma-ir", version = "0.24.0", default-features = false } +rsigma-parser = { path = "../rsigma-parser", version = "0.25.0", default-features = false } +rsigma-ir = { path = "../rsigma-ir", version = "0.25.0", default-features = false } serde = { version = "1", features = ["derive", "rc"] } serde_json = "1" yaml_serde = "0.10" diff --git a/crates/rsigma-ir/Cargo.toml b/crates/rsigma-ir/Cargo.toml index 1efaf4a8..7a79ba3a 100644 --- a/crates/rsigma-ir/Cargo.toml +++ b/crates/rsigma-ir/Cargo.toml @@ -11,7 +11,7 @@ homepage.workspace = true [dependencies] base64 = "0.23" -rsigma-parser = { path = "../rsigma-parser", version = "0.24.0", default-features = false } +rsigma-parser = { path = "../rsigma-parser", version = "0.25.0", default-features = false } ciborium = "0.2" serde = { version = "1", features = ["derive"] } serde_json = "1" diff --git a/crates/rsigma-lsp/Cargo.toml b/crates/rsigma-lsp/Cargo.toml index 76d353d8..25f9d1b4 100644 --- a/crates/rsigma-lsp/Cargo.toml +++ b/crates/rsigma-lsp/Cargo.toml @@ -14,8 +14,8 @@ name = "rsigma-lsp" path = "src/main.rs" [dependencies] -rsigma-parser = { path = "../rsigma-parser", version = "0.24.0" } -rsigma-eval = { path = "../rsigma-eval", version = "0.24.0" } +rsigma-parser = { path = "../rsigma-parser", version = "0.25.0" } +rsigma-eval = { path = "../rsigma-eval", version = "0.25.0" } tower-lsp-server = "0.23" tokio = { version = "1", features = ["rt-multi-thread", "macros", "io-std", "time", "sync"] } log = "0.4" diff --git a/crates/rsigma-mcp/Cargo.toml b/crates/rsigma-mcp/Cargo.toml index 4fd9e952..11fa600a 100644 --- a/crates/rsigma-mcp/Cargo.toml +++ b/crates/rsigma-mcp/Cargo.toml @@ -16,10 +16,10 @@ default = [] http = ["dep:axum", "rmcp/transport-streamable-http-server"] [dependencies] -rsigma-parser = { path = "../rsigma-parser", version = "0.24.0" } -rsigma-eval = { path = "../rsigma-eval", version = "0.24.0" } -rsigma-convert = { path = "../rsigma-convert", version = "0.24.0", features = ["sigma-cli"] } -rsigma-runtime = { path = "../rsigma-runtime", version = "0.24.0" } +rsigma-parser = { path = "../rsigma-parser", version = "0.25.0" } +rsigma-eval = { path = "../rsigma-eval", version = "0.25.0" } +rsigma-convert = { path = "../rsigma-convert", version = "0.25.0", features = ["sigma-cli"] } +rsigma-runtime = { path = "../rsigma-runtime", version = "0.25.0" } rmcp = { version = "3.0", features = ["server", "macros", "transport-io"] } schemars = "1" serde = { version = "1", features = ["derive"] } diff --git a/crates/rsigma-runtime/Cargo.toml b/crates/rsigma-runtime/Cargo.toml index 91933b60..4be760e1 100644 --- a/crates/rsigma-runtime/Cargo.toml +++ b/crates/rsigma-runtime/Cargo.toml @@ -24,9 +24,9 @@ daachorse-index = ["rsigma-eval/daachorse-index"] stix-enrich = ["dep:rstix", "rstix/store", "rstix/store-fs"] [dependencies] -rsigma-parser = { path = "../rsigma-parser", version = "0.24.0" } -rstix = { path = "../rstix", version = "0.24.0", optional = true, default-features = false, features = ["serde"] } -rsigma-eval = { path = "../rsigma-eval", version = "0.24.0", features = ["parallel"] } +rsigma-parser = { path = "../rsigma-parser", version = "0.25.0" } +rstix = { path = "../rstix", version = "0.25.0", optional = true, default-features = false, features = ["serde"] } +rsigma-eval = { path = "../rsigma-eval", version = "0.25.0", features = ["parallel"] } tokio = { version = "1", features = ["rt-multi-thread", "sync", "macros", "io-util", "io-std", "process", "fs"] } serde = { version = "1", features = ["derive"] } serde_json = "1" diff --git a/crates/rsigma-runtime/tests/golden/ocsf/correlation.json b/crates/rsigma-runtime/tests/golden/ocsf/correlation.json index 7bc6becd..6d98cd02 100644 --- a/crates/rsigma-runtime/tests/golden/ocsf/correlation.json +++ b/crates/rsigma-runtime/tests/golden/ocsf/correlation.json @@ -30,7 +30,7 @@ "product": { "name": "rsigma", "vendor_name": "rsigma", - "version": "0.24.0" + "version": "0.25.0" }, "uid": "00000000-0000-4000-8000-000000000002", "version": "1.1.0" diff --git a/crates/rsigma-runtime/tests/golden/ocsf/detection.json b/crates/rsigma-runtime/tests/golden/ocsf/detection.json index 90e4270e..f8572b34 100644 --- a/crates/rsigma-runtime/tests/golden/ocsf/detection.json +++ b/crates/rsigma-runtime/tests/golden/ocsf/detection.json @@ -53,7 +53,7 @@ "product": { "name": "rsigma", "vendor_name": "rsigma", - "version": "0.24.0" + "version": "0.25.0" }, "uid": "00000000-0000-4000-8000-000000000002", "version": "1.1.0" diff --git a/crates/rsigma-runtime/tests/golden/ocsf/incident_open.json b/crates/rsigma-runtime/tests/golden/ocsf/incident_open.json index d7d83835..1957134a 100644 --- a/crates/rsigma-runtime/tests/golden/ocsf/incident_open.json +++ b/crates/rsigma-runtime/tests/golden/ocsf/incident_open.json @@ -35,7 +35,7 @@ "product": { "name": "rsigma", "vendor_name": "rsigma", - "version": "0.24.0" + "version": "0.25.0" }, "uid": "00000000-0000-4000-8000-000000000001", "version": "1.1.0" diff --git a/crates/rsigma-runtime/tests/golden/ocsf/incident_resolved.json b/crates/rsigma-runtime/tests/golden/ocsf/incident_resolved.json index 167dc334..5ff7da0b 100644 --- a/crates/rsigma-runtime/tests/golden/ocsf/incident_resolved.json +++ b/crates/rsigma-runtime/tests/golden/ocsf/incident_resolved.json @@ -35,7 +35,7 @@ "product": { "name": "rsigma", "vendor_name": "rsigma", - "version": "0.24.0" + "version": "0.25.0" }, "uid": "00000000-0000-4000-8000-000000000001", "version": "1.1.0" diff --git a/crates/rsigma-runtime/tests/golden/ocsf/risk_incident.json b/crates/rsigma-runtime/tests/golden/ocsf/risk_incident.json index 3b2debea..a52ca306 100644 --- a/crates/rsigma-runtime/tests/golden/ocsf/risk_incident.json +++ b/crates/rsigma-runtime/tests/golden/ocsf/risk_incident.json @@ -54,7 +54,7 @@ "product": { "name": "rsigma", "vendor_name": "rsigma", - "version": "0.24.0" + "version": "0.25.0" }, "uid": "00000000-0000-4000-8000-000000000001", "version": "1.1.0" diff --git a/crates/rstix/README.md b/crates/rstix/README.md index 7ddd02a9..b8fa0df9 100644 --- a/crates/rstix/README.md +++ b/crates/rstix/README.md @@ -19,11 +19,11 @@ Six optional Cargo features extend the default `serde` bundle model (each implie ```toml [dependencies] -rstix = "0.24.0" +rstix = "0.25.0" # Pattern Engine + Validation Pipeline: -# rstix = { version = "0.24.0", features = ["pattern", "validate"] } +# rstix = { version = "0.25.0", features = ["pattern", "validate"] } # Graph, marking, store (combine as needed): -# rstix = { version = "0.24.0", features = ["graph", "marking", "store"] } +# rstix = { version = "0.25.0", features = ["graph", "marking", "store"] } ``` This library is part of [rsigma]. diff --git a/docs/content/cli/engine/daemon.md b/docs/content/cli/engine/daemon.md index 92c59f4e..141099dd 100644 --- a/docs/content/cli/engine/daemon.md +++ b/docs/content/cli/engine/daemon.md @@ -48,8 +48,8 @@ For narrative coverage see [Streaming Detection](../../guide/streaming-detection | Flag | Default | Description | |------|---------|-------------| -| `--output ` | `stdout` | Detection sink. Schemes: `stdout`, `file://`, `nats://:/`, `otlp(s)://:` (OTLP/gRPC), `otlphttp(s)://:` (OTLP/HTTP, posts to `/v1/logs`); the `s` variants use TLS. On Unix, `unix:///path/to.sock` writes NDJSON to a local collector socket. Repeatable for fan-out. Query params: `?on_full=drop` (best-effort), `?format=ndjson\|ocsf` (line sinks only; see [OCSF Findings](../../guide/ocsf-findings.md)), `?compression=gzip` (OTLP), `?timeout=` {{ added "unreleased" }} (OTLP per-attempt export deadline, default `10s`), and for TLS `?ca=`, `?client_cert=`, `?client_key=` (PEM paths, the last two for mutual TLS) and `?tls_domain=` (SNI override). OTLP schemes require the `daemon-otlp` build. | -| `--dlq ` | unset | Dead-letter queue for events that fail parsing or sink delivery. Same schemes as `--output`. When unset, parse failures are counted on `rsigma_events_parse_errors_total` and discarded, and delivery failures are discarded with a rate-limited `warn` log that carries how many were dropped. {{ added "unreleased" }} | +| `--output ` | `stdout` | Detection sink. Schemes: `stdout`, `file://`, `nats://:/`, `otlp(s)://:` (OTLP/gRPC), `otlphttp(s)://:` (OTLP/HTTP, posts to `/v1/logs`); the `s` variants use TLS. On Unix, `unix:///path/to.sock` writes NDJSON to a local collector socket. Repeatable for fan-out. Query params: `?on_full=drop` (best-effort), `?format=ndjson\|ocsf` (line sinks only; see [OCSF Findings](../../guide/ocsf-findings.md)), `?compression=gzip` (OTLP), `?timeout=` {{ added "0.25.0" }} (OTLP per-attempt export deadline, default `10s`), and for TLS `?ca=`, `?client_cert=`, `?client_key=` (PEM paths, the last two for mutual TLS) and `?tls_domain=` (SNI override). OTLP schemes require the `daemon-otlp` build. | +| `--dlq ` | unset | Dead-letter queue for events that fail parsing or sink delivery. Same schemes as `--output`. When unset, parse failures are counted on `rsigma_events_parse_errors_total` and discarded, and delivery failures are discarded with a rate-limited `warn` log that carries how many were dropped. {{ added "0.25.0" }} | | `--include-event` | off | Embed the full event JSON in every detection match. | | `--match-detail ` | `off` | Match-detail verbosity: `off` (field + value only), `summary` (adds matcher kind, selection, case sensitivity, and reports keyword/absence matches), or `full` (also records the matched pattern). Also settable via `daemon.engine.match_detail`. See [Evaluating Rules](../../guide/evaluating-rules.md#match-detail). | | `--pretty` | off | Pretty-print JSON output. | @@ -61,7 +61,7 @@ For narrative coverage see [Streaming Detection](../../guide/streaming-detection Each sink serializes in its own wire format. The default is RSigma's native NDJSON; `?format=ocsf` emits [OCSF Detection Finding](../../guide/ocsf-findings.md) (class 2004) JSON instead, one finding per line. The parameter is accepted on `stdout`, `file://`, `nats://`, and `unix://` sinks only, and `format=` on an OTLP spec, the `--dlq`, or the `daemon.api.audit.sink` fails startup, as does an unknown value. -Each `--output` sink runs its own bounded queue and worker: results are delivered with bounded exponential-backoff retry, and a result is sent to the DLQ only after retries are exhausted. Fan-out is isolated up to each sink's queue depth, so a slow sink does not immediately stall the others; a slow durable sink eventually applies backpressure, the cost of preserving at-least-once delivery. An acknowledgment is released to the source only once every sink has committed the result, and `?on_full=drop` opts a sink out of that contract in favor of never stalling. A sink commits a failed result by handing it to the DLQ writer, so the ack does not wait for the DLQ sink's write; a failed DLQ write is logged and counted on `rsigma_dlq_write_failures_total`. {{ added "unreleased" }} The file and `unix://` sinks write each result's lines in one go and do not repeat lines a failed attempt already wrote when they retry it; a line a failed file write cut off is ended before anything else is appended. {{ added "unreleased" }} +Each `--output` sink runs its own bounded queue and worker: results are delivered with bounded exponential-backoff retry, and a result is sent to the DLQ only after retries are exhausted. Fan-out is isolated up to each sink's queue depth, so a slow sink does not immediately stall the others; a slow durable sink eventually applies backpressure, the cost of preserving at-least-once delivery. An acknowledgment is released to the source only once every sink has committed the result, and `?on_full=drop` opts a sink out of that contract in favor of never stalling. A sink commits a failed result by handing it to the DLQ writer, so the ack does not wait for the DLQ sink's write; a failed DLQ write is logged and counted on `rsigma_dlq_write_failures_total`. {{ added "0.25.0" }} The file and `unix://` sinks write each result's lines in one go and do not repeat lines a failed attempt already wrote when they retry it; a line a failed file write cut off is ended before anything else is appended. {{ added "0.25.0" }} Acknowledgment guarantees differ by transport. NATS is durable at-least-once: results un-acked at shutdown are redelivered on restart. The HTTP and OTLP push receivers are at-most-once on ingest, since the request is accepted as soon as the event is queued. @@ -94,13 +94,13 @@ The webhooks file (also settable as the `daemon.output.webhooks` path list) decl | Flag | Default | Description | |------|---------|-------------| | `--api-addr ` | `0.0.0.0:9090` | Bind address for `/healthz`, `/readyz`, `/metrics`, `/api/v1/*`, and (with the `daemon-otlp` feature) `/v1/logs`. A TCP `host:port`, or on Unix `unix:///path/to.sock` for a permission-gated local socket. A `unix://` address cannot be combined with TLS (the socket file is the trust boundary) and is exempt from the plaintext and unauthenticated bind refusals. | -| `--allow-plaintext` | off | Permit plaintext on a non-loopback TCP `--api-addr`. Without this flag (and without `--tls-cert`/`--tls-key`) the daemon refuses to start on any public address, in every build. {{ added "unreleased" }} Loopback (`127.0.0.0/8`, `::1`) always allows plaintext for local development. Config-file equivalent: `daemon.api.tls.allow_plaintext`. | -| `--allow-unauthenticated` {{ added "unreleased" }} | off | Permit an unauthenticated API on a non-loopback TCP `--api-addr`. Without this flag the daemon refuses to start on a public address unless bearer tokens (`--api-token-env` or `daemon.api.auth`) or client certificates (`--tls-client-ca`) authenticate callers. Use it only when an authenticating proxy or network policy fronts the daemon; a startup warning is logged. Config-file equivalent: `daemon.api.allow_unauthenticated`. | +| `--allow-plaintext` | off | Permit plaintext on a non-loopback TCP `--api-addr`. Without this flag (and without `--tls-cert`/`--tls-key`) the daemon refuses to start on any public address, in every build. {{ added "0.25.0" }} Loopback (`127.0.0.0/8`, `::1`) always allows plaintext for local development. Config-file equivalent: `daemon.api.tls.allow_plaintext`. | +| `--allow-unauthenticated` {{ added "0.25.0" }} | off | Permit an unauthenticated API on a non-loopback TCP `--api-addr`. Without this flag the daemon refuses to start on a public address unless bearer tokens (`--api-token-env` or `daemon.api.auth`) or client certificates (`--tls-client-ca`) authenticate callers. Use it only when an authenticating proxy or network policy fronts the daemon; a startup warning is logged. Config-file equivalent: `daemon.api.allow_unauthenticated`. | | `--api-token-env ` | unset | Require bearer-token authentication on the API, with the named environment variable holding the single accepted token (full `admin` permissions). The flag names the variable, so the secret never appears on the command line; an unset or empty variable fails startup. `GET /healthz` and `GET /readyz` stay open. Mutually exclusive with the `daemon.api.auth` config block, which adds per-token roles and granular `resource:action` permissions; see [HTTP API: Authentication](../../reference/http-api.md#authentication). | ### Authentication -Off by default: without `--api-token-env` or a `daemon.api.auth` config block, every route is open. {{ added "unreleased" }} The daemon accepts that only on loopback, on a `unix://` socket, on a listener that requires client certificates (`--tls-client-ca`), or with `--allow-unauthenticated`; any other non-loopback bind refuses to start. +Off by default: without `--api-token-env` or a `daemon.api.auth` config block, every route is open. {{ added "0.25.0" }} The daemon accepts that only on loopback, on a `unix://` socket, on a listener that requires client certificates (`--tls-client-ca`), or with `--allow-unauthenticated`; any other non-loopback bind refuses to start. For anything beyond the single admin token, configure the `daemon.api.auth` block (config-file-only, like the tap and tail tuning keys): named roles as `resource:action` permission lists, per-token role assignment with env-resolved secrets, and `anonymous_permissions` for requests without a token (for example `["metrics:read"]` keeps Prometheus scraping token-free). Built-in roles: `reader`, `operator`, `ingest`, `admin`. The per-route permission table, role definitions, and failure semantics (401 vs 403, the `rsigma_api_auth_failures_total` counter) are documented in [HTTP API: Authentication](../../reference/http-api.md#authentication); the threat-model discussion is in [Security: Daemon API authentication](../../reference/security.md#daemon-api-authentication). @@ -155,7 +155,7 @@ See [TLS deployment](../../reference/security.md#tls-termination-for-the-api-lis | `--correlation-event-mode ` | `none` | `none`, `full` (deflate-compressed full bodies), `refs` (timestamp + ID only). | | `--max-correlation-events ` | `10` | Cap on stored events per correlation window. | | `--max-state-entries ` | `100000` | Hard cap on correlation state entries across all correlations and group keys. When reached, the stalest entries are evicted to 90% capacity and a warning is logged. | -| `--max-group-entries ` | unset | Cap on retained entries within a single correlation group's window state. Bounds within-window growth of chatty groups; oldest entries are dropped (session windows keep their span anchor). Unset = unbounded, and a correlation with a `timespan` longer than one hour and no `rsigma.max_group_entries` logs a warning at load. {{ added "unreleased" }} | +| `--max-group-entries ` | unset | Cap on retained entries within a single correlation group's window state. Bounds within-window growth of chatty groups; oldest entries are dropped (session windows keep their span anchor). Unset = unbounded, and a correlation with a `timespan` longer than one hour and no `rsigma.max_group_entries` logs a warning at load. {{ added "0.25.0" }} | | `--timestamp-field ` | unset | Field name to prepend to the timestamp extraction list. Repeatable. | | `--timestamp-fallback ` | `wallclock` | Behavior when no timestamp is found: `wallclock` (use wall clock time) or `skip` (skip correlation state for that event). Use `skip` for forensic replay. | @@ -394,7 +394,7 @@ rsigma engine daemon -r rules/ --input http \ TLS uses the bundled public (webpki) roots by default; pass `?ca=` to verify against a private CA instead. Supplying both `?client_cert=` and `?client_key=` enables mutual TLS. Use `?tls_domain=` to override the verified server name when dialing by IP. -{{ added "unreleased" }} +{{ added "0.25.0" }} Every OTLP export attempt is bounded by a deadline, `10s` by default, so a collector that accepts the connection but never answers cannot stall the sink task (and, under the default `on_full=block`, every other sink behind it). A timed-out attempt is retried under `--sink-retry-max` and then routed to the DLQ like any other transient failure. Override the deadline per sink with a humantime duration, for example `otlp://otel-collector:4317?timeout=3s`; zero and unparsable values are rejected at startup. Connection setup is separately capped at 10 seconds, and gRPC channels send HTTP/2 keepalive pings so a silently dropped connection is detected. diff --git a/docs/content/cli/engine/eval.md b/docs/content/cli/engine/eval.md index 11c66ba9..b4c5582b 100644 --- a/docs/content/cli/engine/eval.md +++ b/docs/content/cli/engine/eval.md @@ -58,7 +58,7 @@ The global `--output-format` / `--color` / `--quiet` / `--no-stats` flags apply | Flag | Default | Description | |------|---------|-------------| -| `-o, --output ` | stdout | Write detection and correlation results to a file instead of stdout. The file is created or truncated at start and flushed on exit, including when `--fail-on-detection` exits with code 1. Results use `--output-format` when set and NDJSON otherwise; `--pretty` still applies. The summary and diagnostics stay on stderr. Config key: `eval.output`. {{ added "unreleased" }} | +| `-o, --output ` | stdout | Write detection and correlation results to a file instead of stdout. The file is created or truncated at start and flushed on exit, including when `--fail-on-detection` exits with code 1. Results use `--output-format` when set and NDJSON otherwise; `--pretty` still applies. The summary and diagnostics stay on stderr. Config key: `eval.output`. {{ added "0.25.0" }} | | `--pretty` | off | Pretty-print JSON output. Kept for backwards compatibility; equivalent to `--output-format json` with pretty-printing on. | | `--emit-detections` | off | Emit output for rules that only feed correlations, including correlations referenced by another correlation. By default, referenced rules are emitted only when a referencing correlation has top-level `generate: true`. {{ added "0.24.0" }} | | `--no-detections` | off | Deprecated compatibility flag. Correlation-only detections are suppressed by default. | diff --git a/docs/content/cli/mcp/serve.md b/docs/content/cli/mcp/serve.md index e20b0df8..4fc20276 100644 --- a/docs/content/cli/mcp/serve.md +++ b/docs/content/cli/mcp/serve.md @@ -26,7 +26,7 @@ The command is gated behind the opt-in `mcp` Cargo feature: build from source wi |------|---------|-------------| | `--lint-config ` | lint defaults | A lint config file (`.rsigma-lint.yml`) applied by the `lint_rules` tool (disabled rules, severity overrides, extra tag namespaces). Config key: `mcp.lint_config`. | | `--rules-dir ` | none | Root directory for path arguments in tool calls. Relative paths resolve against it, and any path that leaves it (absolute, `../`, or through a symlink) is refused. Without it, tools can read any file the server's OS user can read, `fix_rules` cannot write, `--http` refuses to start, and stdio prints a startup warning. Config key: `mcp.rules_dir`. | -| `--allow-unconfined-paths` | off | Run without `--rules-dir`, leaving path arguments unconfined. Required to serve `--http` without a root; on stdio it silences the startup warning. `fix_rules` still cannot write without a root. Config key: `mcp.allow_unconfined_paths`. {{ added "unreleased" }} | +| `--allow-unconfined-paths` | off | Run without `--rules-dir`, leaving path arguments unconfined. Required to serve `--http` without a root; on stdio it silences the startup warning. `fix_rules` still cannot write without a root. Config key: `mcp.allow_unconfined_paths`. {{ added "0.25.0" }} | | `--allow-sigma-cli` | off | Allow the `convert_rules` tool to delegate targets without a native backend to an installed [sigma-cli](../../reference/backends/sigma-cli.md), reaching the full pySigma backend set. Delegated calls spawn a subprocess (60s timeout, at most 2 concurrent); `path` and file-based `pipelines` inputs stay confined to `--rules-dir` when it is set. Config key: `mcp.allow_sigma_cli`. | | `--daemon-url ` | none | Base URL of a running rsigma daemon (`http://127.0.0.1:9090`). Registers the Operate-cycle read tools against that API. Unix-socket URLs are unsupported; use TCP loopback. Config key: `mcp.daemon_url`. {{ added "0.22.0" }} | | `--daemon-ca ` | none | Extra root CA (PEM) for a self-signed daemon TLS listener. Config key: `mcp.daemon_ca`. {{ added "0.22.0" }} | @@ -107,7 +107,7 @@ rsigma mcp serve --http 0.0.0.0:9100 --rules-dir /path/to/rules \ The endpoint is `https?:///mcp`. Plaintext binds on non-loopback addresses are refused unless `--allow-plaintext` is set (use it when a sidecar proxy terminates TLS). Native TLS reuses the daemon's rustls loader and is only available when the binary is built with `daemon-tls`. -`--http` refuses to start without `--rules-dir` unless `--allow-unconfined-paths` is set, because any client that reaches the endpoint could otherwise read files outside a rules tree. {{ added "unreleased" }} +`--http` refuses to start without `--rules-dir` unless `--allow-unconfined-paths` is set, because any client that reaches the endpoint could otherwise read files outside a rules tree. {{ added "0.25.0" }} ## Exit codes diff --git a/docs/content/cli/rule/validate.md b/docs/content/cli/rule/validate.md index a672c6cc..5398c437 100644 --- a/docs/content/cli/rule/validate.md +++ b/docs/content/cli/rule/validate.md @@ -14,7 +14,7 @@ rsigma rule validate [OPTIONS] Parses a single Sigma file, or walks a directory and parses every `*.yml`/`*.yaml` Sigma file, with `rsigma-parser`, optionally applies one or more processing pipelines, and compiles each rule with the evaluator's compiler. Reports the counts on stdout (human summary by default). Pass `--output-format json|ndjson|table|csv|tsv` for a structured envelope or `PATH,STATUS,ERRORS` rows. Exits with code `2` if any rule fails to parse or compile. See [Output Formats](../../reference/output.md). -The parser reports two kinds of problems. A parse error drops the document: malformed YAML, a missing condition, conflicting modifiers, an unknown detection reference. A parse warning keeps the document but ignores an invalid value: a `status` or `level` outside the allowed set, a malformed `related` entry, or an unreadable `sigma-version`. Parse warnings are reported but do not fail validation unless you pass `--fail-level warning`. {{ added "unreleased" }} +The parser reports two kinds of problems. A parse error drops the document: malformed YAML, a missing condition, conflicting modifiers, an unknown detection reference. A parse warning keeps the document but ignores an invalid value: a `status` or `level` outside the allowed set, a malformed `related` entry, or an unreadable `sigma-version`. Parse warnings are reported but do not fail validation unless you pass `--fail-level warning`. {{ added "0.25.0" }} Correlation references are checked as compile errors too. A correlation fails validation when it references a rule `id` or `name` that no detection or correlation rule carries. A reference that resolves to more than one rule also fails validation, which happens when rules share the referenced `id` or `name`, or when one rule's `name` equals another rule's `id`. {{ added "0.24.0" }} @@ -31,7 +31,7 @@ For narrative coverage see [Linting Rules](../../guide/linting-rules.md) and [CI | `-p, --pipeline ` | Processing pipeline(s) to apply. Builtin names (`ecs_windows`, `fibratus_windows`, `sysmon`) or YAML file paths. Repeatable. | | `--source ` | External source file(s) or directory of source files. Repeatable. Same standalone `sources:` YAML shape as the daemon's `--source`. Required for meaningful `--resolve-sources` runs after the v0.19.0 removal of pipeline-embedded `sources:`. | | `--resolve-sources` | Resolve and fetch every loaded dynamic source during validation. Sources must be reachable for validation to pass. | -| `--fail-level ` | Lowest parse diagnostic severity that exits `2`: `error` (default) fails on parse errors and compile errors, and `warning` also fails on parse warnings. Same flag name and default as [`rule lint --fail-level`](lint.md). {{ added "unreleased" }} | +| `--fail-level ` | Lowest parse diagnostic severity that exits `2`: `error` (default) fails on parse errors and compile errors, and `warning` also fails on parse warnings. Same flag name and default as [`rule lint --fail-level`](lint.md). {{ added "0.25.0" }} | ## Examples @@ -67,7 +67,7 @@ A YAML syntax error in the file counts as a parse error and exits with code `2`, ### Fail on parse warnings -{{ added "unreleased" }} +{{ added "0.25.0" }} ```bash rsigma rule validate rules/ --fail-level warning @@ -77,7 +77,7 @@ A rule with `status: draft` still parses and compiles, so plain `rule validate` ### Structured output -{{ added "unreleased" }} +{{ added "0.25.0" }} With `--output-format json`, the envelope carries `summary` (including `parse_errors` and `parse_warnings` counts), `parse_errors`, `parse_warnings`, and `compile_errors` arrays of messages. With `ndjson`, `table`, `csv`, or `tsv`, each finding is one row whose `status` is `parse_error`, `parse_warning`, or `compile_error`, and a clean run emits a single `ok` row. diff --git a/docs/content/deployment/docker.md b/docs/content/deployment/docker.md index 6cb126fe..2de7878d 100644 --- a/docs/content/deployment/docker.md +++ b/docs/content/deployment/docker.md @@ -88,7 +88,7 @@ docker run --rm \ Inside a container the API has to bind a non-loopback address to be reachable through the published port, and the daemon refuses a plaintext non-loopback bind unless you pass `--tls-cert`/`--tls-key` or `--allow-plaintext`. The examples on this page use `--allow-plaintext` and publish the port on the host's loopback or behind a TLS-terminating proxy; see [TLS termination](../reference/security.md#tls-termination-for-the-api-listener) to serve TLS from the daemon itself. -{{ added "unreleased" }} +{{ added "0.25.0" }} A non-loopback bind also needs authentication, so the examples pass `--api-token-env RSIGMA_API_TOKEN` and forward the secret from the host with `-e RSIGMA_API_TOKEN` (generate one with `openssl rand -hex 32`). Clients send it as `Authorization: Bearer `; `GET /healthz` and `GET /readyz` stay open for probes. Use the `daemon.api.auth` config block for per-client roles, `--tls-client-ca` for certificate-based authentication, or `--allow-unauthenticated` only when an authenticating proxy fronts the container. See [Daemon API authentication](../reference/security.md#daemon-api-authentication). diff --git a/docs/content/deployment/kubernetes.md b/docs/content/deployment/kubernetes.md index 64037fe9..a587650f 100644 --- a/docs/content/deployment/kubernetes.md +++ b/docs/content/deployment/kubernetes.md @@ -60,7 +60,7 @@ COPY rules/ /etc/rsigma/rules/ ## API token -The pod binds a routable address, so the daemon refuses to start without [bearer-token authentication](../reference/security.md#daemon-api-authentication), client certificates, or an explicit `allow_unauthenticated: true`. {{ added "unreleased" }} The token lives in a Secret and reaches the daemon as an environment variable: +The pod binds a routable address, so the daemon refuses to start without [bearer-token authentication](../reference/security.md#daemon-api-authentication), client certificates, or an explicit `allow_unauthenticated: true`. {{ added "0.25.0" }} The token lives in a Secret and reaches the daemon as an environment variable: ```bash kubectl -n rsigma create secret generic rsigma-api \ diff --git a/docs/content/deployment/systemd.md b/docs/content/deployment/systemd.md index 930c0ad8..8d674e32 100644 --- a/docs/content/deployment/systemd.md +++ b/docs/content/deployment/systemd.md @@ -171,7 +171,7 @@ daemon: key: /etc/rsigma/tls/server.key ``` -The daemon refuses to start on a non-loopback address without TLS or `--allow-plaintext`, and without authentication or `--allow-unauthenticated`. {{ added "unreleased" }} Renewed certificates are picked up on `systemctl reload rsigma`. See [TLS termination](../reference/security.md#tls-termination-for-the-api-listener). +The daemon refuses to start on a non-loopback address without TLS or `--allow-plaintext`, and without authentication or `--allow-unauthenticated`. {{ added "0.25.0" }} Renewed certificates are picked up on `systemctl reload rsigma`. See [TLS termination](../reference/security.md#tls-termination-for-the-api-listener). A routable bind needs [bearer-token authentication](../reference/security.md#daemon-api-authentication) (or client certificates with `tls.client_ca`). Put the secret in the environment file, which only root can read: diff --git a/docs/content/developers/fuzzing.md b/docs/content/developers/fuzzing.md index 89894e30..50e8d6d2 100644 --- a/docs/content/developers/fuzzing.md +++ b/docs/content/developers/fuzzing.md @@ -28,7 +28,7 @@ The workspace ships 28 [`cargo-fuzz`](https://rust-fuzz.github.io/book/cargo-fuz | `fuzz_scorecard_promtext` | Detection scorecard Prometheus text parser. Not in the weekly matrix; run manually. | (default) | | `fuzz_disposition_record` | Disposition record parsing. Not in the weekly matrix; run manually. | (default) | | `fuzz_capture_spool` | Disposition capture spool documents (`parse_manifest`, `parse_provenance`, `parse_corpus_events`). | 65536 | -| `fuzz_otlp_logs_decode` | OTLP log ingestion: protobuf decoding of `ExportLogsServiceRequest` (the body of `POST /v1/logs` and the gRPC `LogsService` message) and `logs_request_to_raw_events`, asserting every converted record is valid JSON. Seeds in `fuzz/seeds/fuzz_otlp_logs_decode/` cover every `AnyValue` variant, many attributes, and nesting on both sides of prost's recursion limit. {{ added "unreleased" }} | 65536 | +| `fuzz_otlp_logs_decode` | OTLP log ingestion: protobuf decoding of `ExportLogsServiceRequest` (the body of `POST /v1/logs` and the gRPC `LogsService` message) and `logs_request_to_raw_events`, asserting every converted record is valid JSON. Seeds in `fuzz/seeds/fuzz_otlp_logs_decode/` cover every `AnyValue` variant, many attributes, and nesting on both sides of prost's recursion limit. {{ added "0.25.0" }} | 65536 | | `fuzz_risk_config` | Risk configuration parsing. Not in the weekly matrix; run manually. | (default) | | `fuzz_lucene_frontend` | Lucene reverse-conversion frontend. Not in the weekly matrix; run manually. | (default) | | `fuzz_stix_pattern` | STIX pattern parsing. | 4096 | diff --git a/docs/content/ecosystem/helr.md b/docs/content/ecosystem/helr.md index 1101d3a3..e278b5ad 100644 --- a/docs/content/ecosystem/helr.md +++ b/docs/content/ecosystem/helr.md @@ -122,7 +122,7 @@ volumes: helr-state: ``` -Events flow Okta -> Helr -> NATS -> rsigma daemon -> stdout (or whatever next sink). Both services expose Prometheus metrics; both ship signed multi-arch images. The Docker hardening flags (`read_only`, `cap_drop: [ALL]`, non-root user `65534`) match the [Docker deployment guidance](../deployment/docker.md) for rsigma. `--allow-plaintext` is required because the compose example binds a non-loopback `--api-addr` without TLS, and the API token is required because a non-loopback bind refuses to serve an unauthenticated API. {{ added "unreleased" }} Events arrive over NATS, so only the metrics and control API use the token. +Events flow Okta -> Helr -> NATS -> rsigma daemon -> stdout (or whatever next sink). Both services expose Prometheus metrics; both ship signed multi-arch images. The Docker hardening flags (`read_only`, `cap_drop: [ALL]`, non-root user `65534`) match the [Docker deployment guidance](../deployment/docker.md) for rsigma. `--allow-plaintext` is required because the compose example binds a non-loopback `--api-addr` without TLS, and the API token is required because a non-loopback bind refuses to serve an unauthenticated API. {{ added "0.25.0" }} Events arrive over NATS, so only the metrics and control API use the token. ## Alternatives to NATS as the glue diff --git a/docs/content/getting-started/troubleshooting.md b/docs/content/getting-started/troubleshooting.md index aaecfe15..3605b618 100644 --- a/docs/content/getting-started/troubleshooting.md +++ b/docs/content/getting-started/troubleshooting.md @@ -30,7 +30,7 @@ The [prebuilt binaries](installation.md#prebuilt-binaries) and the Docker image refusing to bind plaintext on non-loopback address 0.0.0.0:9090; pass --tls-cert/--tls-key to enable TLS, or pass --allow-plaintext to opt out (e.g. when terminating TLS at a sidecar reverse proxy) ``` -The API listens on `0.0.0.0:9090` by default, and every build refuses plaintext on a non-loopback address. {{ added "unreleased" }} Pick one: +The API listens on `0.0.0.0:9090` by default, and every build refuses plaintext on a non-loopback address. {{ added "0.25.0" }} Pick one: - Local use: `--api-addr 127.0.0.1:9090`. Loopback always allows plaintext. - Serve TLS: `--tls-cert` and `--tls-key` (prebuilt binaries and the image; `cargo install` needs `--features daemon-tls`). See [TLS termination](../reference/security.md#tls-termination-for-the-api-listener). @@ -40,7 +40,7 @@ The API listens on `0.0.0.0:9090` by default, and every build refuses plaintext refusing to serve the API without authentication on non-loopback address 0.0.0.0:9090; configure bearer tokens (daemon.api.auth or --api-token-env), require client certificates with --tls-client-ca, or pass --allow-unauthenticated to opt out (e.g. when an authenticating proxy fronts the daemon) ``` -{{ added "unreleased" }} +{{ added "0.25.0" }} A non-loopback bind also needs authentication. Pick one: diff --git a/docs/content/guide/alert-pipeline.md b/docs/content/guide/alert-pipeline.md index bed13a8b..225183a7 100644 --- a/docs/content/guide/alert-pipeline.md +++ b/docs/content/guide/alert-pipeline.md @@ -41,9 +41,9 @@ Two behaviors follow Alertmanager: - A **self-inhibition guard**: a result matching both `source_match` and `target_match` does not inhibit itself. - Inhibition is **non-transitive**: a *silenced* source still inhibits its targets (the source index is updated before silencing), but an *inhibited* target does not become a source. -Tracked sources across all rules are capped by `max_inhibit_sources` (default 100000), so `equal` selectors over high-cardinality values (source IPs, usernames from untrusted logs) cannot grow the store without bound. At the cap, a result that would open a new source is still checked as a target but is not tracked, and already-tracked sources keep refreshing. {{ added "unreleased" }} +Tracked sources across all rules are capped by `max_inhibit_sources` (default 100000), so `equal` selectors over high-cardinality values (source IPs, usernames from untrusted logs) cannot grow the store without bound. At the cap, a result that would open a new source is still checked as a target but is not tracked, and already-tracked sources keep refreshing. {{ added "0.25.0" }} -Metrics: `rsigma_inhibited_total{rule}` (results muted, by rule name), `rsigma_inhibit_sources_active` (currently-active sources), and `rsigma_inhibit_sources_refused_total{rule}` (new sources not tracked because the cap was reached). {{ added "unreleased" }} +Metrics: `rsigma_inhibited_total{rule}` (results muted, by rule name), `rsigma_inhibit_sources_active` (currently-active sources), and `rsigma_inhibit_sources_refused_total{rule}` (new sources not tracked because the cap was reached). {{ added "0.25.0" }} ## Silencing @@ -115,7 +115,7 @@ dedup: | `dedup.resolve_timeout` | `1h` | Idle timeout after which an active alert resolves and is evicted (humantime). | | `dedup.max_active_alerts` | `100000` | Ceiling on concurrently-active alerts. Once reached, a first-fire for a new fingerprint passes through un-deduped instead of growing the store, so a high-cardinality fingerprint cannot exhaust memory. The `rsigma_dedup_store_entries` gauge plateauing at this value signals saturation. | | `max_silences` | `1000` | Ceiling on concurrently-tracked dynamic (API) silences. `POST /api/v1/silences` returns `429` past this. Static silences from the config do not count against it. | -| `max_inhibit_sources` | `100000` | Ceiling on tracked inhibition sources across all `inhibit_rules`. Once reached, a result that would open a new source is not tracked (counted on `rsigma_inhibit_sources_refused_total{rule}`); it can still be inhibited, and existing sources keep refreshing. {{ added "unreleased" }} | +| `max_inhibit_sources` | `100000` | Ceiling on tracked inhibition sources across all `inhibit_rules`. Once reached, a result that would open a new source is not tracked (counted on `rsigma_inhibit_sources_refused_total{rule}`); it can still be inhibited, and existing sources keep refreshing. {{ added "0.25.0" }} | ### Field selectors @@ -156,7 +156,7 @@ When the daemon runs with `--state-db `, the alert pipeline's state is per Restore is window-aware: dedup alerts past `resolve_timeout`, incidents past their `resolve_timeout`, silences past `ends_at`, and inhibition sources past their rule's `duration` are dropped during restore, so stale state never lingers. Deterministic `group_by` incident ids are preserved across the restart. -Restore and hot-reload also enforce the configured caps, so lowering one takes effect at once instead of when the excess ages out. {{ added "unreleased" }} Beyond `dedup.max_active_alerts`, `group.caps.max_open_incidents`, or `max_inhibit_sources`, the least recently seen alerts, incidents, or inhibition sources are evicted (alerts and incidents without a `resolved` record), and each remaining incident is cut to `max_results_per_incident` samples and `max_entities_per_incident` values per selector. The daemon logs a warning with the eviction counts. A snapshot whose version does not match the current build is ignored with a warning (the daemon starts fresh). `--clear-state` skips the restore (and `--keep-state` forces it), matching the correlation-state flags. +Restore and hot-reload also enforce the configured caps, so lowering one takes effect at once instead of when the excess ages out. {{ added "0.25.0" }} Beyond `dedup.max_active_alerts`, `group.caps.max_open_incidents`, or `max_inhibit_sources`, the least recently seen alerts, incidents, or inhibition sources are evicted (alerts and incidents without a `resolved` record), and each remaining incident is cut to `max_results_per_incident` samples and `max_entities_per_incident` values per selector. The daemon logs a warning with the eviction counts. A snapshot whose version does not match the current build is ignored with a warning (the daemon starts fresh). `--clear-state` skips the restore (and `--keep-state` forces it), matching the correlation-state flags. ## Grouping diff --git a/docs/content/guide/cloud-collection-recipes.md b/docs/content/guide/cloud-collection-recipes.md index 4cdd6fd4..c48b03ae 100644 --- a/docs/content/guide/cloud-collection-recipes.md +++ b/docs/content/guide/cloud-collection-recipes.md @@ -57,7 +57,7 @@ daemon: rsigma engine daemon --config /etc/rsigma/rsigma.yaml ``` -The shipper recipes in this guide post to `localhost`, so the daemon binds loopback. {{ added "unreleased" }} To accept shippers on other hosts, bind a routable address and configure TLS (or `tls.allow_plaintext`) and an `ingest` token in `daemon.api.auth`; see [Daemon network exposure](../reference/security.md#daemon-network-exposure). +The shipper recipes in this guide post to `localhost`, so the daemon binds loopback. {{ added "0.25.0" }} To accept shippers on other hosts, bind a routable address and configure TLS (or `tls.allow_plaintext`) and an `ingest` token in `daemon.api.auth`; see [Daemon network exposure](../reference/security.md#daemon-network-exposure). `schema-routing.yml`: diff --git a/docs/content/guide/enrichers.md b/docs/content/guide/enrichers.md index a814dcf1..ea10a39b 100644 --- a/docs/content/guide/enrichers.md +++ b/docs/content/guide/enrichers.md @@ -173,7 +173,7 @@ Per-result `reqwest` request with template-expanded URL, headers, and optional b tags: ["attack.execution", "attack.defense_evasion"] ``` -Event and correlation values are escaped for where they land. In `url`, each `${detection.*}` or `${correlation.*}` value is percent-encoded as one URL component (everything except letters, digits, `-`, `_`, `.`, and `~`), so a field value cannot add path segments, start a query, or reach another path on the host; a value that is exactly `.` or `..` has its dots encoded twice so it cannot act as a dot segment. In `body`, values are JSON-string-escaped, so a value inside a JSON string literal cannot add keys; a placeholder outside a string literal is inserted as raw text. `${ENV_VAR}` values in `url` are inserted verbatim, and header values are not escaped (CR and LF are rejected). {{ added "unreleased" }} +Event and correlation values are escaped for where they land. In `url`, each `${detection.*}` or `${correlation.*}` value is percent-encoded as one URL component (everything except letters, digits, `-`, `_`, `.`, and `~`), so a field value cannot add path segments, start a query, or reach another path on the host; a value that is exactly `.` or `..` has its dots encoded twice so it cannot act as a dot segment. In `body`, values are JSON-string-escaped, so a value inside a JSON string literal cannot add keys; a placeholder outside a string literal is inserted as raw text. `${ENV_VAR}` values in `url` are inserted verbatim, and header values are not escaped (CR and LF are rejected). {{ added "0.25.0" }} Each enricher instance owns its own response cache: two enrichers hitting the same URL with different `Authorization` headers do not share entries. Response bodies are capped at 10 MiB (same default as HTTP dynamic sources). Outbound URLs follow the daemon [`--egress-policy`](../reference/security.md#http-egress-policy-ssrf-defense) (SSRF / cloud-metadata defense at DNS resolution time). diff --git a/docs/content/guide/evaluating-rules.md b/docs/content/guide/evaluating-rules.md index bd2cefca..7226ce00 100644 --- a/docs/content/guide/evaluating-rules.md +++ b/docs/content/guide/evaluating-rules.md @@ -150,7 +150,7 @@ Each match prints one JSON `EvaluationResult` on stdout. Detection and correlati } ``` -`matched_selections` lists, once each, the selections whose match decides the condition, and `matched_fields` comes from those selections only. A selection that matched but did not make the rule fire is left out: `(s1 and s2) or s3` firing on `s3` reports only `s3` even when `s1` matched, and `a and not b` never reports `b`. {{ added "unreleased" }} +`matched_selections` lists, once each, the selections whose match decides the condition, and `matched_fields` comes from those selections only. A selection that matched but did not make the rule fire is left out: `(s1 and s2) or s3` firing on `s3` reports only `s3` even when `s1` matched, and `a and not b` never reports `b`. {{ added "0.25.0" }} ### Including the event body diff --git a/docs/content/guide/input-formats.md b/docs/content/guide/input-formats.md index 5464e1bb..0e92a53c 100644 --- a/docs/content/guide/input-formats.md +++ b/docs/content/guide/input-formats.md @@ -265,9 +265,9 @@ transformations: | Epoch seconds | `1747315800` | | Epoch milliseconds | `1747315800000` (auto-detected when value > 10^12) | -The time-part modifiers (`|minute`, `|hour`, `|day`, `|week`, `|month`, `|year`) read a field value with the same formats. {{ added "unreleased" }} +The time-part modifiers (`|minute`, `|hour`, `|day`, `|week`, `|month`, `|year`) read a field value with the same formats. {{ added "0.25.0" }} -When an event has no parseable timestamp, the correlation engine falls back to the wall clock by default. A value before 1970, a non-finite number, or a date past the representable range is not parseable; the engine tries the next timestamp field instead. {{ added "unreleased" }} Pass `--timestamp-fallback skip` to instead drop the event from correlation state updates (detections still fire). The `skip` mode is the right choice for forensic replay of historical data, where wall-clock substitution would corrupt the temporal windows. +When an event has no parseable timestamp, the correlation engine falls back to the wall clock by default. A value before 1970, a non-finite number, or a date past the representable range is not parseable; the engine tries the next timestamp field instead. {{ added "0.25.0" }} Pass `--timestamp-fallback skip` to instead drop the event from correlation state updates (detections still fire). The `skip` mode is the right choice for forensic replay of historical data, where wall-clock substitution would corrupt the temporal windows. ## When to use which format diff --git a/docs/content/guide/mcp-server.md b/docs/content/guide/mcp-server.md index 0b69659b..1983d9d6 100644 --- a/docs/content/guide/mcp-server.md +++ b/docs/content/guide/mcp-server.md @@ -18,7 +18,7 @@ The server speaks JSON-RPC over **stdio**: stdin and stdout are the transport, s rsigma mcp serve --rules-dir /path/to/rules ``` -`--rules-dir` sets a root so an agent can pass `path` arguments relative to a rules tree, and confines every path argument to it: absolute paths, `../`, and symlinks that leave the root are refused, and so are directory inputs that contain symlinks. Set it whenever the agent processes untrusted content; without it, tools can read any file the server's OS user can read. Without a root, stdio prints a startup warning, `fix_rules` cannot write, and `--http` refuses to start unless `--allow-unconfined-paths` is set. {{ added "unreleased" }} See [Security Hardening](../reference/security.md#mcp-server-tool-surface). `--lint-config` points the `lint_rules` tool at a `.rsigma-lint.yml` (disabled rules, severity overrides, extra tag namespaces). +`--rules-dir` sets a root so an agent can pass `path` arguments relative to a rules tree, and confines every path argument to it: absolute paths, `../`, and symlinks that leave the root are refused, and so are directory inputs that contain symlinks. Set it whenever the agent processes untrusted content; without it, tools can read any file the server's OS user can read. Without a root, stdio prints a startup warning, `fix_rules` cannot write, and `--http` refuses to start unless `--allow-unconfined-paths` is set. {{ added "0.25.0" }} See [Security Hardening](../reference/security.md#mcp-server-tool-surface). `--lint-config` points the `lint_rules` tool at a `.rsigma-lint.yml` (disabled rules, severity overrides, extra tag namespaces). ## Client setup @@ -51,10 +51,10 @@ Fifteen Engineer-cycle tools always register. When `--daemon-url` (or `mcp.daemo | Tool | Input | Output | |------|-------|--------| -| `parse_rule` | `yaml` or `path` | AST as JSON, plus rule/correlation/filter counts, `parse_errors`, and `parse_warnings`. `ok` is false only when a document fails to parse; an ignored invalid value such as `status: draft` is reported in `parse_warnings` instead. {{ added "unreleased" }} | +| `parse_rule` | `yaml` or `path` | AST as JSON, plus rule/correlation/filter counts, `parse_errors`, and `parse_warnings`. `ok` is false only when a document fails to parse; an ignored invalid value such as `status: draft` is reported in `parse_warnings` instead. {{ added "0.25.0" }} | | `parse_condition` | `condition` | The parsed condition expression tree. | | `lint_rules` | `yaml` or file/dir `path` | Findings per file: lint rule id, severity, message, 1-indexed line, `fixable`, and the fix title. | -| `validate_rules` | `yaml` or file/dir `path`, `pipelines`, `resolve_sources`, `fail_level` | Parse + compile + correlation-reference results, with per-rule compile errors and separate `parse_errors` and `parse_warnings`. `fail_level` is `error` (default) or `warning`; at `warning`, a parse warning also sets `ok` to false. {{ added "unreleased" }} | +| `validate_rules` | `yaml` or file/dir `path`, `pipelines`, `resolve_sources`, `fail_level` | Parse + compile + correlation-reference results, with per-rule compile errors and separate `parse_errors` and `parse_warnings`. `fail_level` is `error` (default) or `warning`; at `warning`, a parse warning also sets `ok` to false. {{ added "0.25.0" }} | | `evaluate_events` | rules (`yaml`/`path`), events (`events` array or `events_path` NDJSON), `pipelines`, `match_detail`, `emit_detections`, `enrichers`/`enrichers_path` | Matches with `event_index`, a summary of detection/correlation counts. With `enrichers` the matches are run through an enrichment pipeline first. Rules referenced only by correlations without `generate: true`, detections and correlations alike, produce no matches unless `emit_detections` is `true`. {{ added "0.24.0" }} | | `convert_rules` | rules, `target`, `format`, `pipelines`, `options`, `skip_unsupported` | Backend queries per rule, plus errors and warnings. Native targets (`postgres`/`lynxdb`/`fibratus`) convert in-process; with `--allow-sigma-cli`, any other target is delegated to an installed [sigma-cli](../reference/backends/sigma-cli.md) and the result carries `engine: "sigma-cli"`, a per-line `queries` split, the verbatim `raw` output (authoritative for multi-line formats), and sigma-cli's diagnostics as `warnings`. | | `reverse_convert` | `query`, optional `dialect` (`lucene`), plus metadata the query cannot carry (`title`, `id`, `level`, `status`, `logsource_product`/`category`/`service`). At least one logsource key is required. | Draft Sigma YAML (or `{ "ok": false, ... }` for constructs with no Sigma equivalent or a missing logsource). Same surface as [`rsigma rule reverse`](../cli/rule/reverse.md). {{ added "0.24.0" }} | @@ -218,7 +218,7 @@ rsigma mcp serve --http 127.0.0.1:9100 --rules-dir /path/to/rules - **Auth.** `--auth-token ` (or `RSIGMA_MCP_AUTH_TOKEN`) requires a static bearer token on every request; requests without `Authorization: Bearer ` get `401`. The token is compared in constant time and is flag/env-only (never read from config files). - **TLS.** `--tls-cert`/`--tls-key` terminate TLS in-process using the same rustls loader as the daemon (requires a build with the `daemon-tls` feature). Alternatively terminate TLS at a sidecar proxy and bind plaintext with `--allow-plaintext`. - **Plaintext safety.** Binding plaintext on a non-loopback address is refused unless `--allow-plaintext` is set. -- **Path confinement.** `--http` refuses to start without `--rules-dir` unless `--allow-unconfined-paths` is set. {{ added "unreleased" }} +- **Path confinement.** `--http` refuses to start without `--rules-dir` unless `--allow-unconfined-paths` is set. {{ added "0.25.0" }} The `--http`, `--lint-config`, `--rules-dir`, `--allow-unconfined-paths`, `--daemon-url`, `--daemon-ca`, and `--allow-operate-writes` settings also resolve from the layered config (`mcp` section) and the `RSIGMA_MCP__*` environment layer (for example `RSIGMA_MCP__HTTP_ADDR=127.0.0.1:9100`); the MCP HTTP auth token and the daemon token stay flag/env-only. diff --git a/docs/content/guide/nats-streaming.md b/docs/content/guide/nats-streaming.md index 5a3b7689..11d96a95 100644 --- a/docs/content/guide/nats-streaming.md +++ b/docs/content/guide/nats-streaming.md @@ -78,7 +78,7 @@ What this guarantees: - If the daemon panics mid-processing, NATS redelivers after `ack_wait` (configured on the JetStream consumer). - While a sink is retrying (file write error, NATS publish-ack timeout), the source message stays unacked so a crash still redelivers. -- After retries are exhausted (or a permanent sink error), the result is handed to the [DLQ](#dead-letter-queue) writer and the source message is acked, so the same failure does not loop forever. With `--dlq` set, the writer then persists it; a failed DLQ write is logged and counted on `rsigma_dlq_write_failures_total`, and the result is lost. Without `--dlq`, the result is dropped with a rate-limited warning. {{ added "unreleased" }} +- After retries are exhausted (or a permanent sink error), the result is handed to the [DLQ](#dead-letter-queue) writer and the source message is acked, so the same failure does not loop forever. With `--dlq` set, the writer then persists it; a failed DLQ write is logged and counted on `rsigma_dlq_write_failures_total`, and the result is lost. Without `--dlq`, the result is dropped with a rate-limited warning. {{ added "0.25.0" }} - If a parse error happens before the engine, the failed event is routed to the DLQ (when configured) and the source message is acked, preventing infinite redelivery of unparseable data. What this does NOT guarantee: @@ -110,7 +110,7 @@ rsigma engine daemon -r rules/ --input nats://localhost:4222/events.> \ ### State restore during replay -The daemon stores the last-acked NATS stream sequence and timestamp alongside the SQLite correlation snapshot. It reads the position before taking the snapshots, so the saved state always covers every event up to the stored position; events acked just after it may also be included and are redelivered as duplicates, which the at-least-once model already allows. {{ added "unreleased" }} On restart, `decide_state_restore` compares the replay start point against the stored position: +The daemon stores the last-acked NATS stream sequence and timestamp alongside the SQLite correlation snapshot. It reads the position before taking the snapshots, so the saved state always covers every event up to the stored position; events acked just after it may also be included and are redelivered as duplicates, which the at-least-once model already allows. {{ added "0.25.0" }} On restart, `decide_state_restore` compares the replay start point against the stored position: | Situation | Decision | |-----------|----------| @@ -177,7 +177,7 @@ Each DLQ entry is a JSON object: } ``` -The `rsigma_dlq_events_total` Prometheus counter tracks DLQ volume, including delivery failures dropped for lack of a `--dlq` sink. Alert on rate-of-change to catch upstream encoding regressions early, and on `rsigma_dlq_write_failures_total` to catch a DLQ sink that cannot write. {{ added "unreleased" }} See [Prometheus metrics reference](../reference/metrics.md). +The `rsigma_dlq_events_total` Prometheus counter tracks DLQ volume, including delivery failures dropped for lack of a `--dlq` sink. Alert on rate-of-change to catch upstream encoding regressions early, and on `rsigma_dlq_write_failures_total` to catch a DLQ sink that cannot write. {{ added "0.25.0" }} See [Prometheus metrics reference](../reference/metrics.md). ## Connection lifecycle and reconnects diff --git a/docs/content/guide/otlp-integration.md b/docs/content/guide/otlp-integration.md index cde3a578..f25657e0 100644 --- a/docs/content/guide/otlp-integration.md +++ b/docs/content/guide/otlp-integration.md @@ -31,7 +31,7 @@ With the feature compiled in, the OTLP endpoints are always active, regardless o rsigma engine daemon -r rules/ --input http --api-addr 127.0.0.1:9090 ``` -Agents on other hosts need a routable `--api-addr`, which requires TLS (or `--allow-plaintext`) and authentication: a bearer token with `events:ingest`, client certificates, or `--allow-unauthenticated`. {{ added "unreleased" }} See [Daemon network exposure](../reference/security.md#daemon-network-exposure). +Agents on other hosts need a routable `--api-addr`, which requires TLS (or `--allow-plaintext`) and authentication: a bearer token with `events:ingest`, client certificates, or `--allow-unauthenticated`. {{ added "0.25.0" }} See [Daemon network exposure](../reference/security.md#daemon-network-exposure). ## Endpoints @@ -45,7 +45,7 @@ OTLP is exposed on the same `--api-addr` port as the rest of the daemon's HTTP A Both transports support `Content-Encoding: gzip` (HTTP) or gRPC gzip compression for compressed payloads. -Both transports cap a decompressed request at 4 MiB. Over HTTP, a gzip body that inflates past the cap is rejected with `413 Payload Too Large` and counted as `rsigma_otlp_errors_total{reason="body_too_large"}`; uncompressed HTTP bodies are limited to 2 MiB. Keep agent batches under these limits. {{ added "unreleased" }} +Both transports cap a decompressed request at 4 MiB. Over HTTP, a gzip body that inflates past the cap is rejected with `413 Payload Too Large` and counted as `rsigma_otlp_errors_total{reason="body_too_large"}`; uncompressed HTTP bodies are limited to 2 MiB. Keep agent batches under these limits. {{ added "0.25.0" }} When no `Content-Type` is provided on a POST to `/v1/logs`, protobuf is assumed, matching the OTLP/HTTP specification default. diff --git a/docs/content/guide/performance-tuning.md b/docs/content/guide/performance-tuning.md index 572e5410..971e1496 100644 --- a/docs/content/guide/performance-tuning.md +++ b/docs/content/guide/performance-tuning.md @@ -184,7 +184,7 @@ The improvements below are separate same-build comparisons on the pinned SigmaHQ Correlation state lives in memory unless `--state-db` writes periodic snapshots to SQLite. The hard cap is `max_state_entries`, default 100,000 `(correlation, group-key)` entries across all correlation rules, settable with `--max-state-entries` (or `daemon.correlation.max_state_entries` in the config file). When the cap is hit, the engine evicts the stalest 10% and emits a warning. -The cap bounds the number of groups, not the bytes within one. A single group's window state grows with `timespan` x event rate: 8 bytes per in-window event for `event_count`, 16 for the numeric aggregations (`value_sum`/`value_avg`/`value_percentile`/`value_median`), and roughly 32 bytes plus the value string for `value_count`. `--max-group-entries` (or the per-rule `rsigma.max_group_entries` custom attribute) caps that within-window growth; when a group exceeds it, the oldest entries are dropped, which can only under-count. Session windows always keep their oldest entry as the span anchor so truncation cannot silently extend the `timespan` cap. Unset means unbounded, the historical behavior. A correlation whose `timespan` is longer than one hour and has no cap from either source logs a warning at load naming the rule, since a busy group's window can then reach gigabytes (24h at 10k events/s is about 864M timestamps). {{ added "unreleased" }} The measured shape (see [Benchmarks](../benchmarks.md#window-mode-memory-stress)): 1M unique session keys against the default cap peaked at 39.8 MiB, and a fully chatty `event_count` workload (100 groups sustaining 1 event/s through a 2h session cap) held 6.3 MiB. A live but quiet session group costs ~256 bytes, dominated by the group-key strings. +The cap bounds the number of groups, not the bytes within one. A single group's window state grows with `timespan` x event rate: 8 bytes per in-window event for `event_count`, 16 for the numeric aggregations (`value_sum`/`value_avg`/`value_percentile`/`value_median`), and roughly 32 bytes plus the value string for `value_count`. `--max-group-entries` (or the per-rule `rsigma.max_group_entries` custom attribute) caps that within-window growth; when a group exceeds it, the oldest entries are dropped, which can only under-count. Session windows always keep their oldest entry as the span anchor so truncation cannot silently extend the `timespan` cap. Unset means unbounded, the historical behavior. A correlation whose `timespan` is longer than one hour and has no cap from either source logs a warning at load naming the rule, since a busy group's window can then reach gigabytes (24h at 10k events/s is about 864M timestamps). {{ added "0.25.0" }} The measured shape (see [Benchmarks](../benchmarks.md#window-mode-memory-stress)): 1M unique session keys against the default cap peaked at 39.8 MiB, and a fully chatty `event_count` workload (100 groups sustaining 1 event/s through a 2h session cap) held 6.3 MiB. A live but quiet session group costs ~256 bytes, dominated by the group-key strings. Window modes (`sliding`/`tumbling`/`session`) have identical per-event cost; they differ only in how long entries are retained. `tumbling` resets per-group state at each bucket boundary and is the cheapest under sustained load; `session` retains everything between the first event and the `timespan` cap, so it is the mode to watch on chatty groups. diff --git a/docs/content/guide/streaming-detection.md b/docs/content/guide/streaming-detection.md index 63d5649e..9145d723 100644 --- a/docs/content/guide/streaming-detection.md +++ b/docs/content/guide/streaming-detection.md @@ -131,8 +131,8 @@ The `--output` flag is repeatable, which gives you fan-out for free. Each match | `stdout` | NDJSON to stdout. Default. Line sinks also accept `?format=ocsf` for [OCSF Detection Finding](ocsf-findings.md) JSON. | | `file:///path/to/file.ndjson` | Append NDJSON to a file, rotating only if you wrap it externally (logrotate, etc.). | | `nats://host:port/subject` | Publish via JetStream with server-confirmed persistence. Requires `daemon-nats`. | -| `otlp(s)://host:port` | Export detections over OTLP/gRPC (`s` = TLS). Each attempt has a `10s` deadline, tunable with `?timeout=`. {{ added "unreleased" }} Requires `daemon-otlp`. | -| `otlphttp(s)://host:port` | Export detections over OTLP/HTTP to `/v1/logs` (`s` = TLS). Each attempt has a `10s` deadline, tunable with `?timeout=`. {{ added "unreleased" }} Requires `daemon-otlp`. | +| `otlp(s)://host:port` | Export detections over OTLP/gRPC (`s` = TLS). Each attempt has a `10s` deadline, tunable with `?timeout=`. {{ added "0.25.0" }} Requires `daemon-otlp`. | +| `otlphttp(s)://host:port` | Export detections over OTLP/HTTP to `/v1/logs` (`s` = TLS). Each attempt has a `10s` deadline, tunable with `?timeout=`. {{ added "0.25.0" }} Requires `daemon-otlp`. | | `unix:///path/to.sock` | Write NDJSON to a collector listening on a Unix domain socket (Unix only). Reconnects once on a transient write failure. | Failed deliveries are routed to the dead-letter queue when `--dlq` is configured: @@ -178,7 +178,7 @@ Without `--state-db`, correlation state lives only in memory and is lost on rest rsigma engine daemon -r rules/ --state-db /var/lib/rsigma/state.db ``` -The daemon loads any existing snapshot on startup, saves periodically (every 30 s by default, tunable with `--state-save-interval`), and saves on graceful shutdown. The database is a single SQLite file in WAL journal mode that holds one JSON snapshot row per kind of state (correlation, alert pipeline, risk, and dispositions). Each save captures all of them back to back and writes them in one transaction, so a crash mid-save leaves the previous save intact rather than a mix of old and new state. {{ added "unreleased" }} +The daemon loads any existing snapshot on startup, saves periodically (every 30 s by default, tunable with `--state-save-interval`), and saves on graceful shutdown. The database is a single SQLite file in WAL journal mode that holds one JSON snapshot row per kind of state (correlation, alert pipeline, risk, and dispositions). Each save captures all of them back to back and writes them in one transaction, so a crash mid-save leaves the previous save intact rather than a mix of old and new state. {{ added "0.25.0" }} This means an `event_count` correlation that has seen 4 of 5 required events resumes at 4 after a restart, not 0. @@ -200,7 +200,7 @@ See [NATS Streaming](nats-streaming.md) for the full replay matrix. ## HTTP API -The daemon binds an HTTP API listener on `--api-addr` (default `0.0.0.0:9090`). It serves both REST and Prometheus endpoints, plus OTLP/gRPC and OTLP/HTTP when the feature is enabled. With the optional `daemon-tls` build feature and `--tls-cert`/`--tls-key`, the same listener terminates HTTPS for every protocol on one socket (ALPN negotiates `h2` and `http/1.1`). Every build refuses to start on a non-loopback TCP `--api-addr` without TLS or `--allow-plaintext`, and without authentication (bearer tokens or `--tls-client-ca`) or `--allow-unauthenticated`. {{ added "unreleased" }} Loopback and `unix://` addresses need neither, for local development. See the [TLS reference](../reference/security.md#tls-termination-for-the-api-listener) for the flag table and hot-reload semantics. +The daemon binds an HTTP API listener on `--api-addr` (default `0.0.0.0:9090`). It serves both REST and Prometheus endpoints, plus OTLP/gRPC and OTLP/HTTP when the feature is enabled. With the optional `daemon-tls` build feature and `--tls-cert`/`--tls-key`, the same listener terminates HTTPS for every protocol on one socket (ALPN negotiates `h2` and `http/1.1`). Every build refuses to start on a non-loopback TCP `--api-addr` without TLS or `--allow-plaintext`, and without authentication (bearer tokens or `--tls-client-ca`) or `--allow-unauthenticated`. {{ added "0.25.0" }} Loopback and `unix://` addresses need neither, for local development. See the [TLS reference](../reference/security.md#tls-termination-for-the-api-listener) for the flag table and hot-reload semantics. Auth is off by default. Enable a single admin bearer token with `--api-token-env`, or configure per-token roles in `daemon.api.auth`. Pair non-loopback listeners with TLS or a TLS-terminating proxy so tokens are never sent in cleartext. See [HTTP API: Authentication](../reference/http-api.md#authentication). diff --git a/docs/content/guide/webhooks.md b/docs/content/guide/webhooks.md index 571cda6f..5d991958 100644 --- a/docs/content/guide/webhooks.md +++ b/docs/content/guide/webhooks.md @@ -100,7 +100,7 @@ The `retry.*` and `queue_size` settings override the daemon's global `--sink-*` The `body` is JSON-string-escaped: interpolated values (rule titles, event field strings) that land inside a JSON string literal have their quotes, backslashes, and control characters escaped, so a rule title containing a quote cannot break the payload. Header values are not escaped (CR and LF are rejected). -In `url`, each `${detection.*}` or `${correlation.*}` value is percent-encoded as one URL component (everything except letters, digits, `-`, `_`, `.`, and `~`), so an event value cannot add path segments, start a query, or reach another path on the host. A value that is exactly `.` or `..` has its dots encoded twice so it cannot act as a dot segment. `${ENV_VAR}` values are inserted verbatim, so `${SLACK_WEBHOOK_PATH}` can still hold a path with `/`. {{ added "unreleased" }} +In `url`, each `${detection.*}` or `${correlation.*}` value is percent-encoded as one URL component (everything except letters, digits, `-`, `_`, `.`, and `~`), so an event value cannot add path segments, start a query, or reach another path on the host. A value that is exactly `.` or `..` has its dots encoded twice so it cannot act as a dot segment. `${ENV_VAR}` values are inserted verbatim, so `${SLACK_WEBHOOK_PATH}` can still hold a path with `/`. {{ added "0.25.0" }} Every templated field is validated at startup against the webhook's `kind`: a `${correlation.*}` reference inside a `kind: detection` webhook rejects the daemon with a pointer at the offending field. diff --git a/docs/content/library/convert.md b/docs/content/library/convert.md index d6eb78d7..3b5be3e7 100644 --- a/docs/content/library/convert.md +++ b/docs/content/library/convert.md @@ -120,7 +120,7 @@ impl Backend for MyBackend { The default `convert_rule` lowers the rule to `IrRule`, walks the resulting `IrDetection` and `IrCondition`, and dispatches into `text_*` helpers (e.g. `text_convert_field_str_ir`, `text_convert_ir_pattern`) that consult the config. Only override the leaves that your backend needs to behave differently from pySigma's `TextQueryBackend` default. -The numeric leaves (`convert_field_eq_num`, `convert_field_compare_op`, `convert_keyword_num`) receive a `rsigma_ir::NumericLiteral` rather than an `f64`, so integers above 2^53 reach the backend exactly. Render it with `to_string()` to get the exact digits. {{ added "unreleased" }} +The numeric leaves (`convert_field_eq_num`, `convert_field_compare_op`, `convert_keyword_num`) receive a `rsigma_ir::NumericLiteral` rather than an `f64`, so integers above 2^53 reach the backend exactly. Render it with `to_string()` to get the exact digits. {{ added "0.25.0" }} See [Adding Backends](../developers/adding-backends.md) for the step-by-step walkthrough, the testing pattern, and how to wire a new backend into `rsigma backend convert` if you also want CLI integration. diff --git a/docs/content/library/eval.md b/docs/content/library/eval.md index 22034e79..be0c1e3f 100644 --- a/docs/content/library/eval.md +++ b/docs/content/library/eval.md @@ -38,12 +38,12 @@ serde_json = "1" # only if you use the JsonEvent shim | `Pipeline` | Parsed processing pipeline. Applied to rules at `add_collection` time, in priority order. | | `ConditionSet` | One pipeline condition scope: identifier-keyed conditions, `and`/`or` linking, optional negation, and an optional expression. `TransformationItem` has one set each for rule, detection-item, and field-name conditions. {{ added "0.24.0" }} | | `pipeline::parse_pipeline(&str) -> Result` | Parse a pipeline YAML string. | -| `pipeline::sources::scan_source_templates(&str)` | Find every `${source.}` and `${source..}` template in a string, with its id, path, and byte offset. It is the single reference grammar shared by pipeline parsing and `rsigma-runtime`'s expansion. `is_valid_source_id` checks the id character set (ASCII letters, digits, `_`, `-`) that source declarations must use. {{ added "unreleased" }} | +| `pipeline::sources::scan_source_templates(&str)` | Find every `${source.}` and `${source..}` template in a string, with its id, path, and byte offset. It is the single reference grammar shared by pipeline parsing and `rsigma-runtime`'s expansion. `is_valid_source_id` checks the id character set (ASCII letters, digits, `_`, `-`) that source declarations must use. {{ added "0.25.0" }} | | `TransformedRule` + `transform_rule` / `transform_collection` | Apply pipelines and hand back the rewritten rule, the transformation ids that fired, and the merged `PipelineState`, without compiling or loading. `Engine::transform_rule` / `Engine::transform_collection` do the same over an engine's configured pipelines. | | `apply_filters(&SigmaCollection) -> Vec` | Merge each filter into the detection rules it targets, as pySigma does when a collection loads, and return the rules for pipelines to transform. Filters target rules the same way `Engine::apply_filter` does. Conversion uses it. {{ added "0.24.0" }} | | `LogSourceExtractor` | Derives an event's `LogSource` from configurable fields plus optional static defaults, for conflict-based logsource pruning. Pass to `Engine::set_logsource_extractor`. | -| `ExtractLogSource` | Trait behind logsource pruning, with one method, `extract(&self, &E) -> LogSource`. `LogSourceExtractor` implements it for every event type. Implement it for your own event type to prune by logsource carried outside the matchable fields, such as metadata beside the payload. See [Custom logsource extractors](#custom-logsource-extractors). {{ added "unreleased" }} | -| `Event` trait + `JsonEvent`, `KvEvent`, `MapEvent`, `PlainEvent` | The event shapes the engine consumes. `JsonEvent::into_value` unwraps the JSON value, cloning only when the event borrows it. {{ added "unreleased" }} | +| `ExtractLogSource` | Trait behind logsource pruning, with one method, `extract(&self, &E) -> LogSource`. `LogSourceExtractor` implements it for every event type. Implement it for your own event type to prune by logsource carried outside the matchable fields, such as metadata beside the payload. See [Custom logsource extractors](#custom-logsource-extractors). {{ added "0.25.0" }} | +| `Event` trait + `JsonEvent`, `KvEvent`, `MapEvent`, `PlainEvent` | The event shapes the engine consumes. `JsonEvent::into_value` unwraps the JSON value, cloning only when the event borrows it. {{ added "0.25.0" }} | | `EvaluationResult` | One detection match or correlation firing. Composes a `RuleHeader` (rule metadata, custom attributes, optional enrichments) and a `ResultBody::Detection(DetectionBody)` / `ResultBody::Correlation(CorrelationBody)` payload. Serializes to one flat JSON object per result. | | `RuleHeader`, `DetectionBody`, `CorrelationBody` | The three composable structs behind `EvaluationResult`. `RuleHeader` carries the fields shared between kinds (`rule_title`, `rule_id`, `level`, `tags`, `custom_attributes`, and an optional `enrichments` map); the body variants carry the kind-specific fields. | | `ResultBody` | `#[serde(untagged)]` enum that picks the kind-specific payload. Use `EvaluationResult::as_detection() / as_correlation()` accessors or pattern match on `result.body` to read its fields. | @@ -198,10 +198,10 @@ Pipeline transformations can write `rsigma.*` attributes that the engine consume | `Engine::set_bloom_max_bytes(usize)` | Per-engine bloom budget. Default 1 MiB. | | `Engine::set_cross_rule_ac(bool)` | Toggle the cross-rule Aho-Corasick pre-filter. Requires the `daachorse-index` feature. Pays off only on very large pure-substring rule sets. | | `Engine::set_logsource_extractor(Option)` | Opt into conflict-based logsource pruning: skip rules whose `product`/`service`/`category` (and custom dimensions) conflict with the event's. Off by default, fail-open. Pays off on large mixed-product rule sets. | -| `Engine::with_logsource_extractor(extractor)` | Create an empty engine that prunes with a custom `ExtractLogSource` implementation. {{ added "unreleased" }} | +| `Engine::with_logsource_extractor(extractor)` | Create an empty engine that prunes with a custom `ExtractLogSource` implementation. {{ added "0.25.0" }} | | `Engine::evaluate_pruned(&event, &LogSource)` | Evaluate with a caller-resolved event logsource for conflict-based pruning, bypassing the engine's own extractor. Used by `SchemaRouter` to feed a per-event logsource resolved from explicit fields plus the recognized schema's implied logsource. | | `Engine::evaluate_batch(&[events])` (with `parallel`) | Batch evaluation. With the `parallel` feature, rayon parallelizes across events internally. | -| `Engine::set_retain_hir(bool)` | Keep each loaded rule's HIR so `save_hir` can write it. Off by default, since nothing on the evaluation path reads it; enable it before loading rules. {{ added "unreleased" }} | +| `Engine::set_retain_hir(bool)` | Keep each loaded rule's HIR so `save_hir` can write it. Off by default, since nothing on the evaluation path reads it; enable it before loading rules. {{ added "0.25.0" }} | | `Engine::save_hir()` / `load_hir(&[u8])` | Serialize the engine's lowered rules to a versioned HIR cache blob and rebuild from one, a restart cache that skips parse, pipeline, and lowering. Captures rules added via the parsed-rule paths in post-pipeline, pre-filter form; re-apply filters after `load_hir`. `save_hir` returns `EvalError::HirCache` unless `set_retain_hir(true)` was on for every such rule. Backed by [`rsigma-ir`](ir.md)'s `cache`. | Schema classification and routing live alongside the engine: `SchemaClassifier::classify` and `classify_with_ambiguity` recognize an event's schema from declarative `SchemaSignature` predicates, `explain` reports why, and `validate_schema_config` statically checks a config. `SchemaRouter` builds one engine per pipeline-set, routes each event to its schema's engine (deriving the event's logsource from the schema for pruning), feeds one shared correlation store, and reports a per-schema pruning summary via `schema_pruning_summary`. See the [Schema Signatures reference](../reference/schema-signatures.md) and the [Schema Routing](../guide/schema-routing.md) guide. @@ -212,7 +212,7 @@ Decision matrix in [Performance Tuning](../guide/performance-tuning.md). Verifie ### Custom logsource extractors -{{ added "unreleased" }} +{{ added "0.25.0" }} `LogSourceExtractor` reads the event's logsource from its fields. When your event type carries the logsource somewhere else, such as metadata from a transport envelope, implement `ExtractLogSource` for that type. `Engine`, `CorrelationEngine`, and `SchemaRouter` are generic over the extractor type, which defaults to `LogSourceExtractor`, so existing code keeps compiling. The extractor runs once per event with static dispatch. @@ -244,7 +244,7 @@ Prefer an extractor over computing the logsource yourself and calling `evaluate_ `EvalError` from `thiserror`. Variants include `Parser` (re-exports the parser errors), `InvalidRegex`, `InvalidCidr`, `IncompatibleValue` (a value the modifier cannot use, such as a `cidr` with host bits set), `InvalidModifiers`, `UnknownRuleRef` (correlation references a rule that wasn't added), `CorrelationCycle`, and `Base64`. Each carries enough context to point operators at the offending rule. -Processing pipelines that fail to parse or validate, or that fail while being applied to a rule (including a deliberate `rule_failure` or `detection_item_failure`), return `EvalError::Pipeline`. The enum is `#[non_exhaustive]`, so a `match` on it needs a wildcard arm. {{ added "unreleased" }} +Processing pipelines that fail to parse or validate, or that fail while being applied to a rule (including a deliberate `rule_failure` or `detection_item_failure`), return `EvalError::Pipeline`. The enum is `#[non_exhaustive]`, so a `match` on it needs a wildcard arm. {{ added "0.25.0" }} ## See also diff --git a/docs/content/library/ir.md b/docs/content/library/ir.md index 7116ade5..dfb6e5bf 100644 --- a/docs/content/library/ir.md +++ b/docs/content/library/ir.md @@ -35,7 +35,7 @@ The crate is sync-only (no tokio/reqwest). | `IrMatcher::Encoded` + `IrEncoding` | Explicit encoding transforms (`base64`, `wide`, `windash`, …) over a wildcard-aware `pattern`. {{ added "0.24.0" }} | | `encoding::expand_encoded` / `encoding::expand_encoded_detections` / `encoding::expand_encoded_matcher` | Replay an `Encoded` matcher into the plain string matches its encodings produce (windash variants, UTF-16, base64 and base64offset), for one value, a whole detection map, or one matcher. Eval compiles them and conversion renders them, so both agree on the variants. `encoding::MAX_WINDASH_DASHES` caps windash expansion. {{ added "0.24.0" }} | | `IrMatcher::Expand` | A `\|expand` value with placeholders left for match time, with the string operator in `op`. {{ added "0.24.0" }} | -| `IrNumber` / `NumericLiteral` | A numeric matcher value. `NumericLiteral` keeps a rule integer as an exact `i64` and a fractional value as an `f64`, and its `Display` renders integers digit for digit. {{ added "unreleased" }} | +| `IrNumber` / `NumericLiteral` | A numeric matcher value. `NumericLiteral` keeps a rule integer as an exact `i64` and a fractional value as an `f64`, and its `Display` renders integers digit for digit. {{ added "0.25.0" }} | | `IrCorrelation` / `IrFilter` | Correlation and filter HIR. | | `IrRuleMetadata` | Metadata superset used when projecting eval `RuleHeader`. | | `lower_rule` / `lower_detection` / `lower_condition` | AST → HIR. | diff --git a/docs/content/library/parser.md b/docs/content/library/parser.md index 280542d5..59573581 100644 --- a/docs/content/library/parser.md +++ b/docs/content/library/parser.md @@ -31,7 +31,7 @@ The crate has no RSigma dependencies and pulls in `yaml_serde` 0.10 (the maintai | `parse_sigma_yaml(&str) -> Result` | The main entry point. Accepts single-document or multi-document YAML (`---` separator). | | `parse_sigma_file(&Path)`, `parse_sigma_directory(&Path)` | File and directory variants. | | `SigmaCollection` | A bag of `SigmaRule`, `CorrelationRule`, and `FilterRule` parsed from one input, plus the `diagnostics` found while parsing it. | -| `CollectionDiagnostic`, `DiagnosticKind`, `DiagnosticSeverity` | One parse error or parse warning, with its kind, severity, message, document index, file path, and source location. See [Error handling](#error-handling). {{ added "unreleased" }} | +| `CollectionDiagnostic`, `DiagnosticKind`, `DiagnosticSeverity` | One parse error or parse warning, with its kind, severity, message, document index, file path, and source location. See [Error handling](#error-handling). {{ added "0.25.0" }} | | `SigmaRule` | A detection rule (`title`, `id`, `sigma_version`, `logsource`, `detection`, `condition`, …). | | `CorrelationRule` | A correlation rule (`correlation:` block plus shared metadata). | | `FilterRule` | A filter rule (`filter:` block plus shared metadata). | @@ -151,7 +151,7 @@ assert!(check_regex("(?`). Variants cover YAML syntax errors, missing required fields, invalid modifiers and modifier combinations (`InvalidModifiers`), invalid values and detections, condition-expression errors, and the size/depth caps above. `Condition` errors carry a `SourceLocation` so callers can highlight the offending location in source, and `kind()` returns the variant as a payload-free `SigmaParserErrorKind`. -The parse functions return `Err` only when they cannot start, for example when a file cannot be read. Problems inside the input are recorded on the returned collection as `CollectionDiagnostic` values in `SigmaCollection::diagnostics`, so one bad document does not hide the others: {{ added "unreleased" }} +The parse functions return `Err` only when they cannot start, for example when a file cannot be read. Problems inside the input are recorded on the returned collection as `CollectionDiagnostic` values in `SigmaCollection::diagnostics`, so one bad document does not hide the others: {{ added "0.25.0" }} - `severity` is `Error` when a document, a file, or the rest of the YAML stream was dropped, and `Warning` when a document was kept with an invalid `status`, `level`, `related` entry, or `sigma-version` ignored. - `kind` says what failed: `YamlSyntax`, `NotAMapping`, `CollectionAction`, `Document(SigmaParserErrorKind)`, `Metadata`, or `File(SigmaParserErrorKind)`. diff --git a/docs/content/library/runtime.md b/docs/content/library/runtime.md index ea79dbfc..c21cb989 100644 --- a/docs/content/library/runtime.md +++ b/docs/content/library/runtime.md @@ -42,7 +42,7 @@ tokio = { version = "1", features = ["rt-multi-thread", "sync", "macros", "io-ut | `LogProcessor` | An ordered batch processor that parses formatted inputs in parallel before locking its live `RuntimeEngine`. `process_batch_with_format_detailed` returns `BatchProcessOutcome`, including input indices that failed parsing for single-parse DLQ routing. The daemon glues this to its bounded mpsc plumbing. | | `EventSource` trait, `Sink` trait | The plug-in surfaces for inputs and outputs. Built-in: `StdinSource`, `StdoutSink`, `FileSink`, and `NatsSource`/`NatsSink` under the `nats` feature. | | `spawn_source(source) -> mpsc::Receiver` | Convenience helper that runs an `EventSource` on its own task. | -| `EventFilter` | Type alias for `dyn Fn(Value) -> Vec`: optional jq/JSONPath-style pre-extraction applied to each input line. The filter takes the parsed line by value, so it can move payloads out instead of cloning them. {{ added "unreleased" }} | +| `EventFilter` | Type alias for `dyn Fn(Value) -> Vec`: optional jq/JSONPath-style pre-extraction applied to each input line. The filter takes the parsed line by value, so it can move payloads out instead of cloning them. {{ added "0.25.0" }} | | `MetricsHook` trait + `NoopMetrics` | The plug-in surface that the daemon uses to wire `prometheus` counters. Reimplement to ship metrics elsewhere (Datadog, OpenTelemetry, custom registry). | | `input::parse_line(...)` | Format-aware line parser. Auto-detects JSON, syslog, plain text; honors format hints. | | `EvtxFileReader` (feature `evtx`) | Streaming `.evtx` reader. | @@ -52,10 +52,10 @@ tokio = { version = "1", features = ["rt-multi-thread", "sync", "macros", "io-ut | `Enricher` trait + `EnrichmentPipeline` | Post-evaluation enrichment surface. Drives the five primitives (`TemplateEnricher`, `LookupEnricher`, `HttpEnricher`, `CommandEnricher`, `StixEnricher` with `stix-enrich`) and any bespoke types registered via `register_builtin`. | | `EnricherKind`, `OnError`, `Scope`, `EnrichError`, `EnrichErrorKind` | Configuration types: declared kind, error policy, scope filter, and the typed error returned by `Enricher::enrich`. | | `HttpResponseCache` (re-exported from `enrichment::http_cache`) | `(method, url, body_hash)`-keyed in-memory response cache with TTL and lazy eviction. Each `HttpEnricher` instance owns its own. | -| `enrichment::{render_template, render_template_json, render_template_url}` | Render a `${detection.*}`, `${correlation.*}`, or `${ENV_VAR}` template against an `EvaluationResult`. `render_template` substitutes verbatim, `render_template_json` JSON-string-escapes every value, and `render_template_url` percent-encodes result values as single URL components while leaving `${ENV_VAR}` values verbatim. Use the variant that matches where the rendered text lands. `render_template_url` {{ added "unreleased" }} | +| `enrichment::{render_template, render_template_json, render_template_url}` | Render a `${detection.*}`, `${correlation.*}`, or `${ENV_VAR}` template against an `EvaluationResult`. `render_template` substitutes verbatim, `render_template_json` JSON-string-escapes every value, and `render_template_url` percent-encodes result values as single URL components while leaving `${ENV_VAR}` values verbatim. Use the variant that matches where the rendered text lands. `render_template_url` {{ added "0.25.0" }} | | `register_builtin(name, factory) -> Result<(), String>` | Process-global, append-only registry hook. External crates use it to ship a bespoke Rust-coded enricher type addressable via `type: ` in the daemon's enrichers config. Reserved names (`template` / `lookup` / `http` / `command`) and duplicate registrations are rejected. | | `enrichment::config::{load_enrichers_file, build_enrichers, build_enrichers_full, EnricherResources, EnrichersFile}` | YAML loader for an enrichers config, shared by the daemon and the MCP server. Validates template namespaces, scopes, and bespoke `type:` values. | -| `alert_pipeline::{AlertPipeline, DedupStore, Selector, AlertPipelineFile, load_alert_pipeline_file, parse_alert_pipeline_config, build_alert_pipeline}` | Post-engine alert-processing layer. Deduplicates results by a configurable fingerprint with an `active -> resolved` lifecycle. `AlertPipeline` is the validated, swappable config; `DedupStore` is the sink-task-owned active-alert state. `AlertPipeline::restore` trims restored state to the pipeline's caps, and `AlertPipeline::enforce_caps` applies the same trim when a reloaded pipeline replaces another, returning a `CapTrim` with the eviction counts. {{ added "unreleased" }} | +| `alert_pipeline::{AlertPipeline, DedupStore, Selector, AlertPipelineFile, load_alert_pipeline_file, parse_alert_pipeline_config, build_alert_pipeline}` | Post-engine alert-processing layer. Deduplicates results by a configurable fingerprint with an `active -> resolved` lifecycle. `AlertPipeline` is the validated, swappable config; `DedupStore` is the sink-task-owned active-alert state. `AlertPipeline::restore` trims restored state to the pipeline's caps, and `AlertPipeline::enforce_caps` applies the same trim when a reloaded pipeline replaces another, returning a `CapTrim` with the eviction counts. {{ added "0.25.0" }} | | `dispositions::{Disposition, DispositionStore, DispositionConfig, DispositionSnapshot, Verdict, Numerator, parse_dispositions, triage_feed}` | Triage feedback loop. `parse_dispositions` parses a POST body or source payload (object, array, or NDJSON); `Disposition::from_raw` validates one record; `DispositionStore` keeps rolling per-rule verdict counts and computes the false-positive ratio; `triage_feed` renders the `rule scorecard --triage` JSON shape. | | `capture::{CaptureRing, CaptureConfig, CaptureRingSink, AdmittedEvent, CapturedEvent, event_digest}` | Byte-bounded in-memory ring of admitted `group_by` detection events. Used by the daemon capture hook after inhibition/silencing and before dedup. | | `capture_spool::{BundleManifest, parse_manifest, parse_provenance, parse_corpus_events, bundle_id, disposition_identity}` | Versioned disposition-bundle documents. Readers reject unknown major versions and malformed NDJSON with bounded input sizes. | @@ -64,7 +64,7 @@ tokio = { version = "1", features = ["rt-multi-thread", "sync", "macros", "io-ut | `ocsf::{detection_finding, incident_finding, risk_incident_finding, FindingSource, SystemSource, OCSF_VERSION}` | Output-only OCSF Detection Finding (class 2004) serializer for the three emitted shapes. Each entry point has a `*_with` variant taking a [`FindingSource`] so the minted `metadata.uid` and the serialization clock can be injected in tests. Selected on a sink with `?format=ocsf`; see [OCSF Findings](../guide/ocsf-findings.md) for the full mapping. | | `SinkFormat` | The non-exhaustive wire format a line-oriented sink serializes into (`Ndjson`, the default, or `Ocsf`). Set with `StdoutSink::with_format` and its siblings; `Sink::format` reports it. | | `IncidentEnvelope`, `FormattedIncidentEnvelope` | `IncidentEnvelope` retains the native NDJSON line and NATS subject as its source-compatible public shape. `FormattedIncidentEnvelope` wraps it with alternate per-format lines for mixed-format delivery. | -| `OtlpSink`, `OtlpProtocol`, `OtlpClientTls`, `DEFAULT_OTLP_TIMEOUT` (feature `otlp`) | OTLP log-record exporter over gRPC or HTTP protobuf, with optional gzip and client TLS. Every export attempt is bounded by `DEFAULT_OTLP_TIMEOUT` (10 seconds); `OtlpSink::with_timeout` overrides it, and an expired attempt surfaces as a retryable `RuntimeError::Io` with `ErrorKind::TimedOut`. {{ added "unreleased" }} | +| `OtlpSink`, `OtlpProtocol`, `OtlpClientTls`, `DEFAULT_OTLP_TIMEOUT` (feature `otlp`) | OTLP log-record exporter over gRPC or HTTP protobuf, with optional gzip and client TLS. Every export attempt is bounded by `DEFAULT_OTLP_TIMEOUT` (10 seconds); `OtlpSink::with_timeout` overrides it, and an expired attempt surfaces as a retryable `RuntimeError::Io` with `ErrorKind::TimedOut`. {{ added "0.25.0" }} | | `io::webhook::{WebhooksFile, WebhookConfig, WebhookSink, build_webhooks, load_webhooks_file}` | YAML loader and template-driven HTTP output sink. Validates `kind`, template namespaces, and retry/rate-limit bounds; the daemon drives it as a lossy (`on_full=drop`) leaf on the async delivery layer. | The full pipeline architecture, source resolution flow, and dynamic-pipeline contract are in [the crate README](https://github.com/timescale/rsigma/blob/main/crates/rsigma-runtime/README.md) and the [Architecture reference](../reference/architecture.md). @@ -106,7 +106,7 @@ async fn main() -> Result<(), Box> { For a streaming daemon shape, wire an `EventSource` (`StdinSource`, `NatsSource`, …) through `spawn_source` into your own loop, batch the channel into `Vec`, and feed it to `process_batch_lines`. The daemon code in `rsigma-cli/src/daemon/server.rs` is the reference implementation. -Sinks work on both multi-threaded and current-thread Tokio runtimes; on a current-thread runtime the stdout and file sinks write inline. `load_rules` is synchronous, so it can only resolve dynamic pipeline sources on a multi-threaded runtime and returns an error on a current-thread one, leaving the previous engine in place. {{ added "unreleased" }} +Sinks work on both multi-threaded and current-thread Tokio runtimes; on a current-thread runtime the stdout and file sinks write inline. `load_rules` is synchronous, so it can only resolve dynamic pipeline sources on a multi-threaded runtime and returns an error on a current-thread one, leaving the previous engine in place. {{ added "0.25.0" }} ## Hot-reload diff --git a/docs/content/reference/conditions.md b/docs/content/reference/conditions.md index 3b109658..5d7d90f6 100644 --- a/docs/content/reference/conditions.md +++ b/docs/content/reference/conditions.md @@ -18,7 +18,7 @@ detection: Keyword values are strings in the specification, so a number matches its decimal text: `4624` matches the numeric value `4624` and the string `logon 4624 succeeded`. The evaluator searches numeric event values as their JSON text, so `4624` also matches `46240`. -A YAML boolean keyword is the word: `true` matches `"true"` and `flag is TRUE now`, and converts as the term `true`. {{ added "unreleased" }} Keyword search covers string and numeric event values, not JSON booleans, so neither `true` nor `'true'` matches the boolean value `{"a": true}`; match a boolean field with `a: true` instead. +A YAML boolean keyword is the word: `true` matches `"true"` and `flag is TRUE now`, and converts as the term `true`. {{ added "0.25.0" }} Keyword search covers string and numeric event values, not JSON booleans, so neither `true` nor `'true'` matches the boolean value `{"a": true}`; match a boolean field with `a: true` instead. ## Field-less values with modifiers @@ -56,4 +56,4 @@ A selector that matches no detection name, such as `all of zzz*`, or `1 of *` wh `any of` is accepted as a synonym of `1 of`, as in pySigma. A count above one, such as `2 of selection_*`, is an extension beyond the specification: the evaluator matches when at least that many of the selected detections match. Backends cannot express a count, so conversion rejects the rule with `'2 of' quantifier not supported in conversion`. -`0 of` is a parse error, because it would match every event. Write `not 1 of selection_*` to require that none of the selected detections match. {{ added "unreleased" }} +`0 of` is a parse error, because it would match every event. Write `not 1 of selection_*` to require that none of the selected detections match. {{ added "0.25.0" }} diff --git a/docs/content/reference/configuration.md b/docs/content/reference/configuration.md index 00a79d43..4dee5251 100644 --- a/docs/content/reference/configuration.md +++ b/docs/content/reference/configuration.md @@ -194,7 +194,7 @@ Run [`rsigma config init`](../cli/config/init.md) to scaffold a full, commented | `daemon.schema` | `engine daemon` | Schema classification and routing (`observe`, `discover`, `routing`, `partition_rules`, `config`, `on_unknown`). All opt-in. See [Schema Routing](../guide/schema-routing.md). | | `daemon.logsource_routing` | `engine daemon` | Conflict-based logsource pruning (`enabled`, `field_map`, `event_logsource`). `strict` is accepted and ignored. Opt-in. See [Logsource-Aware Evaluation](../guide/logsource-routing.md). | | `eval` | `engine eval` | Mirrors the eval flag surface. | -| `eval.output` | `engine eval` | Write results to this file instead of stdout (`-o, --output`). The CLI flag takes precedence. {{ added "unreleased" }} | +| `eval.output` | `engine eval` | Write results to this file instead of stdout (`-o, --output`). The CLI flag takes precedence. {{ added "0.25.0" }} | | `eval.schema` | `engine eval` | Schema routing for one-shot eval (`routing`, `config`, `on_unknown`). `observe`/`discover` have no effect here. | | `eval.logsource_routing` | `engine eval` | Conflict-based logsource pruning for one-shot eval (`enabled`, `field_map`, `event_logsource`). `strict` is accepted and ignored. | | `backtest` | `rule backtest` | `rules`, `corpus`, `expectations`, `unexpected`, `pipelines`, and the syslog input knobs. `unexpected` has no compiled default so the expectations-file default can apply. | @@ -203,7 +203,7 @@ Run [`rsigma config init`](../cli/config/init.md) to scaffold a full, commented | `visibility` | `rule visibility` | `mapping` (logsource/field to ATT&CK data-source table path or URL; unset uses the bundled default) and `fail_on_blind_spots`. `rules` and `observed` are intentionally absent (they are invocation-specific CLI arguments). | | `doc` | `rule doc` | `fail_on_missing` (the CI gate). The ADS bar itself (enforced statuses and required sections) lives in `.rsigma-lint.yml` under an `ads:` block, not here. See [Detection Strategy](../guide/detection-strategy.md). | | `hygiene` | `rule hygiene` | `rules`, the optional sources (`metrics`, `metrics_window`, `fields`), the thresholds (`silent_threshold`, `stale_threshold`, `noisy_threshold`), and `fail_on`. See [Rule Hygiene](../guide/rule-hygiene.md). | -| `mcp` | `mcp serve` | `mcp.http_addr` (the `--http` bind address; unset means stdio), `mcp.lint_config`, `mcp.rules_dir`, `mcp.allow_unconfined_paths` {{ added "unreleased" }} (run without `rules_dir`; required to serve `http_addr` without a root; off by default), `mcp.allow_sigma_cli` (let `convert_rules` delegate non-native targets to an installed sigma-cli; off by default), `mcp.daemon_url`, `mcp.daemon_ca`, and `mcp.allow_operate_writes`. The MCP HTTP auth token and the daemon token are secrets and stay flag/env-only. Inert unless built with the `mcp` feature. | +| `mcp` | `mcp serve` | `mcp.http_addr` (the `--http` bind address; unset means stdio), `mcp.lint_config`, `mcp.rules_dir`, `mcp.allow_unconfined_paths` {{ added "0.25.0" }} (run without `rules_dir`; required to serve `http_addr` without a root; off by default), `mcp.allow_sigma_cli` (let `convert_rules` delegate non-native targets to an installed sigma-cli; off by default), `mcp.daemon_url`, `mcp.daemon_ca`, and `mcp.allow_operate_writes`. The MCP HTTP auth token and the daemon token are secrets and stay flag/env-only. Inert unless built with the `mcp` feature. | `daemon.correlation.emit_detections` defaults to `false`, so detections and correlations referenced only by correlations do not produce standalone output unless a referencing correlation has top-level `generate: true`. The inverse `daemon.correlation.no_detections` key remains accepted for compatibility but is deprecated, and the daemon prints a warning when it is set. A layer that sets only `no_detections` overrides `emit_detections` from lower layers, while `emit_detections` wins when one layer sets both. {{ added "0.24.0" }} diff --git a/docs/content/reference/dynamic-sources.md b/docs/content/reference/dynamic-sources.md index bb0c9882..87d60ab9 100644 --- a/docs/content/reference/dynamic-sources.md +++ b/docs/content/reference/dynamic-sources.md @@ -71,7 +71,7 @@ sources: The full Rust type lives at [`rsigma_eval::pipeline::sources::DynamicSource`](https://github.com/timescale/rsigma/blob/main/crates/rsigma-eval/src/pipeline/sources.rs). The parser is at [`rsigma_eval::pipeline::parsing`](https://github.com/timescale/rsigma/blob/main/crates/rsigma-eval/src/pipeline/parsing.rs). -A source ID is one or more ASCII letters, digits, `_`, or `-` (for example `ip_blocklist`, `threat-intel`, or `1st_feed`), which is exactly what a `${source.}` reference can name. A declaration with any other character, such as a dot or a space, fails to load with an error naming the ID. {{ added "unreleased" }} +A source ID is one or more ASCII letters, digits, `_`, or `-` (for example `ip_blocklist`, `threat-intel`, or `1st_feed`), which is exactly what a `${source.}` reference can name. A declaration with any other character, such as a dot or a space, fails to load with an error naming the ID. {{ added "0.25.0" }} ### Collision semantics diff --git a/docs/content/reference/http-api.md b/docs/content/reference/http-api.md index e4cb2fb7..29d38d6f 100644 --- a/docs/content/reference/http-api.md +++ b/docs/content/reference/http-api.md @@ -51,7 +51,7 @@ Endpoint details are split across three pages: ## Authentication -Bearer-token authentication is off by default, so loopback and `unix://` deployments are untouched. {{ added "unreleased" }} A non-loopback TCP `--api-addr` refuses to start without it unless the listener requires client certificates (`--tls-client-ca`) or `--allow-unauthenticated` is passed (see [Daemon network exposure](security.md#daemon-network-exposure)). Enable it either with the `--api-token-env ` flag (a single token with full `admin` permissions, read from the named environment variable) or with the `daemon.api.auth` config block for per-token roles: +Bearer-token authentication is off by default, so loopback and `unix://` deployments are untouched. {{ added "0.25.0" }} A non-loopback TCP `--api-addr` refuses to start without it unless the listener requires client certificates (`--tls-client-ca`) or `--allow-unauthenticated` is passed (see [Daemon network exposure](security.md#daemon-network-exposure)). Enable it either with the `--api-token-env ` flag (a single token with full `admin` permissions, read from the named environment variable) or with the `daemon.api.auth` config block for per-token roles: ```yaml daemon: @@ -240,7 +240,7 @@ The endpoint returns `200 OK` for any source ID regardless of whether that ID is OTLP log ingestion over HTTP. Accepts `application/x-protobuf` or `application/json`, optionally gzip-encoded. Returns `application/x-protobuf` (or `application/json` matching the request) with the standard OTLP `ExportLogsServiceResponse`. Requires the daemon to be built with `daemon-otlp`. -The request body is limited to 2 MiB as sent. A gzip-encoded body is also limited to 4 MiB after decompression, the same limit the gRPC transport applies, and a larger body is rejected with `413 Payload Too Large`. {{ added "unreleased" }} +The request body is limited to 2 MiB as sent. A gzip-encoded body is also limited to 4 MiB after decompression, the same limit the gRPC transport applies, and a larger body is rejected with `413 Payload Too Large`. {{ added "0.25.0" }} ### gRPC `LogsService/Export` diff --git a/docs/content/reference/lint-rules.md b/docs/content/reference/lint-rules.md index 9f03a858..fa452d1b 100644 --- a/docs/content/reference/lint-rules.md +++ b/docs/content/reference/lint-rules.md @@ -104,7 +104,7 @@ Apply to detection rules (`detection:` block + `condition:`). | `null_in_value_list` | `warning` | none | A `null` literal appears inside a list of values. Sigma's semantics around `null` in lists are spec-ambiguous; this flags the case so the author can be explicit. | | `empty_value_list` | `warning` | none | A detection item with a list value is empty. | | `condition_references_unknown` | `error` | none | The `condition:` expression references a selection name that is not in `detection:`, or a selector such as `1 of filter_*` matches none. Each unresolved reference is a separate finding. | -| `invalid_condition` | `error` | none | The `condition:` expression does not parse, for example `selection andd filter`, an unclosed parenthesis, or `0 of selection_*`. The finding carries the parser's error and location. Conditions that use the deprecated aggregation syntax report `deprecated_aggregation_syntax` instead. {{ added "unreleased" }} | +| `invalid_condition` | `error` | none | The `condition:` expression does not parse, for example `selection andd filter`, an unclosed parenthesis, or `0 of selection_*`. The finding carries the parser's error and location. Conditions that use the deprecated aggregation syntax report `deprecated_aggregation_syntax` instead. {{ added "0.25.0" }} | | `deprecated_aggregation_syntax` | `warning` | none | The condition uses the deprecated aggregation pipe syntax (`condition: selection \| count() > 5`). Use the modern `correlation:` block instead. | | `deprecated_detection_timeframe` | `warning` | none | The `detection:` block has a v1.x `timeframe:` key, which has no effect. Use a correlation rule with a `timespan` instead. {{ added "0.24.0" }} | | `flattened_array_correlation` | `warning` | none | Two or more sibling keys share a quantified array prefix (e.g. `connections[any].protocol` and `connections[any].ip`). Each opens an independent scope, so they do **not** correlate on the same array element. Use an object-scope block (`connections[any]:` with the fields nested) to require one element to satisfy all of them. See [Array Matching](../guide/array-matching.md). | @@ -158,7 +158,7 @@ These also apply to detection rules but sit apart from the core detection-block |------|----------|-----|-------------| | `single_value_all_modifier` | `warning` | yes | A single-value detection item carries the `\|all` modifier, which is a no-op. The fix removes the redundant modifier. | | `all_with_re` | `warning` | yes | The `\|all` modifier is combined with `\|re`, which is not meaningful (regex matching is inherently set-style). The fix removes `\|all`. | -| `incompatible_modifiers` | `error` | none | The modifier chain on a field is one the parser rejects, with the parser's reason: more than one operator (e.g. `\|contains\|startswith` or `\|re\|contains`), two UTF-16 encodings, `base64` with `base64offset`, a value transformation with a non-string operator, a regex flag without or before `re`, or a string modifier before `fieldref`. `fieldref` may be followed by one of `contains`, `startswith`, or `endswith`, and by `neq`. Chains with an unknown or repeated modifier are left to the parser's error. Reported as an `error` rather than a `warning`. {{ added "unreleased" }} | +| `incompatible_modifiers` | `error` | none | The modifier chain on a field is one the parser rejects, with the parser's reason: more than one operator (e.g. `\|contains\|startswith` or `\|re\|contains`), two UTF-16 encodings, `base64` with `base64offset`, a value transformation with a non-string operator, a regex flag without or before `re`, or a string modifier before `fieldref`. `fieldref` may be followed by one of `contains`, `startswith`, or `endswith`, and by `neq`. Chains with an unknown or repeated modifier are left to the parser's error. Reported as an `error` rather than a `warning`. {{ added "0.25.0" }} | | `invalid_related_id` | `warning` | none | `related[].id` is not a valid UUID. | | `invalid_related_type` | `error` | none | `related[].type` is not one of `derived`, `obsolete`, `merged`, `renamed`, `similar`. | | `related_missing_required` | `error` | none | `related[]` entry is missing the required `id:` or `type:` field. | diff --git a/docs/content/reference/metrics.md b/docs/content/reference/metrics.md index 16b4fcad..9105cc83 100644 --- a/docs/content/reference/metrics.md +++ b/docs/content/reference/metrics.md @@ -31,7 +31,7 @@ These always show up. They cover ingest, matches, queue depth, back-pressure, re | `rsigma_pipeline_latency_seconds` | histogram | none | End-to-end latency from event dequeue to sink send. | | `rsigma_batch_size` | histogram | none | Number of events processed per batch. | | `rsigma_dlq_events_total` | counter | none | Events routed to the dead-letter queue, counted whether or not a `--dlq` sink is configured. | -| `rsigma_dlq_write_failures_total` | counter | none | Dead-letter queue entries the `--dlq` sink failed to write. Each one is a lost event that was already acked. {{ added "unreleased" }} | +| `rsigma_dlq_write_failures_total` | counter | none | Dead-letter queue entries the `--dlq` sink failed to write. Each one is a lost event that was already acked. {{ added "0.25.0" }} | | `rsigma_sink_queue_depth` | gauge | `sink` | Results buffered in each sink's delivery queue. | | `rsigma_sink_retries_total` | counter | `sink` | Sink delivery retries after a retryable failure. | | `rsigma_sink_dropped_total` | counter | `sink` | Results dropped because a lossy sink's queue was full (`?on_full=drop`). | @@ -99,7 +99,7 @@ Exposed when the daemon is built with `daemon`. The fixed label sets on `rsigma_ | `rsigma_silences_active` | gauge | none | Currently-active silences. | | `rsigma_inhibited_total` | counter | `rule` | Results muted by an inhibition rule, by rule name. | | `rsigma_inhibit_sources_active` | gauge | none | Currently-active inhibition sources. | -| `rsigma_inhibit_sources_refused_total` | counter | `rule` | Results that would have opened a new inhibition source but were not tracked because `max_inhibit_sources` was reached, by rule name. Appears once a source is first refused. {{ added "unreleased" }} | +| `rsigma_inhibit_sources_refused_total` | counter | `rule` | Results that would have opened a new inhibition source but were not tracked because `max_inhibit_sources` was reached, by rule name. Appears once a source is first refused. {{ added "0.25.0" }} | | `rsigma_alert_pipeline_duration_seconds` | histogram | none | Alert-pipeline stage duration in seconds. | ## Risk-based alerting (9 metrics) diff --git a/docs/content/reference/modifiers.md b/docs/content/reference/modifiers.md index 7f350478..4e3e7f58 100644 --- a/docs/content/reference/modifiers.md +++ b/docs/content/reference/modifiers.md @@ -57,11 +57,11 @@ A `cidr` value is an IPv4 or IPv6 network written as `address/prefix`, such as ` ## Numbers -An integer value, with or without `gt`, `gte`, `lt`, or `lte`, matches and compares exactly, including above 2^53, where a 64-bit float can no longer represent every integer. `pid: 9007199254740993` matches the event value `9007199254740993` and the string `"9007199254740993"`, but not `9007199254740992`. Against a float event value, the comparison is exact too: `9007199254740992.0` equals `9007199254740992` and is less than `9007199254740993`. A value with a fractional part compares as a 64-bit float. Backends render the integer digits exactly, and an integer keyword searches for exactly those digits. {{ added "unreleased" }} +An integer value, with or without `gt`, `gte`, `lt`, or `lte`, matches and compares exactly, including above 2^53, where a 64-bit float can no longer represent every integer. `pid: 9007199254740993` matches the event value `9007199254740993` and the string `"9007199254740993"`, but not `9007199254740992`. Against a float event value, the comparison is exact too: `9007199254740992.0` equals `9007199254740992` and is less than `9007199254740993`. A value with a fractional part compares as a 64-bit float. Backends render the integer digits exactly, and an integer keyword searches for exactly those digits. {{ added "0.25.0" }} ## Time parts -`minute`, `hour`, `day`, `week` (ISO week), `month`, and `year` read the field as a timestamp in UTC and match the extracted component, so `ts|hour: 12` matches `2024-07-10 12:30:00.500`. The field may be an RFC 3339 string, `YYYY-MM-DD` followed by `T` or a space and `HH:MM:SS` with an optional fractional second, or an epoch number in seconds or milliseconds (values above 10^12 are milliseconds). These are the formats the correlation engine accepts for event timestamps. A value in any other format never matches. {{ added "unreleased" }} +`minute`, `hour`, `day`, `week` (ISO week), `month`, and `year` read the field as a timestamp in UTC and match the extracted component, so `ts|hour: 12` matches `2024-07-10 12:30:00.500`. The field may be an RFC 3339 string, `YYYY-MM-DD` followed by `T` or a space and `HH:MM:SS` with an optional fractional second, or an epoch number in seconds or milliseconds (values above 10^12 are milliseconds). These are the formats the correlation engine accepts for event timestamps. A value in any other format never matches. {{ added "0.25.0" }} ## `exists` diff --git a/docs/content/reference/security.md b/docs/content/reference/security.md index b2d16b42..6b642a58 100644 --- a/docs/content/reference/security.md +++ b/docs/content/reference/security.md @@ -9,7 +9,7 @@ For the SECURITY policy and disclosure process, see [`SECURITY.md`](../security- | Limit | Constant | Value | Scope | Behaviour on overrun | |-------|----------|-------|-------|----------------------| | Single event size | `MAX_LINE_BYTES` | 1 MiB | `engine daemon` HTTP/stdin event ingest | Line rejected with a `413`-equivalent error in HTTP mode; counted in `rsigma_events_parse_errors_total`. | -| Decompressed OTLP/HTTP body | `OTLP_HTTP_MAX_DECOMPRESSED_BYTES` | 4 MiB | `POST /v1/logs` with `Content-Encoding: gzip` (`daemon-otlp`) | Request rejected with `413` before decoding; counted as `rsigma_otlp_errors_total{reason="body_too_large"}`. The gRPC transport applies the same 4 MiB decoded-message limit. {{ added "unreleased" }} | +| Decompressed OTLP/HTTP body | `OTLP_HTTP_MAX_DECOMPRESSED_BYTES` | 4 MiB | `POST /v1/logs` with `Content-Encoding: gzip` (`daemon-otlp`) | Request rejected with `413` before decoding; counted as `rsigma_otlp_errors_total{reason="body_too_large"}`. The gRPC transport applies the same 4 MiB decoded-message limit. {{ added "0.25.0" }} | | Condition expression length | `MAX_CONDITION_LEN` | 64 KiB | Rule parser | Rule rejected at parse time with an `InvalidCondition` error. | | Condition expression depth | `MAX_CONDITION_DEPTH` | 64 | Rule parser | Same. | | JSON event traversal depth | `MAX_NESTING_DEPTH` | 64 | Keyword search inside nested JSON | Traversal stops; deeper fields are not matched against keyword detections. | @@ -72,7 +72,7 @@ Every external parser rsigma ships uses panic-free libraries: | Pipeline YAML | `yaml_serde` 0.10 | Same. | | Input event JSON | `serde_json` | Bounded by `MAX_LINE_BYTES` (1 MiB) for streaming sources. | | HTTP request bodies | `reqwest` with explicit size limit | Bounded by `MAX_SOURCE_RESPONSE_BYTES` (10 MiB). | -| OTLP requests | `prost` + `tonic` | The OTLP receiver enforces upstream size limits at the HTTP/gRPC layer. Protobuf decoding stops at prost's 100-message recursion limit, which bounds `AnyValue` nesting at about 49 levels before conversion runs; a unit test fails if a dependency disables that limit, and the `fuzz_otlp_logs_decode` harness covers the decode and conversion path. {{ added "unreleased" }} | +| OTLP requests | `prost` + `tonic` | The OTLP receiver enforces upstream size limits at the HTTP/gRPC layer. Protobuf decoding stops at prost's 100-message recursion limit, which bounds `AnyValue` nesting at about 49 levels before conversion runs; a unit test fails if a dependency disables that limit, and the `fuzz_otlp_logs_decode` harness covers the decode and conversion path. {{ added "0.25.0" }} | | EVTX records | `evtx` crate | Streaming parse; bounded record-by-record memory usage regardless of file size. | | CEF | `cef-parser` | Bounded line size via the input format machinery. | @@ -108,7 +108,7 @@ The `engine daemon` HTTP and gRPC listeners share one socket. With the optional - Bind to a private network segment that the SOC controls. - On Unix, bind a domain socket with `--api-addr unix:///run/rsigma/api.sock`. Access is gated by the socket file's permissions (created `0600`), so it never leaves the host and needs no TLS. This is the simplest hardened shape for a co-located admin/metrics client or a sidecar that proxies the socket. -{{ added "unreleased" }} +{{ added "0.25.0" }} To prevent accidental exposure, every build refuses to start on a non-loopback TCP `--api-addr` in two cases: @@ -155,7 +155,7 @@ OTLP receiver authentication has two options: mTLS (`--tls-client-ca`) so every ## Daemon API authentication -Bearer-token authentication with `resource:action` permissions gates the daemon API at the application layer. Without a `daemon.api.auth` config block or the `--api-token-env` flag, every route is open. {{ added "unreleased" }} The daemon accepts that only on loopback, on a `unix://` socket, on a listener that requires client certificates, or with an explicit `--allow-unauthenticated` (see [Daemon network exposure](#daemon-network-exposure)). When enabled, every route except the liveness probes (`GET /healthz`, `GET /readyz`) requires a token whose permission set covers the route's permission; the per-route mapping is in the [HTTP API endpoint summary](http-api.md#endpoint-summary). +Bearer-token authentication with `resource:action` permissions gates the daemon API at the application layer. Without a `daemon.api.auth` config block or the `--api-token-env` flag, every route is open. {{ added "0.25.0" }} The daemon accepts that only on loopback, on a `unix://` socket, on a listener that requires client certificates, or with an explicit `--allow-unauthenticated` (see [Daemon network exposure](#daemon-network-exposure)). When enabled, every route except the liveness probes (`GET /healthz`, `GET /readyz`) requires a token whose permission set covers the route's permission; the per-route mapping is in the [HTTP API endpoint summary](http-api.md#endpoint-summary). Design properties: @@ -206,7 +206,7 @@ Verdict-driven capture ([Verdict-Driven Corpora](../guide/verdict-to-corpus.md)) The MCP server ([`rsigma mcp serve`](../cli/mcp/serve.md)) treats every tool argument as untrusted, because an agent can be steered by prompt-injected content it is asked to process (a third-party rule set, an issue being summarized). - **`--rules-dir` is a confinement boundary.** When it is set, every path argument (`path`, `events_path`, `enrichers_path`, and file-based `pipelines` in every tool) is canonicalized and refused if it leaves the directory, whether it is absolute, uses `../`, or resolves through a symlink. A missing file outside the root is refused the same way as an existing one, so callers cannot probe for files. Directory inputs that contain a symlink anywhere in the tree are refused. `fix_rules` with `write: true` only writes to files inside the root. -- **Without `--rules-dir`, paths are unconfined.** The server can read any file its OS user can read. Always set `--rules-dir` when the server is reachable by an agent that processes untrusted content. Without a root, `fix_rules` refuses `write: true`, stdio prints a startup warning, and `--http` refuses to start unless `--allow-unconfined-paths` is set. {{ added "unreleased" }} +- **Without `--rules-dir`, paths are unconfined.** The server can read any file its OS user can read. Always set `--rules-dir` when the server is reachable by an agent that processes untrusted content. Without a root, `fix_rules` refuses `write: true`, stdio prints a startup warning, and `--http` refuses to start unless `--allow-unconfined-paths` is set. {{ added "0.25.0" }} - **No process execution or network egress from enrichers.** `evaluate_events` builds only `template` enrichers. `command`, `http`, `lookup`, and `stix` enrichers are refused; configure them on the daemon instead. - **sigma-cli delegation** (`--allow-sigma-cli`, off by default) spawns a subprocess with the rules root as its working directory. Only bare identifiers (letters, digits, `_`, `-`) are passed through as sigma-cli pipeline names; anything else is treated as a path and confined. diff --git a/fuzz/Cargo.lock b/fuzz/Cargo.lock index 9959a048..c1b6a3c9 100644 --- a/fuzz/Cargo.lock +++ b/fuzz/Cargo.lock @@ -2183,7 +2183,7 @@ dependencies = [ [[package]] name = "rsigma-convert" -version = "0.24.0" +version = "0.25.0" dependencies = [ "log", "regex", @@ -2197,7 +2197,7 @@ dependencies = [ [[package]] name = "rsigma-eval" -version = "0.24.0" +version = "0.25.0" dependencies = [ "ahash", "aho-corasick", @@ -2236,7 +2236,7 @@ dependencies = [ [[package]] name = "rsigma-ir" -version = "0.24.0" +version = "0.25.0" dependencies = [ "base64 0.23.1", "ciborium", @@ -2249,7 +2249,7 @@ dependencies = [ [[package]] name = "rsigma-parser" -version = "0.24.0" +version = "0.25.0" dependencies = [ "fancy-regex", "globset", @@ -2267,7 +2267,7 @@ dependencies = [ [[package]] name = "rsigma-runtime" -version = "0.24.0" +version = "0.25.0" dependencies = [ "arc-swap", "async-trait", @@ -2321,7 +2321,7 @@ dependencies = [ [[package]] name = "rstix" -version = "0.24.0" +version = "0.25.0" dependencies = [ "base64 0.23.1", "email_address",