Introduce Spectral Sentinel — an online subspace anomaly detector

[da2ce7]

A sentinel is not a judge. It stands watch, keeps its bearings, and reports what has changed.

That is the idea behind Spectral Sentinel. It does not decide whether a pattern is dangerous, important, or actionable. It measures structure in a stream, learns what has been ordinary so far, and returns statistical readouts when new observations depart from the learned geometry.

The "spectral" part is literal: each selected region is modelled with low-rank subspace trackers, and a second tier models the spectrum of those scores across related cells. The "sentinel" part is restraint: the crate observes, scores, and reports, but policy stays with the host.

---

Summary

This PR adds torrust-sentinel to the workspace: a library crate for hierarchical online subspace anomaly detection over positionally structured observation streams. It is rebuilt as a single commit on the current develop (after #884), so it sits on the MSRV-1.89 floor of ADR-T-011 and follows the per-crate versioning of ADR-T-012.

It is built on top of Mudlark. Mudlark provides the adaptive spatial substrate — regions that receive more observation volume earn finer resolution, quiet regions remain coarse. Spectral Sentinel uses that structure to decide where statistical trackers are worth maintaining. It selects significant V-Tree entries, closes them under G-tree ancestry so every selected cell has a complete ancestor chain back to the root, and scores incoming batches against learned subspace models at every selected scale.

The simplest way to think about it: Mudlark decides where the stream has shape; Sentinel measures whether the recent shape still looks like what that region has learned to expect.

The crate is deliberately policy-free. Reports carry raw measurements — four scoring axes (novelty, displacement, surprise, coherence), maturity, baselines, CUSUM drift accumulators, geometry, contour summaries, and health snapshots. They do not encode threat levels, recommended actions, or decisions. The host reads the measurements and decides what they mean.

The core invariant is feed-forward: every input value updates Mudlark with exactly one unit of observation volume. Anomaly scores never flow back into the spatial index. That keeps spatial adaptation driven by traffic structure, not by the detector's own conclusions. Temporal policy is host-controlled: Sentinel never applies decay automatically.

A second analysis tier — coordination trackers — runs at internal G-tree nodes whose subtrees both contribute competitive cells. It scores cross-cell patterns of the four axes, so a coordinated shift that no single cell would flag still surfaces in the report.

Contents

The addition is substantial, but almost entirely self-contained within packages/sentinel.

• A new torrust-sentinel crate (1.0.0) with a narrow public surface exposed through flat crate-root re-exports
• SpectralSentinel<C, V, N> as the generic engine, with Sentinel128 and Sentinel64 aliases for the common domain widths
• SentinelConfig, NoiseSchedule, and SvdStrategy for host-controlled measurement parameters, with structured ConfigError / ConfigErrors / ConfigWarning validation rather than panics
• Public readout types for batch, cell, coordination, contour, health, maturity, geometry, baseline, score, and analysis-set summaries — ordered deterministically by GNodeId
• Analysis-set selection over Mudlark's G-V Graph: top-K competitive V-entries plus ancestor closure into the investment set
• Per-cell subspace trackers scoring novelty, displacement, surprise, and coherence; EWMA baselines with upper-tail clipping; CUSUM drift accumulators with a separate slow EWMA
• Hierarchical coordination trackers scoring cross-cell score patterns at G-tree internal nodes
• Automatic synthetic noise warm-up for every newly created tracker, plus a deferred staging area and optional background warming thread so cell creation does not block the ingest hot path
• Brand incremental SVD for subspace evolution, with a naive thin-SVD path used both as a fallback for small/numerically sensitive cases and as a debug-mode oracle
• Optional serde support (off by default; pulls in torrust-mudlark/serde)
• A Criterion benchmark suite (15 bench functions across 8 groups: encoding, ingest, auxiliary, convergence, scaling, temporal, analysis)
• 21 architecture decision records covering measurement-not-opinions, the feed-forward invariant, Mudlark integration, deterministic ordering, analysis-set recomputation, automatic noise injection, scoring geometry, decay semantics, routing, degenerate-dimension guards, test budgets, warm-up convergence, visibility, cell-creation performance, Brand SVD, deferred warm-up, generic domain parameters, investment-set terminology, and the clip-pressure / mean-centred-variance EWMA refinements
• README, public API reference, algorithm document, and implementation notes (~4.7k lines of crate-level documentation total)
• 642 tests with all features enabled (631 in the default configuration) across crate-level tests (src/tests/), integration tests (tests/) and doc-tests, with the README compiled as a doc-test via #[cfg(doctest)] include_str!
• Two pedagogy integration tests (pedagogy.rs, pedagogy_advanced.rs) written to be read end-to-end as a walkthrough of the public surface

Changes outside sentinel

• Cargo.toml — packages/sentinel added as a workspace member
• packages/mudlark — 1.0.1 → 1.1.0: monotonic structural mutation counters (splits, evictions, restorations) recorded at the sites that already maintain the node and terminal counts, and a semi_internal_count() accessor; both additive, with a changelog entry
• Cargo.lock — 69 entries added for the dependency closure (faer, rand_distr, plus dev-only criterion and tracing-subscriber); the only moved entry is torrust-mudlark following its bump
• AGENTS.md — adds Sentinel's S- cross-reference prefix to the package table and ADR examples

Manifest under ADR-T-012

The dependency on the sibling torrust-mudlark pins version = "1.1.0" beside its path, because cargo publish writes that requirement into the published manifest and the report reads the structural mutation counters that arrive with that minor. faer, tracing and criterion name the 0.x line the sources are written against (0.24, 0.1, 0.8) instead of a bare 0, for the reason #884 gave for the root's requirements. cargo publish --dry-run -p torrust-sentinel stops at resolution because torrust-mudlark is not on crates.io yet; that is the publication order ADR-T-012 documents, and torrust-mudlark itself dry-runs cleanly.

Reviewing this

The best starting point is the public surface:

packages/sentinel/src/lib.rs → packages/sentinel/docs/api.md → packages/sentinel/README.md

From there, the main implementation path is src/sentinel/mod.rs for the orchestrator, src/analysis_set.rs for competitive selection and ancestor closure, src/sentinel/tracker.rs for per-cell scoring, src/sentinel/{cusum,staging,warming_thread}.rs for drift and warm-up, and src/maths/ for the SVD plumbing.

For a focused review, I would look at:

• the public API shape and the flat crate-root re-exports
• configuration validation, defaults, and the structured error/warning types
• the feed-forward Mudlark integration (Δ = 1 per observation, scores never feed back)
• report semantics, deterministic ordering, and what is and isn't part of the public surface
• tracker warm-up, the deferred staging area, and the optional background warming thread
• the Brand SVD fallback boundary (small d, narrow rank gaps) and the debug-mode oracle path
• integration tests that assert invariants through the public API only

The pedagogy tests are intended to be readable end-to-end; running cargo test -p torrust-sentinel --test pedagogy -- --nocapture produces a narrated walk through the public surface.

Verification

On the rebuilt commit: cargo fmt --check clean; cargo clippy --workspace --all-targets --all-features -- -D warnings clean under the workspace lint table; the crate's 565 tests and 15 doc-tests pass; the whole workspace passes (2,370 tests, none failed); cargo audit keeps the vulnerability count of develop (the one rsa advisory with no fixed release) and adds a single allowed unmaintained-crate warning, RUSTSEC-2024-0436 (paste, a proc-macro pulled by faer through gemm).

Notes

• Ships at 1.0.0: the public surface documented in docs/api.md is covered by semver guarantees from this release onwards. A sibling crate consumes it through a version-beside-path pin, the same discipline this manifest applies to torrust-mudlark, so the version a consumer pins is the one the manifest declares.
• MSRV 1.89, inherited from the workspace (ADR-T-011).
• No unsafe code; #![forbid(unsafe_code)] at the crate root.
• AGPL-3.0-only, inherited from the workspace. Unlike Mudlark, no linking exception is shipped with this crate.
• Default features: none. serde is opt-in.
• The crate measures only. Interpretation and response remain external host policy.
• Temporal policy is host-controlled: Sentinel never applies decay automatically.
• Configuration prefers structured errors over panics.
• Sentinel docs and ADRs use the S- cross-reference prefix added in this PR.

Review fixes

Since the previous head, nine commits on top of the three original ones fix every finding an automated review of the package produced and a code-level verification confirmed: the configuration validation refuses non-numbers, an unrepresentable depth-buffer headroom and a coordinate width below the tracker minimum (two additive error variants); the full-width cell owns the domain maximum and the root leaves the analysis candidates before the capacity cut; staged cells warm by their real volume and a pass with no competitive scores retires every coordination context; health and batch reports count the online sets, populate the semi-internal count from the graph (one additive mudlark accessor, and the headroom arithmetic in mudlark saturates instead of wrapping), count the whole contour and order coordination reports by depth then identifier; the geometric schedule reports its true maximum, a failed corrective factorisation reports failure so the dispatcher falls back, and the unread round scores are gone. Prose follows the code (the z-score denominator, the live-tracker figure, the open bit-source trait, the implemented dimension guard), and the one exact float equality in the invariants suite compares bit patterns. Nothing on the public surface is removed or reshaped.

Second round of review fixes

Five further commits fix every finding of a second automated review at the previous head. The warming thread's shutdown transition is made under the staging lock its wait is paired with, so a shutdown can no longer be lost between the worker reading its predicate and sleeping on it, which left the join — reached from Drop — waiting forever; the u128 centred-bit conversion caps the requested width at the type's own instead of indexing past its backing array; the centred bit vector gains a validated constructor and a length accessor so an implementation of the open bridge trait outside the crate can return the value its impl must produce; the online summary reports the investment count over the whole selection, warming cells included, as its contract states. The test support's four-bit generator refuses a nibble at sixteen or above (which shifted every set bit out of the coordinate and aliased the range sixteen below), a six-bit generator carries the sprays that claim sixty-four distinct ranges, and the ordering, budget and concentration witnesses assert the documented order, the structure's own budget and a report below the root. The upper bounds of analysis entries and coordination contexts document the top-of-domain exception, the analysis set's full field is named as the investment set it is, the thread-safety plan states the Send + Sync the crate asserts statically, and section-mark references with no referent leave the record and the test banners. Public surface: three additive constant functions on the centred bit vector; nothing is removed or reshaped. Verified on stable 1.98 (the toolchain this repository lints and tests with), 1.89.0 and nightly.

Third round of review fixes

Four further commits fix every finding of a third automated review at the previous head. The headroom a depth pair demands was computed with one checked step and three unchecked ones around it, so a creation depth of zero beside an eviction depth at the top of the range overflowed inside the very method that promises to hand back its faults as values; the computation now lives in a helper whose every step is checked, and any overflow reports the existing structured error for a buffer too large to honour, with the widest pair a budget can clear pinned as accepted. The centred-bit vector holds at most 128 values, but the coordinate trait it is fed from is open to wider types and the only width guard compared the tracker's dimension with the coordinate's declared bits, so a 200-wide tracker over a 256-bit coordinate was admitted and fed from a 128-slot vector; the sentinel now refuses a width above the vector's ceiling with a configuration error naming the width and the maximum, and the ceiling is documented on the bit source, on the bit vector and in the crate docs. The prefix generators in the test support guarded their ranges with assertions that release builds compile out; all three sites assert unconditionally. The dimension guard's doc said widths at or below the minimum are refused where the predicate refuses only widths below it, and now names the side of the boundary that is kept; a bare section ordinal in the exponential-average module is replaced by the sense it carried. Public surface: one additive configuration-error variant; nothing is removed or reshaped. Verified on stable 1.98 (the toolchain this repository lints and tests with), 1.89.0 and nightly.

Fourth round of review fixes

Three further commits correct every finding of a fourth automated review at the previous head; all eight are documentation, and no Rust moves. The two warming modes draw from two different generators: synchronous warming drains from the sentinel's own generator, while background warming draws from a second one seeded on the worker and promotes whatever the worker has finished at each ingest, against a map the main thread is concurrently writing. Four sites promised bit-for-bit reproducibility across runs without saying which mode delivers it; each now scopes the claim to synchronous warming on a fixed build, in one identical clause, and names the background-mode interleaving as the second source of randomness that reaches the scores. Three lifecycle records described mechanisms that no longer run where they said: the cell-width rejection lives in the suffix-width filter applied at runtime rather than in configuration validation, and its effective range is stated; creation schedules noise injection rather than performing it, now that the injection itself is deferred; and the bounded per-ingest work of the deferred warm-up record is stated for the background mode it holds in, with the default mode's in-line drain named beside it. The dimension guard's record said cells at or below the minimum are excluded where the filter keeps a cell at it, and now carries the same words as the constant's own documentation. Verified on stable 1.98 and nightly, with the crate's rustdoc and doc tests, since the README is the crate's front-page documentation.

Fifth round of review fixes

Three further commits correct every finding of a fifth automated review at the previous head. Construction asked the operating system for the background warming thread and aborted the host when the request was refused, over a resource limit that has nothing to do with the configuration's correctness; the request now returns the environment's own account as a configuration error naming the setting, and it arrives alone because the thread is asked for only once validation has passed. Reset, which has no error channel, keeps the sentinel running and warms cells synchronously instead, recording the refusal as a warning: the warm-up dispatch keys on whether a thread is present rather than on the flag, so the fallback is complete and every report is produced as before. Seeding one baseline from another copied the numbers but only ever raised warmth, so a receiver seeded from a cold source stayed warm over placeholder statistics and the cold path that replaces them never ran again; warmth is now part of what is handed over, in both directions, with a witness that fails at the previous head. Three tests and their prose claimed more, or other, than the engine guarantees: the determinism test compared three lengths and a few means where it now compares whole reports figure by figure with equal bit patterns; the coordination-report ordering test asserted ascending handle where the producer sorts by depth and then handle, and both prose statements of the handle-only order are corrected with it; and the reproducibility claim at the top of the determinism suite is scoped to synchronous warming on a fixed build, which is the configuration those tests share. Public surface: one additive configuration-error variant, and the configuration-error enumeration is marked non-exhaustive ahead of first publication so a later refusal is additive too; the warming-thread handle whose signature changed is crate-internal. Verified on stable 1.98 (the whole workspace lints clean; the crate's tests pass), 1.89.0 and nightly, with the crate's rustdoc and doc tests.

Later rounds of review fixes

The remaining rounds of automated review, each verified at code level before a change was made, are answered by the commits after the fifth round. The CUSUM allowance now follows the algorithm text, κσ·√v_slow, with the denominator-protection constant kept out of it; the geometric noise schedule saturates an unrepresentable exponent instead of wrapping it, so a public caller with an arbitrarily deep argument still lands on the floor; coordination contexts are retained by online competitive membership rather than by which cells happened to score in the batch, so a quiet batch no longer destroys a context that the next joint batch would have to re-warm. Every section reference in source and tests uses the qualified §ALGO S-N form and points at the section that carries the cited content; the clip-pressure implementation plan moved to docs/plans/ so the ADR identifier it borrowed resolves to one record; the implementation guide describes the lazy coordination warm-up the code performs; a failed warming worker is recorded rather than allowed to take reset() down; the report's geometry record describes the model that produced the scores beside it; the compile-time width bound is named where a reader would look for it in the error list; and a test comment that cited a document the package never contained now derives its tolerance in place. Finally, the structural mutation counts in the contour snapshot are read from the spatial layer's own counters instead of being inferred from node and terminal deltas, an inference that a last-child eviction falsified; that is what the Mudlark minor bump carries.

[codecov]

Codecov Report

❌ Patch coverage is 91.76115% with 218 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.37%. Comparing base (843aaff) to head (bab66fa).

Files with missing lines	Patch %	Lines
packages/sentinel/src/sentinel/mod.rs	91.34%	53 Missing and 21 partials ⚠️
packages/sentinel/src/config.rs	78.02%	49 Missing ⚠️
packages/sentinel/src/maths/bench_tracing.rs	0.00%	45 Missing ⚠️
packages/sentinel/src/sentinel/tracker.rs	96.13%	13 Missing and 4 partials ⚠️
packages/sentinel/src/maths/mod.rs	91.25%	13 Missing and 1 partial ⚠️
packages/sentinel/src/maths/brand_svd.rs	91.46%	6 Missing and 1 partial ⚠️
packages/sentinel/src/sentinel/staging.rs	98.56%	3 Missing and 3 partials ⚠️
packages/sentinel/src/sentinel/warming_thread.rs	96.05%	3 Missing ⚠️
packages/sentinel/src/analysis_set.rs	98.33%	2 Missing ⚠️
packages/sentinel/src/maths/naive_svd.rs	97.61%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #827      +/-   ##
===========================================
+ Coverage    68.52%   72.37%   +3.85%     
===========================================
  Files          161      175      +14     
  Lines        13111    15757    +2646     
  Branches     13111    15757    +2646     
===========================================
+ Hits          8984    11404    +2420     
- Misses        3853     4048     +195     
- Partials       274      305      +31     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Copilot wasn't able to review this pull request because it exceeds the maximum number of lines (20,000). Try reducing the number of changed lines and requesting a review from Copilot again.

[Copilot]

🟡 Changes recommended

Coordination lifecycle semantics conflict across implementation and specification, and several public documentation references are stale or ambiguous.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (6)

packages/sentinel/src/sentinel/mod.rs:1289

• §5.5 documents coherence scoring, not coordination-context pruning. This should point to the lifecycle pruning section and use the package-qualified source-reference form required by AGENTS.md:72-83.
  packages/sentinel/src/sentinel/mod.rs:1532
• This repeats the stale coherence-section reference for lifecycle pruning. Source references must be package-qualified (AGENTS.md:72-83), and the relevant contract is §ALGO S-7.7.2.
  packages/sentinel/src/sentinel/tracker.rs:105
• This source cross-reference is missing the required document and Sentinel package qualifiers (AGENTS.md:72-83).
  packages/sentinel/src/sentinel/tracker.rs:653
• This source cross-reference is missing the required document and Sentinel package qualifiers (AGENTS.md:72-83).
  packages/sentinel/src/sentinel/tracker.rs:785
• This source cross-reference is missing the required document and Sentinel package qualifiers (AGENTS.md:72-83).
  packages/sentinel/src/sentinel/tracker.rs:834
• This source cross-reference is missing the required document and Sentinel package qualifiers (AGENTS.md:72-83).

• Files reviewed: 54/101 changed files
• Comments generated: 10
• Review effort level: Balanced

[Copilot]

🟡 Changes recommended

Structural mutation counts rely on an invalid Mudlark eviction identity, causing debug panics and incorrect release reports.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (4)

packages/sentinel/src/sentinel/tracker.rs:493

• This bare source-code section reference violates the package-qualified format required by AGENTS.md:94-100.
  packages/sentinel/src/sentinel/tracker.rs:651
• This bare source-code section reference violates the package-qualified format required by AGENTS.md:94-100.
  packages/sentinel/src/sentinel/tracker.rs:785
• This bare source-code section reference violates the package-qualified format required by AGENTS.md:94-100.
  packages/sentinel/src/sentinel/tracker.rs:833
• This bare source-code section reference violates the package-qualified format required by AGENTS.md:94-100.

• Files reviewed: 54/101 changed files
• Comments generated: 3
• Review effort level: Balanced

[Copilot]

🔵 Needs a closer look

The shared seeded fixture never crosses its split threshold, leaving several coordination tests vacuous.

Review details

• Files reviewed: 54/101 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

🔵 Needs a closer look

The large public numerical API and concurrent warming lifecycle warrant final human review despite extensive documentation and tests.

Review details

• Files reviewed: 63/110 changed files
• Comments generated: 1
• Review effort level: Balanced

[Copilot]

🟡 Changes recommended

Maturity decays inconsistently with the learned model, infinite epsilon can disable scoring, and several public references are inaccurate.

Get a fresh assessment by requesting another Copilot review.

Review details

• Files reviewed: 63/110 changed files
• Comments generated: 5
• Review effort level: Balanced

[Copilot]

🔵 Needs a closer look

The new statistical engine, numerical routines, concurrency lifecycle, and broad public API require final human validation despite no confirmed blocking defect in the reviewed changes.

Review details

• Files reviewed: 63/110 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[da2ce7]

ACK 4c052f8