fix(fault_manager): keep the near-miss series when a fault is cleared

[bburda]

Summary

Why

Nothing in the fault manager recorded how often a fault code nearly confirmed. Two behaviours
combined to destroy that information. The debounce counter was updated in place, so every report
overwrote the previous state and no series was ever built. And clear_fault deleted what had been
captured for the fault, so acknowledging the fault removed the rest.

This is data being destroyed, not a feature that was missing. Any later analysis of recurring
faults, for example how often a code approaches confirmation and whether that rate is growing,
needs this series, and it cannot be rebuilt after the fact. Every day a deployment runs without
it, more of that history is gone for good. That is why this is a fix and not an enhancement.

What a near miss is

A FAILED report that moves the debounce counter but does not confirm the fault. A PASSED report
moves the counter in the healing direction, so it is not a near miss. The predicate lives in one
place, is_near_miss(), so both storage backends agree.

The near-miss series

1. New near_misses table, one row appended per near miss, never updated in place. Each row holds
   the timestamp, the counter after the report, the confirmation threshold it was measured
   against, the severity, the reporting source, and the fault status the report left behind.
2. resulting_status is what makes the series readable. The HEALED latch holds the status the
   whole way from the healing threshold down to the confirmation threshold, so every report on the
   way back into a fault that does confirm also moves the counter without confirming. Entries
   recording PREFAILED are approaches from a resting state; entries recording HEALED are a
   counter walking back down under the latch. Without the field the two are indistinguishable and
   the question the series exists for is not answerable.
3. Entries are kept and evicted in arrival order, not by timestamp. Reporters carry their own
   clocks, so a report can arrive with a timestamp behind one already stored; ordering by timestamp
   would let such a report evict itself the moment it was written.
4. near_miss.max_per_fault, default 200, 0 means unlimited, evicting the oldest entries.
   That is the opposite of snapshots.max_per_fault, which keeps the earliest, because a series
   that stops growing at boot cannot show whether the rate is changing. Applying the bound also
   trims what is already stored, and the node warns with the number of entries that dropped,
   because a mistyped value deletes history that cannot be recovered.
5. clear_fault no longer touches the series, and neither does the startup reclassification of
   HEALED faults, which is the second place that removed captured data.
6. The fault row and the near-miss row commit together, so a failure on the append cannot leave
   the debounce counter advanced with the near miss missing. Only FAILED reports take that
   transaction: a PASSED report writes at most one row, and taking the writer lock for it would
   make a heal heartbeat fail with SQLITE_BUSY where before it could not.
7. InMemoryFaultStorage keeps the same series, and a test drives one report sequence through both
   backends and compares what they store.

Snapshot retention, now that the switch exists on main

snapshots.retain_on_clear landed separately, so this branch no longer adds it. What it does add
is the handling that switch still needs.

8. The startup reclassification of HEALED faults deleted their snapshots regardless of the
   setting. Retention therefore held only until the next restart, which then removed exactly what
   it was set to keep, and the backends disagreed: the in-memory one kept the snapshots, SQLite
   dropped them. Both follow the switch now.
9. ~/get_fault served the freeze-frame only when no snapshots remained, which was the signal
   that acknowledgement had removed them. With retention on they never run out, so the frame - the
   state at the most recent confirmation - stayed hidden behind snapshots of earlier occurrences.
10. ~/get_snapshots returns one entry per topic and now serves the newest capture of that
    topic. It let the last row processed win, and the backends return these in opposite orders, so
    SQLite served the oldest value under the newest captured_at.

Two other corrections along the way

The SQLite backend brings a stored debounce counter back into the reporting config's band before
applying a report and the in-memory one did not. Per-entity threshold overrides make that
reachable in one process, and it offset the whole series and the report at which a fault confirms.

Before enabling anything

With the default confirmation_threshold: -1 the first FAILED report confirms the fault at once,
so there is never a near miss to store. The series only fills where debounce is configured.

With snapshots.retain_on_clear on, read snapshots.max_per_fault as a cap for the whole life of
the database rather than for one fault cycle. It rejects new snapshots and keeps the earliest, and
the rejection is silent, so a code that has reached the cap records nothing on later occurrences.
Raise it, or set it to 0, if you need snapshots from every occurrence.

The near-miss bound is per fault code, not per database. Fault codes are unbounded in cardinality,
so the bound caps what any single code costs, not the total.

Known limitation

There is no service or REST surface for the series. It is read through FaultStorage or from the
database file, so nothing can query it operationally yet and no end-to-end test can exercise it.
Adding one needs a new service definition, which is deliberately out of scope here.

---

Issue

• closes [BUG] Clearing a fault destroys the record of how often it nearly happened #630

---

Type

• Bug fix
• New feature or tests
• Breaking change
• Documentation only

---

Testing

Full package suite: 739 tests, 0 errors, 0 failures (unit, integration and linters).

colcon build --symlink-install && source install/setup.bash
./scripts/test.sh all --packages-select ros2_medkit_fault_manager

Near-miss series, SQLite: appended and not overwritten, the confirming report is not counted,
survives clear_fault, continues across a reactivation, survives closing and reopening the
database, a report after reopen extends the series in order, a PASSED report is not counted, a
CRITICAL immediate confirm is not counted, confirmation_threshold: -1 records nothing, a FAILED
report under the HEALED latch is counted and is distinguishable from an approach, each entry
carries its own threshold and severity and source and status, arrival order wins over timestamp
order, the bound keeps the newest, a bound of 1 works, the bound is per fault code, 0 is unlimited,
SIZE_MAX is unlimited rather than emptying the table, applying a smaller bound trims and reports
how many it dropped, applying an unlimited bound keeps everything, an unknown code returns empty,
a PASSED report on an unknown fault writes nothing, the table and the status column are both
created on a database from an older build, and the series survives HEALED reclassification.

The in-memory backend has the same contract covered, and one parity test drives an identical
sequence with mixed per-entity thresholds through both backends and compares the stored series.

Snapshot retention through the startup reclassification: kept when configured, dropped by default,
unrelated faults untouched, and the same contract on both backends.

Read path, driven through the real ~/get_snapshots and ~/get_fault services: the newest
snapshot of a topic wins, and the freeze-frame stays visible behind retained snapshots.

Node level: near_miss.max_per_fault reaches storage, the default is 200, a negative value falls
back to the default instead of becoming SIZE_MAX, 0 gives unlimited, snapshots are deleted on
clear by default, snapshots.retain_on_clear reaches storage, and it is applied before the startup
reclassification rather than after it.

Every behaviour above was checked by mutation: reverting each fix in turn fails only the tests that
encode it, and flipping either default fails exactly the default-behaviour tests.

---

Checklist

• Breaking changes are clearly described (there are none: the parameter defaults to current
  behaviour, and the schema changes are additive with a migration)
• Tests were added or updated if needed
• Docs were updated if behavior or public API changed (package README,
  docs/config/fault-manager.rst, docs/tutorials/snapshots.rst, default parameters file)

[Copilot]

Pull request overview

This PR fixes fault-manager data loss by introducing an append-only “near-miss” series that records each FAILED report that advances debounce without confirming, and ensures the series is retained across clear_fault and startup HEALED reclassification. It implements the feature consistently across SQLite and in-memory storage backends, adds a new retention parameter (near_miss.max_per_fault, default 200), and provides comprehensive unit tests plus documentation updates.

Changes:

• Add near-miss storage (new near_misses table in SQLite; in-memory series) and retrieval API (FaultStorage::get_near_misses).
• Add retention control (set_max_near_misses_per_fault) with oldest-first eviction, configured via near_miss.max_per_fault.
• Add extensive tests for both backends and document the new behavior and configuration.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
src/ros2_medkit_fault_manager/src/sqlite_fault_storage.cpp	Creates near_misses table, appends near-miss rows on qualifying FAILED reports, trims per-fault retention, and preserves near-miss rows on clear/reclassify paths.
src/ros2_medkit_fault_manager/src/fault_storage.cpp	Implements in-memory near-miss recording, retention eviction, and retrieval; adds shared is_near_miss() helper.
src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_storage.hpp	Extends storage API with NearMissRecord, get_near_misses(), and retention setter; updates clear_fault contract docs.
src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/sqlite_fault_storage.hpp	Declares near-miss API overrides and adds storage member for retention bound.
src/ros2_medkit_fault_manager/src/fault_manager_node.cpp	Declares/applies near_miss.max_per_fault parameter and clamps negative values to the default.
src/ros2_medkit_fault_manager/test/test_sqlite_storage.cpp	Adds SQLite near-miss series tests (append-only, not counting confirming/critical/PASSED, survives clear/reopen/reclassify, retention bounds, schema creation).
src/ros2_medkit_fault_manager/test/test_fault_manager.cpp	Adds node-parameter tests for near-miss retention and in-memory backend near-miss contract tests.
src/ros2_medkit_fault_manager/README.md	Documents near-miss semantics, retention, and persistence notes (no REST/service surface yet).
docs/config/fault-manager.rst	Adds configuration documentation for near-miss retention parameter and behavior.
src/ros2_medkit_fault_manager/config/fault_manager.yaml	Adds commented default config entry and explanation for near_miss.max_per_fault.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.