diff --git a/docs/observability.md b/docs/observability.md index 675a6db..1fddc27 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -548,8 +548,11 @@ So now, if Prometheus stops evaluating, Alertmanager dies, or this host loses outbound network, something external notices — in principle. That is the failure [#214](https://github.com/Gerrrt/HomeLab/issues/214) lived through from the other direction, and the heartbeat is cited as the answer to it in #214's -own resolution. It is armed and not yet proven: nobody has watched the check go -red, which is [#288](https://github.com/Gerrrt/HomeLab/issues/288). +own resolution. It is armed and proven: on 2026-09-09 Alertmanager was stopped +for 18 minutes, the check went red and emailed, and the first ping after the +restart landed within two minutes; the daily route was confirmed the same +sitting ([#288](https://github.com/Gerrrt/HomeLab/issues/288), times in the +runbook). `check_alert_channels.py --live` reports the destination on every deploy, classifying the heartbeat's host as a watcher or a push service. A push service diff --git a/docs/roadmap.md b/docs/roadmap.md index b51702d..83f1f7f 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -1080,7 +1080,7 @@ them name the condition that would change the answer. deliberately sends no test notification. A check that depends on the delivery path inherits the blind spot that made this last ten hours. Every assertion reads a file. The dead man's switch is the other half of the - answer and remains untested — #288. + answer; it was armed and then tripped on purpose on 2026-09-09 — #288. All three failure paths were exercised rather than assumed: a `url_file` with no renderer, a rendered-but-empty file, and a container that cannot diff --git a/docs/runbooks/verify-the-alert-path.md b/docs/runbooks/verify-the-alert-path.md index 692d416..3d17bb0 100644 --- a/docs/runbooks/verify-the-alert-path.md +++ b/docs/runbooks/verify-the-alert-path.md @@ -42,10 +42,10 @@ travel, but nothing machine-checks its absence — you do. > absence, and passes. From 2026-09-07 to this date the URL pointed at `ntfy.sh` > like every other receiver, and there was no dead man's switch, only a > heartbeat nobody was waiting on -> ([#359](https://github.com/Gerrrt/HomeLab/issues/359)). **The drill below has -> still not been run** — that is [#288](https://github.com/Gerrrt/HomeLab/issues/288), -> and until it has, "the check goes red when the stack dies" is a claim, not an -> observation. +> ([#359](https://github.com/Gerrrt/HomeLab/issues/359)). **The drill below was +> run the same day** — [#288](https://github.com/Gerrrt/HomeLab/issues/288) — +> so "the check goes red when the stack dies" is an observation, with times, +> under *Confirming it actually works*. The watcher has to live somewhere other than the monitoring host. A watcher on this host fails at the same moment as the thing it is watching, which is not @@ -117,6 +117,25 @@ it. Nothing enforces that from here, which is why it is written down. ## Confirming it actually works +> **Done 2026-09-09**, both halves in one sitting, read off the monitoring host +> with the phone in hand ([#288](https://github.com/Gerrrt/HomeLab/issues/288)): +> +> | | | +> | --- | --- | +> | `docker stop alertmanager` | 03:07:37 UTC, seconds after a ping went out | +> | Check DOWN, email received | 03:25:58 UTC — 18 minutes; period 5m + grace 15m says 20 at most | +> | `docker start alertmanager` | 03:25:58 UTC, ready 5 s later | +> | First ping after restart | 03:27:57 UTC — 2 minutes, inside one `repeat_interval` | +> | Second Watchdog route at 2m, `make reload` | 03:28:29 UTC | +> | Watchdog on the normal ntfy channel | 03:28 UTC, on the phone | +> | Route back to 24h, `make reload` | 03:33:16 UTC | +> +> Zero delivery failures across the whole window. One thing the daily half +> showed on the way: a route with `repeat_interval` below the root's +> `group_interval` (5m) repeats on the 5m tick, not at its own interval — the +> same coupling the heartbeat route works around above. It does not affect the +> 24h route, and the first notification still went out immediately. + Do not trust a green check you have never seen go red. ```bash diff --git a/scripts/check_alert_channels.py b/scripts/check_alert_channels.py index 89da3cc..564f05b 100755 --- a/scripts/check_alert_channels.py +++ b/scripts/check_alert_channels.py @@ -47,8 +47,8 @@ assertion here reads a file, and none of them needs a notification to succeed. The dead man's switch is the other half of the answer and is not this: it is what -notices when the whole path is down, and whether it actually works has never been -tested (#288). +notices when the whole path is down. It was tripped on purpose on 2026-09-09 and +did (#288); the times are in docs/runbooks/verify-the-alert-path.md. Usage: scripts/check_alert_channels.py [--files] [--live] [STACK] """