From 0d3bc372db97f0a419c0540ac3e77ab780bf6cdb Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Wed, 9 Sep 2026 03:34:21 +0000 Subject: [PATCH] docs(runbook): the dead man's switch was tripped on purpose, and it worked (#288) A dead man's switch nobody has seen go red is indistinguishable from one that does not work. On 2026-09-09, with the phone in hand: Alertmanager stopped at 03:07:37 UTC seconds after a ping; the healthchecks.io check went DOWN and emailed at 03:25:58, eighteen minutes in against a 5m period and 15m grace; Alertmanager started at 03:25:58 and the first ping after the restart landed at 03:27:57, inside one repeat_interval. Then the daily half: the second Watchdog route lowered to 2m and reloaded at 03:28:29, the notification on the normal ntfy channel at 03:28, the route back to 24h and reloaded at 03:33:16. Zero delivery failures across the window. One observation from the daily half, recorded rather than fixed: a route whose repeat_interval is below the root's 5m group_interval repeats on the 5m tick, the same coupling the heartbeat route already documents. It does not touch the 24h route. The runbook carries the times under "Confirming it actually works" in the house banner form, the watcher banner says the drill was run the same day, and observability.md, the roadmap and check_alert_channels.py's docstring stop saying it was never tested. Refs #288. Co-Authored-By: Claude Fable 5.1 --- docs/observability.md | 7 +++++-- docs/roadmap.md | 2 +- docs/runbooks/verify-the-alert-path.md | 27 ++++++++++++++++++++++---- scripts/check_alert_channels.py | 4 ++-- 4 files changed, 31 insertions(+), 9 deletions(-) diff --git a/docs/observability.md b/docs/observability.md index 675a6db..1fddc27 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -548,8 +548,11 @@ So now, if Prometheus stops evaluating, Alertmanager dies, or this host loses outbound network, something external notices — in principle. That is the failure [#214](https://github.com/Gerrrt/HomeLab/issues/214) lived through from the other direction, and the heartbeat is cited as the answer to it in #214's -own resolution. It is armed and not yet proven: nobody has watched the check go -red, which is [#288](https://github.com/Gerrrt/HomeLab/issues/288). +own resolution. It is armed and proven: on 2026-09-09 Alertmanager was stopped +for 18 minutes, the check went red and emailed, and the first ping after the +restart landed within two minutes; the daily route was confirmed the same +sitting ([#288](https://github.com/Gerrrt/HomeLab/issues/288), times in the +runbook). `check_alert_channels.py --live` reports the destination on every deploy, classifying the heartbeat's host as a watcher or a push service. A push service diff --git a/docs/roadmap.md b/docs/roadmap.md index b51702d..83f1f7f 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -1080,7 +1080,7 @@ them name the condition that would change the answer. deliberately sends no test notification. A check that depends on the delivery path inherits the blind spot that made this last ten hours. Every assertion reads a file. The dead man's switch is the other half of the - answer and remains untested — #288. + answer; it was armed and then tripped on purpose on 2026-09-09 — #288. All three failure paths were exercised rather than assumed: a `url_file` with no renderer, a rendered-but-empty file, and a container that cannot diff --git a/docs/runbooks/verify-the-alert-path.md b/docs/runbooks/verify-the-alert-path.md index 692d416..3d17bb0 100644 --- a/docs/runbooks/verify-the-alert-path.md +++ b/docs/runbooks/verify-the-alert-path.md @@ -42,10 +42,10 @@ travel, but nothing machine-checks its absence — you do. > absence, and passes. From 2026-09-07 to this date the URL pointed at `ntfy.sh` > like every other receiver, and there was no dead man's switch, only a > heartbeat nobody was waiting on -> ([#359](https://github.com/Gerrrt/HomeLab/issues/359)). **The drill below has -> still not been run** — that is [#288](https://github.com/Gerrrt/HomeLab/issues/288), -> and until it has, "the check goes red when the stack dies" is a claim, not an -> observation. +> ([#359](https://github.com/Gerrrt/HomeLab/issues/359)). **The drill below was +> run the same day** — [#288](https://github.com/Gerrrt/HomeLab/issues/288) — +> so "the check goes red when the stack dies" is an observation, with times, +> under *Confirming it actually works*. The watcher has to live somewhere other than the monitoring host. A watcher on this host fails at the same moment as the thing it is watching, which is not @@ -117,6 +117,25 @@ it. Nothing enforces that from here, which is why it is written down. ## Confirming it actually works +> **Done 2026-09-09**, both halves in one sitting, read off the monitoring host +> with the phone in hand ([#288](https://github.com/Gerrrt/HomeLab/issues/288)): +> +> | | | +> | --- | --- | +> | `docker stop alertmanager` | 03:07:37 UTC, seconds after a ping went out | +> | Check DOWN, email received | 03:25:58 UTC — 18 minutes; period 5m + grace 15m says 20 at most | +> | `docker start alertmanager` | 03:25:58 UTC, ready 5 s later | +> | First ping after restart | 03:27:57 UTC — 2 minutes, inside one `repeat_interval` | +> | Second Watchdog route at 2m, `make reload` | 03:28:29 UTC | +> | Watchdog on the normal ntfy channel | 03:28 UTC, on the phone | +> | Route back to 24h, `make reload` | 03:33:16 UTC | +> +> Zero delivery failures across the whole window. One thing the daily half +> showed on the way: a route with `repeat_interval` below the root's +> `group_interval` (5m) repeats on the 5m tick, not at its own interval — the +> same coupling the heartbeat route works around above. It does not affect the +> 24h route, and the first notification still went out immediately. + Do not trust a green check you have never seen go red. ```bash diff --git a/scripts/check_alert_channels.py b/scripts/check_alert_channels.py index 89da3cc..564f05b 100755 --- a/scripts/check_alert_channels.py +++ b/scripts/check_alert_channels.py @@ -47,8 +47,8 @@ assertion here reads a file, and none of them needs a notification to succeed. The dead man's switch is the other half of the answer and is not this: it is what -notices when the whole path is down, and whether it actually works has never been -tested (#288). +notices when the whole path is down. It was tripped on purpose on 2026-09-09 and +did (#288); the times are in docs/runbooks/verify-the-alert-path.md. Usage: scripts/check_alert_channels.py [--files] [--live] [STACK] """