Skip to content

docs(runbook): the dead man's switch was tripped on purpose, and it worked (#288) #409

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Gerrrt merged 1 commit into from
Sep 9, 2026
Merged

docs(runbook): the dead man's switch was tripped on purpose, and it worked (#288) #409

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions docs/observability.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -548,8 +548,11 @@ So now, if Prometheus stops evaluating, Alertmanager dies, or this host loses
outbound network, something external notices — in principle. That is the
failure [#214](https://github.com/Gerrrt/HomeLab/issues/214) lived through from
the other direction, and the heartbeat is cited as the answer to it in #214's
own resolution. It is armed and not yet proven: nobody has watched the check go
red, which is [#288](https://github.com/Gerrrt/HomeLab/issues/288).
own resolution. It is armed and proven: on 2026-09-09 Alertmanager was stopped
for 18 minutes, the check went red and emailed, and the first ping after the
restart landed within two minutes; the daily route was confirmed the same
sitting ([#288](https://github.com/Gerrrt/HomeLab/issues/288), times in the
runbook).

`check_alert_channels.py --live` reports the destination on every deploy,
classifying the heartbeat's host as a watcher or a push service. A push service
Expand Down
2 changes: 1 addition & 1 deletion docs/roadmap.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1080,7 +1080,7 @@ them name the condition that would change the answer.
deliberately sends no test notification. A check that depends on the
delivery path inherits the blind spot that made this last ten hours. Every
assertion reads a file. The dead man's switch is the other half of the
answer and remains untested — #288.
answer; it was armed and then tripped on purpose on 2026-09-09 — #288.

All three failure paths were exercised rather than assumed: a `url_file`
with no renderer, a rendered-but-empty file, and a container that cannot
Expand Down
27 changes: 23 additions & 4 deletions docs/runbooks/verify-the-alert-path.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,10 +42,10 @@ travel, but nothing machine-checks its absence — you do.
> absence, and passes. From 2026-09-07 to this date the URL pointed at `ntfy.sh`
> like every other receiver, and there was no dead man's switch, only a
> heartbeat nobody was waiting on
> ([#359](https://github.com/Gerrrt/HomeLab/issues/359)). **The drill below has
> still not been run** — that is [#288](https://github.com/Gerrrt/HomeLab/issues/288),
> and until it has, "the check goes red when the stack dies" is a claim, not an
> observation.
> ([#359](https://github.com/Gerrrt/HomeLab/issues/359)). **The drill below was
> run the same day** — [#288](https://github.com/Gerrrt/HomeLab/issues/288) —
> so "the check goes red when the stack dies" is an observation, with times,
> under *Confirming it actually works*.

The watcher has to live somewhere other than the monitoring host. A watcher on
this host fails at the same moment as the thing it is watching, which is not
Expand Down Expand Up @@ -117,6 +117,25 @@ it. Nothing enforces that from here, which is why it is written down.

## Confirming it actually works

> **Done 2026-09-09**, both halves in one sitting, read off the monitoring host
> with the phone in hand ([#288](https://github.com/Gerrrt/HomeLab/issues/288)):
>
> | | |
> | --- | --- |
> | `docker stop alertmanager` | 03:07:37 UTC, seconds after a ping went out |
> | Check DOWN, email received | 03:25:58 UTC — 18 minutes; period 5m + grace 15m says 20 at most |
> | `docker start alertmanager` | 03:25:58 UTC, ready 5 s later |
> | First ping after restart | 03:27:57 UTC — 2 minutes, inside one `repeat_interval` |
> | Second Watchdog route at 2m, `make reload` | 03:28:29 UTC |
> | Watchdog on the normal ntfy channel | 03:28 UTC, on the phone |
> | Route back to 24h, `make reload` | 03:33:16 UTC |
>
> Zero delivery failures across the whole window. One thing the daily half
> showed on the way: a route with `repeat_interval` below the root's
> `group_interval` (5m) repeats on the 5m tick, not at its own interval — the
> same coupling the heartbeat route works around above. It does not affect the
> 24h route, and the first notification still went out immediately.

Do not trust a green check you have never seen go red.

```bash
Expand Down
4 changes: 2 additions & 2 deletions scripts/check_alert_channels.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,8 +47,8 @@
assertion here reads a file, and none of them needs a notification to succeed.

The dead man's switch is the other half of the answer and is not this: it is what
notices when the whole path is down, and whether it actually works has never been
tested (#288).
notices when the whole path is down. It was tripped on purpose on 2026-09-09 and
did (#288); the times are in docs/runbooks/verify-the-alert-path.md.

Usage: scripts/check_alert_channels.py [--files] [--live] [STACK]
"""
Expand Down