Watch the alerting dead man's switch trip, once

[Gerrrt]

Split out of #120, which is closed: the three notification secrets are filled in, the five-receiver config is live, and each receiver reads its own non-empty secret. What #120 could not close is its own last instruction:

Then, once, actually watch it trip. A dead man's switch nobody has seen go red is indistinguishable from one that does not work.

The mechanism is built and verifiable from the repository. The drill is not, and there is no record of it having been run. That is a different kind of task with a different definition of done, which is why it is here rather than holding #120 open.

What exists today

• watchdog.rules.yaml emits the always-firing Watchdog alert.
• alertmanager.yaml:61-67 routes it to the heartbeat receiver with group_wait: 0s, group_interval: 1m, repeat_interval: 5m, continue: true.
• A second Watchdog route sends 24h to the real alert channel — the half that catches a webhook which is well-formed, reachable, returns 200 and points at nothing (No dead man's switch on the notification path #67's actual failure).
• ALERTMANAGER_HEARTBEAT_URL is set and the external check is green.

A green check that has never been seen go red is the thing this issue exists to fix.

The procedure

Already written, in docs/runbooks/verify-the-alert-path.md under "Confirming it actually works":

docker stop alertmanager
# wait out the grace window — 15m by default
# the external check must report DOWN and notify you
docker start alertmanager
# it must return to green within one repeat_interval

Nothing needs to be written. This is execution, and then a line somewhere saying it happened.

The runbook's second paragraph there covers the daily half without waiting a day: temporarily lower repeat_interval on the second Watchdog route, make reload, and check the real channel.

Pick the window deliberately

docker stop alertmanager for the length of the grace window means the estate has no alert delivery for ~15 minutes. Prometheus keeps evaluating and Alertmanager will deliver what is still firing when it comes back, but anything that resolves inside the window is never notified. Two constraints from the runbook's own "timing is coupled" section apply:

• Not Sunday 03:30 — homelab-backup-volumes quiesces the stack then, which stops Prometheus evaluating and therefore stops the heartbeat. Overlapping the two makes the result unreadable.
• The external check's notification must not be this stack's webhook. If it is, the test cannot report its own failure.

Done when

• The external check has been observed DOWN and sent its notification.
• It returned green within one repeat_interval after docker start.
• The 24h route to the real channel confirmed by the temporary-repeat_interval method.
• The date recorded — verify-the-alert-path.md is the natural place, the way restore-the-firewall.md carries "until that has been done once, this runbook is a hypothesis".

That last box is what stops this being re-litigated in six months, and it is the same gap #92 has: a mechanism built, a drill never run, and nothing in the repository that says which.

[Gerrrt]

Folding in #312, which was split out of #120 independently 42 minutes after this
one and describes the same drill. Closing that as the duplicate and keeping this
one, because its "done when" is the more complete of the two — it carries the
24h-route check, which #312 omits. Everything #312 had that this issue did not is
below, so nothing is lost by closing it.

Updated state of the mechanism, verified 2026-09-04

This supersedes the numbers in the body above, all of it post-#284:

• Four receivers on four distinct URLs, all four rendered and mounted.
• group_interval: 1m, repeat_interval: 5m on the heartbeat route. Before
  fix(alertmanager): heartbeat pinged at half its advertised rate (#120) #284 it pinged at 600s, which put the 15m grace window under a single
  missed ping — the switch could not have tripped reliably even in principle.
• Measured inter-ping gaps: 300s / 310s / 351s.
• Zero clientError across every ping. Note precisely what that does and does
  not prove: the endpoint exists and accepts posts. It says nothing about
  whether anything alarms when the pings stop, which is the entire question this
  issue exists to answer.

Two settings only you can read back

Both live on the external check, are invisible from this host, and were sized
against a repeat_interval that was wrong until #284:

1. Period 5m, grace 15m. If the check was created before fix(alertmanager): heartbeat pinged at half its advertised rate (#120) #284 it may have
   been retuned to cope with the 600s cadence — a grace stretched to 20m+ to stop
   false pages. That compensation is now wrong in the other direction and should
   go back to 15m.
2. The check's own notification must not land on this stack's ntfy topic.
   If both go to the same topic, a deleted topic takes out the alert and the
   warning about the alert together — the No dead man's switch on the notification path #67 failure, one level up. Email is
   fine here; it fails independently. This is already implied by the body's
   "must not be this stack's webhook", but the ntfy-topic version is the one that
   actually bites.

Scheduling

Beyond the Sunday 03:30 exclusion above: best run when you are near the phone and
not on the clock. The estate has no alert delivery for the ~15 minutes the switch
is stopped.

Done when — merged list

• The external check observed DOWN inside the grace window, with its
  notification arriving somewhere that is not this stack's webhook or
  ntfy topic.
• Returned green within one repeat_interval after docker start.
• Period and grace confirmed at 5m / 15m.
• The 24h route to the real channel confirmed by the
  temporary-repeat_interval method.
• The date recorded in docs/runbooks/verify-the-alert-path.md, the way
  restore-the-firewall.md carries "until that has been done once, this
  runbook is a hypothesis".

Nothing here is actionable from this host — it needs a human watching a phone —
so this stays open until someone runs it.

🤖 Generated with Claude Code

[Gerrrt]

Blocked on #359. Checked the
preconditions before running the drill, and it cannot pass as written.

heartbeat_url points at ntfy.sh, the same service as the three real alert
channels (distinct topics, so not the "same topic" trap this issue's runbook
warns about). ntfy is a push service — it delivers what it is sent and has no
notion of an expected interval, so it cannot notice that a heartbeat stopped
arriving. There is no external check to observe going DOWN.

So the first two acceptance boxes here have nothing to test:

• The external check has been observed DOWN and sent its notification.
• It returned green within one repeat_interval after docker start.

The heartbeat itself is healthy — roughly 5 webhook notifications per 30
minutes, matching repeat_interval: 5m. It is being delivered to a topic nobody
is waiting on.

I did not run the drill. docker stop alertmanager for the 15-minute grace
window would take alert delivery down for a quarter of an hour to demonstrate a
silence that is already demonstrable from the configuration at no cost. If the
outage is wanted anyway to document it, that is a deliberate choice rather than
this issue's default.

The third box — the 24h route to the real channel via a temporary
repeat_interval — is also not something I can close alone: lowering it proves
Alertmanager sent the notification, which is exactly #67's
trap of a webhook that returns 200 and points at nothing. Confirming receipt
needs a person looking at the channel.

This issue stays open and becomes runnable once #359 puts a real watcher in
place.

🤖 Generated with Claude Code

[Gerrrt]

Checked again on 2026-09-08, from the monitoring host, with the repository's
own tool:

$ python3 scripts/check_alert_channels.py --files --live observability
  WARN  heartbeat_url points at ntfy.sh, which is a PUSH service and cannot detect absence.

So this is where #359 left it: #359 closed on detecting the gap (the
deploy-time warning above, #382), not on filling it. There is still no
external watcher, and the first two boxes here still have nothing to observe.
The heartbeat itself is healthy — five pings per 25 minutes to a topic nobody
is waiting on.

The whole procedure, in order, for the day it is run

Everything below is off this host or needs a person watching a phone; none of
it is a repository change.

1. Create the watcher. One check on a service that lives off this host and
   expects a ping — healthchecks.io, Cronitor, Dead Man's Snitch, or a
   self-hosted Uptime Kuma push monitor; check_alert_channels.py --live
   recognises all of them and will turn the warning into a pass. Period 5m,
   grace 15m, named so a 3am notification explains itself.
2. Point its notification somewhere that is not this stack. Not this
   stack's webhook, not any of its four ntfy topics. Email is fine; it fails
   independently. A watcher that reports through the thing it watches cannot
   report its own failure — that is No dead man's switch on the notification path #67 one level up.
3. Install the URL. make secrets-edit from the main checkout,
   ALERTMANAGER_HEARTBEAT_URL → the new check's ping URL, then
   make render && make reload. make up's deploy-time check should now say
   PASS for the heartbeat destination; the check should go green within one
   repeat_interval (5m).
4. Pick the window. Not Sunday 03:30 (homelab-backup-volumes stops
   Prometheus and the heartbeat with it, and the result is unreadable). Near
   the phone, not on the clock: the estate has no alert delivery for the
   ~15 minutes the switch is stopped.
5. Trip it. docker stop alertmanager from the main checkout; wait out the
   grace; the check must report DOWN and its notification must arrive on the
   channel from step 2. docker start alertmanager; it must return green
   within one repeat_interval.
6. The 24h half, in the same sitting. Lower repeat_interval on the
   second Watchdog route in alertmanager.yaml (the one to default,
   currently 24h), make reload, confirm the Watchdog notification arrives
   on the normal ntfy channel, put it back, make reload. Mind the runbook's
   own warning: a grace under 10m turns every make reload into a page, so do
   this after step 5 has settled, not during it.
7. Write the date down. docs/runbooks/verify-the-alert-path.md, in the
   house banner form the other runbooks use —
   > **Done YYYY-MM-DD.** … — replacing the This has not been done
   paragraph, and naming which watcher service and where its notification
   goes (not the URL). That line is what stops this being re-litigated.

A recommendation on splitting it

Step 6 is runnable today without a watcher, and I chose not to run it alone.
Real alerts already arrive on that channel daily, so it proves little on its
own; and a runbook that says "half of the drill was done on the 8th and the
other half later" is exactly the kind of record that gets re-litigated. One
sitting, one date. If you would rather have the 24h half proven now, it is a
five-minute job from the main checkout and I can do it on request.

🤖 Generated with Claude Code