chore(secrets): point the heartbeat at a watcher that expects it (#407)

docs/observability.md

@@ -532,30 +532,35 @@ Neither half substitutes for the other. The heartbeat proves delivery to a
 notification travels the identical URL your warnings travel, but nothing
 machine-checks its absence.
 
-**The heartbeat half is configured and is not yet a dead man's switch, and this
-paragraph is the honest version of the table above.** All four receivers point
-at `ntfy.sh`, the heartbeat included. ntfy is a push service: it delivers what it
-is sent and has no notion of an expected interval, so it cannot notice a ping
-that never arrived — and absence is the entire signal. The pings are being
-delivered to a topic nobody is waiting on
-([#359](https://github.com/Gerrrt/HomeLab/issues/359)).
-
-So today, if Prometheus stops evaluating, Alertmanager dies, or this host loses
-outbound network, **nothing external notices**. That is the failure
-[#214](https://github.com/Gerrrt/HomeLab/issues/214) lived through from the other
-direction, and the heartbeat is cited as the answer to it in #214's own
-resolution — an answer that is wired up but not yet armed.
-
-`check_alert_channels.py --live` now reports it on every deploy, classifying the
-heartbeat's destination host as a watcher or a push service. It is a **warning
-rather than a failure** on purpose: it cannot be fixed from this repository — it
-needs an account on a watcher service and a decision about where its notification
-goes — and a deploy-time check that is permanently red for a known reason stops
-being read, which this repository has already written down about
-`.gitleaksignore`. Closing it is four steps in
+**The heartbeat half became a dead man's switch on 2026-09-09.** Until then all
+four receivers pointed at `ntfy.sh`, the heartbeat included, and ntfy is a push
+service: it delivers what it is sent and has no notion of an expected interval,
+so it cannot notice a ping that never arrived — and absence is the entire
+signal. The pings were being delivered to a topic nobody was waiting on
+([#359](https://github.com/Gerrrt/HomeLab/issues/359)), and they were also
+spending almost all of ntfy.sh's free daily budget, so real alerts were refused
+at the end of every day ([#407](https://github.com/Gerrrt/HomeLab/issues/407)).
+The heartbeat now pings a healthchecks.io check, period 5m and grace 15m, which
+emails when a ping does not arrive; the three real channels stay on ntfy and
+have the budget to themselves.
+
+So now, if Prometheus stops evaluating, Alertmanager dies, or this host loses
+outbound network, something external notices — in principle. That is the
+failure [#214](https://github.com/Gerrrt/HomeLab/issues/214) lived through from
+the other direction, and the heartbeat is cited as the answer to it in #214's
+own resolution. It is armed and not yet proven: nobody has watched the check go
+red, which is [#288](https://github.com/Gerrrt/HomeLab/issues/288).
+
+`check_alert_channels.py --live` reports the destination on every deploy,
+classifying the heartbeat's host as a watcher or a push service. A push service
+is a **warning rather than a failure** on purpose: it cannot be fixed from this
+repository — it needs an account on a watcher service and a decision about where
+its notification goes — and a deploy-time check that is permanently red for a
+known reason stops being read, which this repository has already written down
+about `.gitleaksignore`. It read `ntfy.sh` and warned from 2026-09-07 to
+2026-09-09; it reads `hc-ping.com` and passes since. That was the four steps in
 [`runbooks/verify-the-alert-path.md`](runbooks/verify-the-alert-path.md), and
-[#288](https://github.com/Gerrrt/HomeLab/issues/288)'s drill becomes runnable
-once it is done.
+[#288](https://github.com/Gerrrt/HomeLab/issues/288)'s drill is runnable now.
 
 The watcher lives off this host by necessity — a watcher here fails at the same
 moment as the thing it is watching. Setting it up, the coupling between

docs/runbooks/verify-the-alert-path.md

@@ -33,12 +33,19 @@ travel, but nothing machine-checks its absence — you do.
 
 ## Setting up the external watcher
 
-> **This has not been done.** As of 2026-09-07 the heartbeat URL points at
-> `ntfy.sh` exactly as every other receiver does, and ntfy cannot detect absence — so
-> there is no dead man's switch, only a heartbeat nobody is waiting on
-> ([#359](https://github.com/Gerrrt/HomeLab/issues/359)). `make up` warns about
-> it on every deploy. The steps below are what closes it, and step 2 is the one
-> that matters.
+> **Done 2026-09-09.** One check on healthchecks.io, period 5m, grace 15m,
+> notifying by email — not through this stack's ntfy topics, and not through
+> ntfy.sh at all, since the whole house shares one public address and one free
+> daily budget there ([#407](https://github.com/Gerrrt/HomeLab/issues/407)).
+> The ping URL is in `ALERTMANAGER_HEARTBEAT_URL`; `check_alert_channels.py
+> --live` reads its destination as `hc-ping.com`, a service that watches for
+> absence, and passes. From 2026-09-07 to this date the URL pointed at `ntfy.sh`
+> like every other receiver, and there was no dead man's switch, only a
+> heartbeat nobody was waiting on
+> ([#359](https://github.com/Gerrrt/HomeLab/issues/359)). **The drill below has
+> still not been run** — that is [#288](https://github.com/Gerrrt/HomeLab/issues/288),
+> and until it has, "the check goes red when the stack dies" is a claim, not an
+> observation.
 
 The watcher has to live somewhere other than the monitoring host. A watcher on
 this host fails at the same moment as the thing it is watching, which is not

secrets/observability.sops.yaml

@@ -30,7 +30,7 @@ SNMP_COMMUNITY_ILO: ENC[AES256_GCM,data:RMutxt3mfxMlthO8IQp2wA9x54+SLQuF,iv:fPtL
 ALERTMANAGER_WEBHOOK_URL: ENC[AES256_GCM,data:e7vSRxtJxyxmoYpyxFrnNEGwPUosVY03nyLkJTn6+OmJHRZD6g/r06gOA2RcLqJm,iv:7bfgIKZ0rlapBZiTIjoJGCa+GOCACFqk4wtd3lORJFs=,tag:tjegmtIVAydI+I109g8BFw==,type:str]
 ALERTMANAGER_URGENT_WEBHOOK_URL: ENC[AES256_GCM,data:t3Tia1YDbC4Ks3SdCIm1YhvycwDDTR11kc7eB0mlCq3Eo3X/h1wpyA==,iv:+IZtM/SSJvNrYaOUEymKzURPAbSPLaxxZgTcTlNq1xE=,tag:UY81yVGjmQwaLVJuGCvjlg==,type:str]
 ALERTMANAGER_SECURITY_WEBHOOK_URL: ENC[AES256_GCM,data:wwPyCXZN+O/tQ/aZlTi36z5uZKbm7h2rQ8BjMJM6LQfViU6y7JWBjw==,iv:nQPbRF5w/K08jbhPRDHLbWizcpp63cEDXzckWutGla4=,tag:IK5CqwoUT6GL7qFQ23hTxQ==,type:str]
-ALERTMANAGER_HEARTBEAT_URL: ENC[AES256_GCM,data:KTLoTl4sCi5PHeOFJeYXGigQccHemGQN99ZQ39W+HCh1oPkJ3QsxXw==,iv:kGeOct/7KsqpVasUa8hZ80oWZe1l+L+4X2s/gmJo18k=,tag:KPyCldYWwL7QosHgVNOGlQ==,type:str]
+ALERTMANAGER_HEARTBEAT_URL: ENC[AES256_GCM,data:MLyUW0Il41PdYdQGJ8gP3/1tS5hF7RuFK6s+gC5uGjonhzR+oci17pfUmxWGKR8gq8v2kPgR9oc=,iv:R6MFSHEa8VlsMubSwnssfWnNV4rZj1o4KnV87yr5ZhU=,tag:2MkqtXOL7nZOZ3BJxxZ9Ag==,type:str]
 #ENC[AES256_GCM,data:ibikwgcTWigPNcgg0t3XIZXw9hH35iOL0wCrZT+igu9AwKq4gnI6jXkB6uoGtiVkbrZcHKi4i3flEG8=,iv:4Io1MYaGR3B2hBwC850ZcCYS+JbefgpkwIPtgkdTgcE=,tag:4GPC0EJpVymFIYU/oItxgQ==,type:comment]
 GRAFANA_RENDERER_TOKEN: ENC[AES256_GCM,data:Di7QsFgUnTpafqP8HMDBC8VcrjrU6co9,iv:6NvJz7n2UxORywAd/PvjmEf8hpeTWyqTXMyJ6GuITw0=,tag:fkskNXA66wNSyhLOmCUJcg==,type:str]
 sops:
@@ -48,8 +48,8 @@ sops:
             emRhdUUyUGdHMWFlSVE3alNuekp3YTQKAN5Dt+v1PnZlFDZgN2NQ09dGiIfMr1S9
             pIUfYyxRjC2qM5prt3Lhfcp0qvvvCsrm857wvxteXNb1jAfGHQoQHQ==
             -----END AGE ENCRYPTED FILE-----
-    lastmodified: "2026-08-30T04:28:03Z"
-    mac: ENC[AES256_GCM,data:hOLZByVet9cieIoH9nmhcUViHg/i57k3ItVMU8geFKDWpYb3zX4suvCfD3FDTrTsHTDtKyd2Eq2rpfzUKarIcgYCKlc7OroQHj2nLi7ZGh0PxCpRCUyE30kWRzeUmOPS4YcOhoV7w5AJ2iZRSFLLgzarzhSwCB3ESywZ7hLoBeQ=,iv:K/UvOIpqNAHM8zAkv8WdQq4Z9CkybRcdDI3MmE93CKY=,tag:6OBgHOI8Tjc2mCeUIPELQA==,type:str]
+    lastmodified: "2026-09-09T02:41:05Z"
+    mac: ENC[AES256_GCM,data:Yupg0xGZOP/NzoQcMUmQYf7i/0lyYR42Ik5xx4VM2s30PuLgwj0wxzhec+3Oljr5Q9GgILUWrpI7x14KZcSUKjteFqgSqvJdncMUGw4ZYGsWueKocyZXswlbspEM3S6VM9rd/bUu9f20Z/aLjdPxAQqyis3uNrVUmQlqEwYLGYE=,iv:Z4UEAOsx6Qn2sAFv+ION4gFh9KNuylhI46EKHk72uqo=,tag:sKDfmTE7d12effAC4Rcfww==,type:str]
     pgp: []
     unencrypted_suffix: _unencrypted
     version: 3.9.4