From 1844d13637bd84e8f35c28b5f529546c20c870fd Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Wed, 9 Sep 2026 02:45:10 +0000 Subject: [PATCH] chore(secrets): point the heartbeat at a watcher that expects it (#407) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every receiver posted to ntfy.sh, the heartbeat included. ntfy is a push service and cannot notice a ping that never arrives (#359), and the 5-minute heartbeat was spending about 240 of ntfy.sh's 250 free messages a day, so real alerts were refused at the end of every day (#407) — nine criticals on 2026-09-08 among them. ALERTMANAGER_HEARTBEAT_URL now points at a healthchecks.io check, period 5m, grace 15m, notifying by email rather than through ntfy, since the whole house shares one public address and one ntfy budget. Set from the main checkout with make secrets-edit; re-encrypted here, never decrypted in the open. check_alert_channels.py --live reads the destination as hc-ping.com and passes. Measured after the reload: 20 webhook deliveries in two hours, zero failures, and ntfy's 429s stopped. The runbook's "this has not been done" banner becomes "done 2026-09-09", and observability.md says the switch is armed and not yet proven — the drill is #288, which this unblocks. Refs #288, #359, #407. Co-Authored-By: Claude Fable 5.1 --- docs/observability.md | 51 ++++++++++++++------------ docs/runbooks/verify-the-alert-path.md | 19 +++++++--- secrets/observability.sops.yaml | 6 +-- 3 files changed, 44 insertions(+), 32 deletions(-) diff --git a/docs/observability.md b/docs/observability.md index 0e01e6e..675a6db 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -532,30 +532,35 @@ Neither half substitutes for the other. The heartbeat proves delivery to a notification travels the identical URL your warnings travel, but nothing machine-checks its absence. -**The heartbeat half is configured and is not yet a dead man's switch, and this -paragraph is the honest version of the table above.** All four receivers point -at `ntfy.sh`, the heartbeat included. ntfy is a push service: it delivers what it -is sent and has no notion of an expected interval, so it cannot notice a ping -that never arrived — and absence is the entire signal. The pings are being -delivered to a topic nobody is waiting on -([#359](https://github.com/Gerrrt/HomeLab/issues/359)). - -So today, if Prometheus stops evaluating, Alertmanager dies, or this host loses -outbound network, **nothing external notices**. That is the failure -[#214](https://github.com/Gerrrt/HomeLab/issues/214) lived through from the other -direction, and the heartbeat is cited as the answer to it in #214's own -resolution — an answer that is wired up but not yet armed. - -`check_alert_channels.py --live` now reports it on every deploy, classifying the -heartbeat's destination host as a watcher or a push service. It is a **warning -rather than a failure** on purpose: it cannot be fixed from this repository — it -needs an account on a watcher service and a decision about where its notification -goes — and a deploy-time check that is permanently red for a known reason stops -being read, which this repository has already written down about -`.gitleaksignore`. Closing it is four steps in +**The heartbeat half became a dead man's switch on 2026-09-09.** Until then all +four receivers pointed at `ntfy.sh`, the heartbeat included, and ntfy is a push +service: it delivers what it is sent and has no notion of an expected interval, +so it cannot notice a ping that never arrived — and absence is the entire +signal. The pings were being delivered to a topic nobody was waiting on +([#359](https://github.com/Gerrrt/HomeLab/issues/359)), and they were also +spending almost all of ntfy.sh's free daily budget, so real alerts were refused +at the end of every day ([#407](https://github.com/Gerrrt/HomeLab/issues/407)). +The heartbeat now pings a healthchecks.io check, period 5m and grace 15m, which +emails when a ping does not arrive; the three real channels stay on ntfy and +have the budget to themselves. + +So now, if Prometheus stops evaluating, Alertmanager dies, or this host loses +outbound network, something external notices — in principle. That is the +failure [#214](https://github.com/Gerrrt/HomeLab/issues/214) lived through from +the other direction, and the heartbeat is cited as the answer to it in #214's +own resolution. It is armed and not yet proven: nobody has watched the check go +red, which is [#288](https://github.com/Gerrrt/HomeLab/issues/288). + +`check_alert_channels.py --live` reports the destination on every deploy, +classifying the heartbeat's host as a watcher or a push service. A push service +is a **warning rather than a failure** on purpose: it cannot be fixed from this +repository — it needs an account on a watcher service and a decision about where +its notification goes — and a deploy-time check that is permanently red for a +known reason stops being read, which this repository has already written down +about `.gitleaksignore`. It read `ntfy.sh` and warned from 2026-09-07 to +2026-09-09; it reads `hc-ping.com` and passes since. That was the four steps in [`runbooks/verify-the-alert-path.md`](runbooks/verify-the-alert-path.md), and -[#288](https://github.com/Gerrrt/HomeLab/issues/288)'s drill becomes runnable -once it is done. +[#288](https://github.com/Gerrrt/HomeLab/issues/288)'s drill is runnable now. The watcher lives off this host by necessity — a watcher here fails at the same moment as the thing it is watching. Setting it up, the coupling between diff --git a/docs/runbooks/verify-the-alert-path.md b/docs/runbooks/verify-the-alert-path.md index 5afd272..692d416 100644 --- a/docs/runbooks/verify-the-alert-path.md +++ b/docs/runbooks/verify-the-alert-path.md @@ -33,12 +33,19 @@ travel, but nothing machine-checks its absence — you do. ## Setting up the external watcher -> **This has not been done.** As of 2026-09-07 the heartbeat URL points at -> `ntfy.sh` exactly as every other receiver does, and ntfy cannot detect absence — so -> there is no dead man's switch, only a heartbeat nobody is waiting on -> ([#359](https://github.com/Gerrrt/HomeLab/issues/359)). `make up` warns about -> it on every deploy. The steps below are what closes it, and step 2 is the one -> that matters. +> **Done 2026-09-09.** One check on healthchecks.io, period 5m, grace 15m, +> notifying by email — not through this stack's ntfy topics, and not through +> ntfy.sh at all, since the whole house shares one public address and one free +> daily budget there ([#407](https://github.com/Gerrrt/HomeLab/issues/407)). +> The ping URL is in `ALERTMANAGER_HEARTBEAT_URL`; `check_alert_channels.py +> --live` reads its destination as `hc-ping.com`, a service that watches for +> absence, and passes. From 2026-09-07 to this date the URL pointed at `ntfy.sh` +> like every other receiver, and there was no dead man's switch, only a +> heartbeat nobody was waiting on +> ([#359](https://github.com/Gerrrt/HomeLab/issues/359)). **The drill below has +> still not been run** — that is [#288](https://github.com/Gerrrt/HomeLab/issues/288), +> and until it has, "the check goes red when the stack dies" is a claim, not an +> observation. The watcher has to live somewhere other than the monitoring host. A watcher on this host fails at the same moment as the thing it is watching, which is not diff --git a/secrets/observability.sops.yaml b/secrets/observability.sops.yaml index 3600fc4..66ca678 100644 --- a/secrets/observability.sops.yaml +++ b/secrets/observability.sops.yaml @@ -30,7 +30,7 @@ SNMP_COMMUNITY_ILO: ENC[AES256_GCM,data:RMutxt3mfxMlthO8IQp2wA9x54+SLQuF,iv:fPtL ALERTMANAGER_WEBHOOK_URL: ENC[AES256_GCM,data:e7vSRxtJxyxmoYpyxFrnNEGwPUosVY03nyLkJTn6+OmJHRZD6g/r06gOA2RcLqJm,iv:7bfgIKZ0rlapBZiTIjoJGCa+GOCACFqk4wtd3lORJFs=,tag:tjegmtIVAydI+I109g8BFw==,type:str] ALERTMANAGER_URGENT_WEBHOOK_URL: ENC[AES256_GCM,data:t3Tia1YDbC4Ks3SdCIm1YhvycwDDTR11kc7eB0mlCq3Eo3X/h1wpyA==,iv:+IZtM/SSJvNrYaOUEymKzURPAbSPLaxxZgTcTlNq1xE=,tag:UY81yVGjmQwaLVJuGCvjlg==,type:str] ALERTMANAGER_SECURITY_WEBHOOK_URL: ENC[AES256_GCM,data:wwPyCXZN+O/tQ/aZlTi36z5uZKbm7h2rQ8BjMJM6LQfViU6y7JWBjw==,iv:nQPbRF5w/K08jbhPRDHLbWizcpp63cEDXzckWutGla4=,tag:IK5CqwoUT6GL7qFQ23hTxQ==,type:str] -ALERTMANAGER_HEARTBEAT_URL: ENC[AES256_GCM,data:KTLoTl4sCi5PHeOFJeYXGigQccHemGQN99ZQ39W+HCh1oPkJ3QsxXw==,iv:kGeOct/7KsqpVasUa8hZ80oWZe1l+L+4X2s/gmJo18k=,tag:KPyCldYWwL7QosHgVNOGlQ==,type:str] +ALERTMANAGER_HEARTBEAT_URL: ENC[AES256_GCM,data:MLyUW0Il41PdYdQGJ8gP3/1tS5hF7RuFK6s+gC5uGjonhzR+oci17pfUmxWGKR8gq8v2kPgR9oc=,iv:R6MFSHEa8VlsMubSwnssfWnNV4rZj1o4KnV87yr5ZhU=,tag:2MkqtXOL7nZOZ3BJxxZ9Ag==,type:str] #ENC[AES256_GCM,data:ibikwgcTWigPNcgg0t3XIZXw9hH35iOL0wCrZT+igu9AwKq4gnI6jXkB6uoGtiVkbrZcHKi4i3flEG8=,iv:4Io1MYaGR3B2hBwC850ZcCYS+JbefgpkwIPtgkdTgcE=,tag:4GPC0EJpVymFIYU/oItxgQ==,type:comment] GRAFANA_RENDERER_TOKEN: ENC[AES256_GCM,data:Di7QsFgUnTpafqP8HMDBC8VcrjrU6co9,iv:6NvJz7n2UxORywAd/PvjmEf8hpeTWyqTXMyJ6GuITw0=,tag:fkskNXA66wNSyhLOmCUJcg==,type:str] sops: @@ -48,8 +48,8 @@ sops: emRhdUUyUGdHMWFlSVE3alNuekp3YTQKAN5Dt+v1PnZlFDZgN2NQ09dGiIfMr1S9 pIUfYyxRjC2qM5prt3Lhfcp0qvvvCsrm857wvxteXNb1jAfGHQoQHQ== -----END AGE ENCRYPTED FILE----- - lastmodified: "2026-08-30T04:28:03Z" - mac: ENC[AES256_GCM,data:hOLZByVet9cieIoH9nmhcUViHg/i57k3ItVMU8geFKDWpYb3zX4suvCfD3FDTrTsHTDtKyd2Eq2rpfzUKarIcgYCKlc7OroQHj2nLi7ZGh0PxCpRCUyE30kWRzeUmOPS4YcOhoV7w5AJ2iZRSFLLgzarzhSwCB3ESywZ7hLoBeQ=,iv:K/UvOIpqNAHM8zAkv8WdQq4Z9CkybRcdDI3MmE93CKY=,tag:6OBgHOI8Tjc2mCeUIPELQA==,type:str] + lastmodified: "2026-09-09T02:41:05Z" + mac: ENC[AES256_GCM,data:Yupg0xGZOP/NzoQcMUmQYf7i/0lyYR42Ik5xx4VM2s30PuLgwj0wxzhec+3Oljr5Q9GgILUWrpI7x14KZcSUKjteFqgSqvJdncMUGw4ZYGsWueKocyZXswlbspEM3S6VM9rd/bUu9f20Z/aLjdPxAQqyis3uNrVUmQlqEwYLGYE=,iv:Z4UEAOsx6Qn2sAFv+ION4gFh9KNuylhI46EKHk72uqo=,tag:sKDfmTE7d12effAC4Rcfww==,type:str] pgp: [] unencrypted_suffix: _unencrypted version: 3.9.4