chore(secrets): point the heartbeat at a watcher that expects it (#407) - #408
Merged
Conversation
Every receiver posted to ntfy.sh, the heartbeat included. ntfy is a push service and cannot notice a ping that never arrives (#359), and the 5-minute heartbeat was spending about 240 of ntfy.sh's 250 free messages a day, so real alerts were refused at the end of every day (#407) — nine criticals on 2026-09-08 among them. ALERTMANAGER_HEARTBEAT_URL now points at a healthchecks.io check, period 5m, grace 15m, notifying by email rather than through ntfy, since the whole house shares one public address and one ntfy budget. Set from the main checkout with make secrets-edit; re-encrypted here, never decrypted in the open. check_alert_channels.py --live reads the destination as hc-ping.com and passes. Measured after the reload: 20 webhook deliveries in two hours, zero failures, and ntfy's 429s stopped. The runbook's "this has not been done" banner becomes "done 2026-09-09", and observability.md says the switch is armed and not yet proven — the drill is #288, which this unblocks. Refs #288, #359, #407. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This was referenced Sep 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The heartbeat now pings a healthchecks.io check instead of ntfy.sh. This is the fix #359 asked for and the one that empties #407's problem in one step: ntfy's free budget goes back to the three real channels.
Secret:
ALERTMANAGER_HEARTBEAT_URLre-keyed to the check's ping URL. Set by the operator withmake secrets-edit; the encrypted file is the only thing in this diff. Value length checked (56 bytes), never printed.Verified on the live stack after
make render && make reload:check_alert_channels.py --live:heartbeat_url points at hc-ping.com, which watches for absence— PASS, where it warned before.Docs:
verify-the-alert-path.md's watcher banner becomes Done 2026-09-09 with the email-not-ntfy reasoning;observability.md's dead man's switch paragraphs say armed and not yet proven. The drill that proves it is #288, now runnable.Checks:
make check-docs,scripts/lint.sh,check_sops_rules.py,check-sops-encrypted.sh.Refs #288, #359, #407.
🤖 Generated with Claude Code