ntfy.sh's free daily quota is spent by the heartbeat — alert delivery is dead at the end of every day

[Gerrrt]

Found on 2026-09-08 while checking the hosts after the #110 rack visit, from
Alertmanager's own log:

urgent/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts:
unexpected status code 429: {"code":42908,"http":429,"error":"limit reached: daily
message quota reached; increase your limits with a paid plan, see https://ntfy.sh"}

What is happening

All four receivers post to ntfy.sh from one address, so they share one
visitor budget, and ntfy.sh's published free-tier limit
is 250 messages per day. The heartbeat route alone sends one every
repeat_interval of 5m — about 240 a day, measured at 10 webhook
attempts an hour on a quiet stack. That is 96% of the budget spent on a ping
that, per #359, nothing is even waiting for. What is left for real alerts is
roughly ten messages a day.

So the last part of every day has no alert delivery at all, and any incident
earlier in the day brings that window forward. Read from
alertmanager_notifications_failed_total{reason="rateLimited"}:

Window (UTC)	Refused
2026-09-07 23:30 → 2026-09-08 00:10	46
2026-09-08 23:10 → still refusing at 23:30	57 and counting

The budget refills around 00:10 UTC. Today it ran out earlier than yesterday
because the #110 switch move at 19:01 produced a burst of resolved
notifications.

What it cost tonight

A second network loss for the monitoring host at 23:04–23:09 (rearranging at
the rack, after #110's move) fired InstanceDown for all four SNMP devices
and EndpointUnreachable for five endpoints — nine criticals to urgent.
Their notifications from 23:10 onward were refused, and
AlertmanagerNotificationsFailing fired at 23:08 and is still firing — an
alert about the delivery path that cannot travel the delivery path, which is
#214's recursion exactly. Nothing external notices, because the heartbeat
watcher #359 asks for does not exist.

check_alert_channels.py --live cannot see this: every URL is present,
readable and reachable, and a 200 an hour ago says nothing about a 429 now.

What closes it

Any one of these; the first is already the fix #359 asks for and #288 waits on:

1. Move the heartbeat off ntfy to a service that expects a ping —
   healthchecks.io, Cronitor, an Uptime Kuma push monitor. That removes ~240
   messages a day from the ntfy budget in one step and gives the estate a dead
   man's switch, which it does not have. ALERTMANAGER_HEARTBEAT_URL in the
   secret, make render && make reload. After that the three real channels
   have the whole 250 to themselves.
2. Self-hosted ntfy on the sensitive tier (Sensitive tier: ntfy, to bring the notification path in-house #136), which has no such limit —
   but it is behind Build the sensitive-tier host on the ProDesk (ADR-0034) #404, and an in-house endpoint cannot reach a phone that is
   not in the house, which Sensitive tier: ntfy, to bring the notification path in-house #136 already notes.
3. A paid ntfy plan, or a longer heartbeat repeat_interval — the latter
   trades the budget for a slower dead man's switch and reopens fix(alertmanager): heartbeat pinged at half its advertised rate (#120) #284's
   arithmetic (external grace ≥ 2 × repeat_interval).

Worth adding regardless: AlertmanagerNotificationsFailing's description
should name reason="rateLimited" as a cause to check first, because "the
webhook is unreachable" is what a reader assumes and it was not the case.

Related

• The heartbeat receiver is not a dead man's switch — nothing watches for its absence #359 — the heartbeat is not a dead man's switch (closed on detection)
• Watch the alerting dead man's switch trip, once #288 — the drill, blocked on the same missing watcher
• Every alert destination was unreachable for 10.5 hours on 2026-08-31 #214 — the recursive failure this reproduces
• fix(alertmanager): heartbeat pinged at half its advertised rate (#120) #284 — where the heartbeat cadence became 5m, which is when this started

[Gerrrt]

Fix landed on the live stack on 2026-09-09 and is in
#408: the heartbeat pings a
healthchecks.io check instead of ntfy.sh. That is option 1 above, and it does
two things at once — the ~240 messages a day come off the ntfy budget, and the
estate gets the dead man's switch #359 asked for.

Measured after make reload: 20 webhook deliveries in two hours, zero
failures, and no 429 since. The three real channels now have the whole 250 to
themselves, which at the stack's normal alert volume is not a limit anyone
will meet.

Left open on this issue, small: AlertmanagerNotificationsFailing's
description should name reason="rateLimited" as the first thing to check,
because "the webhook is unreachable" is what a reader assumes and it was not
the case here. One annotation edit; I will fold it into the next rules change
rather than open a PR for a sentence.

🤖 Generated with Claude Code