feat(alerts): AlertmanagerNotificationsFailing names the reason, and has a test (#407) - #410
Merged
Merged
Conversation
…has a test (#407) On 2026-09-08 this rule fired with reason="rateLimited": every receiver was reachable and answering, and ntfy.sh was refusing with 429 because the day's free message budget was spent. The summary said "cannot deliver via webhook", which reads as an unreachable URL and sent the reader to the wrong place. The reason label was the one thing that told the two failure modes apart, so it is now in the summary, and the description says what each reason means before the reader goes looking for a URL. The rule had no unit test — the file header listed it among the four that "are validated for syntax only, which is exactly the standing #63 had". It has one now: the firing case is the incident's own shape and asserts the rendered summary carries the reason; the quiet case pins the increase() form against the counter Alertmanager exports at zero forever for every integration. promtool passes; coverage counts in observability.md move to 58 tested and 20 not. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The last open item on #407:
AlertmanagerNotificationsFailingnow puts thereasonlabel in its summary —Alertmanager cannot deliver via webhook: rateLimited— and its description says what each reason means before the reader goes looking for a URL. On 2026-09-08 the reason was the only thing that distinguished "ntfy refused for quota" from "the webhook is down", and the old summary pointed at the wrong one.Also adds the unit test the rule never had (the test file's header listed it among the untested four): a firing case in the incident's shape that asserts the rendered summary, and a quiet case that pins the
increase()form against a counter Alertmanager exports at zero for every integration forever.Checks:
promtool test rulesvia the pinned image (SUCCESS),make check-docs(coverage counts 58 / 20),scripts/lint.sh.After merge:
make reloadfrom the main checkout so Prometheus re-reads the rule.Closes #407 — the heartbeat move in #408 was the fix; this was the annotation it left open.
🤖 Generated with Claude Code