Repository navigation
Dispatcher: give up on ntfy 404s at once, and on rows that never progress - #377
Merged
Merged
Conversation
…ress Two follow-ups to the channel auto-disable work. An ntfy publish cannot get a 404 from ntfy itself, since publishing to a topic creates it; a 404 means the configured server URL does not reach an ntfy publish endpoint at all. It was classed transient and retried for a day before the failure rule gave up. It is now permanent, the same as a vanished webhook: the channel is disabled and the owner told. Other 4xx answers stay transient because the owner can fix them. A row whose processing raised was neither delivered nor dropped, so its claim lease expired and it was re-claimed at every expiry forever with the error only in the logs. The claim query now counts each re-claim of an expired lease in a new reclaim_count column (a clean delivery failure releases its claim and never matches), and past NOTIFICATIONS_DEAD_LETTER_AFTER_RECLAIMS, default 5, the row is marked done with the reason in last_error, logged with its row and channel ids, and counted in sheaf_notifications_outbox_dead_lettered_total.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #331, the two follow-ups from #326.
ntfy 404 is permanent
Publishing to an ntfy topic creates it, so ntfy itself never answers a publish with 404. One means the configured server URL does not route to an ntfy publish endpoint (wrong path, a proxy in front, something that is not ntfy), which no retry is going to change. The handler now classes 404 and 410 as permanent, the same as a webhook that has gone away: the channel is disabled on the spot and its owner told, instead of a day of backoff before the continuous-failure rule gives up. Auth, rate-limit and topic-name errors stay transient, since the owner can fix those.
Dead-lettering rows that never progress
A row whose processing raises is neither delivered nor dropped: its lease expires and the claim query picks it up again, forever, with the traceback in the logs and nothing else changing. The claim query now notices when a row comes back with an expired lease (the only way a row returns without having recorded an outcome, since a clean delivery failure releases its claim and schedules its retry) and increments a new
reclaim_countcolumn. PastNOTIFICATIONS_DEAD_LETTER_AFTER_RECLAIMS(default 5) the row is given up on:delivered_atset the way_dropdoes it, the reason inlast_error, an error log carrying the row and channel ids as the pointer back to the per-attempt tracebacks, and a newsheaf_notifications_outbox_dead_lettered_totalcounter. A worker that dies mid-batch bumps the count on its rows too, which is fine: five consecutive deaths on one row is itself the signal.Migration adds the integer column with a constant default and the usual lock timeout. Documented in METRICS.md and the changelog.
Verification