Skip to content

Dispatcher: give up on ntfy 404s at once, and on rows that never progress - #377

Merged
SiteRelEnby merged 2 commits into
mainfrom
fix/notification-dispatch-followups
Nov 3, 2026
Merged

SiteRelEnby merged 2 commits into
mainfrom
fix/notification-dispatch-followups

Conversation

@SiteRelEnby

Copy link
Copy Markdown
Contributor

Closes #331, the two follow-ups from #326.

ntfy 404 is permanent

Publishing to an ntfy topic creates it, so ntfy itself never answers a publish with 404. One means the configured server URL does not route to an ntfy publish endpoint (wrong path, a proxy in front, something that is not ntfy), which no retry is going to change. The handler now classes 404 and 410 as permanent, the same as a webhook that has gone away: the channel is disabled on the spot and its owner told, instead of a day of backoff before the continuous-failure rule gives up. Auth, rate-limit and topic-name errors stay transient, since the owner can fix those.

Dead-lettering rows that never progress

A row whose processing raises is neither delivered nor dropped: its lease expires and the claim query picks it up again, forever, with the traceback in the logs and nothing else changing. The claim query now notices when a row comes back with an expired lease (the only way a row returns without having recorded an outcome, since a clean delivery failure releases its claim and schedules its retry) and increments a new reclaim_count column. Past NOTIFICATIONS_DEAD_LETTER_AFTER_RECLAIMS (default 5) the row is given up on: delivered_at set the way _drop does it, the reason in last_error, an error log carrying the row and channel ids as the pointer back to the per-attempt tracebacks, and a new sheaf_notifications_outbox_dead_lettered_total counter. A worker that dies mid-batch bumps the count on its rows too, which is fine: five consecutive deaths on one row is itself the signal.

Migration adds the integer column with a constant default and the usual lock timeout. Documented in METRICS.md and the changelog.

Verification

  • Pure tests for the ntfy classification: 404 and 410 permanent; 401, 403, 413, 429, 500, 502 transient; 200 success.
  • A database test against the stack inserts four outbox rows for a real channel (fresh, stale lease below the cap, stale lease at the cap, and a cleanly retrying row with failed attempts) and runs the claim query: the first two and the retrying row are claimed with the expected counts, the one at the cap is terminal with the reason recorded and is not handed back.
  • Existing dispatcher, aggregation and enum-migration tests pass on the rebuilt stack. Ruff clean.

…ress

Two follow-ups to the channel auto-disable work.

An ntfy publish cannot get a 404 from ntfy itself, since publishing to
a topic creates it; a 404 means the configured server URL does not reach
an ntfy publish endpoint at all. It was classed transient and retried
for a day before the failure rule gave up. It is now permanent, the same
as a vanished webhook: the channel is disabled and the owner told. Other
4xx answers stay transient because the owner can fix them.

A row whose processing raised was neither delivered nor dropped, so its
claim lease expired and it was re-claimed at every expiry forever with
the error only in the logs. The claim query now counts each re-claim of
an expired lease in a new reclaim_count column (a clean delivery failure
releases its claim and never matches), and past
NOTIFICATIONS_DEAD_LETTER_AFTER_RECLAIMS, default 5, the row is marked
done with the reason in last_error, logged with its row and channel ids,
and counted in sheaf_notifications_outbox_dead_lettered_total.
@SiteRelEnby
SiteRelEnby enabled auto-merge November 3, 2026 23:40
@SiteRelEnby
SiteRelEnby merged commit d557fc7 into main Nov 3, 2026
24 checks passed
@SiteRelEnby
SiteRelEnby deleted the fix/notification-dispatch-followups branch November 3, 2026 23:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

notification cleanup followups

1 participant