Skip to content

fix(daemon): log delivery failures dropped without a DLQ sink - #615

Merged
mostafa merged 5 commits into
mainfrom
fix/dlq-silent-drop
Oct 18, 2026
Merged

mostafa merged 5 commits into
mainfrom
fix/dlq-silent-drop

Conversation

@mostafa

@mostafa mostafa commented Oct 18, 2026

Copy link
Copy Markdown
Member

Fixes #586.

When a sink exhausted its retries and --dlq was unset (the default), the daemon counted the dropped result and logged "DLQ entry counted but no sink configured" once, at debug. Every later dropped alert left only a counter increment.

Changes

  • The DLQ writer logs dropped delivery failures at warn when no DLQ sink is configured. The first drop warns immediately; later drops are folded into at most one warning every 10 seconds, carrying a dropped count and the last error, and any remainder is flushed on shutdown.
  • A failed write to a configured DLQ sink is counted on the new rsigma_dlq_write_failures_total counter, next to the existing warning.
  • The daemon's delivery-failure bridge and rsigma-runtime's delivery worker log a failure they could not queue instead of discarding the send result.
  • Ack timing is unchanged, and the docs now describe it accurately: a sink commits a failed result by handing it to the DLQ writer, so the source ack does not wait for the DLQ sink's write. This is corrected in the delivery module docs, DeliveryFailure, Dispatcher::spawn, cli/engine/daemon.md, and guide/nats-streaming.md. The --dlq flag docs now say what happens to parse failures and delivery failures when it is unset.

Tests

  • delivery_failures_without_dlq_log_one_rate_limited_warning (daemon integration): three events go to an unreachable webhook with no --dlq, and the test asserts exactly one WARN line with dropped: 1 and the delivery error once all three failures are counted. It fails on main.
  • dropped_failure_warnings_are_rate_limited unit-tests the interval and flush logic with explicit instants.
  • terminal_failure_with_closed_dlq_counts_and_acks covers the runtime path where the DLQ receiver is gone.
  • DaemonProcess in the CLI test helpers now keeps the daemon's stderr lines so tests can assert on logs.

@mostafa
mostafa merged commit 7166e37 into main Oct 18, 2026
19 checks passed
@mostafa
mostafa deleted the fix/dlq-silent-drop branch October 18, 2026 18:35
@mostafa mostafa mentioned this pull request Oct 19, 2026
12 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Terminal delivery failures are dropped silently when no DLQ sink is configured

1 participant