Skip to content

fix(observe): keep the flush chain alive on large requeues and cap close() at ~2s - #711

Merged
LinuxDevil merged 2 commits into
mainfrom
obs/o2-reporter-requeue
Oct 24, 2026
Merged

LinuxDevil merged 2 commits into
mainfrom
obs/o2-reporter-requeue

Conversation

@LinuxDevil

Copy link
Copy Markdown
Owner

When the observe daemon was unreachable the reporter dropped each failed batch for good (seen live: daemon stopped 3.5 s into a run, 0 of 10 runs recorded). Failed batches are now requeued at the front with exponential backoff (1 s base, 30 s cap), bounded by maxBuffered (oldest evicted and counted in dropped). The flush chain can no longer be broken by a throw, and close() keeps its ~2 s bound by aborting an in-flight regular flush and sending everything in one final attempt. Live check: 30-run agent with a 5 s outage recorded 30/30 (was 17/30).

Ticket: O2 (Forge Observe vs LangSmith evaluation follow-up)

Changes

  • fix(observe): keep the flush chain alive on large requeues and cap close() at ~2s
  • fix(observe): requeue failed ingest batches with backoff instead of dropping

Verification: focused vitest, tsc --noEmit, eslint on changed files (details in the review package); full suite and fallow run in CI.

🤖 Generated with Claude Code

LinuxDevil and others added 2 commits October 24, 2026 17:21
…ropping

A failed POST now puts its batch back at the front of the outbox, trims
to maxBuffered from the oldest end (evictions count in `dropped`), and
backs off min(2^n * 1000, 30000) ms before the next size or interval
flush. close() makes one forced final attempt and counts what it could
not send as dropped. A full buffer now evicts the oldest event instead
of rejecting the newest, so the deregister needs no special case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ose() at ~2s

- failed(): rebuild with concat instead of unshift(...batch); a batch as
  large as maxBuffered overflowed the call stack, rejected the chain link,
  and stalled every later flush with unhandled rejections.
- Each chain link now swallows its own errors, so no future throw can
  break the chain.
- maxBuffered is clamped to at least 1 so the deregister survives.
- close() aborts a regular flush still in flight (its batch is requeued
  and rides along in the final attempt) and no longer starts a size flush
  ahead of it, so the exit flush stays near its documented ~2s bound.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@LinuxDevil
LinuxDevil merged commit e008d4a into main Oct 24, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant