Repository navigation
fix(observe): cap ingest batches by bytes and drop batches the daemon refuses - #716
Merged
Merged
Conversation
An uncaught exception or unhandled rejection killed the process with the reporter's buffer unsent. The reporter now watches uncaughtExceptionMonitor (which also fires, origin 'unhandledRejection', for a rejection that crashes) and sends what it holds - the batch an in-flight POST carried included - in one request that blocks the dying process for at most 1s. An async fetch from the monitor never opens its socket (Node exits as soon as the monitor returns; measured 0 connections), so the POST runs in a throwaway worker thread while the main thread sleeps on Atomics.wait. No uncaughtException or unhandledRejection listener is added, so Node's crash output, exit code and --unhandled-rejections mode are untouched; a script with its own uncaughtException listener survives and keeps reporting. close() removes the hook. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The daemon rejects a batch over OBSERVE_MAX_BATCH_EVENTS (1000), but the outbox handed out everything pending (up to 5000): after an outage with a larger backlog every retry got a 400 and was requeued forever, and a crash with more than 1000 pending lost all of it. - outbox.take() returns at most 1000 events, oldest first; a flush keeps sending while its last batch was full, so a backlog drains at once instead of one batch per interval. close()'s final attempt shares one ~2s deadline across its batches; a failure drops it and the rest. - The crash flush sends successive <=1000-event POSTs from the one worker inside the same 1s budget; blockingPost now returns how many bodies were delivered and the undelivered rest is counted in dropped. - A capture callback (process.setUncaughtExceptionCaptureCallback: REPL, domains) counts as surviving the throw, like an uncaughtException listener. - The builtin lookup sits inside blockingPost's try; getBuiltinModule is shared from builtins.ts by identity.ts and blockingPost.ts. - docs: "up to 1 s per reporter", batch cap, paragraph fix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… refuses With content capture on by default (O6), 1000 events can pass the daemon's 5MB body cap: every retry got a 413 and was requeued forever. - Each event is serialized once when queued; a batch stops growing before its body would pass BATCH_BUDGET_BYTES (4MB), measured incrementally alongside the 1000-event cap. One event over the budget but under the hard cap goes out alone; one over the hard cap is dropped and counted on arrival (warnOnce). The POST body is joined from those strings. - A 400/413 answer drops and counts that batch instead of requeueing it. - The crash flush cuts its batches the same way; blockingPost reports per body, and the worker skips a body the server refuses instead of stopping. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Telemetry buffered when an agent process died of an uncaught exception or rejection was lost (seen live). An async send started from
uncaughtExceptionMonitornever connects before Node exits, so the crash path makes one blocking POST from a worker thread (≤1 s per reporter) while Node's original error output and exit code are kept. Processes that handle the exception themselves keep reporting. Also fixes a follow-up to #711: batches are now capped at the daemon's 1000-event and 4 MB limits (oversized events and batches the daemon refuses with 400/413 are dropped and counted instead of retried forever).Ticket: O3 (Forge Observe vs LangSmith evaluation follow-up)
Changes
Verification: focused vitest,
tsc --noEmit, eslint on changed files (details in the review package); full suite and fallow run in CI.🤖 Generated with Claude Code