Bound the workerd collector's memory, ingest queue and stored spans - #5
Merged
Merged
Conversation
…nd per export Malformed exports get 400; store failures get 500 or 503, which exporters retry. Refusal counts persist in the collector's storage and include declared bytes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The workerd collector runs inside Executor's self-host process and shares its memory limit. Its memory and store are now bounded by configuration.
MOTEL_OTEL_MAX_PENDING_INGESTexports (default 16) are read at once, each at mostMOTEL_OTEL_MAX_INGEST_BYTES(default 16 MiB). Beyond that the collector answers 429 withRetry-After: 1or 413.queueFull,tooLarge,invalid,storeFailed) with its declared bytes. The counts are kept in the durable object's storage, so they survive eviction and restarts.GET /api/ingestreports them. Storage failures are logged each time with their cause; other refusals at most once a minute.MOTEL_OTEL_MAX_SPANS(new, default 1,000,000) andMOTEL_OTEL_MAX_DB_SIZE_MB, within a writer-time budget (MOTEL_OTEL_RETENTION_PASS_BUDGET_MS, default 500). The size bound is also enforced after each export, so the database file exceeds it by at most one export rather than by what arrives between passes. Before, a pass evicted at most one batch, and an Executor soak store reached 1.7 GB against a 1 GB bound.Limits: workerd owns the SQLite write-ahead log, and
PRAGMA journal_size_limit/wal_checkpointare not authorized from a worker, so the WAL beside the file is not bounded by this configuration. An exporter that exhausts its retries on 429 drops what it buffers next without sending it, and the collector cannot count that.In an Executor soak (30 minutes, ~480 spans/s, inspector sampling), the collector isolate's V8 heap on 89fbfd3 levelled at 87–92 MiB total from minute 10 to minute 25, and a forced collection left 17.5 MiB live. On 2f5bd17 it grew from 71 to 380 MiB over 14 minutes.
Tests are in the PR stacked above this one.