Repository navigation
mailer v1.2.0: a persistent queue, MongoDB outbox, templates and an audit of message states - #4
Merged
Merged
Conversation
The queue to come computes the delay of an attempt and classifies what a transport throws as ./retry does, and fingerprints a message as ./idempotency does, so these move out of the two decorators into src/internal/, with no change of behavior: computeBackoff, retryAfterOf and isReceiptError (backoff.js); isProgrammerError and receiptOfThrown (thrown.js); fingerprintOf (fingerprint.js, moved from ./idempotency); and isWithin (path.js), which the file loader of ./templates shares with fromPath. The existing suites are the gate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…and an audit of states
The reliability layer of the roadmap, on the transport of 1.1, all
additive.
./queue: createMailQueue({ driver, audit, logger }) enqueues a message as
a JSON job, a template with its context or content rendered already,
checked when it is enqueued as createMessage checks a message; its id is
a UUID of namespace, tenant and idempotencyKey, so the same message
enqueued again is one job and another under the key is refused
(queue.key_reused), and its Message-ID and Content-IDs are fixed for
every attempt. createMailJobHandler, the worker for any driver, renders,
resolves attachments by reference, sends through the transport the
application composed, and makes of the receipt an outcome: a permanent
failure, a missing template or a bug fails at once (a queue that retries
everything sent them three times), a temporary one is retried never
sooner than Retry-After, an operator's failure after five minutes, and a
message of unknown delivery is never sent blindly again. The contracts
(QueueDriver, QueueSource, QueueOutcome, MessageState, AuditEvent,
AuditSink) are types of the root; createPollingDriver makes a driver of
a store that lends its jobs, the outbox to come, and createMemoryQueue is
one in memory for tests.
./bullmq: createBullMQDriver over the Queue and Worker the application
passes in: jobs under their id, priorities from 1 (0 runs before every
prioritized job), retention by time, the delay of each retry read back
by a backoff strategy of the worker's, failures as UnrecoverableError and
releases that spend no attempt, all by the names BullMQ reads, so it is
imported nowhere. Checked against BullMQ 6.3.16 and, in the opt-in
tests/bullmq suite, on BullMQ and Valkey.
./templates: createTemplateRenderer, a Renderer of templates by name over
EJS, Handlebars, Pug, MJML, Liquid, Nunjucks, Eta, Mustache and React
Email passed in: names never paths, variants by locale and part, the
subject in front matter, a layout, includes read through the loader, each
template compiled once, CSS inlined without fetching a stylesheet, and the
text derived. The cases where @nestjs-modules/mailer went wrong have
tests: a callback called twice, an async EJS that hung, a broken MJML,
global Handlebars helpers and partials read on every send, Pug options in
the locals, absolute template paths.
./audit: sinks of the states of a message (memory, logger, combined),
stateOf, and auditEventOfDeliveryEvent for the events of ./webhooks.
./render gains RenderOutput.subject and renderFields.
The engines, bullmq and ioredis are devDependencies for the tests; src/
requires none of them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tency store
./mongodb keeps the mail queue in a collection of the MongoDB driver the
application passes in, so a queue needs no Redis, and a job is written in
the transaction of its cause: createMongoOutbox({ collection }) is a
QueueSource, read by the polling driver of ./queue.
- A job enqueued with { transaction: session } is written in it by an
upsert, so a duplicate under its key does not abort the transaction, as
an insert failing on the key would.
- A job is lent by one atomic findOneAndUpdate (priority, then time) under
a token every later write is fenced by; a worker whose lease ran out
writes nothing, and its job goes to another as redelivered, its attempt
run again.
- A rate limit of the whole queue pauses every worker through a document of
its own; changeStream: true wakes the workers on a replica set.
- ensureIndexes() makes the indexes of the ready jobs and the leases, and a
TTL of finished jobs, which is the window of duplicates.
createMongoAuditSink writes each event once by its id, in the transaction
it is given, and orders the events of one time and attempt by their state
(rankOf in ./audit). createMongoIdempotencyStore makes each step of
IdempotencyTransport atomic across processes: a free key is taken by an
upsert only a free key meets, so of concurrent reserves one holds it.
The driver is a devDependency only: a collection in memory for the unit
suite, tsd against mongodb 7, and pnpm test:mongodb on a replica set of
one (transactions committed and aborted, workers of several processes,
leases, concurrent reserves, the change stream).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The version is 1.2.0 in package.json and src/version.js, and the [Unreleased] entry of CHANGELOG.md becomes 1.2.0 (2026-10-18), with its Tooling and comparison link. The status of README, CLAUDE.md, the docs and llms.txt names what 1.2 adds instead of calling it unreleased; the comparison gives this package 1.2.0 and its test count; and the threat model of SECURITY.md covers the jobs of the queue, the templates and the audit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The hash of a job covers every field but enqueuedAt and itself, so a field added to version 1 changed the hash an older worker recomputed, and during a rolling deploy that worker failed the job for good (queue.hash_mismatch). The fields of version 1 are now a fixed set: a field beyond it makes the job wait for a worker that reads it (queue.unsupported_version). mac, kid and trace are reserved outside the hash for the signature and the trace context. The set of fields takes ./queue from 14.4 to 14.6 KB min+gzip: its budget goes from 14.5 to 15 KB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A job carried a SHA-256 of its fields, which tells a job that changed in its store but not one that was forged: whoever can write to Redis or MongoDB can recompute it, and the worker sent their message from the application's domain. signingKeys, on the queue and its workers (and on prepareMailJob and parseMailJob), sign each job with an HMAC-SHA256 of the same canonical form (kid, mac, outside the hash) and refuse, unsent, a job none of them signed (queue.signature_invalid), compared in constant time. The first key signs and each verifies, for rotation. The keyring takes ./queue from 14.6 to 15.3 KB min+gzip: its budget goes from 15 to 15.5 KB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A job held its message in clear in Redis or MongoDB, its addresses, its context (a link of a password reset among it) and its attachments, for as long as the job was kept and in every backup. A codec of the application's (encode, decode, such as AES-GCM under its key or KMS), on the queue and its workers, seals those fields as enc; the version, the hash, by which a duplicate is still told, the signature and the trace context stay in clear, and no other field may stand beside enc. A worker that cannot open a sealed job gives it back (queue.codec_missing, queue.codec_failed) unless the codec says its failure is for good (retryable: false). The checks of the queue's options move out of createMailQueue, which had grown past 80 lines. The envelope takes ./queue from 15.3 to 15.8 KB min+gzip: its budget goes from 15.5 to 16 KB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The id of a job, a SHA-256 of its namespace, tenant and key, goes into the audit, the logs, the provider's idempotency key and the Message-ID, where addresses do not; but a key such as welcome:ada@example.com is found again from it by hashing the addresses one knows. idSecret (32 bytes at least) derives the id with an HMAC instead (toKeyedUuid). The examples keyed by a reset token now key by the id of the request, and the docs say a key is an id, never an address or a secret. ./queue stays within its budget (16.0 of 16 KB min+gzip). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
settledWithin was written twice (the polling driver, the BullMQ driver), and the ranks of priorities and the shape of a job request twice (the memory queue, the MongoDB outbox). They move to src/internal/settle.js and src/internal/priority.js; the outbox keeps its own rule that an id has no colon. The modules of their own take ./queue from 16.0 to 16.1 KB min+gzip: its budget goes from 16 to 16.5 KB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The memory queue and the MongoDB outbox were each tested on their own, so nothing kept their semantics one, and a source of an application's (an outbox in Postgres) had nothing to be held to. tests/helpers/queue/ source-contract.js is the contract as tests: the order of priorities, when a job is due, acknowledgements and failures, retries, releases that spend no attempt, leases that run out and are extended, fencing by the token, and the pause. It runs on the memory queue and on the outbox over a fake collection in pnpm test, and on a replica set in pnpm test:mongodb. The memory queue gives its source, as the outbox does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each reserve of the memory queue looked through every open job twice, once for the leases that ran out and once for the next job, so draining its default 100000 jobs was quadratic (30000 took 6.6 s). The jobs waiting are on a heap by when they are due, those due on one by the order they run in, and the leases on one by when they run out (src/internal/heap.js), stale nodes skipped by a version; the ids of finished jobs are dropped from the head of their list rather than shifted. The contract of a source holds. The heap takes ./queue from 16.1 to 16.5 KB min+gzip: its budget goes from 16.5 to 17 KB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ack, retry, fail and release of a source are fenced by the token of the lease, but the outbox ignored matchedCount and the memory queue returned nothing, so a run whose lease ran out between its send and its settling was dropped silently, though another worker held the job and could send it again. They give false when nothing was written, and the polling driver logs queue.settle_fenced at warn. QueueSource types it as boolean | void, so a source of an application's that gives nothing still works. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A job whose lease ran out was lent again with its attempt counted down, so a job that killed its worker (out of memory) or blocked its event loop came back for ever and took every worker down in turn; BullMQ has maxStalledCount, the outbox and the memory queue had nothing. Each source counts the runs that end unsettled (redeliveries, never counted down; BullMQ's stalledCounter), and the worker gives up a job past maxRedeliveries (3) before it renders, as delivery_unknown (queue.redelivery_exhausted), since one of those runs may have sent it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When extend() threw (the store out of reach), the polling driver only logged it, and the run went on past its lease: another worker took the job and sent it beside this one, the token fencing only the writes to the store, not the send. The driver keeps when the lease was last extended, and when the next keep would come after the lease runs out, it aborts the run with QueueLeaseLostError, before it sends. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ocks The outbox wrote leases and took them back by the clock of each worker, so a worker whose clock was behind wrote a lease another already saw as run out, and the job ran in both. A lease is taken back clockSkewTolerance- Milliseconds (5000) after it ran out by the taker's clock, and the lease must be more than three times longer. The opt-in suite registers its cleanup before anything that may throw, so a failed setup no longer holds the run open. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A finished job kept its data, the message with its addresses, context and
attachments (a reset link among them), for seven days in the outbox and a
day (a week if failed) in Redis, and in their backups. retainData, on the
memory queue, the outbox and the BullMQ driver, drops it once the job is
done with: 'failed' (the default) keeps the data of failed jobs only, for a
retry by hand, 'all' keeps every one and 'none' none. The version and the
hash stay, so an enqueue again is still told a duplicate. The outbox unsets
data; the BullMQ processor writes { v, hash } through job.updateData, and a
store that refuses it leaves the send acknowledged and warns
(MAILER_QUEUE_RETAIN_FAILED).
./bullmq goes from 3.4 to 3.7 KB min+gzip: its budget goes from 3.5 to 4 KB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The window in which an enqueue again is a duplicate was a day on BullMQ, a week in the outbox and 10000 jobs in memory, so the promise of "once per key" changed with the driver. dedupeWindowSeconds (a day) keeps a completed job and its key, and retainFailedSeconds (a week) a failed one, with the same names and defaults on the memory queue, the outbox (retainFinishedSeconds is gone) and BullMQ (removeOnComplete and removeOnFail, which jobOptions may still set). The outbox drops a job by a TTL on expireAt rather than finishedAt; the memory queue keeps its count bound, the oldest finished dropped first. The checks the queue and its drivers share move to src/internal/queue-options.js, so that the entries with no queue (./webhooks, ./azure, ./audit) carry none of them. ./queue goes from 16.8 to 17.2 KB min+gzip and ./mongodb from 5.5 to 5.6: their budgets go from 17 to 17.5 and from 5.5 to 6 KB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
enqueueMany threw queue.key_reused at the first job whose key was used before for another message, after the driver had enqueued every job, so the queued events of the jobs after it were never written and the caller could not tell which were enqueued. It tells every other job to the sink first, then refuses with the jobIds refused and the answers of the others; a single enqueue's refusal carries its jobId too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A partial was loaded with the partials it names, and the promise of its load was the one returned for its name: a partial that named itself, or one that named it back, waited for itself, and the render never settled, the pending promise kept until clear(). A partial is now registered as soon as it is compiled, as Handlebars finds partials when it renders, and a template walks the partials it names, and theirs, each once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Partials were registered by the name as written, so `./footer` written in a/x and in b/y was one partial, and b/y rendered the footer of a/x when a/x was compiled first. Each name a template writes is looked up from it, registered under the name of the template it resolves to, and renamed to it in the AST of the template that writes it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nsaction With a driver that writes no transaction (BullMQ, the memory queue), the queued event was told after the job was enqueued, and a strict sink that failed made the enqueue reject though the job was in the queue: a caller that retried without a key enqueued the message twice. Without a transaction the event is told first (written once by its id), so a strict sink that fails leaves nothing enqueued. A driver declares transactional (the outbox does), and one that does not refuses a transaction with a TypeError rather than ignore it, which let the job go without the application's writes it was meant to go with. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A job of a later version, or sealed for a codec the worker lacks, was retried after five minutes, each try an attempt, so during a rolling deploy longer than its attempts the job failed for good before a worker that reads it came up. It is released instead (its attempts not spent) until it expires or has waited a week; a codec that fails, which may be an outage of its key, is still retried within the attempts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The failed lookups of an environment were one list, cut to the window of each render, so of two concurrent renders one took the other's missing include and the other lost its own; and a missing include was a failure of the loader, which `ignore missing` does not skip. A lookup that finds nothing now tells Nunjucks there is no such template, which it fails on itself unless `ignore missing`, and the TemplateNotFoundError is made again from the name its error gives, with the names tried kept by name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A job whose run died was lent again and sent again, the redelivery only logged: behind IdempotencyTransport that is deduped, but a transport with no idempotency key (SES, SMTP) sent the message twice when the run before died after its send. redelivered: 'delivery_unknown' makes such a run of unknown delivery (queue.redelivered), unless the audit sink can tell that the run before died before it submitted: AuditSink.has(id), on the memory, MongoDB and combined sinks, asked for the submitted event of the attempt, whose id is deterministic. The default stays 'send'. ./queue goes from 17.2 to 17.6 KB min+gzip: its budget goes from 17.5 to 18 KB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A loader's failure came back as it was, naming no template, and under Nunjucks, which drops the error it wraps, and Liquid, which keeps it as `originalError`, it became a TemplateRenderError, which a queue fails at once; Liquid's missing include was liquidjs's ENOENT, a TemplateRenderError too. What a loader throws is now a TemplateLoadError (MAILER_TEMPLATE_LOAD) naming the template, its error the cause, retryable unless that error is a bug or says it is not, when `retryable` is left unset; Nunjucks keeps its causes (`dev: true`), Liquid's become causes, and liquidjs's last lookup, `fallback`, tells a missing template by the name written. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A job BullMQ failed itself, stalled more than maxStalledCount, was only logged (queue.job.lost), so its audit trail ended at submitted though its run may have sent it. A driver's consume() takes the sink to tell (QueueConsumeOptions.audit, which a mail queue's worker passes), and the BullMQ driver appends delivery_unknown (queue.lost) for such a job, by its correlationId, which a sealed job now keeps in clear for it. The audit guide shows how to find the messages to reconcile. ./bullmq goes from 3.8 to 4.2 KB min+gzip: its budget goes from 4 to 4.5 KB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Handlebars `noEscape`, Nunjucks `autoescape: false`, a Liquid `outputEscape` other than 'escape' (`undefined` included, which replaced the default), Eta `autoEscape: false` or `escapeFunction`, and an EJS `escape` went through to the engine, which then wrote every value of a context raw into the HTML. Each throws a TypeError unless the engine is given `allowUnescaped: true`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… audit A template that failed to render put its error message, which may hold what a helper threw, an excerpt of the template or a value of the context, into the failure of the job in the store and into the audit event. Only the template, the phase and the engine go there now; the full text goes to the log at debug, its addresses redacted. A template not found keeps its message, which names only templates. The renderer is given the job's signal (MailRenderInput.signal), so a worker that stops can stop a render that hears it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… a template The guide and the types named Liquid, with its limits, the engine for templates a tenant writes, and the engine set none, liquidjs having none of its own: a template could loop or allocate until the process fell over. parseLimit (1 MiB), renderLimit (1000 ms) and memoryLimit (1e8) are on by default, each overridden by the options of the engine. The guide has a matrix of who may write a template in each engine, the front-matter subject included. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When no job was ready, reserve read when the next one is due, sorted by
availableAt alone, which the index of the ready jobs (by priority first)
cannot give: MongoDB scanned and sorted every ready job on each empty poll
of each lane, a long schedule of delayed jobs making it O(n). ensureIndexes
makes { state, availableAt } (mailer_due); the opt-in suite checks by
explain() that the plan has no sort.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
./otel instrumented a transport alone: the worker of a queue had logs and
an audit but no span and no metric, so the backlog, the attempts and the
failures of the queue could not be graphed or alerted on, nor a send traced
back to the request that enqueued it. instrumentHandler(handler, {
tracerProvider, meterProvider, withSpan }) wraps any queue handler: a span
mailer.queue.process per run, linked to the producer's trace, and the
metrics mailer.queue.jobs (by outcome and code), mailer.queue.lag,
mailer.queue.job.duration and mailer.queue.redeliveries. The mail queue's
traceContext gives each job the producer's traceparent as its trace, outside
its hash and signature; withSpan lets the application make the spans of an
instrumented transport children of the job's, with no @opentelemetry/api
required here. The preparation of a job moves out of createMailQueue.
./otel goes from 5.4 to 6.1 KB min+gzip: its budget goes from 5.5 to 6.5
KB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Liquid, Eta and Pug look an include up, then read it, and without a cache each read asks the loader: an include deleted in between is a missing template, which no test reached. Eta's cacher keeps define and get alone, the two Eta calls, as the engine registers no template of its own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…kipped pnpm test:mongodb and pnpm test:bullmq skip their tests without a store, so a release could pass them unrun, and no CI runs them. pnpm test:stores runs both, serial, with a timeout per test, and fails when MONGODB_URL or BULLMQ_REDIS_URL is not given. It is a step of the release, by hand. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bench/outbox.js runs the outbox on a replica set (MONGODB_URL), a handler that acknowledges at once, for 1 and 4 workers of 1 and 10 lanes, with and without a change stream, and counts the round trips of each job through a proxy of the collection. The guide of the outbox gives its numbers, against those of the code before the review (4 round trips a job, 837 jobs a second for 1 x 10, now 2 and 2308), and when to take BullMQ instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CLAUDE.md still said the outbox was to implement the idempotency store, that scheduling through the queue was to come and nothing of the audit was done, and named no page of 1.2; the release skill asked for test:e2e and test:mailpit but not the stores, though CLAUDE.md said it did. Both now say what 1.2 holds, and the release process adds the gates no CI runs (test:stores, Bun and Deno where installed, the bench of the outbox), the merge before the tag on main, and the manual publish. A commit per subpath and per fix is a rule of the conventions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…be replaced The tests of the file loader that replace functions of node:fs, the first unit tests to, would fail rather than test anything on a runtime whose node:fs is not writable: they tell whether each function was replaced, and skip, saying so, where it was not. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The codes of the queue, its drivers and its worker (the failure of a job, the messages of its log records, its process warnings) were spread over the code, and a dashboard or an alert had no list to go by. The queue guide has the table, and tests/unit/codes.test.js fails on a code of the code it does not list, or one it lists that the code no longer has. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The cells of @alexify/mailer did not name the MongoDB store of keys, the spans and metrics of the queue's worker, the signed and sealed jobs or the tenants in turn, and its count of tests was that of before the review; SECURITY.md did not say that a store drops the message of a job done with, nor how the id of a job could name a person and what idSecret does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ueue keeps The example of the queue enqueued unsigned jobs, which whoever can write to Redis could forge; it signs them. A paragraph names what 1.2 adds around the queue: the message dropped once done with, sealing, the limit of runs that end unsettled, tenants in turn and the spans and metrics of the worker. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…export The fixes of the review were added to the ./queue entry one sentence after another, until it held the audit, the outbox and the drivers in one paragraph; the entry is now by subpath, ./queue in parts (jobs, the security of the store, the worker, sources and drivers, the memory queue, retention and tenants). It names the exports it left out: MAIL_JOB_NAME, MAIL_JOB_VERSION, MailQueueError, QueueLeaseLostError, MAILER_BACKOFF_TYPE, createMemoryLoader, isFinalState, stateOfDeliveryEvent and the root types of the queue and of JSON. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The coverage gate of functions (100) failed: resume() of the memory queue and of the outbox was reached only through their sources, and the size getter of the heap had no caller. The tests of stats() resume a tenant's pause through the drivers, and the getter is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fixes of ./templates made in their own worktree (claude/vigilant-keller-683183): Handlebars, Nunjucks, Liquid and EJS bugs, escaping that cannot be turned off silently, Liquid limits, bounds, timeouts and signals, telemetry and the loop rules. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The templates wave of the review, made in a worktree that was not to touch the budgets, takes ./templates from 11.6 to 15.7 KB min+gzip: the bounds of includes, depth and output, the timeout and signal of a render, the structured errors and TemplateLoadError, the shared environments of the engines, the bounded caches of the engines, the locales and the logger. Its budget goes from 12 to 16 KB, the smallest half-KB step that passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The templates wave made a render that timed out (TemplateRenderError, phase timeout) and a loader out of reach (TemplateLoadError) retryable, but settleThrown failed every error of a template code at once, before it looked at retryable, so a hung loader failed the job for good. An error that says it is retryable is retried first; a broken template, a template not found and a bug of the loader still fail the job. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The ./templates entry gains what its wave added and fixed before release: escaping that cannot be turned off silently, Liquid's limits and the bounds of a render, the timeout and signal of a render and TemplateLoadError, one rule of includes in every engine, bounded caches, locales and the logger, and the bugs of Handlebars, Nunjucks, Liquid and the layout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
createMailQueue took now as given, while every other option of the queue and its worker is checked when it is made, so a clock that was not a function failed at the first enqueue. It is checked with the others. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The driver counted the delay of a scheduled job, and the processor the time of a release with a delay, from Date.now(), while the request's enqueuedAt and availableAt came from the queue's own clock, so a queue with a clock of its own (tests, a clock kept in step otherwise) scheduled by another. createBullMQDriver and createBullMQProcessor take now, the driver passing its own to the processors it makes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The defaults (attempts, backoff, the delay of an operator's failure, redeliveries, leases and polls, retention and data, fairness, the skew of clocks, BullMQ's priorities, the lease of IdempotencyTransport and the timeout of a render) were each in the docs of their option, some only in the code. The queue guide lists them in one table, each read from the code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The entry of 1.2.0 is dated the day it is prepared for release (to move to the day of the merge if that is another). The API page names instrumentHandler of ./otel and TemplateLoadError of ./templates, and llms.txt the spans and metrics of the worker, the signed and sealed jobs and the tenants in turn. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The test of a sealed job that waited past a week for a worker with its codec dated the job by Date.now() and the handler by a fixed clock, so it passed only while the real day was within a week of the fixed one. The job is dated by the same fixed clock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A job's id, and the Message-ID and Content-IDs of a job enqueued under a key, are derived from that key so that the same key is the same job, and were UUIDs of its hash whatever generateId the queue was given: an application whose Message-IDs are ULIDs got UUIDs from its keyed jobs. deriveId(kind, seed), on the queue and prepareMailJob, makes them of a seed, a SHA-256 in hexadecimal of what they derive from (an HMAC under idSecret for the job id), in a format of the application's. It is a name of its own, not generateId: a generator of new ids (ulid()) would make the same message enqueued again another job, sent twice. The queue checks that the same seed gives the same id, and that a job id is one every driver takes (1 to 128 printable characters, no colon, not all digits) and the other ids the left half of a Message-ID. By default the ids are the UUIDs of before. ./queue goes from 18.9 to 19.3 KB min+gzip: its budget goes from 19 to 19.5 KB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…nc and load loadSync resolved the directory and the file with fs.realpathSync, which keeps the spelling it is given (its case, a short name of Windows such as the RUNNER~1 of a CI runner), and load with fsp.realpath, the system's. Whichever ran first fixed the root, so a load after a loadSync was refused as leading out of the directory. The sync path uses fs.realpathSync.native. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…it using Three tests declared their worker with `await using`, which Node 22 does not parse, so queue.test.js and queue-security.test.js failed whole on Node 22 in CI (Linux, Windows and the coverage job) while passing on Node 24. CLAUDE.md says so, and how a test compares a path with the template loader's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…o does not Deno keeps a method mocked with t.mock.method after its test, so the open that threw EACCES in one test of the file loader threw in the three after it. The helper that replaces node:fs restores every mock in t.after; CLAUDE.md says so next to the timers Deno keeps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The reliability layer of the roadmap, on the transport of 1.1, and the fixes of a code review
of it (maintainability, performance, security, scalability, resilience, governance and
observability) made before its release. Everything is additive: nothing of 1.1 changes, and the
package still has no runtime dependency (BullMQ, MongoDB, the template engines and the
OpenTelemetry providers are passed in).
New subpaths:
./queue,./bullmq,./mongodb,./templates,./audit;./otelgainsinstrumentHandler,./rendergainsrenderFields.What 1.2.0 adds
./queue:createMailQueue({ driver })withenqueue,enqueueManyandcreateWorker({ transport, renderer }). A job is JSON checked when it is enqueued, its idderived from namespace, tenant and
idempotencyKey(a UUID of their hash, or your format withderiveId(kind, seed)), its Message-ID fixed for every attempt. Theworker fails a permanent failure at once, retries a temporary one never sooner than
Retry-After, and never sends again a message of unknown delivery. A memory queue for tests../bullmq: a driver over BullMQ'sQueueandWorker, passed in../mongodb: the transactional outbox (a job written in the transaction of its cause), itsaudit sink and the store of
IdempotencyTransport../templates: a renderer of templates by name over nine engines passed in (EJS,Handlebars, Pug, MJML, Liquid, Nunjucks, Eta, Mustache, React Email).
./audit: the states of a message and their sinks, with no recipient in any event.Fixed and hardened by the review, before release
Security
signingKeys, rotated): the SHA-256 of a job tells a change,but whoever writes to Redis or MongoDB could recompute it and have the worker send their mail.
codecseals the fields of a job in its store (an envelope encryption of yours), and everydriver drops the message of a job done with (
retainData), so a reset link does not wait aweek in the store and its backups.
idSecretderives job ids with an HMAC.allowUnescaped), Liquid has limits bydefault, a render is bounded (includes, depth, output), loaders get names alone.
Resilience
maxRedeliveries(3), not lent againfor ever; a run whose lease the store does not extend stops before the lease runs out; a
settle the token fenced off is told; leases allow for the skew of workers' clocks.
redelivered: 'delivery_unknown'stops a blind second send after a submit (SES, SMTP withoutIdempotencyTransport), withAuditSink.has.enqueueManyaudits every job before it refuses a reused key; a strict audit sink no longerreports an enqueued job as failed.
render has a timeout and a signal, and a timeout or a loader outage is retried.
Performance and scalability
bench/outbox.js),an index for the next due job, a change stream that wakes one lane per ready change and is
reopened when it fails. The memory queue on heaps. Batched audit writes and BullMQ reads.
fairness: 'tenant'(high jobs first, then tenants in turn) and a rate-limit pause of onetenant (
rateLimitScope: 'tenant',resume()); one retention on every driver(
dedupeWindowSeconds,retainFailedSeconds).Observability
instrumentHandlerof./otel: a span per run, linked to the producer's trace(
traceContext), andmailer.queue.jobs,mailer.queue.lag,mailer.queue.job.duration,mailer.queue.redeliveries.stats(),failed()andretryFailed(id)on the outbox and the memory queue; jobs BullMQgives up are audited as
delivery_unknown; one table of codes, kept in step by a test.Governance and maintainability
QueueSource, run on the memory queue, a fake collection and a replica set.pnpm test:storesruns the MongoDB and BullMQ suites and fails without their stores, a manualstep of the release; CLAUDE.md, the release skill, the comparison and SECURITY.md are true to
1.2. Release and publishing stay manual; GitHub CI is unchanged.
Testing
pnpm test: 2702 tests, 2701 passing, 1 knowntodo(JMAP on Node 24).test:coverage(98/90/100),test:runtime,test:types,check:dts,lint,format:check,size --check,check:pack(346 files),docs:build, comparison drift: all pass.pnpm test:storeson a MongoDB 7 replica set and Valkey 8 (Docker): 38 and 5 passing.pnpm test:mailpit: 24 passing.bench/outbox.js: 2 round trips a job.test:bunandtest:deno(not installed here; CI runs them),test:e2e(needs theproviders' sandbox credentials).
After the merge
main; if the merge is not on 2026-10-20, move the date of[1.2.0]inCHANGELOG.md to its day.
v1.2.0on the merge commit ofmain, then push the tag.pnpm run releaseby hand (prepublishOnlyruns the gates again).🤖 Generated with Claude Code