diff --git a/.claude/skills/npm-publish/SKILL.md b/.claude/skills/npm-publish/SKILL.md index 2b09346..bb5aca0 100644 --- a/.claude/skills/npm-publish/SKILL.md +++ b/.claude/skills/npm-publish/SKILL.md @@ -193,6 +193,19 @@ pnpm run lint && pnpm run format:check && pnpm run test:coverage && pnpm run tes Must exit 0. If it fails, fix and re-run. +Then run the stores, which no CI runs and a release may not skip: + +```bash +docker compose -f tests/mongodb/docker-compose.yml up -d --wait +docker compose -f tests/bullmq/docker-compose.yml up -d --wait +MONGODB_URL='mongodb://localhost:27027/?replicaSet=rs0' BULLMQ_REDIS_URL=redis://localhost:6390 pnpm test:stores +``` + +`test:stores` fails without either variable, rather than skipping. When the outbox changed, run +its bench (`MONGODB_URL=… node bench/outbox.js`) and compare with the numbers in +`docs/guide/outbox.md`. Run `pnpm test:bun` and `pnpm test:deno` where Bun and Deno are +installed. + Then ask the user whether to run the opt-in suites against the real services, which only they can: `pnpm test:e2e` with the sandbox credentials of the providers (each suite is skipped until its variables are set; `tests/e2e/providers.e2e.test.js` names those of the providers of 1.1), @@ -200,6 +213,9 @@ and `pnpm test:mailpit` against the local Mailpit. A release that changed a prov its error handling should have run that provider's suite. Report which suites ran and which were skipped. +The release is published by hand (`pnpm run release`), after the merge into `main` and the tag +on the commit of `main`; no workflow publishes. + ## Step 8: Summary & Next Steps Present a summary to the user: diff --git a/CHANGELOG.md b/CHANGELOG.md index e3ab669..4ee3bb2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,171 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [1.2.0] - 2026-10-20 + +The reliability layer of the roadmap, on the transport of 1.1: a persistent queue, on BullMQ or a +transactional outbox in MongoDB, whose workers render templates of the engine the application +uses and know a permanent failure from a temporary one, and the audit of every state of a +message. Everything is additive: nothing of 1.1 changes, and the package still has no runtime +dependency, BullMQ, MongoDB and the template engines being passed in. + +### Added + +- **`./queue`**: `createMailQueue({ driver, audit, logger })` with `enqueue`, `enqueueMany` and + `createWorker({ transport, renderer, resolveAttachment, checkJob, classify })`. + - **Jobs.** A job is a template with its context, rendered by the worker, or content rendered + already, as JSON checked when it is enqueued (`prepareMailJob`, `parseMailJob`): its fields as + `createMessage` checks them, its context refused with the path to what JSON would change, + attachments as bytes (bounded) or references the worker resolves. Its id is derived from the + namespace, tenant and `idempotencyKey`, a UUID of their hash or what `deriveId(kind, seed)` + makes of it (as for the Message-ID and Content-IDs of a job under a key; `generateId` makes + those it draws), so the same message enqueued again is one job, and another under the key is + refused (`MailQueueError` `queue.key_reused`, with the `jobIds` + refused and, from `enqueueMany`, the `answers` of the other jobs, all of them enqueued and + audited first). Its Message-ID and Content-IDs are fixed for every attempt. `scheduledAt` is a + delay of the queue and `expiresAt` a deadline. The fields of a job of version 1 + (`MAIL_JOB_VERSION`, named `MAIL_JOB_NAME`) are fixed: a later version, or a field this one + does not know, makes the job wait for a worker that reads it (`queue.unsupported_version`, + given back with its attempts not spent, for a week or until it expires) rather than fail its + hash during a rolling deploy. + - **Security of the store.** `signingKeys` sign each job with an HMAC-SHA256 (`kid`, `mac`), and + a worker given them sends no job they did not sign (`queue.signature_invalid`): the SHA-256 of + a job tells a change, but whoever writes to the store can recompute it; the first key signs and + each verifies, for rotation. A `codec` (`encode`, `decode`, such as AES-GCM under the + application's key) seals the fields of each job, its addresses, context and attachments, + leaving the version, hash, correlation id, signature and trace context in clear; a worker + without it gives a sealed job back (`queue.codec_missing`), one whose codec fails retries it + (`queue.codec_failed`). `idSecret` derives the id of a job from its key with an HMAC, so an id + in the audit and the logs cannot be traced back to a key that names a person. + - **The worker** (`createMailJobHandler`, for any driver) fails a permanent failure, a missing + template or a bug at once, retries a temporary one never sooner than `Retry-After`, an + operator's failure (`auth`, `configuration`) after five minutes, and never sends again a + message of unknown delivery. It gives up a job whose runs ended unsettled more than + `maxRedeliveries` (3) times, as `delivery_unknown`, so a job that kills or blocks its worker + does not take every worker down in turn for ever; with `redelivered: 'delivery_unknown'`, a + run again of an attempt whose run before died is of unknown delivery rather than a second send, + unless the audit sink tells (`has`) that it died before its submit. `rateLimitScope` makes a + rate limit wait for the job, the queue or the job's tenant. The renderer is given the job's + `signal`; a template that fails to render leaves only its name, phase and engine in the store + and the audit, its text in the log at `debug`. Every failure, retry and escape is logged with + its code and duration (the codes in one table of the queue guide). + - **Sources and drivers.** `createPollingDriver(source)` makes a driver of a store that lends its + jobs (an outbox): it keeps the lease of a run, aborts a run whose lease the store does not + extend before the lease runs out (`QueueLeaseLostError`), logs a settle the token fenced off + (`queue.settle_fenced`; the settling writes of a source give `false`), and wakes one lane per + change of its source. A job request carries its `tenantId` and a job gives back its + `redeliveries`, never counted down. A driver declares `transactional`; one that is not refuses + a transaction, and without one the `queued` event is told before the job, so a strict sink + that fails leaves nothing enqueued. `AuditSink.appendMany` takes the `queued` events of an + `enqueueMany` in one write. `traceContext` gives each job the producer's W3C `traceparent`. + - **The memory queue**, `createMemoryQueue()`, for tests and a single process: its jobs on heaps + (a reserve costs O(log n)), its `source` held to the same contract as the outbox + (`tests/helpers/queue/source-contract.js`), with `stats()`, `failed()`, `retryFailed(id)` and + `resume()`. + - **Retention and tenants**, the same on every driver: a completed job is kept + `dedupeWindowSeconds` (a day, its window of duplicates), a failed one `retainFailedSeconds` (a + week), and the message of a job done with is dropped (`retainData`: `'failed'` keeps the failed + ones, `'all'`, `'none'`), its version and hash kept so a duplicate is still told. + `fairness: 'tenant'` (memory queue, outbox) takes a high job of any tenant first, then the + tenants in turn, so one tenant's backlog holds no other's mail. +- **`./bullmq`**: `createBullMQDriver({ bullmq: { Queue, Worker }, connection })`, with BullMQ + passed in: jobs under their id, priorities from 1, retention by time, the delay of each retry + read back by a backoff strategy of the worker's, failures as `UnrecoverableError` and releases + that spend no attempt, delays counted from the clock it is given (`now`); a batch read back 32 + jobs at once to tell its duplicates. + `createBullMQProcessor` and `createBullMQBackoffStrategy` (of the type `MAILER_BACKOFF_TYPE`) + serve a Worker of your own. A job BullMQ gives up itself (stalled too often) is told to the + queue's audit sink as `delivery_unknown` (`queue.lost`). `pnpm test:bullmq` runs it on BullMQ + and Valkey. +- **`./mongodb`**: the transactional outbox, `createMongoOutbox({ collection })`, a `QueueSource` + in a collection of the MongoDB driver passed in. + - A job enqueued with `{ transaction: session }` is written in that transaction (an upsert, so a + duplicate does not abort it), lent by one atomic `findOneAndUpdate` under a token every later + write is fenced by, and given to another worker as `redelivered` when its lease runs out, + `clockSkewToleranceMilliseconds` (5 s) after by the taker's clock. + - An idle poll is two round trips: leases are taken back once a third of a lease, the pauses read + once a poll interval, and the next due time read from its own index; its optional change + stream wakes one lane per change after which a job may be ready, and is opened again when it + fails. `bench/outbox.js` measures it (2 round trips a job, some 2300 jobs a second for one + worker of ten lanes on a laptop). + - A pause of the queue or of one tenant (`rateLimitScope: 'tenant'`), and `resume()`; `stats()` + (jobs by state, the age of the oldest job due, the pauses), `failed({ limit, after })` and + `retryFailed(id)`; `ensureIndexes()` makes its indexes and the TTL on `expireAt`. + - `createMongoAuditSink` (each event once by its id, in the transaction; `has`, `appendMany` in + one `bulkWrite`, and `eventsOfProviderMessage(id)`, which finds the events a webhook filed + under the provider's id of a message) and `createMongoIdempotencyStore` (each step atomic, for + `IdempotencyTransport` across processes) go with it. `pnpm test:mongodb` runs them on a replica + set. +- **`./templates`**: `createTemplateRenderer({ engine, loader, layout, partials, globals, + defaultLocale, inlineCss })`, a `Renderer` of templates by name, over nine engines passed in + (`createEjsEngine`, `createHandlebarsEngine`, `createPugEngine`, `createMjmlEngine`, + `createLiquidEngine`, `createNunjucksEngine`, `createEtaEngine`, `createMustacheEngine`, + `createReactEmailEngine`): names never paths (`createFileLoader` reads nothing outside its + directory, `createMemoryLoader` holds them by name), variants by locale and part, the subject in + front matter, a layout, includes read through the loader, each template compiled once, CSS inlined + by juice or css-inline without fetching a stylesheet (`createCssInliner`), and the text derived + from the HTML. `TemplateNotFoundError` and `TemplateRenderError` have a stable `code`. + - **Safety.** An engine option that turns escaping off (Handlebars `noEscape`, Nunjucks + `autoescape: false`, a Liquid `outputEscape`, Eta `autoEscape: false` or `escapeFunction`, EJS + `escape`) throws unless the engine is given `allowUnescaped: true`. Liquid has `parseLimit` + (1 MiB), `renderLimit` (1000 ms) and `memoryLimit` (1e8) on by default, and the guide says who + may write a template in each engine. A render is bounded: `maxIncludes` (10000), + `maxIncludeDepth` (32), `maxOutputBytes` (10 MiB). A loader is given template names alone, + none with a segment starting with a dot, and the file loader reads a file through the + descriptor its size came from. + - **Time and failures.** `render({ ..., signal })` stops at the abort of its signal, with its + reason, and `renderTimeoutMilliseconds` (30000) fails a render that hangs with a retryable + `TemplateRenderError` of phase `timeout`, what it waited for read or compiled anew; + `loader.load(name, { signal })` is told when to stop. A loader that fails is a + `TemplateLoadError` (`MAILER_TEMPLATE_LOAD`), retryable unless its error is a bug; every error + tells `phase`, `engine`, `template`, `requested` and `locale`. + - **Includes.** One rule in every engine: `/x` from the root, `./x` against the template it is + written in, a bare name from the root then the partials, with the engine's extensions; EJS and + Mustache read nested relative includes against their own template, and a bare Pug include is + read from the root, then the partials (`include ./footer` for the file beside). + - **Caches and telemetry.** What each engine keeps is bounded by the renderer's cache; an EJS + include is compiled once (`ejs.cache`, per renderer, forgotten at `clear()`), and Mustache + parses a template once (it needs `mustache.Context`). `locales` names the locales templates + are written in; `logger` hears `templates.compiled` and `templates.rendered` at `debug`. + - **Fixed before release:** a Handlebars partial that names itself hung the render, and + `./footer` of two directories was one partial; concurrent Nunjucks renders took each other's + missing include, and `ignore missing` failed; Liquid's missing include was a + `TemplateRenderError`; a renderer with a layout and React Email sent every message without the + layout, which now throws when the renderer is made; the file loader refused every `load` after + a `loadSync` when its `dir` was spelled otherwise than on disk (its case, a short name of + Windows). +- **`./audit`**: the states of a message (`queued`, `rendered`, `submitted`, `accepted`, + `delivered`, `bounced`, `complained`, `failed`, `delivery_unknown`) and their sinks: + `createMemoryAuditSink`, `createLoggerAuditSink`, `combineAuditSinks` (with `has` and + `appendMany` where a sink of them has them), `stateOf`, `isFinalState`, and + `auditEventOfDeliveryEvent` and `stateOfDeliveryEvent` for the events of `./webhooks`. +- **`./otel`**: `instrumentHandler(handler, { tracerProvider, meterProvider, withSpan })`, a span + `mailer.queue.process` per run of a job, linked to the producer's trace (the `traceparent` the + mail queue's `traceContext` gives each job as `trace`), and the metrics `mailer.queue.jobs` (by + outcome and code), `mailer.queue.lag`, `mailer.queue.job.duration` and + `mailer.queue.redeliveries`. +- **`./render`**: `RenderOutput.subject` and `renderFields(renderer, input)`, which gives + `{ subject, content }`. +- **The root**: the contracts of the reliability layer as types (`QueueDriver`, `QueueSource`, + `QueueJob`, `QueueJobRequest`, `QueueLease`, `QueueOutcome`, `QueueHandler`, `QueueConsumer`, + `QueueConsumeOptions`, `QueueEnqueueOptions`, `MessageState`, `AuditEvent`, `AuditSink`, + `JsonValue`, `JsonObject`), with no code. + +### Tooling + +- **Opt-in suites on the real services:** `pnpm test:bullmq` on BullMQ and Valkey + (`tests/bullmq/docker-compose.yml`) and `pnpm test:mongodb` on a MongoDB replica set + (`tests/mongodb/docker-compose.yml`), outside the gates, as `test:mailpit` is; + `pnpm test:stores` runs both and fails without their stores, a manual step of every release. +- **`bench/outbox.js`**: the throughput of the outbox on a replica set, by workers and lanes. +- **One table of the codes of the queue** in its guide, which `tests/unit/codes.test.js` keeps in + step with the code. +- **devDependencies for the tests:** the nine template engines, juice and css-inline, BullMQ, + ioredis and the MongoDB driver; `src/` requires none of them, as `tests/unit/dependencies.test.js` + checks. +- **Shared internals:** the backoff, the classification of thrown errors, the fingerprint of a + message and the path check moved to `src/internal/`, with no change of behavior. + ## [1.1.0] - 2026-10-18 The features of three libraries of the [comparison](https://mailerat.vercel.app/guide/comparison) @@ -616,6 +781,7 @@ for code written against upyo 0.7: `./resend`, `./sendgrid`, `./mailgun`, `./plunk` and `./jmap` used to retry no 4xx at all, and waited without jitter, the first three without a cap. -[unreleased]: https://github.com/Alexis-Technologies/mailer/compare/v1.1.0...HEAD +[unreleased]: https://github.com/Alexis-Technologies/mailer/compare/v1.2.0...HEAD +[1.2.0]: https://github.com/Alexis-Technologies/mailer/compare/v1.1.0...v1.2.0 [1.1.0]: https://github.com/Alexis-Technologies/mailer/compare/v1.0.0...v1.1.0 [1.0.0]: https://github.com/Alexis-Technologies/mailer/releases/tag/v1.0.0 diff --git a/CLAUDE.md b/CLAUDE.md index 6c07249..46c3b05 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -19,15 +19,23 @@ Nodemailer, emailjs or Upyo, made of two parts: `docs/comparison.json` has: a persistent queue and outbox, an append-only audit log, one model of delivery events, suppression, idempotency and failover between providers. 1.1 has the model of delivery events with the verified webhooks of fifteen providers (`./webhooks`) and the - idempotency decorator whose store interface the outbox is to implement (`./idempotency`). + idempotency decorator (`./idempotency`), whose store the outbox of 1.2 implements. + 1.2 adds the queue (`./queue`, a driver-neutral contract, its worker and a memory queue; + `./bullmq`, the BullMQ driver; `./mongodb`, the transactional outbox with its audit sink and + idempotency store), templates by name over nine engines passed in (`./templates`), the + audit of the states of a message (`./audit`) and the spans and metrics of the worker + (`instrumentHandler` of `./otel`). Jobs are signed (HMAC) and may be sealed (a codec), their + message dropped once done with; sources hold one contract (leases, redelivery, fencing, a + limit of runs that end unsettled), with tenants in turn and a pause per tenant. It has zero runtime dependencies and runs on Node.js ≥ 22, and on Bun and Deno through their Node compatibility (see "Platform"; no browser entry). -**Status: 1.1 is the transport and the first parts of the reliability layer.** The root entry -exports the message model and the transport contract of upyo, with the delivery events of 1.1; -each transport, decorator and helper is a subpath, and their API is under semver from 1.0.0. The -queue, the audit log and suppression have no API yet; their direction is in "Roadmap" below. +**Status: 1.2 is the transport and the reliability layer on it.** The root entry exports the +message model and the transport contract of upyo, with the delivery events of 1.1 and the +contracts of the reliability layer of 1.2 as types; each transport, decorator, driver and helper +is a subpath, and their API is under semver from 1.0.0. The crypto-shredding of the audit and +suppression have no API yet; their direction is in "Roadmap" below. It is maintained by Alexis Technologies and set up like the sibling libraries `@alexify/metaschema` and `@alexify/protoarray` (the scaffold follows protoarray's @@ -45,6 +53,9 @@ pnpm test:bun # tests/unit and tests/runtime on Bun (bun te pnpm test:deno # tests/unit and tests/runtime on Deno (deno test --parallel) pnpm test:mailpit # opt-in: tests/mailpit against a local Mailpit (compose file there) pnpm test:e2e # opt-in: tests/e2e against real providers, from env variables +pnpm test:bullmq # opt-in: tests/bullmq on BullMQ and Valkey (compose file there) +pnpm test:mongodb # opt-in: tests/mongodb on a MongoDB replica set (compose file there) +pnpm test:stores # both above, failing (not skipping) without MONGODB_URL and BULLMQ_REDIS_URL node --test tests/unit/mailer.test.js # a single file node --test --test-name-pattern="Bundle" tests/unit/*.test.js # filter by name pnpm run test:coverage # c8 over src/, gated at 98 lines/statements, 90 branches, 100 functions @@ -816,7 +827,9 @@ src/ receipts (provider.js, refusal.js), message fields (headers.js, raw.js, schedule.js, tracking.js, capabilities.js, batch.js, base64.js, uuid.js), the decorators of 1.1 (gate.js, callback.js, options.js, decorator.js), - the base of the transports of 1.1 (provider-transport.js), and abort.js, + the base of the transports of 1.1 (provider-transport.js), what the + decorators and the queue share (backoff.js, thrown.js, fingerprint.js, + path.js), the driver of a QueueSource (polling-driver.js), and abort.js, iterate.js, warn.js, redact.js, redact-addresses.js, sleep.js, log.js tests/ unit/*.test.js node:test suites, one per area (-.test.js for a subpath) @@ -855,15 +868,27 @@ The root entry is the contract every subpath shares, ported from `@upyo/core` (` `retryAfterMilliseconds`; `classifyHttpStatus` and `classifyReceiptError` classify. A transport reports a failure in a receipt rather than by throwing. -The contracts of the reliability layer, still to be designed (Phase 0): - -- **`QueueDriver`**: enqueue, reserve, ack, retry with a delay; drivers for an outbox in MongoDB, - BullMQ and an in-memory one for tests. -- **`AuditSink`**: append-only records of state changes; sinks for MongoDB, stdout/OTel logs and a - custom one. +The contracts of the reliability layer (Phase 0), types of the root with no code, so that +`./queue`, `./bullmq`, `./audit` and a store's driver share them without requiring each other: + +- **`QueueDriver`**: `enqueue`, `enqueueMany`, `consume(handler) → QueueConsumer`; a + `QueueHandler` makes of each run of a job a `QueueOutcome` (`ack`, `retry` after a delay, + `fail`, `release` without spending the attempt), and never throws. BullMQ is a driver + (`./bullmq`); a store that lends its jobs (an outbox) is a **`QueueSource`** (`reserve`, + `extend`, `ack`, `retry`, `fail`, `release`, fenced by the token of the lease), which + `createPollingDriver` (`src/internal/polling-driver.js`) makes a driver of, as it does the + memory queue. The queue retries, not the transport: a worker's chain has `retries: 1` and no + `RetryTransport`. +- **A mail job** (`./queue`): JSON, `v: 1`, a template with its context or content rendered + already, its id a UUID of namespace, tenant and `idempotencyKey`, its Message-ID and + Content-IDs fixed (derived from the id under a key of the caller's), and a SHA-256 of all but + `enqueuedAt` that tells a duplicate from a key used again. +- **`AuditSink`**: `append(event, { transaction })`, append-only, an event written once by its + deterministic `id`; sinks in memory, of a logger, and combined (`./audit`), and in MongoDB + (`./mongodb`). The crypto-shredding of recipients is next. - **The states and events of a message:** `queued → rendered → submitted → accepted → delivered | bounced | complained | failed | delivery_unknown`, with correlationId, tenantId, attempt, the - provider's message id and a hash of the payload. + provider's message id and the hashes of the payload and of the message; no recipient. - **`DeliveryEvent`** (root types, 1.1), what `./webhooks` reads from a provider: one per message and recipient, its `id` the same on every redelivery, its `messageId` that of the receipt. `accepted`, `delivered`, `bounced` and `complained` are the states above; `rejected` is @@ -888,14 +913,18 @@ model of states and events. `tenantId` is in the context from the start. providers. - **A persistent queue:** the outbox in MongoDB (transactional outbox, no Redis needed) and a BullMQ driver (Valkey) for throughput and delays; priorities (auth codes before newsletters). + (Done in 1.2: `./queue`, `./bullmq` and `./mongodb`, the outbox as a `QueueSource` with its + audit sink and idempotency store, and the memory queue.) - **The audit log** of every state, with a keyed HMAC plus an encrypted block for the recipient (crypto-shredding), so the right to erasure does not fight an append-only log. 3–6 records per - message; optional hash chain for tamper evidence. + message; optional hash chain for tamper evidence. (Partly done in 1.2: `./audit` and the + MongoDB sink, events with no recipient; the crypto-shredding and the hash chain are next.) - **Idempotency:** a deterministic key (tenant + purpose + recipient + window) deduplicated in the outbox. At least once with deduplication, never a promise of exactly once. A provider without an idempotency key (SES) leads to `delivery_unknown` and reconciliation from events, not to a blind - retry. `./idempotency` (1.1) reserves keys in a store of `reserve`, `complete` and `release`, - which the outbox is to implement. + retry. `./idempotency` (1.1) reserves keys in a store of `reserve`, `complete` and `release`; + `createMongoIdempotencyStore` (1.2) is that store in MongoDB, and `redelivered: + 'delivery_unknown'` of the worker stops a blind retry where no key is taken. - **Errors classified:** permanent (5xx, rejected address) are not retried; temporary ones are, with jitter and `Retry-After`, under an attempt limit. - **The correctness suite:** golden `.eml` files, DKIM checked by an independent verifier in CI, @@ -916,8 +945,9 @@ store of buckets that processes share). **Phase 3, multi-tenant (P1–P2).** Provider credentials per tenant (bring your own SES), domains and DKIM per tenant, quotas; List-Unsubscribe one-click (RFC 8058, done in 1.1) and the split -between transactional and bulk mail; scheduled sending as a queue delay (1.1 schedules through the -providers that can); a `Renderer` interface with an optional React Email adapter (no template +between transactional and bulk mail (tenants in turn and a pause per tenant in 1.2); scheduled +sending as a queue delay (done in 1.2, `scheduledAt`; 1.1 schedules through the providers that +can); a `Renderer` interface with an optional React Email adapter (no template engine of its own; done in 1.1, `./render`); more providers (done in 1.1). **Decisions:** @@ -934,6 +964,12 @@ engine of its own; done in 1.1, `./render`); more providers (done in 1.1). more would be fifteen more parts of the API to keep under semver, for less than a kilobyte each. If that cost is ever measured to matter, the entries can come later without a change to the one there is. +- **One `./templates` entry, not one per engine.** The nine adapters weigh under a kilobyte + each, the engines are passed in, and the core (loaders, variants, front matter, cache) is most + of the entry, as the verifiers of `./webhooks` are of theirs. +- **Templates are named, never pathed, and read through the loader.** No engine reads a file + itself: includes, partials and layouts go through the renderer's loader, which holds them to + its root; the one exception, MJML's `mj-include`, is off unless asked for. - **Eighteen providers, then few new ones.** The eight HTTP providers of upyo came with the port; 1.1 adds the ten most used of unemail and sently (Postmark, Brevo, Mailjet, MailerSend, SparkPost, ZeptoMail, SMTP2GO, Azure Communication Services, Scaleway and Cloudflare). Each @@ -954,8 +990,9 @@ accepted but never read (Upyo `validateSsl`); no CRLF sanitization and `os.EOL` inbound mail; the management APIs of the providers (domains, templates, statistics), which the official SDKs cover. -**To check before designing:** whether per-tenant fairness (groups) is a paid BullMQ Pro feature; -whether Fly.io restricts outgoing SMTP (it would make HTTP-first a requirement). +**To check before designing:** whether Fly.io restricts outgoing SMTP (it would make HTTP-first +a requirement). Checked: per-tenant fairness (groups) is a feature of BullMQ Pro only; the type +definitions of BullMQ 6 have no group options. ## Platform @@ -1015,6 +1052,8 @@ separate entry with no `node:*`, and a platform test would enforce it as in prot - Built-ins with the `node:` prefix; relative requires always end in `.js`. - Self-descriptive code; comments explain *why*, not what. - Conventional Commits (`feat(scope):`, `fix:`, `docs:`, `chore:`, `ci:`, `test:`, `refactor!:`). + A commit per subpath and per fix: a feature that spans subpaths lands as a commit for each, so + that a review reads one at a time (the 1.2 commit of four subpaths, 88 files, was not). ## Skills @@ -1065,7 +1104,23 @@ change. **This file wins** if they ever disagree. /mailpit sendmail"` for the sendmail case) and `tests/e2e/` (`pnpm test:e2e`, each suite skipped until its provider's variables are set; Stalwart for JMAP in `tests/e2e/jmap/`; the providers of 1.1 in one table, `providers.e2e.test.js`, which also checks that refused credentials are a - non-retryable `auth` failure). The release skill asks for them before a release. + non-retryable `auth` failure) and `tests/bullmq/` (`pnpm test:bullmq`, serial; + `docker compose -f tests/bullmq/docker-compose.yml up -d` and + `BULLMQ_REDIS_URL=redis://localhost:6390`: duplicates, retries after `Retry-After`, priorities, + delays, a release on close and a Worker without the strategy, on BullMQ itself). + `pnpm test:stores` runs `tests/mongodb` and `tests/bullmq` and fails without their variables, + rather than skipping: it is a step of every release (see "Release process"), as GitHub CI does + not run the stores. The release skill asks for all of them before a release. +- **The template engines, BullMQ, ioredis and mongodb are devDependencies** for the tests of + `./templates`, `./bullmq` and `./mongodb` (the real engines, a fake BullMQ in + `tests/helpers/bullmq/`, a collection in memory in `tests/helpers/mongodb/`, and tsd against + the real classes); `src/` requires none of them. `tests/mongodb/` (`pnpm test:mongodb`, + serial; `docker compose -f tests/mongodb/docker-compose.yml up -d --wait` and + `MONGODB_URL='mongodb://localhost:27027/?replicaSet=rs0'`) runs the outbox, the audit sink and + the store on a replica set: transactions committed and aborted, workers of several processes, + leases, concurrent reserves of a key, the change stream, the plan of the next-due query + (`explain()`), and the contract of a source (`tests/helpers/queue/source-contract.js`), which + `pnpm test` also runs on the memory queue and a fake collection, whole and under fairness. - **Platform-dependent tests:** the mock SMTP server takes a client that has gone as no failure: a reset (`ECONNRESET`; macOS and Windows send an RST where Linux sends a FIN) and a reply written after the client closed (`EPIPE` on Linux, `ECONNABORTED` on Windows), which the 221 to @@ -1073,7 +1128,11 @@ change. **This file wins** if they ever disagree. catches a new one, and Docker (`node:22-bookworm-slim`) reproduces Linux. The JMAP test of a 401 to a raw message of unknown size, which streams, is `todo` on Node 24 (known upstream bug 14). `tests/unit/export-parity.test.js` also checks that every d.ts using `AsyncDisposable` - references `esnext.disposable` itself. + references `esnext.disposable` itself. Node 22 parses no `await using` (Node 24 does), so no + file it loads may hold one: a test closes what it opens with `t.after`, and a file with one + fails whole on Node 22 alone, in CI, never on a Node 24 at hand. A path compared with what the + loader of `./templates` gives is resolved with `fs.realpathSync.native`, as the plain one keeps + the short names of Windows (the temporary directory of a runner). - **`tests/runtime/`** runs the package as a consumer does, by its name, with no mock: every entry required and imported by name; SMTP over plain TCP, implicit TLS, STARTTLS and an HTTP CONNECT proxy, with DKIM checked by `tests/helpers/mime/verify-dkim.js`; an HTTP provider and @@ -1082,8 +1141,8 @@ change. **This file wins** if they ever disagree. they use `node:test` and `node:assert` alone, never `t.mock`. - **Bun and Deno** (`pnpm test:bun`, `pnpm test:deno`) run each file in a fresh global (`--parallel`), as `node --test` runs each in a process of its own; in one global, the mocks of - one file reach the next. Deno keeps mocked timers enabled after a test (`t.after` resets - them), a TLS server reads no SNI as `null` on Deno and `undefined` on Bun (`false` on Node), the + one file reach the next. Deno keeps mocked timers enabled and mocked methods in place after a + test (`t.after` resets and restores them, `t.mock.restoreAll()`), a TLS server reads no SNI as `null` on Deno and `undefined` on Bun (`false` on Node), the JMAP tests of a lost answer are skipped on Deno, whose `fetch` sends it again, and the import by name of `export-parity` spawns the node CLI, which Deno is not. @@ -1113,19 +1172,26 @@ published to npm, and lint/format do not cover it. examples in CommonJS on `@alexify/mailer/`. `/transports/logger` is new (pino-like logger) and `/transports/otel` passes the providers in. The pages of 1.1 are new: `/messages/render`, `/guide/webhooks`, a page per provider and decorator, and Bun and Deno in - getting started. One sidebar (`guide` in `config.mts`) covers the guide, the messages, the - transports and the decorators. + getting started. Those of 1.2 are `/guide/queue`, `/guide/outbox`, `/guide/audit`, + `/messages/templates` and `/transports/bullmq` (a "Reliability" group). One sidebar (`guide` + in `config.mts`) covers the guide, the messages, the transports and the decorators. - When behavior changes, update README and the matching docs page together. Keep example outputs in the docs true to the code. ## Release process -Manual, as in the sibling repositories. The `npm-publish` skill walks through it step by step: +Manual, as in the sibling repositories: no workflow publishes, and GitHub CI is not changed for +it. The `npm-publish` skill walks through it step by step: 1. Bump `version` in `package.json` and in `src/version.js` (`tests/unit/version.test.js` fails while they differ). 2. Turn `## [Unreleased]` in CHANGELOG.md into `## [X.Y.Z] - YYYY-MM-DD` and add its link reference. -3. Commit, then tag `vX.Y.Z`. -4. Run `pnpm run release` (`pnpm publish`, gated by `prepublishOnly`, which includes the size +3. Run the gates no CI runs: `pnpm test:stores` on the compose files of `tests/mongodb` and + `tests/bullmq`; `pnpm test:bun` and `pnpm test:deno` where they are installed; the bench of the + outbox (`MONGODB_URL=… node bench/outbox.js`) when the outbox changed; and, as before, + `pnpm test:mailpit` and the `pnpm test:e2e` suites of the providers that changed. +4. Commit, merge into `main` (a squash merge, as #2 and #3), then tag `vX.Y.Z` on the commit of + `main`, never on the feature branch. +5. Run `pnpm run release` (`pnpm publish`, gated by `prepublishOnly`, which includes the size gate and the tarball check; `publishConfig.access` is public). diff --git a/README.md b/README.md index b3cc0d2..4105e18 100644 --- a/README.md +++ b/README.md @@ -9,17 +9,21 @@ **Email for Node.js, with a reliability layer.** Its own SMTP client, MIME builder, DKIM signing and eighteen HTTP providers, and on top of them what the other libraries leave to the -application: one model of delivery events, idempotency, and next a persistent queue and outbox, an -audit log and suppression. Zero runtime dependencies. +application: a persistent queue, on BullMQ or a transactional outbox in MongoDB, whose workers +render templates of the engine you use and know a permanent failure from a temporary one, an +audit of every state of a message, one model of delivery events and idempotency; next +suppression. Zero runtime dependencies. -> **Status: 1.1 is the transport layer and the first parts of the reliability layer.** 1.0 ported +> **Status: 1.2 is the transport layer and the reliability layer on it.** 1.0 ported > the transport from [upyo](https://github.com/dahlia/upyo) 0.7.0; 1.1 adds what > [nodemailer](https://github.com/nodemailer/nodemailer), > [unemail](https://github.com/productdevbook/unemail) and > [sently](https://github.com/omqkhafi/sently) did better: sendmail, ten more HTTP providers, > attachments from paths and URLs, one-click List-Unsubscribe, verified webhooks into one model of > delivery events, and decorators for circuit breaking, rate limits, idempotency, hooks and -> mirroring. The persistent queue, the audit log and suppression are being designed. +> mirroring. 1.2 adds the queue (`./queue`, with a BullMQ driver in `./bullmq` and a +> transactional outbox in `./mongodb`), templates by name over nine engines (`./templates`) and +> the audit of the states of a message (`./audit`). Suppression is next. ### 📖 [Read the documentation →](https://mailerat.vercel.app/) @@ -35,9 +39,11 @@ audit log and suppression. Zero runtime dependencies. read for an attachment unless you ask. - **What happens after `send()`.** The webhooks of fifteen providers, each verified before it is read and taken as handled only once your handler commits it, give one model of delivery events - (delivered, bounced, complained, …). Next: a persistent queue and outbox, an append-only audit - log and suppression. None of the - [fourteen libraries compared](https://mailerat.vercel.app/guide/comparison) has the three. + (delivered, bounced, complained, …). A queue on BullMQ or a store of yours sends each message + once per key, retries what may pass and fails what will not, and an audit sink hears every state + from `queued` to `bounced`; in MongoDB, the job is written in the transaction of its cause. + Next: suppression. None of the + [fourteen libraries compared](https://mailerat.vercel.app/guide/comparison) has them. - **Sent once, or known to be unknown.** Errors are classified as permanent or temporary, and a failure after which the provider may have taken the message is marked `deliveryUnknown`: `./retry` and `./pool` never send it again. `./idempotency` reserves a key before the send, @@ -47,9 +53,10 @@ audit log and suppression. Zero runtime dependencies. golden `.eml` messages and fuzzing of headers, `multipart/related`, dot-stuffing and internationalized addresses are planned. - **Zero dependencies.** Built on `node:*` modules and `fetch`. What a part needs from your - application is passed in: the OpenTelemetry providers, a pino-like logger, the `render` of React - Email, and later the driver of a queue or a database (BullMQ, MongoDB). CommonJS with ESM named - imports and hand-written TypeScript typings, no build step. + application is passed in: the OpenTelemetry providers, a pino-like logger, BullMQ's classes, + your template engine (EJS, Handlebars, Pug, MJML, Liquid, Nunjucks, Eta, Mustache, React Email) + and the collections of MongoDB. CommonJS with ESM named imports and hand-written TypeScript + typings, no build step. ## Installation @@ -92,7 +99,9 @@ if (!receipt.successful) console.error(receipt.errorMessages); | --- | --- | | `@alexify/mailer` | messages, attachments, calendar parts, the `Transport` contract, receipts, delivery events | | `/mime` | MIME composition and DKIM signing | -| `/attachments`, `/render` | attachments from a path, a URL, a data URI or a stream; templates and React Email | +| `/attachments`, `/render` | attachments from a path, a URL, a data URI or a stream; the content of a renderer, React Email | +| `/templates` | templates by name over EJS, Handlebars, Pug, MJML, Liquid, Nunjucks, Eta, Mustache or React Email: locales, layouts, subjects, CSS inlined | +| `/queue`, `/bullmq`, `/mongodb`, `/audit` | the mail queue and its worker, the BullMQ driver, the MongoDB outbox (and its audit and idempotency stores), the states of a message and their sinks | | `/smtp`, `/sendmail`, `/jmap` | SMTP, sendmail and JMAP transports | | `/ses`, `/resend`, `/sendgrid`, `/mailgun`, `/postmark`, `/brevo`, `/mailjet`, `/mailersend`, `/sparkpost`, `/mailtrap`, `/maileroo`, `/lettermint`, `/zeptomail`, `/smtp2go`, `/azure`, `/scaleway`, `/cloudflare`, `/plunk` | HTTP providers | | `/webhooks` | verified webhooks of fifteen providers, read into delivery events | @@ -104,6 +113,44 @@ UUID) and a pino-like `logger` for what happens inside it, and every HTTP transp such as one bound to an undici `Agent` or a proxy; see [your own fetch and identifiers](https://mailerat.vercel.app/guide/getting-started#your-own-fetch-and-identifiers). +Through a queue, the worker renders the template and sends the message, and the queue retries +what may pass: + +```js +const { Queue, Worker } = require('bullmq'); +const ejs = require('ejs'); +const { createMailQueue } = require('@alexify/mailer/queue'); +const { createBullMQDriver } = require('@alexify/mailer/bullmq'); +const { createEjsEngine, createFileLoader, createTemplateRenderer } = require('@alexify/mailer/templates'); + +const queue = createMailQueue({ + driver: createBullMQDriver({ bullmq: { Queue, Worker }, connection: { host: 'localhost' } }), + // Whoever can write to Redis cannot make the worker send a message of theirs. + signingKeys: [{ id: '2026-10', secret: process.env.MAIL_JOB_KEY }], +}); +await queue.enqueue({ + kind: 'template', + template: 'reset-password', // emails/reset-password.ejs, its subject in its front matter + context: { url }, + from: 'no-reply@example.com', + to: user.email, + idempotencyKey: `reset-password:${resetRequest.id}`, // an id, never an address or a token +}); + +queue.createWorker({ + transport, + renderer: createTemplateRenderer({ + engine: createEjsEngine({ ejs }), + loader: createFileLoader({ dir: 'emails' }), + }), +}); +``` + +A job holds its message only until it is done with (`retainData`), and may be sealed by a codec +of yours; a job that kills its worker is given up rather than sent round for ever; tenants can +take turns (`fairness: 'tenant'` on the outbox), and `instrumentHandler` of `/otel` gives the +worker its spans and metrics. See the [queue guide](https://mailerat.vercel.app/guide/queue). + The same names import from ESM (`import { SmtpTransport } from '@alexify/mailer/smtp'`). The [API reference](https://mailerat.vercel.app/api/exports) lists every member, and the [guide](https://mailerat.vercel.app/guide/getting-started) every transport. diff --git a/SECURITY.md b/SECURITY.md index 77cfa7a..242bda4 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -28,9 +28,9 @@ timeline once the report is confirmed. ## Threat model -The transport layer, the delivery events and their webhooks are built; the queue and the audit -log are being designed, and this section will be completed with them. The principles all of it is -designed around: +The transport layer, the delivery events and their webhooks, the queue with its drivers, +templates and the audit of states are built; suppression and the crypto-shredding of the audit +are next. The principles all of it is designed around: - **Message content is untrusted data.** Addresses, display names, subjects, headers, file names and bodies can come from users. A CR or LF in any header field is rejected rather than encoded @@ -70,6 +70,30 @@ designed around: match, compared in constant time; within a window of time where a time is signed, and once by its id. An SNS message (SES, Scaleway) is taken only from a listed topic, signed with the certificate of an SNS host or of the trust chain the application pins. +- **A job of the queue is untrusted when it is read.** Whoever can write to its store (Redis, + MongoDB) could have written it, so the worker checks its shape and its hash before it acts on + it, and `checkJob` can refuse a sender. The hash, a SHA-256, tells a job that changed but not + one that was forged, since a writer can recompute it: with `signingKeys`, each job carries an + HMAC-SHA256 under the application's key, and the worker sends no job that one of its keys did + not sign. A job names no file or URL: an attachment is its bytes + or a reference that the application's `resolveAttachment` reads. A job's data is stored as it is, + a token of a reset link among it, unless a `codec` of the application's seals it (an envelope + encryption: the store then holds the version, the hash and the signature in clear and the + rest encrypted); once a job is done with, every driver drops its message (`retainData`, the + failed jobs' kept for a retry by hand), so a reset link does not wait in the store and its + backups. The id of a job is a hash of its key, which goes into the audit and the logs: a key + names what the message tells of, never an address or a secret, and `idSecret` derives the id + with an HMAC so that it cannot be traced back. The store is the application's to restrict + (Redis ACLs and TLS, MongoDB roles). +- **Templates are named, never pathed.** A template, include, partial or layout is read only + through the loader of the renderer, by a name with no `..` or leading `/`; the file loader + compares real paths, so a symbolic link cannot lead out of its directory. No engine reads a file + itself, MJML's `mj-include` is off unless asked for, output is escaped by default (Liquid + included), and the CSS inliner fetches no remote stylesheet. Templates are code: EJS, Eta, Pug + and Nunjucks run JavaScript, so templates that tenants write belong to Liquid, Handlebars or + Mustache. +- **The audit holds no recipient.** An event names its message by `correlationId`, with the + hashes of its job and message; the text of an error in it has its addresses redacted. - **The audit log can forget a person** (planned). Recipients are stored as a keyed hash and an encrypted block whose key can be destroyed, so a request to erase someone's data does not require editing an append-only log. diff --git a/bench/bench.js b/bench/bench.js index 4a1a33d..1c34894 100644 --- a/bench/bench.js +++ b/bench/bench.js @@ -30,7 +30,7 @@ const { const { composeMessage } = require('../src/mime/index.js'); const { inlineDataUrls } = require('../src/attachments/index.js'); const { htmlToText } = require('../src/render/index.js'); -const { fingerprintOf } = require('../src/idempotency/fingerprint.js'); +const { fingerprintOf } = require('../src/internal/fingerprint.js'); const { gatedSendMany } = require('../src/internal/gate.js'); const { createMailtrapWebhookVerifier, diff --git a/bench/outbox.js b/bench/outbox.js new file mode 100644 index 0000000..2f290e7 --- /dev/null +++ b/bench/outbox.js @@ -0,0 +1,118 @@ +/** + * The throughput of the MongoDB outbox: how many jobs a second its workers + * take and settle, and how many round trips each job costs, for 1 and 4 + * workers of 1 and 10 lanes, with and without a change stream. The handler + * acknowledges at once, so what is measured is the queue alone, not a send. + * + * Run (manual only, outside the gates): + * docker compose -f tests/mongodb/docker-compose.yml up -d --wait + * MONGODB_URL='mongodb://localhost:27027/?replicaSet=rs0' node bench/outbox.js [jobs] + * + * Each worker is an outbox of its own over the collection, as a process of + * its own would be, its caches and its change stream apart from the others. + */ + +const { randomUUID } = require('node:crypto'); +const { MongoClient } = require('mongodb'); +const { createMongoOutbox } = require('../src/mongodb/index.js'); + +const URL = process.env.MONGODB_URL; +const JOBS = Number(process.argv[2] ?? 2000); + +// A collection that counts the round trips made through it. +const counting = (collection) => { + const counts = { total: 0 }; + return { + counts, + collection: new Proxy(collection, { + get(target, property) { + const value = target[property]; + if (typeof value !== 'function') return value; + return (...args) => { + counts.total += 1; + counts[property] = (counts[property] ?? 0) + 1; + return value.apply(target, args); + }; + }, + }), + }; +}; + +const requestOf = (index) => ({ + id: randomUUID(), + name: 'mail', + data: { v: 1, n: index }, + hash: `h${index}`, + priority: index % 10 === 0 ? 'high' : 'normal', + maxAttempts: 5, + enqueuedAt: Date.now(), + unique: true, +}); + +const run = async (db, { workers, lanes, changeStream }) => { + const raw = db.collection(`outbox_${randomUUID().replaceAll('-', '')}`); + const { collection, counts } = counting(raw); + const setup = createMongoOutbox({ collection: raw }); + await setup.ensureIndexes(); + for (let index = 0; index < JOBS; index++) await setup.source.enqueue(requestOf(index)); + let done = 0; + let finished; + const all = new Promise((resolve) => { + finished = resolve; + }); + const handler = async () => { + done += 1; + if (done === JOBS) finished(); + return { type: 'ack' }; + }; + counts.total = 0; + const started = performance.now(); + const consumers = []; + for (let worker = 0; worker < workers; worker++) { + const outbox = createMongoOutbox({ collection, changeStream, pollIntervalMilliseconds: 50 }); + consumers.push(outbox.consume(handler, { concurrency: lanes })); + } + await all; + const seconds = (performance.now() - started) / 1000; + for (const consumer of consumers) await consumer.close({ timeoutMilliseconds: 1000 }); + await raw.drop(); + return { jobsPerSecond: Math.round(JOBS / seconds), roundTrips: counts.total / JOBS }; +}; + +const main = async () => { + if (URL === undefined || URL === '') { + console.error('bench/outbox.js needs MONGODB_URL: a replica set, as for pnpm test:mongodb.'); + process.exit(1); + } + const client = new MongoClient(URL); + await client.connect(); + const db = client.db(`mailer_bench_${randomUUID().replaceAll('-', '')}`); + const rows = []; + try { + for (const changeStream of [false, true]) { + for (const workers of [1, 4]) { + for (const lanes of [1, 10]) { + const result = await run(db, { workers, lanes, changeStream }); + rows.push({ workers, lanes, changeStream, ...result }); + } + } + } + } finally { + await db.dropDatabase(); + await client.close(); + } + console.log(`${JOBS} jobs, acknowledged at once\n`); + console.log('| workers | lanes | change stream | jobs/s | round trips a job |'); + console.log('| --- | --- | --- | --- | --- |'); + for (const row of rows) { + console.log( + `| ${row.workers} | ${row.lanes} | ${row.changeStream ? 'yes' : 'no'} | ` + + `${row.jobsPerSecond} | ${row.roundTrips.toFixed(1)} |`, + ); + } +}; + +main().catch((error) => { + console.error(error); + process.exit(1); +}); diff --git a/docs/.vitepress/config.mts b/docs/.vitepress/config.mts index 3fa712f..d41304f 100644 --- a/docs/.vitepress/config.mts +++ b/docs/.vitepress/config.mts @@ -68,6 +68,7 @@ const guide = [ { text: 'Composing messages', link: '/messages/compose' }, { text: 'Attachments', link: '/messages/attachments' }, { text: 'Rendering templates', link: '/messages/render' }, + { text: 'Templates by name', link: '/messages/templates' }, { text: 'Webhooks and events', link: '/guide/webhooks' }, { text: 'Calendar invitations', link: '/messages/calendar' }, { text: 'MIME composition', link: '/messages/mime' }, @@ -115,6 +116,15 @@ const guide = [ { text: 'Tee', link: '/transports/tee' }, ], }, + { + text: 'Reliability', + items: [ + { text: 'The queue', link: '/guide/queue' }, + { text: 'BullMQ', link: '/transports/bullmq' }, + { text: 'MongoDB outbox', link: '/guide/outbox' }, + { text: 'The audit', link: '/guide/audit' }, + ], + }, ]; // https://vitepress.dev/reference/site-config diff --git a/docs/api/exports.md b/docs/api/exports.md index 9890b60..b6b509a 100644 --- a/docs/api/exports.md +++ b/docs/api/exports.md @@ -41,6 +41,10 @@ model of delivery events. - **Delivery events:** the `DeliveryEvent`, `DeliveryEventType` and `DeliveryBounce` types, which `./webhooks` gives. See [webhooks and events](/guide/webhooks). +- **The reliability layer:** the `QueueDriver`, `QueueSource`, `QueueJob`, + `QueueJobRequest`, `QueueOutcome` and `QueueHandler` types every driver + shares, and `MessageState`, `AuditEvent` and `AuditSink`, as types only. See + [the queue](/guide/queue) and [the audit](/guide/audit). ## Subpaths @@ -48,7 +52,12 @@ model of delivery events. | --- | --- | --- | | `@alexify/mailer/mime` | `composeMessage`, `MimeAttachmentReplayError` | [MIME](/messages/mime) | | `@alexify/mailer/attachments` | `fromPath`, `fromUrl`, `fromDataUri`, `fromStream`, `inlineDataUrls`, `isPublicUrl` | [Attachments](/messages/attachments#files-urls-data-uris-and-streams) | -| `@alexify/mailer/render` | `renderContent`, `createReactEmailRenderer`, `htmlToText` | [Rendering templates](/messages/render) | +| `@alexify/mailer/render` | `renderContent`, `renderFields`, `createReactEmailRenderer`, `htmlToText` | [Rendering templates](/messages/render) | +| `@alexify/mailer/templates` | `createTemplateRenderer`, `createFileLoader`, `createMemoryLoader`, `createCssInliner`, `createEjsEngine`, `createHandlebarsEngine`, `createPugEngine`, `createMjmlEngine`, `createLiquidEngine`, `createNunjucksEngine`, `createEtaEngine`, `createMustacheEngine`, `createReactEmailEngine`, `TemplateNotFoundError`, `TemplateRenderError`, `TemplateLoadError` | [Templates by name](/messages/templates) | +| `@alexify/mailer/queue` | `createMailQueue`, `createMailJobHandler`, `createMemoryQueue`, `createPollingDriver`, `prepareMailJob`, `parseMailJob`, `MailQueueError`, `QueueLeaseLostError`, `MAIL_JOB_NAME`, `MAIL_JOB_VERSION` | [The queue](/guide/queue) | +| `@alexify/mailer/bullmq` | `createBullMQDriver`, `createBullMQProcessor`, `createBullMQBackoffStrategy`, `MAILER_BACKOFF_TYPE` | [BullMQ](/transports/bullmq) | +| `@alexify/mailer/mongodb` | `createMongoOutbox`, `createMongoAuditSink`, `createMongoIdempotencyStore` | [MongoDB outbox](/guide/outbox) | +| `@alexify/mailer/audit` | `createMemoryAuditSink`, `createLoggerAuditSink`, `combineAuditSinks`, `stateOf`, `stateOfDeliveryEvent`, `auditEventOfDeliveryEvent`, `isFinalState` | [The audit](/guide/audit) | | `@alexify/mailer/webhooks` | `createSesWebhookVerifier`, `createResendWebhookVerifier`, `createSendGridWebhookVerifier`, `createMailgunWebhookVerifier`, `createPostmarkWebhookVerifier`, `createMailerSendWebhookVerifier`, `createMailtrapWebhookVerifier`, `createBrevoWebhookVerifier`, `createMailjetWebhookVerifier`, `createSparkPostWebhookVerifier`, `createLettermintWebhookVerifier`, `createMailerooWebhookVerifier`, `createSmtp2goWebhookVerifier`, `createAzureWebhookVerifier`, `createScalewayWebhookVerifier`, `createMemoryReplayStore`, `WebhookVerificationError` | [Webhooks](/guide/webhooks) | | `@alexify/mailer/smtp` | `SmtpTransport` and its error classes (`SmtpResponseError`, `SmtpAuthError`, …), `isSmtpResponseProviderDetails`, `createHttpProxyConnector` | [SMTP](/transports/smtp) | | `@alexify/mailer/sendmail` | `SendmailTransport` | [Sendmail](/transports/sendmail) | @@ -74,7 +83,7 @@ model of delivery events. | `@alexify/mailer/mock` | `MockTransport` | [Mock](/transports/mock) | | `@alexify/mailer/pool` | `PoolTransport`, `RoundRobinStrategy`, `WeightedStrategy`, `PriorityStrategy`, `SelectorStrategy` | [Pool](/transports/pool) | | `@alexify/mailer/retry` | `RetryTransport`, `createRetryTransport` | [Retry](/transports/retry) | -| `@alexify/mailer/otel` | `OpenTelemetryTransport`, `createOpenTelemetryTransport`, `createEmailAttributeExtractor`, `createErrorClassifier`, `defaultErrorClassifier` | [OpenTelemetry](/transports/otel) | +| `@alexify/mailer/otel` | `OpenTelemetryTransport`, `createOpenTelemetryTransport`, `createEmailAttributeExtractor`, `createErrorClassifier`, `defaultErrorClassifier`, `instrumentHandler` | [OpenTelemetry](/transports/otel) | | `@alexify/mailer/logger` | `LoggerTransport` | [Logger](/transports/logger) | | `@alexify/mailer/breaker` | `CircuitBreakerTransport` | [Circuit breaker](/transports/breaker) | | `@alexify/mailer/ratelimit` | `RateLimitTransport` | [Rate limit](/transports/ratelimit) | diff --git a/docs/comparison.json b/docs/comparison.json index 9fc5a75..fa63a16 100644 --- a/docs/comparison.json +++ b/docs/comparison.json @@ -339,9 +339,9 @@ "text": "`signal` + timeout over the body" }, "queue": { - "mark": "🔜", - "text": "MongoDB outbox, BullMQ driver", - "note": "On the [roadmap](/guide/why#the-plan) of the reliability layer: designed, with no code yet." + "mark": "✅", + "text": "MongoDB outbox (transactional), BullMQ driver, memory queue", + "note": "Permanent failures fail at once, `Retry-After` is a minimum, and a delivery of unknown outcome is never sent blindly again; in MongoDB, a job is written in the transaction of its cause. Jobs are signed (HMAC) and may be sealed, their message dropped once done with; tenants take turns, and a job that kills its worker is given up." }, "attachments": { "mark": "✅", @@ -373,8 +373,9 @@ "text": "RFC 8058 one-click, DKIM-signed" }, "templates": { - "mark": "⚠️", - "text": "`Renderer` interface, no engine by design" + "mark": "✅", + "text": "9 engines passed in: EJS, Handlebars, Pug, MJML, Liquid, Nunjucks, Eta, Mustache, React Email", + "note": "Names, never paths; locales, layouts, front-matter subjects, CSS inlined without remote fetches, text derived." }, "reactEmail": { "mark": "✅", @@ -385,8 +386,8 @@ "text": "native: Resend, Postmark, Mailjet, Mailtrap, JMAP; concurrency elsewhere" }, "scheduled": { - "mark": "⚠️", - "text": "native in 8 providers; queue delay planned", + "mark": "✅", + "text": "queue delay; native in 8 providers", "note": "A transport that cannot schedule refuses the message before sending it, rather than sending it at once." }, "tags": { @@ -399,7 +400,7 @@ }, "idempotency": { "mark": "✅", - "text": "decorator with reserved keys + 5 providers" + "text": "decorator with reserved keys, a MongoDB store + 5 providers" }, "dkim": { "mark": "✅", @@ -465,7 +466,7 @@ }, "otel": { "mark": "✅", - "text": "decorator: spans + metrics" + "text": "decorator: spans + metrics, and those of the queue's worker" }, "hooks": { "mark": "✅", @@ -476,9 +477,9 @@ "text": "`./mock`" }, "auditLog": { - "mark": "🔜", - "text": "append-only, crypto-shredding", - "note": "On the [roadmap](/guide/why#the-plan) of the reliability layer: designed, with no code yet." + "mark": "⚠️", + "text": "state model + sinks, MongoDB among them; crypto-shredding next", + "note": "Each state of a message, from the queue to the webhooks, one event per change, written once by its id. The keyed HMAC and encrypted recipients are on the [roadmap](/guide/why#the-plan)." }, "types": { "mark": "✅", @@ -494,13 +495,13 @@ "text": "MIT" }, "version": { - "text": "1.1.0" + "text": "1.2.0" }, "lastCommit": { "text": "2026-10 (this release)" }, "tests": { - "text": "167 / ~2340" + "text": "183 / ~2690" }, "maintainer": { "text": "1 author, Alexis Technologies" diff --git a/docs/guide/audit.md b/docs/guide/audit.md new file mode 100644 index 0000000..efe2525 --- /dev/null +++ b/docs/guide/audit.md @@ -0,0 +1,123 @@ +# The audit + +`@alexify/mailer/audit` keeps what happened to each message: one event per +change of its state, from the queue to the provider's webhooks, all with its +`correlationId`. A sink is anything with `append(event)`; the package gives +one in memory, one of your logger, and the mapping of [delivery +events](/guide/webhooks). + +## States + +``` +queued ─▶ rendered ─▶ submitted ─▶ accepted ─▶ delivered │ bounced │ complained + ▲ │ + └──── queued again ◀───┤ (a retry) + ├─▶ failed (no attempt again) + └─▶ delivery_unknown (it may have been taken) +``` + +`stateOf(events)` reads the state of a message from its events whatever their +order: a `delivered` from a webhook settles a `delivery_unknown`, a `bounced` +comes after a `delivered`. `isFinalState` is true for `delivered`, `bounced`, +`complained` and `failed`. + +## An event + +```js +{ + id: '…', // the same for the same change: a sink writes it once + state: 'accepted', + at: '2026-10-18T09:00:00.000Z', + source: 'worker', // 'producer', 'worker' or 'webhook' + correlationId: '…', + jobId: '…', tenantId: 'acme', attempt: 1, + payloadHash: '…', // the job as enqueued + messageHash: '…', // the message as rendered + messageId: '…@example.com', provider: 'ses', providerMessageId: '…', + priority: 'high', template: 'reset-password', tags: ['auth'], + error: { category, code, message, retryable }, // for a failure, redacted + retryAt: '…', // for a retry +} +``` + +An event holds no recipient: the message is found by its `correlationId`. Its +`id` is the same however often it is told, and a sink that can tell whether it +holds one (`has(id)`: the memory and MongoDB sinks, and combined sinks when one +of them can) lets a worker know a run that submitted before it died. `appendMany(events)` +writes a batch in one call where a sink can (the MongoDB sink in one +`bulkWrite`); `enqueueMany` tells the sink its batch so. + +## Sinks + +```js +const { + combineAuditSinks, + createLoggerAuditSink, + createMemoryAuditSink, +} = require('@alexify/mailer/audit'); + +const audit = combineAuditSinks(createLoggerAuditSink(logger), myDatabaseSink); +``` + +- `createMemoryAuditSink({ maxEvents })` holds the events, with + `eventsOf(correlationId)` and `stateOf(correlationId)`: for tests. +- `createLoggerAuditSink(logger, { levels })` writes each event as a record + `mail.`: `failed` at error; `delivery_unknown`, `bounced` and + `complained` at warn. +- `combineAuditSinks(...sinks)` tells each in turn, whatever another threw. +- `createMongoAuditSink({ collection })` of [`./mongodb`](/guide/outbox), each + event once by its id, in the transaction of the enqueue. +- Your own: `{ append(event, { transaction }) }`. An outbox writes the + `queued` event in the transaction of the enqueue; a sink keyed by `id` + writes an event once however often it is told. + +## Webhooks + +```js +const { auditEventOfDeliveryEvent } = require('@alexify/mailer/audit'); + +const { events, commit } = await verifier.verify(request); +for (const event of events) { + const audited = auditEventOfDeliveryEvent(event); + if (audited !== undefined) await audit.append(audited); +} +await commit(); +``` + +`accepted`, `delivered`, `bounced` and `complained` are states; `rejected` is +`failed`; `deferred` keeps a message accepted; an open, a click and an +unsubscribe are engagement, not states. The queue writes `correlationId` into +the metadata of each message, which the providers give back with its events. + +A webhook's event names its message by the `correlationId` the message's +metadata gave back, which every HTTP provider of `./webhooks` returns; without +it (an SMTP relay, a provider that drops metadata) it is filed under the +provider's id of the message. The MongoDB sink's +`eventsOfProviderMessage(providerMessageId)` finds both: the events filed under +that id and those of the message whose `accepted` event carries it (an index of +`ensureIndexes()`). + +## Messages to reconcile + +A message whose last state is `submitted` long after its send, or +`delivery_unknown`, may have gone or not: its worker died after the request, +a timeout came after the provider took it, or BullMQ gave the job up after it +stalled too often (the driver tells the queue's sink `delivery_unknown`, code +`queue.lost`). Find them, then settle each from the provider's webhooks or its +dashboard rather than send it again. On MongoDB: + +```js +const stale = await db + .collection('mail_audit') + .aggregate([ + { $sort: { correlationId: 1, at: 1, attempt: 1, rank: 1 } }, + { $group: { _id: '$correlationId', state: { $last: '$state' }, at: { $last: '$at' } } }, + { + $match: { + state: { $in: ['submitted', 'delivery_unknown'] }, + at: { $lt: new Date(Date.now() - 15 * 60_000) }, + }, + }, + ]) + .toArray(); +``` diff --git a/docs/guide/comparison.md b/docs/guide/comparison.md index d28af74..d721479 100644 --- a/docs/guide/comparison.md +++ b/docs/guide/comparison.md @@ -70,7 +70,7 @@ it. Rate limiting✅ token bucket per key, FIFO, shared store✅ rateLimit in the pool❌⚠️ sendMany only✅ token bucket⚠️ pool + sendBulk❌❌—❌⚠️ contact importer only❌⚠️ 429 → error❌⚠️ adaptive retry Circuit breaker✅ one probe in half-open❌❌❌✅ half-open without a single probe⚠️ cooldownMs❌❌—❌❌❌❌❌❌ Timeout / AbortSignal✅ signal + timeout over the body⚠️ timeouts, no AbortSignal⚠️ timeout, no signal✅ signal + timeout⚠️ signal for HTTP, not SMTP⚠️ SMTP connect only⚠️ socket timeouts✅ signal on connect and send—⚠️ signal, no timeout⚠️ global signal only⚠️ timeout, no signal⚠️ 180 s timeout⚠️ hard 10 s✅ abortSignal + requestTimeout -Persistent queue (survives a restart)🔜 MongoDB outbox, BullMQ driver3❌❌❌🔜 planned❌❌⚠️ sample outbox only—❌❌❌❌❌❌ +Persistent queue (survives a restart)✅ MongoDB outbox (transactional), BullMQ driver, memory queue3❌❌❌🔜 planned❌❌⚠️ sample outbox only—❌❌❌❌❌❌ Messages Attachments✅ path, URL, data URI, stream, bytes, Blob4✅ path, URL, Buffer, stream, data-URI⚠️ path, stream, no URL⚠️ bytes, Blob, File, factory⚠️ bytes, string, URL⚠️ bytes + path⚠️ base64 only⚠️ string, bytes, Blob⚠️ base64 only⚠️ base64 or URL⚠️ base64, helper for Buffer✅ Buffer, Blob, stream, 50 MB✅ base64✅ string / Buffer✅ Uint8Array or Raw MIME Streaming large attachments⚠️ SMTP, sendmail; HTTP providers buffer⚠️ SMTP yes; DKIM and SES buffer✅⚠️ SMTP only❌❌❌❌❌❌❌⚠️ accepts a stream❌❌❌ @@ -79,44 +79,44 @@ it. iCalendar✅ every METHOD✅ icalEvent + METHOD⚠️ manual✅ every METHOD🔜 planned❌❌✅⚠️ as an attachment❌❌⚠️ as an attachment⚠️ as an attachment⚠️ as an attachment⚠️ through Raw MIME Unicode / IDN✅ punycode + SMTPUTF8✅ punycode + SMTPUTF8⚠️ Q-encoding, no punycode⚠️ SMTPUTF8, no automatic punycode⚠️ RFC 2047/2231⚠️ one RFC 2047 word⚠️ RFC 2047⚠️ SMTPUTF8, no punycode⚠️ Subject as one B-word⚠️ UTF-8 JSON⚠️ UTF-8 JSON❌❌⚠️ UTF-8 JSON⚠️ Charset, no IDN List-Unsubscribe✅ RFC 8058 one-click, DKIM-signed✅ list.unsubscribe❌⚠️ manual header✅ RFC 8058 oneClick❌⚠️ headers⚠️ headers❌⚠️ headers, topics⚠️ headers, asm⚠️ headers⚠️ headers⚠️ headers✅ ListManagementOptions -Built-in templates⚠️ Renderer interface, no engine by design❌❌❌⚠️ Renderer interface✅ {{var}} + plugins❌❌❌✅ Templates API✅ dynamic templates✅ Templates API✅ Templates API✅ Templates API✅ Templates API +Built-in templates✅ 9 engines passed in: EJS, Handlebars, Pug, MJML, Liquid, Nunjucks, Eta, Mustache, React Email5❌❌❌⚠️ Renderer interface✅ {{var}} + plugins❌❌❌✅ Templates API✅ dynamic templates✅ Templates API✅ Templates API✅ Templates API✅ Templates API React Email✅ render passed in❌❌❌✅✅❌❌❌✅❌❌❌❌❌ Batch / bulk✅ native: Resend, Postmark, Mailjet, Mailtrap, JMAP; concurrency elsewhere❌❌✅ native: Resend, Mailtrap, JMAP✅ sendBatch⚠️ sendBulk, SendGrid bug⚠️ queue per session✅—✅ up to 100, no attachments✅ personalizations⚠️ recipient-variables✅ batch + bulk API✅ batchSend, 500✅ SendBulkEmail -Scheduled sending⚠️ native in 8 providers; queue delay planned5❌❌❌⚠️ 8 drivers❌❌❌—✅✅✅ up to 3 days❌⚠️ campaigns only❌ +Scheduled sending✅ queue delay; native in 8 providers6❌❌❌⚠️ 8 drivers❌❌❌—✅✅✅ up to 3 days❌⚠️ campaigns only❌ Tags / metadata✅ metadata in 17 providers, tags in 13⚠️ headers; tags on SES only⚠️ headers⚠️ tags in 7 providers✅⚠️ headers only⚠️ headers⚠️ headers⚠️ headers✅✅✅✅✅✅ Open/click tracking✅ per message in 10 providers❌❌⚠️ config only✅ 10 drivers❌❌❌—⚠️ per domain✅✅✅⚠️ per domain✅ per-message override -Idempotency key✅ decorator with reserved keys + 5 providers❌❌⚠️ Resend, Lettermint, Maileroo✅ middleware + providers⚠️ decorator; batch bypasses it❌❌—✅❌❌❌❌❌ +Idempotency key✅ decorator with reserved keys, a MongoDB store + 5 providers❌❌⚠️ Resend, Lettermint, Maileroo✅ middleware + providers⚠️ decorator; batch bypasses it❌❌—✅❌❌❌❌❌ Security and authentication DKIM signing in the library✅ RSA + Ed25519, streaming, over-signing✅ RSA-SHA256❌✅ RSA + Ed25519, streaming✅ RSA + Ed25519, SMTP only⚠️ body canonicalization bug❌✅ RSA❌⚠️ keys through the API⚠️ whitelabel, generic⚠️ keys through the API⚠️ keys through the API⚠️ domain DNS records⚠️ Easy / BYO DKIM OAuth2 for SMTP✅ refresh, OAUTHBEARER✅ refresh + service account⚠️ bearer token only✅ refresh, OAUTHBEARER⚠️ callback, no refresh✅ refresh client❌⚠️ no refresh——————— ARC❌❌❌❌❌❌❌❌❌—————— DSN✅✅❌✅❌❌✅✅——————— Proxy✅ SMTP: HTTP CONNECT or any connect; HTTP: fetch⚠️ HTTP CONNECT, SOCKS through a module❌❌⚠️ custom fetch❌❌❌—⚠️ baseUrl only⚠️ axios proxy✅⚠️ custom fetch + undici⚠️ environment only⚠️ through requestHandler -verify() / connection check✅ SMTP, sendmail, JMAP, 13 HTTP providers6✅⚠️ VRFY, not a check⚠️ SMTP/JMAP only⚠️ isAvailable(), nominal✅❌⚠️ probe() without auth———⚠️ domain DNS only⚠️ getServer()⚠️ getAllAccounts()⚠️ GetAccount +verify() / connection check✅ SMTP, sendmail, JMAP, 13 HTTP providers7✅⚠️ VRFY, not a check⚠️ SMTP/JMAP only⚠️ isAvailable(), nominal✅❌⚠️ probe() without auth———⚠️ domain DNS only⚠️ getServer()⚠️ getAllAccounts()⚠️ GetAccount SMTP AUTH mechanismsPLAIN, LOGIN, XOAUTH2, OAUTHBEARERPLAIN, LOGIN, CRAM-MD5, XOAUTH2CRAM-MD5, LOGIN, PLAIN, XOAUTH2PLAIN, LOGIN, XOAUTH2, OAUTHBEARERPLAIN, LOGIN, CRAM-MD5, XOAUTH2PLAIN, LOGIN, CRAM-MD5, XOAUTH2PLAIN, LOGIN, CRAM-MD5PLAIN, LOGIN, CRAM-MD5, XOAUTH2——————— SMTP TLS policies✅ requireTls, SNI, CA, cert, minVersion✅ REQUIRETLS, SNI, no DANE⚠️ STARTTLS accepts any certificate✅ requireTls, CA, cert, minVersion✅ requireTLS, off by default✅ requireTLS by default with auth⚠️ falls back to plaintext + AUTH✅ tlsPolicy ×4, REQUIRETLS——————— Platform features (events, domains, provider API) -Webhooks: parsing and signature verification✅ 15 providers, verified before parsing7❌❌❌🔜 planned✅ 10 parsers, HMAC in 5❌❌—✅ standardwebhooks✅ ECDSA⚠️ CRUD, no verification⚠️ CRUD + types, no verification✅ HMAC verify❌ SNS signature not in the SDK +Webhooks: parsing and signature verification✅ 15 providers, verified before parsing8❌❌❌🔜 planned✅ 10 parsers, HMAC in 5❌❌—✅ standardwebhooks✅ ECDSA⚠️ CRUD, no verification⚠️ CRUD + types, no verification✅ HMAC verify❌ SNS signature not in the SDK Inbound mail❌ out of scope❌❌❌🔜 planned❌❌❌—✅⚠️ parser with a bug⚠️ routes✅✅❌ legacy client-ses only -Suppression / bounce API🔜 per tenant, before the queue3❌❌❌🔜 planned❌❌❌—✅⚠️ generic client✅✅✅✅ +Suppression / bounce API🔜 per tenant, before the queue9❌❌❌🔜 planned❌❌❌—✅⚠️ generic client✅✅✅✅ Domain management (API)❌ out of scope❌❌❌❌❌❌❌—✅⚠️ generic client✅✅✅✅ identities Address validation⚠️ RFC 5322/6531 syntax⚠️ parser only⚠️ parser only⚠️ RFC 5322/6531 syntax⚠️ regex⚠️ regex❌❌ syntax assert❌❌❌✅ /v4/address/validate❌❌✅ GetEmailAddressInsights Sending log on the provider side—————————✅ logs API⚠️ generic /v3/messages✅ logs API⚠️ events, opens, clicks⚠️ emailLogs⚠️ event destinations + archiving Provider API coverage—————————18 namespaces, ~117 callsonly mail/send typed; ~251 endpoints in USAGE.md21 clients, ~155 calls85 methods~126 calls116 commands Multi-channel: SMS, WhatsApp, push❌ out of scope❌❌❌❌✅ SMS, WhatsApp, webpush, FCM❌❌—❌❌❌❌❌❌ Observability and extensibility -OpenTelemetry✅ decorator: spans + metrics❌❌✅ decorator: spans + metrics⚠️ withLogger⚠️ hooks + consoleObserver❌❌❌❌❌❌❌❌⚠️ middlewareStack +OpenTelemetry✅ decorator: spans + metrics, and those of the queue's worker❌❌✅ decorator: spans + metrics⚠️ withLogger⚠️ hooks + consoleObserver❌❌❌❌❌❌❌❌⚠️ middlewareStack Hooks / middleware✅ ./hooks, decorators, a pino-like logger✅ use(compile|stream)⚠️ logger function⚠️ decorators, LogTape✅ Middleware pipeline✅ hooks + plugins⚠️ logger✅ onEvent⚠️ dump()⚠️ console.error⚠️ setClient❌❌❌✅ logger + middlewareStack Mock / test mode✅ ./mock✅ jsonTransport, streamTransport, Ethereal⚠️ Message#read()✅ @upyo/mock✅ mock + Mailpit⚠️ preview, Mailpit❌❌—❌✅ sandboxMode✅ testmode⚠️ Sandbox server✅ Sandbox API⚠️ TestRenderEmailTemplate -Local audit log / event journal🔜 append-only, crypto-shredding3❌❌❌❌❌❌⚠️ events, not stored—❌❌❌❌❌❌ +Local audit log / event journal⚠️ state model + sinks, MongoDB among them; crypto-shredding next10❌❌❌❌❌❌⚠️ events, not stored—❌❌❌❌❌❌ Quality, license, maturity TypeScript types✅ hand-written, tsd✅ built in✅✅✅✅✅✅✅✅✅ hand-written .d.ts✅✅✅✅ generated Runtime dependencies0000–1 (@noble/hashes)0000422+ (axios)3 (axios)02 (axios, qs)9 Module formatCJS, ESM importESM + CJSESM onlyESM + CJS + JSRESM onlyESM onlyESM + CJSESM + CJS + JSRESM + CJSESM + CJSCJS onlyESM + CJSCJS onlyCJS onlyESM + CJS LicenseMITMIT-0MITMITMITMITMITMITMITMITMITMITMITMITApache-2.0 -Version (2026-10-07)1.1.010.0.165.0.30.7.0 (unreleased; release 0.6.0)0.7.01.2.21.2.10.8.53.0.24 (tag 3.0.28)6.32.18.1.614.0.25.1.04.10.0 (HEAD ahead)3.1147.0 +Version (2026-10-07)1.2.010.0.165.0.30.7.0 (unreleased; release 0.6.0)0.7.01.2.21.2.10.8.53.0.24 (tag 3.0.28)6.32.18.1.614.0.25.1.04.10.0 (HEAD ahead)3.1147.0 Last commit2026-10 (this release)2026-10-072026-08-262026-09-092026-09-022026-08-192025-11-152026-09-082026-01-282026-10-062025-09-192026-10-072026-07-102026-10-072026-10-06 -Tests: files / cases167 / ~234061 / ~133015 / ~153109 / ~130040 / ~77366 / ~5324 / ~18914 / ~2586 / ~3039 / ~44143 / ~45049 files23 (16 integration)55 files2 smoke (the rest in the monorepo) +Tests: files / cases183 / ~269061 / ~133015 / ~153109 / ~130040 / ~77366 / ~5324 / ~18914 / ~2586 / ~3039 / ~44143 / ~45049 files23 (16 integration)55 files2 smoke (the rest in the monorepo) Owner / maintenance risk1 author, Alexis Technologies1 maintainerno SECURITY.md1 main author1 author1 author, 0.1 → 1.2 in ~80 days1 author1 author, 0.xno SECURITY.mdvendorvendor, old toolchainvendorvendorvendorvendor @@ -127,11 +127,14 @@ it. 1. Through npm:@alexify/mailer. Deno’s fetch sends a request again when a reused connection closes before its answer, so the docs give it a fetch of its own (known issues). 2. A failure after which the provider may have taken the message is marked deliveryUnknown: ./retry does not retry it and ./pool does not fail over with it. -3. On the roadmap of the reliability layer: designed, with no code yet. +3. Permanent failures fail at once, Retry-After is a minimum, and a delivery of unknown outcome is never sent blindly again; in MongoDB, a job is written in the transaction of its cause. Jobs are signed (HMAC) and may be sealed, their message dropped once done with; tenants take turns, and a job that kills its worker is given up. 4. Nothing is read implicitly: fromPath and fromUrl are called by the application, fromUrl checks every redirect against allowUrl (no private address by default) and bounds time and bytes. -5. A transport that cannot schedule refuses the message before sending it, rather than sending it at once. -6. The other five have no request that checks a sending key without sending a message. -7. A signature where the provider signs (SNS, Svix, ECDSA, HMAC), its Basic credentials or token where it does not; a window of time, and an id kept as taken only once the handler commits its events. +5. Names, never paths; locales, layouts, front-matter subjects, CSS inlined without remote fetches, text derived. +6. A transport that cannot schedule refuses the message before sending it, rather than sending it at once. +7. The other five have no request that checks a sending key without sending a message. +8. A signature where the provider signs (SNS, Svix, ECDSA, HMAC), its Basic credentials or token where it does not; a window of time, and an id kept as taken only once the handler commits its events. +9. On the roadmap of the reliability layer: designed, with no code yet. +10. Each state of a message, from the queue to the webhooks, one event per change, written once by its id. The keyed HMAC and encrypted recipients are on the roadmap. diff --git a/docs/guide/getting-started.md b/docs/guide/getting-started.md index 355ebfe..bc886de 100644 --- a/docs/guide/getting-started.md +++ b/docs/guide/getting-started.md @@ -4,7 +4,8 @@ Version 1.0 is the transport layer, ported from [upyo](https://github.com/dahlia/upyo) 0.7.0; 1.1 adds sendmail, ten more HTTP providers, verified webhooks into delivery events, and decorators for circuit breaking, rate limits, idempotency, hooks and -mirroring. The queue and the audit log are still being designed. +mirroring; 1.2 adds [the queue](/guide/queue) on BullMQ or a MongoDB outbox, +[templates by name](/messages/templates) and [the audit](/guide/audit). ::: ## Installation @@ -214,7 +215,8 @@ their own unless the part is given a store they share: | Part | What it keeps | In one process | Shared by several | | --- | --- | --- | --- | | [`./ratelimit`](/transports/ratelimit) | the token buckets | at most `maxKeys` buckets | a `store` of `take()`, such as a script on Redis, so that N processes send no more than the one limit | -| [`./idempotency`](/transports/idempotency) | the keys reserved and the receipts | `createMemoryIdempotencyStore()` | a store of `reserve`, `complete` and `release`, the outbox to come among them | +| [`./idempotency`](/transports/idempotency) | the keys reserved and the receipts | `createMemoryIdempotencyStore()` | a store of `reserve`, `complete` and `release`, such as [`createMongoIdempotencyStore()`](/guide/outbox) | +| [`./queue`](/guide/queue) | the jobs | `createMemoryQueue()` | a driver they share: [BullMQ](/transports/bullmq), or the [MongoDB outbox](/guide/outbox) | | [`./webhooks`](/guide/webhooks#time-and-replays) | the ids of the requests taken | `createMemoryReplayStore()` | a store of `reserve`, `commit` and `release`; required, `null` to keep none | | [`./breaker`](/transports/breaker) | the state of the circuit | one per transport | none: each process opens its own circuit on the failures it sees | | [`./pool`](/transports/pool) | the turn of round-robin | one per transport | none | diff --git a/docs/guide/outbox.md b/docs/guide/outbox.md new file mode 100644 index 0000000..239eb1b --- /dev/null +++ b/docs/guide/outbox.md @@ -0,0 +1,175 @@ +# MongoDB outbox + +`@alexify/mailer/mongodb` keeps the [mail queue](/guide/queue) in a collection +of MongoDB: the transactional outbox. A job is written in the transaction of +your own writes, so a message is queued exactly when what it tells of is +committed, with no Redis. The same subpath keeps the [audit](/guide/audit) and +the keys of [`IdempotencyTransport`](/transports/idempotency) in MongoDB. The +package depends on no MongoDB driver: you pass in its collections. + +```js +const { randomUUID } = require('node:crypto'); +const { MongoClient } = require('mongodb'); +const { createMailQueue } = require('@alexify/mailer/queue'); +const { IdempotencyTransport } = require('@alexify/mailer/idempotency'); +const { + createMongoAuditSink, + createMongoIdempotencyStore, + createMongoOutbox, +} = require('@alexify/mailer/mongodb'); + +const client = new MongoClient(process.env.MONGODB_URL); +const db = client.db('app'); + +const outbox = createMongoOutbox({ collection: db.collection('mail_outbox'), changeStream: true }); +const audit = createMongoAuditSink({ collection: db.collection('mail_audit') }); +const store = createMongoIdempotencyStore({ collection: db.collection('mail_idempotency') }); +await Promise.all([outbox.ensureIndexes(), audit.ensureIndexes(), store.ensureIndexes()]); + +const queue = createMailQueue({ driver: outbox, audit }); + +// The producer: the job and its audit event go in the transaction. +const session = client.startSession(); +await session.withTransaction(async () => { + const resetId = randomUUID(); // the key names the request, never its token + await db.collection('resets').insertOne({ _id: resetId, userId: user.id, token }, { session }); + await queue.enqueue( + { + kind: 'template', + template: 'reset-password', + context: { url }, + from: 'no-reply@example.com', + to: user.email, + idempotencyKey: `reset-password:${resetId}`, + }, + { transaction: session }, + ); +}); + +// A worker, in as many processes as you like. +queue.createWorker({ + transport: new IdempotencyTransport(new SesTransport({ ...config, retries: 1 }), { store }), + renderer: templates, + concurrency: 5, +}); +``` + +If the transaction aborts, neither the job nor its `queued` event is written. +A job enqueued again under its key in the transaction is a duplicate, and does +not abort it: the outbox writes with an upsert, never an insert that would +fail on the key. + +## How workers share it + +The outbox is a `QueueSource`, read by the polling driver of `./queue`: + +- a job is lent by one atomic `findOneAndUpdate`, the next ready one by + priority, then time, under a token of its own, for the lease + (`leaseMilliseconds`, a minute), which the worker extends while it runs; +- every later write of that run (done, retried, failed, given back) is fenced + by the token: a worker whose lease ran out writes nothing; +- a job whose lease ran out (its worker died) is ready again, its attempt run + again, `redelivered`; +- a rate limit of the whole queue (`rateLimitScope: 'queue'`) pauses every + worker, and one of a tenant (`'tenant'`) that tenant's jobs alone, through + documents of their own (`mailer:pause`, `mailer:pause:`), which + `outbox.resume()` ends early; +- with `fairness: 'tenant'`, a lane takes a `high` job of any tenant first, + then the tenants with jobs ready in turn (read with `distinct` once a poll + interval): two round trips a job. + +Leases are written by the clock of each worker, so a lease is taken back only +`clockSkewToleranceMilliseconds` (5 s) after it ran out, lest a worker whose +clock is behind still holds it; keep the clocks of the workers in sync (NTP), +within that tolerance, and the lease more than three times longer. + +Workers poll (`pollIntervalMilliseconds`, a second). An idle poll of a lane is +one `findOneAndUpdate` and one read of when the next job is due: a process +takes back the leases that ran out once a third of a lease, and reads the +pause of the queue once a poll interval (a pause of its own at once). With `changeStream: true`, +on a replica set, a change after which a job may be ready (an enqueue, a +retry, a release, a lease taken back) wakes one lane at once, and the poll is +only the fallback; the outbox's own leases and settlings wake none. A stream +that fails is opened again a second later, then up to every 30 seconds +(`queue.change_stream_failed`). + +## Throughput + +`bench/outbox.js` measures the queue alone (a handler that acknowledges at +once) on a replica set of one, MongoDB 7 in Docker on a laptop: numbers to +compare on one machine, not a promise. + +| Workers × lanes | Jobs a second (change stream) | Round trips a job | +| --- | --- | --- | +| 1 × 1 | 354 (261) | 2.1 | +| 1 × 10 | 2308 (2214) | 2.0 | +| 4 × 1 | 1081 (1084) | 2.1 | +| 4 × 10 | 4723 (4707) | 2.1 | + +A job costs its lease and its settling; before 1.2.0 was released, the same +bench gave 4 round trips a job and 837 jobs a second for 1 × 10. A send through +a provider takes tens to hundreds of milliseconds, so the outbox is rarely what +limits a worker: past a few thousand jobs a second, or for delays and rate +limits at that scale, take [BullMQ](/transports/bullmq) on Valkey. + +```bash +MONGODB_URL='mongodb://localhost:27027/?replicaSet=rs0' node bench/outbox.js 2000 +``` + +## Indexes and retention + +`ensureIndexes()` makes: + +| Collection | Index | +| --- | --- | +| outbox | `{ state, rank, availableAt, enqueuedAt }` for the ready jobs; `{ state, leaseUntil }` for the leases; `{ state, availableAt }` for the next one due; `{ state, tenantId, rank, availableAt, enqueuedAt }` for the tenants in turn; a TTL on `expireAt` | +| audit | `{ correlationId, at, attempt, rank }`; `{ jobId }` | +| idempotency | a TTL on `expiresAt` | + +A completed job is kept `dedupeWindowSeconds` (a day), and while it is kept, +an enqueue under its key is a duplicate: the retention is the window of +deduplication. A failed job is kept `retainFailedSeconds` (a week), for a retry +by hand. Both are the same options, with the same defaults, on every driver. A key of the store goes when its time passes. A completed job +keeps no `data`, only its hash (`retainData: 'failed'`, the default; `'all'` +or `'none'`), so a reset link in it does not wait a week in the collection and +its backups. + +## A job as stored + +```js +{ + _id: '1b9d…', // the id of the job: a UUID of namespace, tenant and key + name: 'mail', data: { … }, hash: '…', + priority: 'high', rank: 0, maxAttempts: 5, + state: 'ready', // 'active', 'completed', 'failed' + attempts: 0, availableAt: ISODate(…), enqueuedAt: ISODate(…), + token: null, leaseUntil: null, redelivered: false, + redeliveries: 0, // the runs that ended unsettled, never counted down + failure: { code, reason }, // of the last failed attempt + result: { provider, providerMessageId }, + finishedAt: ISODate(…), + expireAt: ISODate(…), // when the TTL drops it: dedupeWindowSeconds or retainFailedSeconds on +} +``` + +`outbox.get(id)` reads one. `outbox.stats()` gives its jobs by state (`ready`, +`delayed`, `active`, `completed`, `failed`), how long the oldest job due has +waited (`oldestReadyMilliseconds`, the lag of the workers, to alert on) and its +pauses; `outbox.failed({ limit, after })` lists the failed jobs a page at a +time, and `outbox.retryFailed(id)` sends one again, its attempts anew, before +`retainFailedSeconds` drops it. The data of a job is stored as it is, a token of a +reset link among it, unless the queue seals it with a +[codec](/guide/queue#sealed-jobs): restrict who reads the collection. + +## The audit and the keys + +`createMongoAuditSink` writes each event once by its id (an upsert), in the +transaction it is given, with `eventsOf(correlationId)` in their order and +`stateOf(correlationId)`. `createMongoIdempotencyStore` makes each step of +`IdempotencyTransport` atomic, so that of two workers sending one key, one +sends and the other is told it is in flight; a worker that died mid-send +leaves the key reserved, and the attempt again is of unknown delivery, not a +second send. + +`pnpm test:mongodb` runs all three on MongoDB (`tests/mongodb/docker-compose.yml`, +a replica set of one). diff --git a/docs/guide/queue.md b/docs/guide/queue.md new file mode 100644 index 0000000..c504c5d --- /dev/null +++ b/docs/guide/queue.md @@ -0,0 +1,403 @@ +# The queue + +`@alexify/mailer/queue` sends mail through a queue: a producer enqueues a +message as a job, and a worker renders it, sends it through the transport you +composed, and decides from the receipt what the queue does next. The queue is a +driver of your choice: [BullMQ](/transports/bullmq) on Redis or Valkey, the +[MongoDB outbox](/guide/outbox), the memory queue for tests, or an outbox of +your own store (a `QueueSource`). + +``` +producer queue.enqueue(job) ──▶ driver (BullMQ, outbox, memory) +worker driver ──▶ render ──▶ attachments ──▶ createMessage ──▶ transport.send ──▶ receipt + │ + ack · retry after a delay · fail for good · release ◀─────────┘ +``` + +## Enqueue and work + +```js +const { Queue, Worker } = require('bullmq'); +const { createMailQueue } = require('@alexify/mailer/queue'); +const { createBullMQDriver } = require('@alexify/mailer/bullmq'); +const { createLoggerAuditSink } = require('@alexify/mailer/audit'); + +const driver = createBullMQDriver({ bullmq: { Queue, Worker }, connection: { host: 'valkey' } }); +const queue = createMailQueue({ driver, audit: createLoggerAuditSink(logger), logger }); + +// The producer: an HTTP handler, a cron. +await queue.enqueue({ + kind: 'template', + template: 'reset-password', + context: { name: user.name, url }, + locale: user.locale, + from: 'Hirevel ', + to: user.email, + idempotencyKey: `reset-password:${resetRequest.id}`, + priority: 'high', + expiresAt: new Date(Date.now() + 15 * 60_000), +}); + +// The worker, in a process of its own or the same one. +const worker = queue.createWorker({ + transport: new IdempotencyTransport(new SesTransport({ ...config, retries: 1 }), { store }), + renderer: templates, + concurrency: 5, +}); +process.on('SIGTERM', () => worker.close()); +``` + +A job is `{ kind: 'template', template, context, locale }`, which the worker +renders with its `renderer` (such as one of [`./templates`](/messages/templates)), +or `{ kind: 'content', subject, content }`, rendered already, such as with React +Email in the producer. The other fields are those of +[`createMessage`](/messages/compose), plus: + +| Field | What it does | +| --- | --- | +| `idempotencyKey` | the key the job is enqueued once under (one is made when absent) | +| `scheduledAt` | a delay of the queue: not sent before then | +| `expiresAt` | not sent after then: a code of sign-in sent late is of no use | +| `tenantId` | scopes the key of the job | +| `correlationId` | what the events of the message share, from the queue to the webhooks | +| `maxAttempts` | the attempts in all (5 by default) | + +## What a job holds + +A job is JSON, checked when it is enqueued, so a bad address fails the caller, +not the worker: + +- the fields are checked as `createMessage` checks them; +- `context` must be JSON as it is: a `Date`, a `Map`, a class, `NaN` or a cycle is + refused with the path to it (`context.user.createdAt is a Date; pass an ISO + string`), so a template does not meet a string where it expected a date; +- an attachment goes as its bytes (base64, 256 KiB in all by default) or as + `{ ref }`, which the worker's `resolveAttachment(ref, { job, attachment, + signal })` reads, with `fromPath` or `fromUrl` of + [`./attachments`](/messages/attachments). A job names no file or URL itself: + whoever can write to the store could; +- the Message-ID and the Content-IDs are fixed, so every attempt sends the + same message; +- a job is 256 KiB at most. + +A completed job is kept `dedupeWindowSeconds` (a day), the window in which an +enqueue under its key is a duplicate, and a failed one `retainFailedSeconds` (a +week), on every driver. Once a job is done with, the store drops its message +(`retainData`, on the driver): `'failed'` by default keeps the data of the failed jobs only, for a +retry by hand; `'all'` keeps every one, `'none'` none. The version and the hash +of a job stay, so an enqueue again is still told a duplicate. + +A job is of version 1 (`v: 1`), and its fields are those of that version. A +worker that meets a later version, or a field it does not know, gives the job +back for a worker that reads it (`queue.unsupported_version`) instead of +failing it for a hash it cannot recompute: during a rolling deploy, the jobs +the new producers write wait for the new workers. A field is never added to +version 1; a new field is a new version. + +## Signed jobs + +The hash of a job tells a job that changed in its store, but not one that was +forged: whoever can write to Redis or MongoDB can recompute a SHA-256, and a +worker would send their message from your domain, signed with your DKIM key. +Give the queue `signingKeys`, and each job is signed with an HMAC-SHA256 (`kid` +and `mac`); its workers send only jobs that one of the keys signed, and fail +any other with `queue.signature_invalid`, unsent, logged at `error`: + +```js +const queue = createMailQueue({ + driver, + signingKeys: [{ id: '2026-10', secret: process.env.MAIL_JOB_KEY }], // 32 bytes at least +}); +const worker = queue.createWorker({ transport }); // verifies with the queue's keys +``` + +The first key signs and each one verifies. To rotate, put the new key first, +deploy the workers, then the producers, and drop the old key once no job it +signed is left. A worker without keys checks the hash alone, so give every +worker the keys before any producer. + +## Sealed jobs + +A job holds its message in clear: its addresses, its context (a link of a +password reset among it) and its attachments, in Redis or MongoDB for as long +as the job is kept, and in their backups. Give the queue a `codec`, and the +store holds what it made of them: + +```js +const { createCipheriv, createDecipheriv, randomBytes } = require('node:crypto'); +const key = Buffer.from(process.env.MAIL_DATA_KEY, 'base64'); // 32 bytes + +const codec = { + encode(text) { + const iv = randomBytes(12); + const cipher = createCipheriv('aes-256-gcm', key, iv); + const sealed = Buffer.concat([cipher.update(text, 'utf8'), cipher.final()]); + return Buffer.concat([iv, cipher.getAuthTag(), sealed]).toString('base64'); + }, + decode(text) { + const bytes = Buffer.from(text, 'base64'); + const decipher = createDecipheriv('aes-256-gcm', key, bytes.subarray(0, 12)); + decipher.setAuthTag(bytes.subarray(12, 28)); + return Buffer.concat([decipher.update(bytes.subarray(28)), decipher.final()]).toString(); + }, +}; +const queue = createMailQueue({ driver, codec, signingKeys }); +``` + +A sealed job keeps in clear only `v`, `hash` (a duplicate is still told by +it), `enqueuedAt`, `correlationId`, `kid`, `mac` and `trace`; the rest is `enc`. Its workers +take the queue's codec. A worker without it gives the job back, its attempts +not spent, for a worker that can open it (`queue.codec_missing`); one whose +codec throws (a key out of reach) retries it (`queue.codec_failed`), and a +codec that throws an error with `retryable: false` fails it. Encryption is not a signature: a writer who +can seal can still forge, so seal with `signingKeys` too. + +## Once per key + +The id of a job is a UUID of its namespace, tenant and `idempotencyKey`: the +same message enqueued again is the same job (`duplicate: true`), and another +message under the key is refused with `MailQueueError` `queue.key_reused`, its +`jobIds` the jobs refused. `enqueueMany` refuses them only once every other +job of the call is enqueued and told to the audit, and gives what those gave as +the error's `answers`. +The worker gives the message that id as its `idempotencyKey`, so +[`IdempotencyTransport`](/transports/idempotency), with a store the workers +share, and the providers that take a key (Resend, Lettermint, Brevo, Azure) +dedupe it again. A worker that died mid-send leaves the key reserved, and the +attempt again is of unknown delivery (`delivery_unknown`), not a second send. + +The rendering must then be the same for the same context: no `Date.now()` in a +template, or an attempt again is refused as another message. + +With a transport that takes no idempotency key (SES, SMTP) and no +`IdempotencyTransport`, give the worker `redelivered: 'delivery_unknown'`: a +run again of an attempt whose run before died is then of unknown delivery +(`queue.redelivered`), not a second send, unless the audit sink can tell +(`has(id)`, as the memory and MongoDB sinks can) that the run before died +before it submitted the message. + +The id of a job is a hash of its key, and it goes where addresses do not: into +the audit, the logs, the provider's idempotency key and the Message-ID. A key +is an id of what the message tells of (`reset-password:${resetRequest.id}`), +never an address or a token: a SHA-256 of `welcome:ada@example.com` is found +again by hashing the addresses one knows. `idSecret` (32 bytes at least) +derives the id with an HMAC instead; it is the same on every producer and never +rotated, as another secret makes other ids and the same message another job. + +The identifiers a job draws come from `generateId`, as everywhere in the +library: its key when you give none, and its Message-ID and Content-IDs then. +Those it derives, the same for the same key, are UUIDs of a hash: its id and, +under a key of yours, its Message-ID and Content-IDs. `deriveId(kind, seed)` +makes them your way; `seed` is a SHA-256 in hexadecimal of what they derive +from (an HMAC under `idSecret` for the id of the job), and it must give the same +identifier for the same seed, which the queue checks: + +```js +const queue = createMailQueue({ + driver, + generateId: () => ulid(), // drawn: new at each call + deriveId: (kind, seed) => (kind === 'job-id' ? `job_${seed.slice(0, 26)}` : `m${seed.slice(0, 32)}`), +}); +``` + +A generator such as `ulid()` does not fit `deriveId`: the same message enqueued +again would get another id, and so be another job, sent twice. + +## What the queue does with an attempt + +| The attempt | The queue | The state | +| --- | --- | --- | +| accepted | done | `accepted` | +| a failure of unknown delivery (a timeout after the request, a key in flight, the worker stopped mid-send) | failed for good, never sent blindly again | `delivery_unknown` | +| a retryable failure, attempts left | retried after the backoff, never sooner than the provider's `Retry-After` | `queued` | +| a retryable failure, the last attempt | failed (`queue.attempts_exhausted`) | `failed` | +| a refused message (`validation`, `rejected`, `unsupported`) | failed at once | `failed` | +| a failure of the operator (`auth`, `configuration`) | retried after `operatorDelayMilliseconds` (5 minutes) | `queued` | +| a template missing or broken, no subject, no renderer | failed at once (`render.*`); a broken one's reason names the template and where it failed, its text (a helper's error, an excerpt, a value of the context) going to the log at `debug` only | `failed` | +| an outage of the template loader (`TemplateLoadError`) or of `resolveAttachment`, a render that timed out | retried | `queued` | +| a bug: a `TypeError` thrown by the transport | failed at once (`queue.thrown`), logged with its stack | `failed` | +| a job past `expiresAt`, or whose retry would come after it | failed (`queue.expired`) | `failed` | +| a job written by a later version of the worker, or sealed for a codec it lacks | given back, its attempts not spent, for a worker that reads it; failed once it expires or has waited a week | — | +| the worker stops before the send | given back, its attempt not spent | — | +| a job whose runs ended unsettled more than `maxRedeliveries` times (3): it kills or blocks its worker | failed before it renders (`queue.redelivery_exhausted`) | `delivery_unknown` | + +The backoff is exponential with full jitter: 30 seconds, doubled per attempt, +an hour at most (`backoff`). A `Retry-After` is a minimum, with a little jitter +over it, so that a rate limit does not bring every job back at once. +`rateLimitScope: 'queue'` makes a rate limit pause the whole queue where the +driver can. `classify({ receipt, job })` decides in the place of these rules +(`'retry'`, `'fail'`, `'delivery_unknown'`, or `undefined` for the rules), and +`checkJob(data, job)` refuses a job for good by throwing. +`rateLimitScope: 'tenant'` makes a rate limit pause the jobs of the job's +tenant alone, on the memory queue and the outbox; `resume()` of either ends a +pause, of the queue or of a tenant (`{ tenantId }`), before its time. A +`'queue'` pause suits one provider; with failover among several, let +[`./breaker`](/transports/breaker) and [`./ratelimit`](/transports/ratelimit) +take a provider out instead. + +## Tenants in turn + +Jobs go by priority, then time: a tenant who enqueues a million newsletters +holds every other tenant's `normal` mail behind them. `fairness: 'tenant'`, on +the memory queue and the outbox, takes a `high` job of any tenant first (a code +of sign-in waits for no one), then the tenants in turn, a job of each, the jobs +of no tenant as one more. It is exact within a process and close across them. +Without it, a queue per stream (transactional and bulk, each with its own +workers) keeps a campaign from holding a password reset. BullMQ shares a queue +by tenant only in its Pro edition: use a queue per tenant or per stream there. + +**The queue retries; the transport should not.** Each attempt of a +`RetryTransport` inside the worker holds the job and multiplies its attempts: +give the providers `retries: 1` (they retry only what was provably not sent) +and leave `RetryTransport` out. + +## The audit and the log + +The worker tells the [audit sink](/guide/audit) each state: `queued` at the +enqueue (in the transaction of an outbox), `rendered`, `submitted` just before +the send, then `accepted`, `failed` or `delivery_unknown`, and `queued` again +for a retry. A sink that fails is a warning (`MAILER_AUDIT_FAILED`); +`auditStrict: true` makes it retry the job before the send, so that no message +goes unrecorded. Without a transaction, `queued` is told before the job is +enqueued, so a strict sink that fails refuses the enqueue and nothing is +enqueued, rather than a job enqueued and reported as failed. A driver that +writes no transaction (BullMQ, the memory queue) refuses `{ transaction }` with +a `TypeError` rather than ignore it; the outbox is `transactional`. + +Spans and metrics of the worker (`mailer.queue.jobs`, `mailer.queue.lag`, +`mailer.queue.job.duration`, `mailer.queue.redeliveries`) come from +[`instrumentHandler`](/transports/otel#the-worker-of-a-queue) of `./otel`, +linked to the producer's trace that `traceContext` gives each job. + +`logger`, pino-like, hears what the queue does, never an address: the +enqueues and the starts at `debug`, the sends at `info`, the retries, +redeliveries and releases at `warn`, the failures at `error` with their code, +category and reason. + +## Several processes + +A job is shared by every worker of its driver; the key store of +`IdempotencyTransport` must be shared too (`createMongoIdempotencyStore`, or +one on Redis), or a crash mid-send can be sent twice. The breaker and the rate limit of the chain are +per process unless given a shared store (see +[getting started](/guide/getting-started#several-processes)). + +## The memory queue and the polling driver + +`createMemoryQueue()` is a queue in memory for tests and a single process, +with `list()`, `get(id)`, `drain()`, `wake()`, and `stats()`, `failed()` and +`retryFailed(id)` as the outbox has them, and `now` and `sleep` for a +clock of the tests. It is the polling driver over a source in memory: +`createPollingDriver(source)` makes a driver of any `QueueSource`, a store that +lends its jobs (`reserve`, `extend`, `ack`, `retry`, `fail`, `release`), such +as an outbox in your database. It keeps the lease of a job while it runs, gives +a job whose lease ran out to another worker as `redelivered`, and on `close()` +waits for the jobs in flight, then aborts them. When the store does not answer +the extensions of a lease, the run is aborted before the lease runs out +(`QueueLeaseLostError`), so that another worker that takes the job does not +send it beside this one. A source's `ack`, `retry`, +`fail` and `release` are fenced by the token of the lease and give `false` +when it was lost: the driver then logs `queue.settle_fenced` at `warn`, as +another worker holds the job and what this run did may be done again. + +A source of your own is held to the same semantics as the memory queue and the +MongoDB outbox: the order of priorities, leases, redelivery, fencing by the +token, releases that spend no attempt, and the pause. The repository's +`tests/helpers/queue/source-contract.js` is that contract as tests; copy it, +and run it on your source (`createMemoryQueue().source` is one to compare). + +## Defaults + +What the queue, its worker and its drivers do when not told otherwise, in one +place: + +| Option | Of | Default | +| --- | --- | --- | +| `namespace` | the queue | `'mail'` | +| `maxAttempts` | the queue, a job | 5 | +| `maxJobBytes`, `maxInlineAttachmentBytes` | the queue | 256 KiB each | +| `backoff` | the worker | 30 seconds, doubled per attempt, an hour at most, full jitter | +| `operatorDelayMilliseconds` | the worker | 5 minutes (`auth`, `configuration`, a job it cannot read yet) | +| `maxRedeliveries` | the worker | 3 | +| `redelivered` | the worker | `'send'` | +| `rateLimitScope` | the worker | `'job'` | +| `concurrency` | a worker | 1 | +| `pollIntervalMilliseconds` | the memory queue, the outbox | 1 second | +| `leaseMilliseconds` | the memory queue, the outbox | 1 minute, extended every third of it | +| `timeoutMilliseconds` of `close()` | every driver | 30 seconds, then the jobs are aborted | +| `dedupeWindowSeconds` | every driver | a day | +| `retainFailedSeconds` | every driver | a week | +| `retainData` | every driver | `'failed'` | +| `fairness` | the memory queue, the outbox | `'none'` | +| `clockSkewToleranceMilliseconds` | the outbox | 5 seconds | +| `changeStream` | the outbox | `false` | +| `maxJobs`, `retainFinished` | the memory queue | 100000, 10000 | +| `priorities` | BullMQ | `high` 1, `normal` 2, `low` 3 | +| `detectDuplicates` | BullMQ | `true` | +| `leaseMs` | `IdempotencyTransport` | 10 minutes, longer than a lease, so a run that died is not sent again | +| `renderTimeoutMilliseconds` | a template renderer | 30 seconds | + +## Codes + +The codes the queue, its drivers and its worker give: in the `failure` of a job +and the `error` of its audit event, as the message of a log record, or as the +code of a process warning. `tests/unit/codes.test.js` keeps this table and the +code in step. + + + +| Code | Where | What | +| --- | --- | --- | +| `queue.invalid_job` | failure | the data of a job is not that of a mail job | +| `queue.unsupported_version` | failure | written by a later version: given back, then failed after a week | +| `queue.hash_mismatch` | failure | the job is not what was enqueued | +| `queue.signature_invalid` | failure | not signed by a key of `signingKeys` | +| `queue.codec_missing` | failure | sealed, and the worker has no codec: given back, then failed after a week | +| `queue.codec_failed` | failure | the codec could not open it | +| `queue.key_reused` | thrown | a key enqueued before for another message | +| `queue.refused` | failure | `checkJob` threw | +| `queue.expired` | failure | past `expiresAt`, or its retry would be | +| `queue.attempts_exhausted` | failure | a retryable failure on the last attempt | +| `queue.redelivery_exhausted` | failure | its runs ended unsettled more than `maxRedeliveries` times | +| `queue.redelivered` | failure | a run again after a run that may have sent it (`redelivered: 'delivery_unknown'`) | +| `queue.attachment_unresolved` | failure | an attachment by reference, and no `resolveAttachment` | +| `queue.attachment_failed` | failure | `resolveAttachment` threw or gave no content | +| `queue.invalid_message` | failure | the job does not make a message | +| `queue.invalid_receipt` | failure | the transport gave no receipt | +| `queue.aborted` | failure | the worker stopped while the message was sent | +| `delivery_unknown` | failure | the send may have gone (a timeout after the request, a key in flight) | +| `queue.thrown` | failure | a bug of the transport or of the worker | +| `queue.handler_threw` | failure | the handler of a driver threw | +| `queue.invalid_outcome` | failure | the handler gave no outcome | +| `queue.lost` | failure | BullMQ gave the job up itself (stalled too often) | +| `queue.released` | failure | a release without a token, a retry on BullMQ | +| `render.no_renderer` | failure | a template job, and the worker has no renderer | +| `render.no_subject` | failure | neither the template nor the job gives a subject | +| `render.failed` | failure | the template failed to render or was not found | +| `audit.failed` | failure | a strict sink failed before the send: retried | +| `queue.enqueued` | log | a job enqueued | +| `queue.duplicate` | log | a job enqueued again under its key | +| `queue.job.started` | log | a run starts | +| `queue.job.completed` | log | a run accepted | +| `queue.job.retrying` | log | a run to be retried | +| `queue.job.released` | log | a run given back, its attempt not spent | +| `queue.job.failed` | log | a run failed for good | +| `queue.job.redelivered` | log | a run again of an attempt whose run died | +| `queue.render.failed` | log | the text of a failed render, at `debug` | +| `queue.worker.error` | log | what escaped the steps of the worker | +| `queue.audit.failed` | log | the audit sink failed | +| `queue.lease.extend_failed` | log | the store did not extend a lease | +| `queue.lease.lost` | log | a run lost its lease, its result dropped | +| `queue.settle_fenced` | log | a settle the token fenced off: the job may run twice | +| `queue.settle_failed` | log | the store failed a settle: the lease will run out | +| `queue.reserve_failed` | log | the store failed a reserve | +| `queue.close_abandoned` | log | jobs still running past twice the timeout of `close()` | +| `queue.change_stream_failed` | log | the change stream failed, to be opened again | +| `queue.job.lost` | log | BullMQ failed a job itself | +| `queue.job.stalled` | log | BullMQ found a job stalled | +| `MAILER_AUDIT_FAILED` | warning | the audit sink failed | +| `MAILER_QUEUE_HANDLER_THREW` | warning | the handler of a driver threw | +| `MAILER_QUEUE_WORKER_THREW` | warning | an error escaped the steps of the worker | +| `MAILER_QUEUE_RETAIN_FAILED` | warning | BullMQ refused to drop the data of a finished job | + + diff --git a/docs/guide/why.md b/docs/guide/why.md index cb68c36..e7e2941 100644 --- a/docs/guide/why.md +++ b/docs/guide/why.md @@ -3,8 +3,9 @@ ::: tip Status Version 1.0 is the transport layer, ported from [upyo](https://github.com/dahlia/upyo) 0.7.0, and 1.1 adds what nodemailer, unemail and sently did better, delivery events and the decorators of a -closed loop. The queue and the audit log are being designed: the direction below is settled, their -API is not. +closed loop; 1.2 adds the [queue](/guide/queue), on BullMQ or a +[MongoDB outbox](/guide/outbox), [templates by name](/messages/templates) and the +[audit](/guide/audit) of the states of a message. Suppression is next. ::: ## Two parts @@ -19,8 +20,9 @@ of two parts: signing, Amazon SES v2 (signed with SigV4, without the AWS SDK) and seventeen more HTTP providers on `fetch`, JMAP, and decorators for failover, retries, telemetry and logs. 2. **A reliability layer on top of it,** in part here. One model of delivery events read from - verified webhooks, idempotency keys, a circuit breaker and rate limits came in 1.1; a persistent - queue and outbox, an append-only audit log and suppression are being designed. + verified webhooks, idempotency keys, a circuit breaker and rate limits came in 1.1; the queue + with its BullMQ driver and MongoDB outbox, templates and the audit of states in 1.2; the + crypto-shredding of the audit and suppression are next. ## The gap it fills diff --git a/docs/index.md b/docs/index.md index 9e23f7e..d4ac91b 100644 --- a/docs/index.md +++ b/docs/index.md @@ -7,7 +7,7 @@ titleTemplate: Email for Node.js, with a reliability layer hero: name: mailer text: Email for Node.js, with a reliability layer - tagline: SMTP, sendmail, MIME and DKIM of its own, eighteen HTTP providers and JMAP, with failover, retries, rate limits and idempotency around them, and verified webhooks into one model of delivery events. Zero dependencies. Next on top, a persistent queue and an audit log. + tagline: SMTP, sendmail, MIME and DKIM of its own, eighteen HTTP providers and JMAP, with failover, retries, rate limits and idempotency around them, and verified webhooks into one model of delivery events. Zero dependencies. On top, a persistent queue with templates of your engine and an audit of every state. image: src: /logo-mark.svg alt: mailer @@ -45,14 +45,14 @@ features: linkText: Pool, retry, breaker - icon: 🪶 title: Zero dependencies - details: "Built on node:* and fetch. The OpenTelemetry providers, the logger, React Email's render and, later, the queue driver come from your application. Node.js 22+, tested on Bun and Deno, CommonJS with ESM imports, hand-written types." + details: "Built on node:* and fetch. The OpenTelemetry providers, the logger, React Email's render, your template engine and BullMQ come from your application. Node.js 22+, tested on Bun and Deno, CommonJS with ESM imports, hand-written types." link: /api/exports linkText: API - icon: 🧾 - title: Delivery events, then the reliability layer - details: "The webhooks of fifteen providers, each verified before it is read, give one model of delivery events. In design: a persistent queue and outbox, an append-only audit log and suppression. None of the fourteen libraries compared has the three." - link: /guide/webhooks - linkText: Webhooks and events + title: The reliability layer + details: "A queue on BullMQ or a store of yours, whose workers render your templates, fail a permanent failure at once and retry a temporary one after Retry-After; the webhooks of fifteen providers in one model of delivery events; an audit of every state. None of the fourteen libraries compared has them." + link: /guide/queue + linkText: The queue --- ::: tip Status @@ -60,8 +60,9 @@ The transport layer is here, ported from [upyo](https://github.com/dahlia/upyo) what [nodemailer](https://github.com/nodemailer/nodemailer), [unemail](https://github.com/productdevbook/unemail) and [sently](https://github.com/omqkhafi/sently) did better ported in 1.1: one package, with a subpath -per transport and decorator. Delivery events are here too; the queue and the audit log are being -designed. See [Why @alexify/mailer?](/guide/why) for the plan. +per transport and decorator. Delivery events are here too, and, from 1.2, the +[queue](/guide/queue), [templates](/messages/templates) and the [audit](/guide/audit). See +[Why @alexify/mailer?](/guide/why) for the plan. ::: ## A quick look diff --git a/docs/messages/compose.md b/docs/messages/compose.md index db37feb..5937c4e 100644 --- a/docs/messages/compose.md +++ b/docs/messages/compose.md @@ -423,7 +423,9 @@ const transport = new SmtpTransport({ host: 'smtp.example.com', generateId }); ``` `composeMessage()` of `./mime` takes `generateId` in its options too, and every -transport in its configuration. +transport in its configuration. A [queue](/guide/queue#once-per-key) derives +the id of a job, and the Message-ID of a job under a key, from that key rather +than drawing it, so that the same key is the same job: `deriveId` makes those. ### Dropping an inherited header diff --git a/docs/messages/render.md b/docs/messages/render.md index 03af0d0..db17c26 100644 --- a/docs/messages/render.md +++ b/docs/messages/render.md @@ -49,6 +49,26 @@ const content = await renderContent(markdown, '# Hello\n\nYour order has shipped When a renderer gives HTML without text, `renderContent` derives the text with `htmlToText`; `{ autoText: false }` leaves it without one. +## The subject + +A renderer may give the subject of the message with its content, as the +templates of [`./templates`](/messages/templates) do from their front matter. +`renderContent` leaves it out; `renderFields` gives `{ subject, content }`, the +content rendered as `renderContent` renders it: + +```js +const { renderFields } = require('@alexify/mailer/render'); + +const { subject, content } = await renderFields(templates, { + template: 'welcome', + context: { name: user.name }, +}); +const message = createMessage({ from: 'shop@example.com', to: user.email, subject, content }); +``` + +`subject` is undefined when the renderer gives none; a subject that is not a +string fails the render. + ## htmlToText `htmlToText(html)` gives the plain text an HTML body reads as, for the text diff --git a/docs/messages/templates.md b/docs/messages/templates.md new file mode 100644 index 0000000..9e597f8 --- /dev/null +++ b/docs/messages/templates.md @@ -0,0 +1,369 @@ +# Templates by name + +`@alexify/mailer/templates` renders a template by its name, with the engine you +already use: EJS, Handlebars, Pug, MJML, Liquid, Nunjucks, Eta, Mustache or +React Email. The package has no engine of its own and depends on none of them: +you pass the engine in, and the adapter reads its templates through a loader, +caches what it compiles, applies a layout, inlines the CSS, gives the subject +from the template, and derives the text from the HTML. + +A template renderer is a [`Renderer`](/messages/render) of `./render`, so +`renderContent` and `renderFields` take it, and so does the worker of the +[queue](/guide/queue). + +## Quick start + +```js +const path = require('node:path'); +const ejs = require('ejs'); +const { createMessage } = require('@alexify/mailer'); +const { renderFields } = require('@alexify/mailer/render'); +const { + createEjsEngine, + createFileLoader, + createTemplateRenderer, +} = require('@alexify/mailer/templates'); + +const templates = createTemplateRenderer({ + engine: createEjsEngine({ ejs }), + loader: createFileLoader({ dir: path.join(__dirname, 'emails'), extensions: ['.ejs'] }), + layout: 'layouts/letter', + globals: { appUrl: 'https://app.example.com' }, +}); + +const { subject, content } = await renderFields(templates, { + template: 'reset-password', + context: { name: user.name, url }, + locale: user.locale, +}); +const message = createMessage({ from: 'no-reply@example.com', to: user.email, subject, content }); +``` + +`emails/reset-password.ejs`: + +```ejs +--- +subject: Reset your password, <%= name %> +--- +

Hi <%= name %>,

+

Choose a new password

+``` + +`emails/layouts/letter.ejs`, with the header and footer every letter shares: + +```ejs + + + + <%- body %> +
…
+ + +``` + +`render()` gives `{ html, text, subject, template, locale }`: the text is +derived from the HTML unless a text template is there, and `template` names the +file the message came from. The renderer derives the text itself, so +`autoText: false` of `renderFields` leaves it; give `autoText: false` to the +renderer instead. + +## Names, variants and locales + +A template is asked for by a **name**, never a path: `welcome`, +`auth/reset-password`. A name has no `..`, no leading `/` and no `\`; the file +loader also reads nothing a symbolic link leads out of its directory to. + +Each part of a message may have a file of its own, next to the HTML: + +| File | Part | +| --- | --- | +| `welcome.hbs` | the HTML | +| `welcome.text.hbs` | the text, in place of the one derived from the HTML | +| `welcome.subject.hbs` | the subject, when the front matter gives none | +| `welcome.uk.hbs`, `welcome.uk.text.hbs`, … | the same in Ukrainian | + +A locale is looked up with its subtags dropped one by one, then +`defaultLocale` the same way, then the template without a locale: `uk-UA` reads +`welcome.uk-UA.hbs`, `welcome.uk.hbs`, then `welcome.hbs`. The text and subject +are taken at the locale the HTML was found at, so that a Ukrainian body never +goes with an English subject. `variantName(template, { locale, part })` names +the variants another way. + +When the locale comes from a user, name the locales your templates are written +in: with `locales: ['en', 'uk']`, `uk-UA` is looked up as `uk` and any other +locale as none, so that a made-up tag reads no template and keeps no entry of +the cache of its own. + +A name with the extension of an engine (`welcome.hbs`) is that engine's alone; +without one, the engines are tried in their order. + +## Front matter + +A template may start with its subject and layout: + +```hbs +--- +subject: "Welcome, {{name}}" +layout: layouts/marketing +--- +

…

+``` + +The subject is rendered with the template's engine and context, its entities +decoded and its whitespace collapsed. `layout: none` renders the template +without the renderer's `layout`. Any other key is an error, so a typo does not +pass unnoticed. `frontMatter: false` reads none. + +## Layouts, partials and includes + +`layout` names the layout of every HTML template; it is rendered with the +context and `body`, the HTML of the template, raw. A layout has no layout of +its own. Engines that have layouts of their own (Pug's `extends`, Nunjucks's +and Liquid's blocks, Handlebars partial blocks, Eta's `layout()`) keep them. +A React Email template gives its HTML whole, so a renderer with a `layout` +refuses the React Email engine, rather than send its messages without it. + +Every include, partial and layout an engine reads goes through the loader of +the renderer, by one rule, the same in every engine: + +- `/x` is read from the root of the loader; +- `./x` and `../x` against the template it is written in, a partial of a + partial included, where EJS, Mustache and Pug would tell otherwise; +- any other name from the root, then from each directory of `partials`; +- the extensions of the engine are tried when the name has none of them, then + the name as it is. + +A name that leads out of the root is not found. React Email has no includes: +its components import each other. MJML's are those of the engine inside it. + +## Engines + +::: v-pre + +Each engine takes the module of its package; what the adapter sets itself (the +file name includes are read by, the cache, a file system) is refused as an +option rather than accepted and ignored. + +| Engine | Made with | Extensions | Escapes by default | +| --- | --- | --- | --- | +| EJS | `createEjsEngine({ ejs: require('ejs'), options: { async } })` | `.ejs` | `<%= %>`; `<%- %>` is raw | +| Handlebars | `createHandlebarsEngine({ handlebars, helpers, configure })` | `.hbs`, `.handlebars` | `{{ }}`; `{{{ }}}` is raw | +| Pug | `createPugEngine({ pug, options: { doctype } })` | `.pug` | `#{}`, `=`; `!{}` is raw | +| MJML | `createMjmlEngine({ mjml2html: require('mjml'), engine })` | `.mjml`, or `.mjml` before the engine's | — | +| Liquid | `createLiquidEngine({ Liquid: require('liquidjs').Liquid, filters })` | `.liquid` | yes (`outputEscape: 'escape'`); `\| raw` | +| Nunjucks | `createNunjucksEngine({ nunjucks, filters, configure })` | `.njk`, `.nunjucks` | yes (`autoescape`) | +| Eta | `createEtaEngine({ Eta: require('eta').Eta, async })` | `.eta` | `<%= %>`; `<%~ %>` is raw | +| Mustache | `createMustacheEngine({ mustache, tags })` | `.mustache` | `{{ }}`; `{{{ }}}` is raw | +| React Email | `createReactEmailEngine({ render, createElement, components })` | — | JSX | + +Several engines go into one renderer with `engines: [...]`, chosen by +extension, the longest first. + +Output is escaped, and no option of an engine turns that off unless the engine +is given `allowUnescaped: true`: Handlebars `noEscape`, Nunjucks `autoescape: +false`, a Liquid `outputEscape` other than `'escape'`, Eta `autoEscape: false` +or an `escapeFunction`, and an EJS `escape` throw a `TypeError` without it. +The raw tags of each engine (`{{{ }}}`, `<%- %>`, `| raw`, `| safe`) stay +yours to use where a value is HTML you trust. + +Some engines behave a little differently here than they do on their own: + +- **Handlebars** gets an environment per renderer (`handlebars.create()`), so + its helpers and partials stay out of the global one. The partials a template + names are read when it is compiled. A dynamic partial (`{{> (name)}}`) has to + be registered in `configure`. +- **Liquid** escapes output by default, where liquidjs escapes nothing unless + told. It reads own properties only. +- **Mustache** fails on a partial that is not there, where it renders nothing, + and reads `./x` against the partial it is written in, which Mustache does + not tell. Its writer, and the cache of what it parsed, are per renderer and + bounded: a template is parsed once, as it is compiled, and a partial once + per template read, where Mustache keeps what it parses by the whole source. +- **MJML** renders after the engine inside it (`.mjml.hbs`), once the layout is + applied, so the layout of an MJML template is MJML too. A static `.mjml` is + rendered once. What MJML finds wrong under `validationLevel: 'soft'` goes to + `onErrors`, or to a warning (`MAILER_TEMPLATE_MJML`); `'strict'` fails the + render. `mj-include` reads files itself, unchecked, so it is off unless + `includes: true`. Use the engine's partials instead. +- **React Email**: its templates are its components, by name (`welcome.uk` for + a locale), each with its subject: + + ```js + const engine = createReactEmailEngine({ + render: require('@react-email/render').render, + createElement: require('react').createElement, + components: { + welcome: { component: Welcome, subject: ({ name }) => `Welcome, ${name}` }, + }, + }); + const templates = createTemplateRenderer({ engine }); + ``` + +::: + +## Who may write a template + +A template is code that runs in your process, so who writes it decides the +engine. The context is data in every engine, escaped on output; what differs +is what a template itself can do. + +| Engine | A template | Templates a tenant writes | +| --- | --- | --- | +| EJS, Eta | compiles to JavaScript: `<% %>` runs any code, with the rights of the process | no | +| Pug | compiles to JavaScript: `- code` lines and `#{}` run any code | no | +| Nunjucks | compiles to JavaScript and is no sandbox: its own guide calls user-defined templates unsafe, and known escapes reach `Function` through the values a template sees | no | +| React Email | is a component, code of yours | no | +| MJML | is the engine inside it; `mj-include` reads files, so it stays off | as that engine | +| Handlebars | is logic-less: it calls the helpers you register and reads own properties only | yes, with helpers you trust and `knownHelpersOnly` | +| Mustache | is logic-less | yes | +| Liquid | was made for templates users write; parse, render and memory limits are on by default here (`parseLimit` 1 MiB, `renderLimit` 1000 ms, `memoryLimit` 1e8) | yes, the one to choose | + +The subject of a front matter is compiled by the same engine as the template, +under the same rules: a tenant who writes the subject writes a template. + +For templates a tenant writes, keep the bounds of the renderer on: the +includes a render reads, the size of its output, and its timeout (see +[Bounds](#bounds)). A loop of a template that never yields can be stopped by +none of them, only by the engine: Liquid's `renderLimit` is checked as it +renders. + +## CSS + +```js +const templates = createTemplateRenderer({ + engine, + loader, + inlineCss: createCssInliner({ inline: require('juice').default }), + // or: createCssInliner({ inline: require('@css-inline/css-inline').inline }) +}); +``` + +The CSS is inlined after MJML and before the text is derived. The inliner is +given `{ loadRemoteStylesheets: false, keepLinkTags: true }` under your +options, so css-inline fetches no stylesheet on a send and keeps the `` +of a web font for the mail clients that read it; juice fetches nothing itself. +css-inline drops `

x

'; + // Before: css-inline fetched every remote stylesheet on each send and + // dropped the of the font. + for (const inliner of [createCssInliner({ inline: juice }), createCssInliner({ inline })]) { + const inlined = await inliner(html); + assert.match(inlined, /

x<\/p>/); + assert.match(inlined, /fonts\.googleapis\.com/); + } +}); diff --git a/tests/unit/templates-loader.test.js b/tests/unit/templates-loader.test.js new file mode 100644 index 0000000..67e7949 --- /dev/null +++ b/tests/unit/templates-loader.test.js @@ -0,0 +1,272 @@ +// Not ported: tests of the loaders of ./templates, on a directory of their +// own under the system's temporary directory. +const { test } = require('node:test'); +const assert = require('node:assert'); +const fs = require('node:fs'); +const fsp = require('node:fs/promises'); +const os = require('node:os'); +const path = require('node:path'); +const { + createFileLoader, + createMemoryLoader, + createTemplateRenderer, + createEjsEngine, +} = require('../../src/templates/index.js'); +const ejs = require('ejs'); + +// Replaces functions of node:fs for a test, and tells whether it could: a +// runtime whose node:fs is not writable leaves the test to the others. Node +// restores them after the test; Deno keeps them for the tests that follow. +const replace = (t, pairs) => { + t.after(() => t.mock.restoreAll()); + for (const [object, name, implementation] of pairs) { + const before = object[name]; + try { + t.mock.method(object, name, implementation); + } catch { + return false; + } + if (object[name] === before) return false; + } + return true; +}; + +const makeTree = (t) => { + const base = fs.mkdtempSync(path.join(os.tmpdir(), 'mailer-templates-')); + t.after(() => fs.rmSync(base, { recursive: true, force: true })); + const root = path.join(base, 'emails'); + fs.mkdirSync(path.join(root, 'auth'), { recursive: true }); + fs.mkdirSync(path.join(root, 'folder.ejs')); + fs.writeFileSync(path.join(root, 'welcome.ejs'), '

Hi <%= name %>

'); + fs.writeFileSync(path.join(root, 'auth', 'reset.ejs'), '

reset

'); + fs.writeFileSync(path.join(root, 'notes.txt'), 'not a template'); + fs.writeFileSync(path.join(root, 'big.ejs'), 'x'.repeat(2048)); + fs.writeFileSync(path.join(base, 'secret.ejs'), 'secret'); + fs.writeFileSync(path.join(root, '.env'), 'SECRET=1'); + return { base, root }; +}; + +test('Templates: the file loader reads a name under its directory, its BOM dropped', async (t) => { + const { root } = makeTree(t); + const loader = createFileLoader({ dir: root, extensions: ['.ejs'], maxBytes: 1024 }); + const welcome = await loader.load('welcome.ejs'); + assert.strictEqual(welcome.name, 'welcome.ejs'); + assert.strictEqual(welcome.source, '

Hi <%= name %>

'); + assert.strictEqual(welcome.path, fs.realpathSync.native(path.join(root, 'welcome.ejs'))); + assert.strictEqual((await loader.load('auth/reset.ejs')).source, '

reset

'); + assert.strictEqual(loader.loadSync('auth/reset.ejs').source, '

reset

'); + // Missing, a directory, another extension and a name that is not one: none. + for (const name of [ + 'missing.ejs', + 'folder.ejs', + 'notes.txt', + '../secret.ejs', + 'auth/../welcome.ejs', + '/etc/passwd', + // Before: read, as a segment after the first could start with a dot. + 'auth/../.env', + 'auth/.hidden.ejs', + ]) { + assert.strictEqual(await loader.load(name), undefined, name); + assert.strictEqual(loader.loadSync(name), undefined, name); + } + assert.strictEqual(await loader.load('welcome.ejs/x.ejs'), undefined); + assert.strictEqual(loader.loadSync('welcome.ejs/x.ejs'), undefined); + // Over maxBytes. + await assert.rejects(loader.load('big.ejs'), /more than 1024 bytes/); + assert.throws(() => loader.loadSync('big.ejs'), /more than 1024 bytes/); + // Any extension by default. + assert.strictEqual( + (await createFileLoader({ dir: root }).load('notes.txt')).source, + 'not a template', + ); +}); + +test('Templates: the file loader reads one real path, sync or not, however its directory is spelled', async (t) => { + // Before: loadSync resolved with the realpath of JavaScript, which keeps + // the case it is given and the short names of Windows (the temporary + // directory of a runner), and load with the system's, so a load after a + // loadSync was refused as leading out of the directory. + const { root } = makeTree(t); + const spelled = root.replace('mailer-templates-', 'MAILER-TEMPLATES-'); + if (!fs.existsSync(spelled)) { + t.skip('the file system tells case apart'); + return; + } + const real = fs.realpathSync.native(path.join(root, 'welcome.ejs')); + const syncFirst = createFileLoader({ dir: spelled }); + assert.strictEqual(syncFirst.loadSync('welcome.ejs').path, real); + assert.strictEqual((await syncFirst.load('welcome.ejs')).path, real); + const asyncFirst = createFileLoader({ dir: spelled }); + assert.strictEqual((await asyncFirst.load('welcome.ejs')).path, real); + assert.strictEqual(asyncFirst.loadSync('welcome.ejs').path, real); +}); + +test('Templates: the file loader reads no more than maxBytes of a file that grew since its size', async (t) => { + // Before: the size was taken by path and the file read whole by path + // after, so a file that grew in between, or was replaced, was read whole. + const { root } = makeTree(t); + const small = { isFile: () => true, size: 10 }; + const open = fsp.open; + const replaced = replace(t, [ + [fsp, 'stat', async () => small], + [fs, 'statSync', () => small], + [ + fsp, + 'open', + async (...args) => { + const handle = await open(...args); + handle.stat = async () => small; + return handle; + }, + ], + [fs, 'fstatSync', () => small], + ]); + if (!replaced) { + t.skip('node:fs cannot be replaced on this runtime'); + return; + } + const loader = createFileLoader({ dir: root, maxBytes: 1024 }); + await assert.rejects(loader.load('big.ejs'), /more than 1024 bytes/); + assert.throws(() => loader.loadSync('big.ejs'), /more than 1024 bytes/); + // A file that grew within maxBytes is read whole, through the descriptor. + assert.strictEqual((await createFileLoader({ dir: root }).load('big.ejs')).source.length, 2048); + assert.strictEqual(createFileLoader({ dir: root }).loadSync('big.ejs').source.length, 2048); +}); + +test('Templates: the file loader stops at the abort of its signal', async (t) => { + const { root } = makeTree(t); + const loader = createFileLoader({ dir: root }); + const reason = new Error('stop'); + await assert.rejects( + loader.load('welcome.ejs', { signal: AbortSignal.abort(reason) }), + (e) => e === reason, + ); + // Aborted after it found the file, before it opens it. + const controller = new AbortController(); + const realpath = fsp.realpath; + const replaced = replace(t, [ + [ + fsp, + 'realpath', + async (...args) => { + const real = await realpath(...args); + if (args[0].endsWith('welcome.ejs')) controller.abort(reason); + return real; + }, + ], + ]); + if (!replaced) { + t.skip('node:fs cannot be replaced on this runtime'); + return; + } + await assert.rejects( + loader.load('welcome.ejs', { signal: controller.signal }), + (e) => e === reason, + ); + assert.strictEqual((await loader.load('welcome.ejs', {})).source, '

Hi <%= name %>

'); +}); + +test('Templates: the file loader takes a directory it cannot open for none', async (t) => { + // Windows opens no directory: EISDIR, where the others open it and say it + // is no file. + const { root } = makeTree(t); + const denied = (code) => Object.assign(new Error(code), { code }); + let code = 'EISDIR'; + const replaced = replace(t, [ + [ + fsp, + 'open', + async () => { + throw denied(code); + }, + ], + [ + fs, + 'openSync', + () => { + throw denied(code); + }, + ], + ]); + if (!replaced) { + t.skip('node:fs cannot be replaced on this runtime'); + return; + } + const loader = createFileLoader({ dir: root }); + assert.strictEqual(await loader.load('folder.ejs'), undefined); + assert.strictEqual(loader.loadSync('folder.ejs'), undefined); + code = 'EACCES'; + await assert.rejects(loader.load('welcome.ejs'), { code: 'EACCES' }); + assert.throws(() => loader.loadSync('welcome.ejs'), { code: 'EACCES' }); +}); + +test('Templates: the file loader follows no symbolic link out of its directory', async (t) => { + if (process.platform === 'win32') { + t.skip('symbolic links need privileges on Windows'); + return; + } + const { base, root } = makeTree(t); + fs.symlinkSync(path.join(base, 'secret.ejs'), path.join(root, 'link.ejs')); + fs.symlinkSync(path.join(root, 'welcome.ejs'), path.join(root, 'inside.ejs')); + const loader = createFileLoader({ dir: root }); + await assert.rejects(loader.load('link.ejs'), /leads out of/); + assert.throws(() => loader.loadSync('link.ejs'), /leads out of/); + assert.strictEqual((await loader.load('inside.ejs')).source, '

Hi <%= name %>

'); + // A directory reached through a link is its real path. + const linked = path.join(base, 'linked'); + fs.symlinkSync(root, linked); + assert.strictEqual( + (await createFileLoader({ dir: linked }).load('welcome.ejs')).name, + 'welcome.ejs', + ); +}); + +test('Templates: the file loader surfaces what is not a missing file', async (t) => { + const { root } = makeTree(t); + const missingRoot = createFileLoader({ dir: path.join(root, 'nope') }); + await assert.rejects(missingRoot.load('welcome.ejs'), { code: 'ENOENT' }); + assert.throws(() => missingRoot.loadSync('welcome.ejs'), { code: 'ENOENT' }); + // A loader made before its directory exists reads it once it does. + const later = path.join(root, 'later'); + const early = createFileLoader({ dir: later }); + fs.mkdirSync(later); + fs.writeFileSync(path.join(later, 'a.ejs'), 'a'); + assert.strictEqual((await early.load('a.ejs')).source, 'a'); + if (process.platform !== 'win32') { + const loop = path.join(root, 'loop.ejs'); + fs.symlinkSync(loop, loop); + const loader = createFileLoader({ dir: root }); + await assert.rejects(loader.load('loop.ejs'), { code: 'ELOOP' }); + assert.throws(() => loader.loadSync('loop.ejs'), { code: 'ELOOP' }); + } +}); + +test('Templates: the loaders check their options', () => { + assert.throws(() => createFileLoader(), /needs dir/); + assert.throws(() => createFileLoader({ dir: '' }), /needs dir/); + assert.throws(() => createFileLoader({ dir: 'x', extensions: '.ejs' }), /must be an array/); + assert.throws(() => createFileLoader({ dir: 'x', extensions: ['ejs'] }), /starts with its dot/); + assert.throws(() => createFileLoader({ dir: 'x', maxBytes: 0 }), /maxBytes/); + assert.throws(() => createMemoryLoader(), /needs the templates/); + assert.throws(() => createMemoryLoader([]), /needs the templates/); + assert.throws(() => createMemoryLoader({ '../x': 'a' }), /template name/); + const memory = createMemoryLoader({ a: '

', 'b/c': { component: true } }); + assert.deepStrictEqual(memory.load('a'), { name: 'a', source: '

' }); + assert.deepStrictEqual(memory.loadSync('b/c'), { name: 'b/c', source: { component: true } }); + assert.strictEqual(memory.load('z'), undefined); +}); + +test('Templates: a renderer over the file loader renders from disk', async (t) => { + const { root } = makeTree(t); + const renderer = createTemplateRenderer({ + engine: createEjsEngine({ ejs }), + loader: createFileLoader({ dir: root, extensions: ['.ejs'] }), + }); + const output = await renderer.render({ template: 'welcome', context: { name: 'Ada' } }); + assert.deepStrictEqual(output, { + template: 'welcome.ejs', + html: '

Hi Ada

', + text: 'Hi Ada', + }); +}); diff --git a/tests/unit/templates-renderer.test.js b/tests/unit/templates-renderer.test.js new file mode 100644 index 0000000..addcc58 --- /dev/null +++ b/tests/unit/templates-renderer.test.js @@ -0,0 +1,1017 @@ +// Not ported: tests of the template renderer of ./templates, against the fake +// engine of tests/helpers/templates. +const { test } = require('node:test'); +const assert = require('node:assert'); +const { renderContent, renderFields } = require('../../src/render/index.js'); +const { + createCssInliner, + createMemoryLoader, + createTemplateRenderer, + TemplateLoadError, + TemplateNotFoundError, + TemplateRenderError, +} = require('../../src/templates/index.js'); +const { fakeEngine } = require('../helpers/templates/fake-engine.js'); + +const rendererOf = (templates, options = {}) => { + const engine = options.engine ?? fakeEngine(); + const renderer = createTemplateRenderer({ + engine, + loader: createMemoryLoader(templates), + ...options, + }); + return { renderer, engine }; +}; + +test('Templates: a template renders its HTML, its text derived', async () => { + const { renderer } = rendererOf({ 'welcome.tpl': '

Hi {{name}}

' }); + assert.deepStrictEqual(await renderer.render({ template: 'welcome', context: { name: 'A&B' } }), { + template: 'welcome.tpl', + html: '

Hi A&B

', + text: 'Hi A&B', + }); + // An explicit extension names the engine, and the result is the same. + assert.strictEqual( + (await renderer.render({ template: 'welcome.tpl', context: { name: 'x' } })).html, + '

Hi x

', + ); + const { renderer: noText } = rendererOf({ 'a.tpl': '

x

' }, { autoText: false }); + assert.deepStrictEqual(await noText.render({ template: 'a' }), { + template: 'a.tpl', + html: '

x

', + }); +}); + +test('Templates: the subject comes from the front matter, the text from its variant', async () => { + const { renderer } = rendererOf({ + 'reset.tpl': '---\n# a comment\nsubject: "Reset, {{name}} & co"\n---\n

{{url}}

', + 'reset.text.tpl': 'Open {{url}} & go', + }); + const output = await renderer.render({ + template: 'reset', + context: { name: 'Ada', url: 'https://example.com/r?a=1&b=2' }, + }); + assert.strictEqual(output.subject, 'Reset, Ada & co'); + assert.strictEqual(output.text, 'Open https://example.com/r?a=1&b=2 & go'); + assert.strictEqual(output.html, '

https://example.com/r?a=1&b=2

'); + // Its subject variant, its whitespace collapsed. + const { renderer: variant } = rendererOf({ + 'a.tpl': '

x

', + 'a.subject.tpl': ' Hello\n {{name}}\n', + }); + assert.strictEqual( + (await variant.render({ template: 'a', context: { name: 'B' } })).subject, + 'Hello B', + ); + // renderFields and renderContent take the renderer. + assert.deepStrictEqual( + await renderFields(renderer, { template: 'reset', context: { url: 'u' } }), + { + subject: 'Reset, & co', + content: { html: '

u

', text: 'Open u & go' }, + }, + ); + assert.deepStrictEqual(await renderContent(variant, { template: 'a' }), { + html: '

x

', + text: 'x', + }); +}); + +test('Templates: a template of text alone', async () => { + const { renderer } = rendererOf({ + 'note.text.tpl': '---\nsubject: Note\n---\nHi {{name}} & all', + }); + assert.deepStrictEqual(await renderer.render({ template: 'note', context: { name: '' } }), { + template: 'note.text.tpl', + text: 'Hi & all', + subject: 'Note', + }); +}); + +test('Templates: locales fall back by subtag, then the default, then none', async () => { + const { renderer } = rendererOf( + { + 'welcome.tpl': '---\nsubject: Welcome\n---\n

Hi

', + 'welcome.uk.tpl': '---\nsubject: Вітаємо\n---\n

Привіт

', + 'welcome.de.tpl': '

Hallo

', + }, + { defaultLocale: 'de-AT' }, + ); + const uk = await renderer.render({ template: 'welcome', locale: 'uk-UA' }); + assert.deepStrictEqual([uk.template, uk.locale, uk.subject], ['welcome.uk.tpl', 'uk', 'Вітаємо']); + const fr = await renderer.render({ template: 'welcome', locale: 'fr' }); + assert.deepStrictEqual([fr.template, fr.locale], ['welcome.de.tpl', 'de']); + // The subject is that of the locale the body was found at: none here, not + // the English one. + assert.strictEqual(fr.subject, undefined); + const { renderer: plain } = rendererOf({ 'welcome.tpl': '

Hi

' }); + const none = await plain.render({ template: 'welcome', locale: 'EN-us' }); + assert.deepStrictEqual([none.template, none.locale], ['welcome.tpl', undefined]); + await assert.rejects(plain.render({ template: 'welcome', locale: 'not a tag' }), RangeError); +}); + +test('Templates: with locales, a locale asked for is one of them or none', async () => { + // Before: every locale a user chose was looked up, and kept an entry of the + // cache of its own, so made-up tags pushed the templates out of it. + const reads = []; + const loader = createMemoryLoader({ + 'welcome.tpl': '

Hi

', + 'welcome.uk.tpl': '

Привіт

', + }); + const counted = { + load: (name) => { + reads.push(name); + return loader.load(name); + }, + }; + const renderer = createTemplateRenderer({ + engine: fakeEngine(), + loader: counted, + locales: ['uk', 'EN'], + defaultLocale: 'en', + }); + const uk = await renderer.render({ template: 'welcome', locale: 'uk-UA-x-anything' }); + assert.deepStrictEqual([uk.template, uk.locale], ['welcome.uk.tpl', 'uk']); + const before = reads.length; + for (let index = 0; index < 20; index++) { + const other = await renderer.render({ template: 'welcome', locale: `de-x-tag${index}` }); + assert.deepStrictEqual([other.template, other.locale], ['welcome.tpl', undefined]); + } + // The twenty made-up tags were one lookup: the template without a locale. + assert.ok(reads.length - before <= 4, String(reads.length - before)); + assert.ok(!reads.some((name) => name.includes('tag'))); + assert.throws( + () => createTemplateRenderer({ engine: fakeEngine(), loader, locales: 'uk' }), + /locales must be an array/, + ); + assert.throws( + () => createTemplateRenderer({ engine: fakeEngine(), loader, locales: ['not a tag'] }), + RangeError, + ); +}); + +test('Templates: a missing template names what was tried, and is not retryable', async () => { + const { renderer } = rendererOf({}, { defaultLocale: 'en' }); + const error = await renderer.render({ template: 'nope', locale: 'uk' }).catch((e) => e); + assert.ok(error instanceof TemplateNotFoundError); + assert.strictEqual(error.code, 'MAILER_TEMPLATE_NOT_FOUND'); + assert.strictEqual(error.retryable, false); + assert.deepStrictEqual(error.tried, [ + 'nope.uk.tpl', + 'nope.en.tpl', + 'nope.tpl', + 'nope.uk.text.tpl', + 'nope.en.text.tpl', + 'nope.text.tpl', + ]); + const many = new TemplateNotFoundError('x', ['1', '2', '3', '4', '5', '6', '7', '8', '9', '10']); + assert.match(many.message, /8, and 2 more|8 and 2 more/); +}); + +test('Templates: a name is never a path', async () => { + const { renderer } = rendererOf({}); + for (const template of [ + '../secret', + '/etc/passwd', + 'a//b', + 'a/./b', + 'a\\b', + '', + '.hidden', + // Before: a segment after the first could start with a dot. + 'partials/.env', + 'a/.git/config', + 42, + ]) { + await assert.rejects(renderer.render({ template }), TypeError, String(template)); + } + await assert.rejects(renderer.render(null), TypeError); + await assert.rejects(renderer.render({ template: 'a', context: [] }), TypeError); + await assert.rejects(renderer.render({ template: 'a', locale: 1 }), TypeError); +}); + +test('Templates: a loader is given template names alone, whoever asks', async () => { + // Before: a name an engine read, or one variantName gave, reached the loader + // unchecked, such as a database query by name. + const asked = []; + const loader = { + load: (name) => { + asked.push(name); + return name === 'a.tpl' ? { name, source: '

' } : undefined; + }, + loadSync: (name) => { + asked.push(name); + return undefined; + }, + }; + const reader = (read) => ({ + name: 'reader', + extensions: ['.tpl'], + compile: async (source, context) => { + await read(context.includes); + return () => source; + }, + }); + for (const read of [ + (includes) => includes.read('../secret.tpl'), + (includes) => includes.readSync('partials/.env'), + (includes) => includes.read(42), + ]) { + const renderer = createTemplateRenderer({ engine: reader(read), loader }); + await assert.rejects(renderer.render({ template: 'a' }), /must be a template name/); + } + const variants = createTemplateRenderer({ + engine: fakeEngine(), + loader, + variantName: (template) => `../${template}`, + }); + await assert.rejects( + variants.render({ template: 'a' }), + /variantName gives must be a template name/, + ); + assert.deepStrictEqual(asked, [ + 'a.tpl', + 'a.text.tpl', + 'a.subject.tpl', + 'a.tpl', + 'a.text.tpl', + 'a.subject.tpl', + 'a.tpl', + 'a.text.tpl', + 'a.subject.tpl', + ]); +}); + +test('Templates: includes are read through the loader, from the root or the partials', async () => { + const { renderer } = rendererOf( + { + 'emails/welcome.tpl': '{{> header}}

{{name}}

{{> ./sign}}{{> /legal}}', + 'partials/header.tpl': '
', + 'emails/sign.tpl': 'sign', + 'legal.tpl': 'legal', + }, + { partials: ['partials'] }, + ); + assert.strictEqual( + (await renderer.render({ template: 'emails/welcome', context: { name: 'x' } })).html, + '

x

signlegal', + ); + const { renderer: out } = rendererOf({ 'a.tpl': '{{> ../../etc/passwd}}' }); + const error = await out.render({ template: 'a' }).catch((e) => e); + assert.ok(error instanceof TemplateNotFoundError); + assert.deepStrictEqual(error.tried, []); + assert.strictEqual(error.from, 'a.tpl'); +}); + +test('Templates: a layout wraps the HTML, opted out or named by front matter', async () => { + const templates = { + 'layouts/main.tpl': '{{title}}{{{body}}}', + 'layouts/alt.tpl': '{{{body}}}', + 'a.tpl': '

{{name}}

', + 'b.tpl': '---\nlayout: none\n---\n

b

', + 'c.tpl': '---\nlayout: layouts/alt\n---\n

c

', + 'd.tpl': '---\nlayout: layouts/missing\n---\n

d

', + 'layouts/nested.tpl': '---\nlayout: layouts/main\n---\n{{{body}}}', + 'e.tpl': '---\nlayout: layouts/nested\n---\n

e

', + 'f.text.tpl': 'text alone', + }; + const { renderer } = rendererOf(templates, { + layout: 'layouts/main', + globals: { title: 'T' }, + }); + assert.strictEqual( + (await renderer.render({ template: 'a', context: { name: 'x' } })).html, + 'T

x

', + ); + assert.strictEqual((await renderer.render({ template: 'b' })).html, '

b

'); + assert.strictEqual((await renderer.render({ template: 'c' })).html, '

c

'); + const missing = await renderer.render({ template: 'd' }).catch((e) => e); + assert.ok(missing instanceof TemplateNotFoundError); + assert.strictEqual(missing.from, 'd.tpl'); + const nested = await renderer.render({ template: 'e' }).catch((e) => e); + assert.ok(nested instanceof TemplateRenderError); + assert.strictEqual(nested.phase, 'layout'); + // A template of text alone takes no layout. + assert.deepStrictEqual(await renderer.render({ template: 'f' }), { + template: 'f.text.tpl', + text: 'text alone', + }); + // A layout of text alone has no HTML to wrap with. + const { renderer: textLayout } = rendererOf( + { 'a.tpl': '

a

', 'l.text.tpl': 'x' }, + { layout: 'l' }, + ); + const noHtml = await textLayout.render({ template: 'a' }).catch((e) => e); + assert.strictEqual(noHtml.phase, 'layout'); +}); + +test('Templates: globals sit under the context, per input when a function', async () => { + const { renderer } = rendererOf( + { 'a.tpl': '{{who}} {{site}} {{locale}}' }, + { globals: async (input) => ({ who: 'all', site: 'S', locale: input.locale ?? '-' }) }, + ); + assert.strictEqual( + (await renderer.render({ template: 'a', context: { who: 'me' }, locale: 'uk' })).html, + 'me S uk', + ); + const { renderer: bad } = rendererOf({ 'a.tpl': 'x' }, { globals: () => 42 }); + await assert.rejects(bad.render({ template: 'a' }), /globals must give an object/); + // A failure of globals, such as an i18n backend that is down, stays as it is. + const down = new Error('i18n down'); + const { renderer: failing } = rendererOf( + { 'a.tpl': 'x' }, + { globals: () => Promise.reject(down) }, + ); + assert.strictEqual(await failing.render({ template: 'a' }).catch((e) => e), down); + const { renderer: none } = rendererOf({ 'a.tpl': 'x{{y}}' }, { globals: () => undefined }); + assert.strictEqual((await none.render({ template: 'a', context: { y: 1 } })).html, 'x1'); +}); + +test('Templates: each template is compiled once, concurrent renders included', async () => { + const engine = fakeEngine({ async: true }); + const { renderer } = rendererOf( + { 'a.tpl': '---\nsubject: S {{n}}\n---\n

{{n}}

{{> p}}', 'p.tpl': '' }, + { engine }, + ); + const outputs = await Promise.all([ + renderer.render({ template: 'a', context: { n: 1 } }), + renderer.render({ template: 'a', context: { n: 2 } }), + renderer.render({ template: 'a', context: { n: 3 } }), + ]); + assert.deepStrictEqual( + outputs.map((output) => output.subject), + ['S 1', 'S 2', 'S 3'], + ); + assert.strictEqual(engine.compiled.length, 2, 'the body and the subject, once'); + renderer.clear(); + await renderer.render({ template: 'a', context: { n: 4 } }); + assert.strictEqual(engine.compiled.length, 4); + // Without a cache, each render compiles anew. + const fresh = fakeEngine(); + const { renderer: uncached } = rendererOf({ 'a.tpl': '

' }, { engine: fresh, cache: false }); + await uncached.render({ template: 'a' }); + await uncached.render({ template: 'a' }); + assert.strictEqual(fresh.compiled.length, 2); + // A cache of one entry drops the oldest. + const small = fakeEngine(); + const { renderer: bounded } = rendererOf( + { 'a.tpl': 'a', 'b.tpl': 'b' }, + { engine: small, cache: { maxEntries: 1 } }, + ); + await bounded.render({ template: 'a' }); + await bounded.render({ template: 'b' }); + await bounded.render({ template: 'a' }); + assert.strictEqual(small.compiled.length, 3); +}); + +test('Templates: a template that is missing at first is found once it is there', async () => { + const sources = Object.create(null); + const loader = { + load: (name) => (name in sources ? { name, source: sources[name] } : undefined), + }; + const renderer = createTemplateRenderer({ engine: fakeEngine(), loader }); + await assert.rejects(renderer.render({ template: 'late' }), TemplateNotFoundError); + sources['late.tpl'] = '

late

'; + renderer.clear(); + assert.strictEqual((await renderer.render({ template: 'late' })).html, '

late

'); +}); + +test('Templates: every failure tells its step, engine, template and the input it was for', async () => { + // Before: the template and the phase alone, and no engine for a layout or + // an output that was no string, so a worker had to read the message. + const templates = { + 'welcome.uk.tpl': '

{{x}}

', + 'include.tpl': '{{> missing}}', + 'layout.tpl': '---\nlayout: other\n---\n{{{body}}}', + 'other.tpl': '{{{body}}}', + 'framed.tpl': '---\nlayout: layout\n---\n

', + 'lost.tpl': '---\nlayout: gone\n---\n

', + }; + const { renderer } = rendererOf(templates); + const fields = (error) => ({ + name: error.name, + phase: error.phase, + engine: error.engine, + template: error.template, + requested: error.requested, + locale: error.locale, + }); + const failure = (input) => renderer.render(input).catch(fields); + assert.deepStrictEqual(await failure({ template: 'nope', locale: 'uk' }), { + name: 'TemplateNotFoundError', + phase: 'resolve', + engine: undefined, + template: 'nope', + requested: 'nope', + locale: 'uk', + }); + assert.deepStrictEqual(await failure({ template: 'include' }), { + name: 'TemplateNotFoundError', + phase: 'compile', + engine: 'fake', + template: 'missing', + requested: 'include', + locale: undefined, + }); + assert.deepStrictEqual( + await failure({ template: 'welcome', locale: 'uk-UA', context: { explode: true } }), + { + name: 'TemplateRenderError', + phase: 'render', + engine: 'fake', + template: 'welcome.uk.tpl', + requested: 'welcome', + locale: 'uk-UA', + }, + ); + assert.deepStrictEqual(await failure({ template: 'framed' }), { + name: 'TemplateRenderError', + phase: 'layout', + engine: 'fake', + template: 'layout.tpl', + requested: 'framed', + locale: undefined, + }); + assert.deepStrictEqual((await failure({ template: 'lost' })).phase, 'layout'); + // Renders that wait for one compile each get their own input. + const { renderer: shared } = rendererOf( + { 'a.tpl': '{{!syntax' }, + { engine: fakeEngine({ async: true }) }, + ); + const [first, second] = await Promise.all([ + shared.render({ template: 'a', locale: 'uk' }).catch((e) => e), + shared.render({ template: 'a.tpl', locale: 'de' }).catch((e) => e), + ]); + assert.deepStrictEqual( + [first.requested, first.locale, second.requested, second.locale], + ['a', 'uk', 'a.tpl', 'de'], + ); + assert.ok(first.cause instanceof SyntaxError); + assert.match(first.stack, /SyntaxError|TemplateRenderError/); + const load = await createTemplateRenderer({ + engine: fakeEngine(), + loader: { + load: () => { + throw new Error('down'); + }, + }, + }) + .render({ template: 'x' }) + .catch(fields); + assert.deepStrictEqual( + [load.name, load.phase, load.requested], + ['TemplateLoadError', 'resolve', 'x'], + ); + // The parts of an engine that are no strings name it too. + const parts = createTemplateRenderer({ + engine: { name: 'parts', extensions: [], compile: () => () => ({}) }, + loader: createMemoryLoader({ x: '' }), + }); + assert.strictEqual((await parts.render({ template: 'x' }).catch((e) => e)).engine, 'parts'); +}); + +test('Templates: failures are classified by phase, the engine error their cause', async () => { + const { renderer } = rendererOf({ + 'syntax.tpl': '

{{!syntax

', + 'boom.tpl': '

{{x}}

', + 'front.tpl': '---\nsubjct: typo\n---\n

', + 'open.tpl': '---\nsubject: x\n', + 'line.tpl': '---\nnot a pair\n---\n', + 'twice.tpl': '---\nsubject: a\nsubject: b\n---\n', + }); + const syntax = await renderer.render({ template: 'syntax' }).catch((e) => e); + assert.ok(syntax instanceof TemplateRenderError); + assert.deepStrictEqual( + [syntax.phase, syntax.engine, syntax.code], + ['compile', 'fake', 'MAILER_TEMPLATE_RENDER'], + ); + assert.ok(syntax.cause instanceof SyntaxError); + assert.strictEqual(syntax.retryable, false); + const boom = await renderer + .render({ template: 'boom', context: { explode: true } }) + .catch((e) => e); + assert.strictEqual(boom.phase, 'render'); + assert.match(boom.message, /A helper exploded/); + for (const [template, pattern] of [ + ['front', /no key "subjct"/], + ['open', /no closing ---/], + ['line', /not "key: value"/], + ['twice', /subject twice/], + ]) { + const error = await renderer.render({ template }).catch((e) => e); + assert.strictEqual(error.phase, 'front-matter', template); + assert.match(error.message, pattern); + } +}); + +test('Templates: a loader that fails fails as a load of its template, which may pass on a retry', async () => { + // Before: the loader's error came back as it was, naming no template. + const outage = Object.assign(new Error('EMFILE: too many open files'), { code: 'EMFILE' }); + const loader = { + load: (name) => { + if (name === 'b.tpl') throw outage; + return name === 'a.tpl' ? { name, source: '{{> b}}' } : undefined; + }, + }; + const renderer = createTemplateRenderer({ engine: fakeEngine(), loader }); + const include = await renderer.render({ template: 'a' }).catch((e) => e); + assert.ok(include instanceof TemplateLoadError); + assert.deepStrictEqual( + [include.code, include.template, include.retryable, include.cause], + ['MAILER_TEMPLATE_LOAD', 'b.tpl', true, outage], + ); + assert.match(include.message, /The template "b.tpl" failed to load: EMFILE/); + const failing = createTemplateRenderer({ + engine: fakeEngine(), + loader: { load: () => Promise.reject(outage) }, + }); + const main = await failing.render({ template: 'x' }).catch((e) => e); + assert.deepStrictEqual([main.template, main.cause], ['x.tpl', outage]); + // A bug of the loader, or an error that says so, is left for the queue to + // classify by its cause, never marked retryable. + for (const cause of [new TypeError('x is undefined'), { retryable: false, message: 'gone' }]) { + const buggy = createTemplateRenderer({ + engine: fakeEngine(), + loader: { + load: () => { + throw cause; + }, + }, + }); + const error = await buggy.render({ template: 'x' }).catch((e) => e); + assert.ok(error instanceof TemplateLoadError); + assert.ok(!('retryable' in error)); + } + // A loader that throws an error of ./templates throws it as it is. + const refusal = new TemplateNotFoundError('x', []); + const own = createTemplateRenderer({ + engine: fakeEngine(), + loader: { + load: () => { + throw refusal; + }, + }, + }); + const refused = await own.render({ template: 'x' }).catch((e) => e); + assert.ok(refused instanceof TemplateNotFoundError); + assert.deepStrictEqual([refused.template, refused.message], ['x', refusal.message]); +}); + +test('Templates: finish makes the HTML of the outermost template, then the CSS is inlined', async () => { + const order = []; + const engine = fakeEngine({ + finish: (html, context) => { + order.push(`finish ${context.name}`); + return html.replace('', ''); + }, + }); + const inline = (html, options) => { + order.push(`inline ${JSON.stringify(options)}`); + return html.replace('

', '

'); + }; + const { renderer } = rendererOf( + { 'a.tpl': '

a

', 'layout.tpl': '{{{body}}}' }, + { engine, layout: 'layout', inlineCss: createCssInliner({ inline }) }, + ); + const output = await renderer.render({ template: 'a' }); + assert.strictEqual(output.html, '

a

'); + assert.deepStrictEqual(order, [ + 'finish layout.tpl', + 'inline {"loadRemoteStylesheets":false,"keepLinkTags":true}', + ]); + // A failure of the inliner rejects once, as a failure of its phase. + const { renderer: failing } = rendererOf( + { 'a.tpl': '

' }, + { + inlineCss: createCssInliner({ + inline: () => { + throw new Error('bad css'); + }, + }), + }, + ); + const error = await failing.render({ template: 'a' }).catch((e) => e); + assert.strictEqual(error.phase, 'inline-css'); + const { renderer: notString } = rendererOf( + { 'a.tpl': '

' }, + { inlineCss: createCssInliner({ inline: () => 1 }) }, + ); + assert.strictEqual( + (await notString.render({ template: 'a' }).catch((e) => e)).phase, + 'inline-css', + ); + const { renderer: overridden } = rendererOf( + { 'a.tpl': '

' }, + { + inlineCss: createCssInliner({ + inline: (html, o) => `${html}${o.keepLinkTags}`, + options: { keepLinkTags: false }, + }), + }, + ); + assert.strictEqual((await overridden.render({ template: 'a' })).html, '

false'); +}); + +test('Templates: an engine that renders its parts gives HTML, text and subject at once', async () => { + const parts = { + name: 'parts', + extensions: [], + compile: (source) => (scope) => ({ + html: `${scope.n}`, + text: `t${scope.n}`, + subject: source, + }), + }; + const renderer = createTemplateRenderer({ + engine: parts, + loader: createMemoryLoader({ welcome: 'Subject here' }), + }); + assert.deepStrictEqual(await renderer.render({ template: 'welcome', context: { n: 1 } }), { + template: 'welcome', + html: '1', + text: 't1', + subject: 'Subject here', + }); + const empty = createTemplateRenderer({ + engine: { ...parts, compile: () => () => ({}) }, + loader: createMemoryLoader({ x: '' }), + }); + assert.match( + (await empty.render({ template: 'x' }).catch((e) => e)).message, + /neither html nor text/, + ); + const wrong = createTemplateRenderer({ + engine: { ...parts, compile: () => () => ({ html: 1 }) }, + loader: createMemoryLoader({ x: '' }), + }); + assert.match( + (await wrong.render({ template: 'x' }).catch((e) => e)).message, + /html as no string/, + ); + const nothing = createTemplateRenderer({ + engine: { ...parts, compile: () => () => 42 }, + loader: createMemoryLoader({ x: '' }), + }); + assert.match((await nothing.render({ template: 'x' }).catch((e) => e)).message, /gave no string/); + const textNotString = createTemplateRenderer({ + engine: { + name: 'x', + extensions: ['.x'], + compile: (source, context) => () => (context.part === 'text' ? 1 : '

'), + }, + loader: createMemoryLoader({ 'a.x': '', 'a.text.x': '' }), + }); + const notText = await textNotString.render({ template: 'a' }).catch((e) => e); + assert.match(notText.message, /gave no string/); + assert.strictEqual(notText.template, 'a.text.x'); + const notFunction = createTemplateRenderer({ + engine: { ...parts, compile: () => 'nope' }, + loader: createMemoryLoader({ x: '' }), + }); + assert.match( + (await notFunction.render({ template: 'x' }).catch((e) => e)).message, + /gave no function/, + ); +}); + +test('Templates: no layout goes silently unapplied to an engine that gives its parts', async () => { + // Before: a layout was skipped for a template that gave its parts, React + // Email's, and the message went without it. + const parts = { + name: 'parts', + extensions: [], + compile: () => () => ({ html: '' }), + }; + assert.throws( + () => + createTemplateRenderer({ + engine: { ...parts, parts: true }, + loader: createMemoryLoader({}), + layout: 'main', + }), + /parts engine gives its parts at once, which no layout wraps/, + ); + const renderer = createTemplateRenderer({ + engine: parts, + loader: createMemoryLoader({ a: '', main: '' }), + layout: 'main', + }); + const error = await renderer.render({ template: 'a' }).catch((e) => e); + assert.ok(error instanceof TemplateRenderError); + assert.deepStrictEqual([error.phase, error.engine], ['layout', 'parts']); +}); + +test('Templates: engines are chosen by extension, the longest first', async () => { + const hbs = fakeEngine({ name: 'hbs', extensions: ['.hbs'] }); + const mjml = fakeEngine({ name: 'mjml', extensions: ['.mjml.hbs'] }); + const renderer = createTemplateRenderer({ + engines: [hbs, mjml], + loader: createMemoryLoader({ 'a.mjml.hbs': 'mjml', 'b.hbs': 'hbs', 'a.hbs': 'shadow' }), + }); + assert.strictEqual((await renderer.render({ template: 'a.mjml.hbs' })).html, 'mjml'); + assert.strictEqual((await renderer.render({ template: 'b' })).html, 'hbs'); + // Without an extension, the engines are tried in their order. + assert.strictEqual((await renderer.render({ template: 'a' })).html, 'shadow'); + const text = await renderer.render({ template: 'b', context: {} }); + assert.strictEqual(text.text, 'hbs'); +}); + +test('Templates: no part of a message is longer than maxOutputBytes', async () => { + // Before: no bound, so a template that repeated a value gave as much as it + // could hold. + const templates = { + 'a.tpl': '{{x}}', + 'a.text.tpl': '{{y}}', + 'b.tpl': '

', + 'b.subject.tpl': '{{x}}', + 'c.tpl': '{{x}}', + 'layout.tpl': '{{{body}}}{{{body}}}', + }; + const big = async (template, context, options) => { + const { renderer } = rendererOf(templates, { maxOutputBytes: 30, ...options }); + return renderer.render({ template, context }).catch((e) => e); + }; + for (const [template, context] of [ + ['a', { x: 'x'.repeat(31), y: '' }], + ['a', { x: '', y: 'я'.repeat(16) }], + ['b', { x: 'x'.repeat(31) }], + ]) { + const error = await big(template, context); + assert.ok(error instanceof TemplateRenderError, template); + assert.match(error.message, /more than 30 bytes \(maxOutputBytes\)/); + } + // Ten Cyrillic letters are 20 bytes, and need no count. + assert.strictEqual((await big('a', { x: 'я'.repeat(10), y: 'y' })).html, 'я'.repeat(10)); + // The layout counts too. + const layout = await big('c', { x: 'x'.repeat(20) }, { layout: 'layout' }); + assert.strictEqual(layout.phase, 'layout'); + const parts = createTemplateRenderer({ + engine: { name: 'parts', extensions: [], compile: () => () => ({ html: 'x'.repeat(40) }) }, + loader: createMemoryLoader({ x: '' }), + maxOutputBytes: 30, + }); + assert.match((await parts.render({ template: 'x' }).catch((e) => e)).message, /maxOutputBytes/); + const inlined = await big( + 'b', + {}, + { inlineCss: createCssInliner({ inline: () => 'x'.repeat(31) }) }, + ); + assert.strictEqual(inlined.phase, 'inline-css'); + const finished = await big('b', {}, { engine: fakeEngine({ finish: () => 'x'.repeat(31) }) }); + assert.strictEqual(finished.phase, 'finish'); +}); + +test('Templates: a render that hangs times out, and what it waited for is read anew', async () => { + // Before: no timeout, and a read or a compile that never settled was kept + // and waited for by every render after it, until clear(). + let hang = true; + const signals = new Map(); + const loader = { + load: (name, options) => { + signals.set(name, options.signal); + if (name !== 'a.tpl') return undefined; + return hang ? new Promise(() => {}) : { name, source: '

{{n}}

' }; + }, + }; + const renderer = createTemplateRenderer({ + engine: fakeEngine(), + loader, + renderTimeoutMilliseconds: 50, + }); + const error = await renderer.render({ template: 'a', locale: 'uk' }).catch((e) => e); + assert.ok(error instanceof TemplateRenderError); + assert.deepStrictEqual( + [error.phase, error.retryable, error.template, error.requested, error.locale], + ['timeout', true, 'a', 'a', 'uk'], + ); + assert.match(error.message, /did not render in time: It took over 50 ms/); + // The loader is told to stop, at the timeout of its read, which may come a + // tick after that of the render. + await new Promise((resolve) => setTimeout(resolve, 100)); + assert.ok(signals.get('a.tpl').aborted); + assert.strictEqual(signals.get('a.tpl').reason.name, 'TimeoutError'); + assert.ok(!signals.get('a.uk.tpl').aborted); + hang = false; + assert.strictEqual( + (await renderer.render({ template: 'a', context: { n: 1 } })).html, + '

1

', + ); + // A compile that hangs is made anew too. + let stalls = 1; + const engine = fakeEngine(); + const compile = engine.compile; + engine.compile = (source, context) => + stalls-- > 0 ? new Promise(() => {}) : compile(source, context); + const { renderer: stalled } = rendererOf( + { 'b.tpl': '

' }, + { engine, renderTimeoutMilliseconds: 50 }, + ); + assert.strictEqual((await stalled.render({ template: 'b' }).catch((e) => e)).phase, 'timeout'); + await new Promise((resolve) => setTimeout(resolve, 100)); + assert.strictEqual((await stalled.render({ template: 'b' })).html, '

'); +}); + +test('Templates: a render stops at the abort of its signal, with its reason', async () => { + // Before: render took no signal, so a worker shutting down waited for it. + const seen = []; + const loader = { + load: (name, options) => { + seen.push(options.signal); + return name === 'a.tpl' ? new Promise(() => {}) : undefined; + }, + }; + const renderer = createTemplateRenderer({ engine: fakeEngine(), loader }); + const controller = new AbortController(); + const rendering = renderer.render({ template: 'a', signal: controller.signal }); + const reason = new Error('shutting down'); + setTimeout(() => controller.abort(reason), 10); + assert.strictEqual(await rendering.catch((e) => e), reason); + // An aborted signal stops it before it reads anything. + const before = seen.length; + const aborted = AbortSignal.abort(reason); + assert.strictEqual( + await renderer.render({ template: 'a', signal: aborted }).catch((e) => e), + reason, + ); + assert.strictEqual(seen.length, before); + // Without a cache, the reads are the render's own, and its signal stops them. + const uncached = createTemplateRenderer({ engine: fakeEngine(), loader, cache: false }); + const own = new AbortController(); + const pending = uncached.render({ template: 'a', signal: own.signal }); + await new Promise((resolve) => setTimeout(resolve, 10)); + own.abort(reason); + assert.strictEqual(await pending.catch((e) => e), reason); + assert.ok(seen.at(-1).aborted); + // A render that ends first is not touched by a later abort. + const done = new AbortController(); + const { renderer: quick } = rendererOf({ 'q.tpl': '

' }); + assert.strictEqual((await quick.render({ template: 'q', signal: done.signal })).html, '

'); + done.abort(); + // An abort between the steps of a render stops it there. + const later = new AbortController(); + const { renderer: steps } = rendererOf( + { 's.tpl': '

' }, + { globals: () => later.abort(reason) }, + ); + assert.strictEqual( + await steps.render({ template: 's', signal: later.signal }).catch((e) => e), + reason, + ); + await assert.rejects( + renderer.render({ template: 'a', signal: {} }), + /signal must be an AbortSignal/, + ); +}); + +test('Templates: a logger hears each compile and each render at debug, a hit or a miss', async () => { + // Before: nothing told how long a compile or a render took, or whether the + // cache had it. + const records = []; + const logger = { + debug: (properties, message) => records.push({ message, ...properties }), + isLevelEnabled: (level) => level === 'debug', + }; + const { renderer } = rendererOf( + { 'a.tpl': '---\nsubject: S\n---\n

{{n}}

', 'a.uk.tpl': '

uk

' }, + { logger }, + ); + await renderer.render({ template: 'a', context: { n: 1 } }); + await renderer.render({ template: 'a', context: { n: 2 } }); + await renderer.render({ template: 'a', locale: 'uk' }); + const shape = (record) => { + assert.ok(record.durationMs >= 0, record.message); + return { ...record, durationMs: undefined }; + }; + assert.deepStrictEqual(records.map(shape), [ + { + message: 'templates.compiled', + component: 'templates', + template: 'a.tpl', + engine: 'fake', + part: 'html', + durationMs: undefined, + }, + { + message: 'templates.rendered', + component: 'templates', + template: 'a.tpl', + requested: 'a', + locale: undefined, + engine: 'fake', + cache: 'miss', + durationMs: undefined, + }, + { + message: 'templates.rendered', + component: 'templates', + template: 'a.tpl', + requested: 'a', + locale: undefined, + engine: 'fake', + cache: 'hit', + durationMs: undefined, + }, + { + message: 'templates.compiled', + component: 'templates', + template: 'a.uk.tpl', + engine: 'fake', + part: 'html', + durationMs: undefined, + }, + { + message: 'templates.rendered', + component: 'templates', + template: 'a.uk.tpl', + requested: 'a', + locale: 'uk', + engine: 'fake', + cache: 'miss', + durationMs: undefined, + }, + ]); + // Above debug, nothing is built. + const quiet = []; + const { renderer: info } = rendererOf( + { 'a.tpl': '

' }, + { logger: { debug: () => quiet.push(1), isLevelEnabled: () => false } }, + ); + await info.render({ template: 'a' }); + assert.strictEqual(quiet.length, 0); +}); + +test('Templates: the renderer checks its options', () => { + const engine = fakeEngine(); + const loader = createMemoryLoader({}); + const bad = [ + [{}, /needs an engine/], + [{ engine, engines: [engine], loader }, /not both/], + [{ engines: [], loader }, /needs an engine/], + [{ engine: {}, loader }, /must be a template engine/], + [{ engine: { compile() {}, extensions: [] }, loader }, /must have a name/], + [{ engine: { name: 'x', compile() {} }, loader }, /must list its extensions/], + [{ engines: [engine, fakeEngine()], loader }, /Two engines take .tpl/], + [ + { engines: [engine, { name: 'b', extensions: [], compile() {} }], loader }, + /must be the only one/, + ], + [{ engine }, /needs a loader/], + [{ engine, loader, partials: 'p' }, /partials must be an array/], + [{ engine, loader, partials: ['../p'] }, /directory of partials/], + [{ engine, loader, partials: ['p/.hidden'] }, /directory of partials/], + [{ engine, loader, layout: '/abs' }, /layout must be a template name/], + [{ engine, loader, globals: 1 }, /globals must be an object or a function/], + [{ engine, loader, frontMatter: 'yes' }, /frontMatter must be a boolean/], + [{ engine, loader, autoText: 1 }, /autoText must be a boolean/], + [{ engine, loader, cache: 'yes' }, /cache must be a boolean/], + [{ engine, loader, cache: { maxEntries: 0 } }, /cache.maxEntries/], + [{ engine, loader, variantName: 1 }, /variantName must be a function/], + [{ engine, loader, inlineCss: 1 }, /inlineCss must be a function/], + [{ engine, loader, defaultLocale: 1 }, /defaultLocale must be a string/], + [{ engine, loader, maxIncludes: 0 }, /maxIncludes must be a positive whole number/], + [{ engine, loader, maxIncludeDepth: 1.5 }, /maxIncludeDepth must be/], + [{ engine, loader, maxOutputBytes: '1' }, /maxOutputBytes must be/], + [{ engine, loader, renderTimeoutMilliseconds: -1 }, /renderTimeoutMilliseconds must be/], + [{ engine, loader, renderTimeoutMilliseconds: Number.NaN }, /renderTimeoutMilliseconds/], + ]; + for (const [options, pattern] of bad) { + assert.throws(() => createTemplateRenderer(options), pattern, String(pattern)); + } + assert.throws(() => createTemplateRenderer(), /needs an engine/); + createTemplateRenderer({ engine, loader, maxIncludes: Infinity, maxOutputBytes: Infinity }); + createTemplateRenderer({ engine, loader, renderTimeoutMilliseconds: 0 }); + createTemplateRenderer({ engine, loader, renderTimeoutMilliseconds: Infinity }); + assert.throws(() => createCssInliner({}), /needs inline/); + assert.throws(() => createCssInliner({ inline() {}, options: 1 }), /must be an object/); +}); + +test('Templates: variantName names the variants, frontMatter: false reads none', async () => { + const { renderer } = rendererOf( + { 'welcome-uk-html.tpl': '

uk

', 'welcome-uk-subject.tpl': 'S' }, + { variantName: (template, { locale, part }) => `${template}-${locale ?? 'x'}-${part}` }, + ); + const output = await renderer.render({ template: 'welcome', locale: 'uk' }); + assert.deepStrictEqual([output.html, output.subject], ['

uk

', 'S']); + const { renderer: broken } = rendererOf({}, { variantName: () => 1 }); + await assert.rejects( + broken.render({ template: 'a' }), + /variantName gives must be a template name/, + ); + const { renderer: raw } = rendererOf( + { 'a.tpl': '---\nsubject: x\n---\n

' }, + { frontMatter: false }, + ); + const plain = await raw.render({ template: 'a' }); + assert.strictEqual(plain.subject, undefined); + assert.match(plain.html, /^---/); + // A subject that is only whitespace is none. + const { renderer: blank } = rendererOf({ 'a.tpl': '

', 'a.subject.tpl': ' ' }); + assert.strictEqual((await blank.render({ template: 'a' })).subject, undefined); + // A BOM before the front matter, and CRLF lines. + const { renderer: bom } = rendererOf({ 'a.tpl': '---\r\nsubject: "B"\r\n---\r\n

' }); + assert.strictEqual((await bom.render({ template: 'a' })).subject, 'B'); + // A first line that only starts with --- is no front matter. + const { renderer: rule } = rendererOf({ 'a.tpl': '----\n

', 'b.tpl': '---' }); + assert.strictEqual((await rule.render({ template: 'a' })).html, '----\n

'); + assert.strictEqual((await rule.render({ template: 'b' })).html, '---'); +});