Skip to content

Migronaut v2.4: migration logs for your users, and architecture diagrams in the docs - #7

Merged
Alex-Dolid merged 21 commits into
mainfrom
feature/userland-logs
Oct 7, 2026
Merged

Alex-Dolid merged 21 commits into
mainfrom
feature/userland-logs

Conversation

@Alex-Dolid

Copy link
Copy Markdown

Summary

Release 2.4.0: migration logs for the application's users. Every migration gets ctx.logger,
bound to its run, and ctx.run; a line marked userland: true is also emitted as the
migration:log event, for the application to store and show. Migronaut stores none of it — it
captures and correlates, the application keeps. Plus architecture diagrams across the docs site, a
new How It Works page, and a home page that shows what 2.1 – 2.4 added.

Additive: a migration that never touches the new context fields, and a kit with no migration:log
listener, behave as before. Everything new is marked experimental.

What's new

Migration logs

  • ctx.logger — the kit's logger with runId, migration, direction, batch, attempt and,
    on a queue, jobId / groupId bound into every line. Pino's (fields, msg) order works too.
  • ctx.run — the same correlation as a frozen object; attempt is 2 or more when the driver
    retried the transaction and the body ran again.
  • migration:log — emitted only for calls whose fields hold userland: true, whatever the
    logger's level and with logger: null. The payload is { kind, runId, …correlation, level, msg, data, at, seq, truncated? }; data is a bounded, redacted copy — credentials and the values a
    server error quotes are masked, an Error becomes { name, message, code?, codeName? }.
  • Hooks get a logger too: beforeAll / afterAll the run's, the others the migration's.
  • job: { id, groupId? } on up, down and redo (and { id } on runBackgroundSlice).
  • Retried transactions, visible — attempts on migration:success / migration:error, the
    kit's log lines and the migronaut.migration span; migronaut.job.id / migronaut.job.group_id
    on the run span of a run that names a job.

Queue adapter

  • Each job's run is told its job. Its userland: true lines land in the job's own log as one-line
    ✎ rows, matched by job id and run id, so a timed-out body never writes into the next job's
    log.
  • The new userlandLogRows option (createMigrationQueue and both processors, default 1000) caps
    the rows one job's log takes; the rest are counted in one closing row.
  • Background lanes mirror their slice's lines into the lane job's log. Lanes an up drives inline
    (backgroundInline) carry the run's jobId and groupId, so their lines land in that migration
    job's log.

Background migrations

  • ctx.logger is bound to ctx.background — now frozen and typed BackgroundRunInfo, with the
    lane's runId, its jobId and the transaction attempt. A userland: true call emits
    migration:log with kind: 'background'; a dry run marks its lines dryRun: true and emits
    nothing.

Fixed

  • An async event listener that rejects no longer crashes the process: the kit is an
    EventEmitter with captureRejections, and the rejection is a debug line.

Docs

  • Migration Logs guide, with a storage recipe; the examples/migration-service app stores the
    logs and serves them over REST.
  • Diagrams — Mermaid, drawn in the browser by a small theme component (mermaid is a docs-only
    devDependency, loaded only by a page that has a diagram). A new How It Works page (the engine
    and its entry points, the life of a run, the lock, where the state lives, background migrations,
    what a run reports), diagrams in Core Concepts, Transactions, Declared Collections, Document
    Versioning, Background Migrations, the queue adapter and Migration Logs, and four in
    ARCHITECTURE.md.
  • Home page — twelve features that cover 2.1 – 2.4, a "How it fits together" diagram, the
    smaller features in one list; the Why page names the new features too.

Changed (for code typed against 2.3)

  • ctx.background is frozen and typed BackgroundRunInfo. A type change on an experimental
    API:
    its generation is number | undefined (as it always was at runtime for the live drift
    watcher), so strict TypeScript that reads it as a number needs a check.
  • ctx.logger is typed MigrationLogger — every MigronautLogger, and a pino instance, still fits.
    logger? and run? are optional on MigrationContext, so a context built by hand in a test
    still type-checks.
  • The kit's log lines of a run that names a job carry jobId / groupId.

Not in this release

  • No log storage, no kit.logs(), no migronaut logs command, no counters API — keeping the lines,
    and the UI that shows them, is the application's job (the guide has the recipe).
  • The CLI's terminal output is unchanged: the default terminal logger does not print fields.

How to review

The commits are ordered so you can review them one at a time:

  1. 8 feature phases — fix(core): contain rejected async event listeners …
    docs(examples): the migration service stores and serves migration logs;
  2. 7 pre-release review fixes — fix(core): what a migration logs is copied safely …
    fix(core): mask quoted server values in log data strings too;
  3. 5 docs commits — docs: render Mermaid diagrams on the docs site …
    docs: the Why page names what 2.1 to 2.4 added;
  4. chore(release): 2.4.0.

Start with src/core/migration-logger.js (ARCHITECTURE §5 and the "What a migration logs" part of
§6.6), then src/bullmq/processor.js and src/bullmq/background-processor.js. "Things that look
like bugs but aren't" in CLAUDE.md lists the intentional behaviours: userland: true exactly, no
storage, rows routed by job id and run id, a job sealed before its last flush, inline lanes told
apart by their group.

Test plan

  • prepublishOnly gate: lint, format:check, test:coverage, test:types, check:dts —
    2417 tests; coverage 97.89 / 93.44 / 98.52 (gate 90/90/90).
  • Real-BullMQ suite (bullmq-redis.test.js): 75/75 on redis:7-alpine.
  • size, docs:build (no warnings); all 26 docs diagrams render in the production build, in
    light and dark, and on a phone.
  • Sharded-cluster and Atlas Search suites: not affected by this branch.

Alex-Dolid and others added 21 commits October 7, 2026 15:08
An async listener whose promise rejects surfaced as an unhandledRejection,
which ends the process on Node 15+. The kit now extends EventEmitter with
captureRejections and reports the rejection at debug level, the way #emit
already contains a listener that throws.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A migration's context gains ctx.run — the run id, direction, migration,
batch, transaction attempt and actor, frozen — and ctx.logger, the kit's
logger with that correlation bound into every line. A call whose fields
hold userland: true is also emitted as migration:log, for the application
to store and show its users; migronaut stores none of it.

The runner builds a fresh context per attempt, so a transaction that the
driver retries hands its body attempt 2. Hooks get the same: the run's
correlation around the run, the migration's inside it. Event data is a
bounded, redacted copy (redactBounded). Both fields are optional in the
types, so a hand-built context still type-checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
job: { id, groupId? } names the queue job a run works for. It is bound
into ctx.run, every line ctx.logger writes, the migration:log event and
the kit's own lines for that run — nothing is stored. Validated before
the run touches anything (src/utils/job-ref.js holds the limits, for the
queue adapter to share); redo carries it into both halves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The migration processor passes job: { id, groupId } to kit.up/down, so a
migration's ctx.run and its migration:log events name the job that ran
it. Its userland lines are also written into that job's log as '✎ …'
rows next to the processor's own — matched by job id, so a body that
outlived its timeout cannot write into the next job, and sealed before
the job settles, so nothing lands after BullMQ removed it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…m background migrations

ctx.background gains the lane's runId, its queue job and the transaction
attempt, and is frozen; ctx.logger binds it into every line, and a
userland: true call emits migration:log with kind 'background'. The
attempt is counted where the engine runs user code again in a new
transaction: a transactional batch, a transactional step and the drift
watcher's rewrite. runBackgroundSlice takes job: { id } for the lane's
lines. A dry run logs with dryRun: true and emits nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A lane job hands its id to runBackgroundSlice, so the slice's lines and
migration:log events name it, and the lane's userland: true lines are
written into that job's log. Lanes run side by side, so they are routed
by the event's jobId through a map of the lanes in flight, and drained
before the lane moves on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Migration Logs guide: ctx.logger and ctx.run, the userland marker, the
migration:log payload, a recipe for keeping the events in a collection of
your own (indexes, TTL, where to subscribe), what retries, rollbacks,
timeouts and dry runs mean for them, the queue's job log rows and
background migrations. The migration context, hooks, API, BullMQ and
background guides, README, ARCHITECTURE, CLAUDE.md and the changelog
follow. A sync test pins every declared event to the docs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The worker keeps every migration:log event in a migration_logs collection
of the service's own (logs.js: indexed by run, job and migration, a TTL
of 90 days, inserts drained on shutdown), and the API serves them at
GET /migrations/runs/:runId/logs and GET /migrations/jobs/:id/logs. One
MongoClient, injected into the kit, serves both roles. The seed migration
logs what it inserted and removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- an Error in a userland call's data becomes { name, message, code?,
  codeName? }: kept by reference it was stored as {} (its message is not
  enumerable), and a driver error carried the offending document's values
  (keyValue, errorResponse.errmsg) into the application's store and Redis
- other values go by an allowlist: Dates, RegExps and BSON values as they
  are, binary data up to 4096 bytes, a Map as an object, a Set as an array,
  anything else the way JSON.stringify sees it — all within the bounds
- an own __proto__ key stays a key in redactBounded and redactDeep
- the event's msg masks the values a server error quotes (redactOutbound)
- jobRefIssue checks own keys only: constructor/toString/__proto__ no
  longer pass for a limit
- one plain-object predicate (canonical's isPlainObject) in redact and options

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- userlandLogRows (createMigrationQueue and both processors, default
  1000, 0 mirrors none): how many userland rows one job's log takes — a
  lane's per slice; the rest are counted in one closing row, so a
  migration that logs in a loop no longer writes an entry per line into
  Redis. migration:log still carries every line
- userlandRow writes the data's JSON only as far as the row shows it
  (it serialized up to ~4 MB to keep 1 KB, inside the migration's
  transaction), and keeps a row on one line: line breaks show as ⏎,
  control characters are dropped — a message can no longer forge a
  lifecycle row
- a migration job takes a line only from its own run: the job id and the
  run id must both match (a job put back in the queue keeps its id)
- one jobRefOf for both processors; the processor's jobFields, which
  collided with job-ref's, is now jobSummary

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With backgroundInline, the lanes an up drives under its lock had no job:
their userland lines named none, the migration processor dropped them
(kind 'background') and the background processor never knew them — so
they reached no job's log and could not be joined to the job.

- #runInline hands the run's job ({ id, groupId }) to drive, which names
  it on the coordinator's and every lane's deps
- ctx.background, the lanes' lines and their migration:log events carry
  jobId and groupId; each lane keeps a runId of its own (no parent run id)
- the migration processor takes a background line by job id and group:
  the facade shares one kit between both processors, and a background
  queue's lane — which has no group — may have the same id as a migration
  job; the background processor leaves lines with a group alone

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- the runner's attempt count reaches beyond ctx.run: attempts on
  migration:success / migration:error and the kit's line when the driver
  retried the transaction, in a failure's context, and always as
  migronaut.migration.attempts on the migration span
- a run that names a job puts migronaut.job.id / migronaut.job.group_id on
  its run span — the join for a queue of the user's own, without
  bullmq-otel
- a ctx.logger call that is dropped (a field's getter throws) or cut to
  the event's bounds leaves one debug line per run (or lane slice, or dry
  run), with the correlation — before, both vanished without a trace

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- createRunLog (migration-logger.js) holds a run's log side — its job and
  actor, ctx.run made once per direction, the loggers, one counter and one
  set of notices — and backgroundLogs a lane slice's, a watcher's or a dry
  run's; migrator.js keeps one #runLog instead of five pieces, and the
  fallbacks that would have restarted seq mid-run are gone
- migration:log is delivered without the second copy #emit made: its
  payload always names its own run
- runner: onError gets a first attempt's context when the session never
  started (it got one without ctx.run/ctx.logger), and the dead
  session-only context is gone
- types: MigrationLogger for ctx.logger — (msg, fields?), pino's
  (fields, msg?) and an Error, as the runtime accepts; method syntax keeps
  any MigronautLogger, pino or one-argument stub assignable, so a context
  built by hand still type-checks. A background migration:log event's
  runId is required: every emitted one has it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- the guide's recipe indexes and reads a queue job by { jobId, at, seq }:
  a lane job's slices (and an up driving background migrations inline)
  are runs of their own, each numbering from 1; it says how to buffer
  writes, and that the CLI's terminal logger prints the message only
- the example service buffers its log lines and writes them with
  insertMany (bounded buffer, lines over it counted and dropped), and its
  README says the log routes need the auth the example leaves out
- CHANGELOG names the type change on ctx.background.generation; ARCHITECTURE
  and CLAUDE.md describe the run log, the row limit, the job+run routing,
  the inline lanes' group and MigrationLogger

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The guide says an event's data has credentials and the values a server
error quotes masked; only msg and Errors in data got the second. A string
copied into a field — { error: err.message } — kept an E11000's
duplicate key. redactBounded now runs every string through redactOutbound.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A ```mermaid fence becomes <MermaidDiagram>, drawn in the browser by a small theme
component: the brand palette in light and dark, semantic node classes (core, store,
ext, warn, muted), labels laid out under the page's own CSS, lazy loading near the
viewport, and a readable minimum width that scrolls sideways on a phone.

mermaid is a docs-only devDependency, loaded only by a page that has a diagram.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A new How It Works page maps the engine: its entry points, the three flows inside
MigratorKit, the life of a run, the lock, where the state lives, which tool fits
which change, background migrations and what a run reports.

Diagrams join Core Concepts, Transactions, Declared Collections, Document
Versioning, Background Migrations, the queue adapter and Migration Logs.
llms.txt lists the new page and Migration Logs, which it was missing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Twelve features in a 3 x 4 grid: declared collections, Atlas Search, document
versioning, background migrations, OpenTelemetry and migration logs join the
core ones, some of which are merged. Below them, a diagram of how the parts fit
together and the smaller features in one list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 2.1-era box drawing of the mental model becomes a diagram of the entry points,
the engine and its state; the layers get a diagram of what each requires, the
queue adapter its two queues, and background migrations who writes what. The
snapshot line names 2.4.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Search indexes, document versioning, background migrations, the queue adapter and
observability join the reasons to choose migronaut, with a pointer to How It Works.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Migration logs for the application's users: ctx.logger bound to the run,
ctx.run, and the migration:log event for the lines a migration marks
userland: true — mirrored into a queue job's own log, and from background
migrations too. Migronaut stores none of it. Plus architecture diagrams
across the docs site, a How It Works page and a home page for what 2.1 to
2.4 added. See the CHANGELOG.

Gate: lint, format:check, test:coverage (2417 tests; 97.89% lines, 93.44%
branches, 98.52% functions), test:types, check:dts, size, docs:build — and
the opt-in real-Redis BullMQ suite (75/75).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 7, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
migronaut Ready Ready Preview Oct 7, 2026 8:40pm UTC

@Alex-Dolid
Alex-Dolid merged commit a57d7bb into main Oct 7, 2026
12 checks passed
@Alex-Dolid
Alex-Dolid deleted the feature/userland-logs branch October 7, 2026 20:57

This branch was successfully deployed

1 active deployment
Preview — e8cbb129 Deployed Oct 7, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant