Repository navigation
Migronaut v2.4: migration logs for your users, and architecture diagrams in the docs - #7
Merged
Merged
Conversation
An async listener whose promise rejects surfaced as an unhandledRejection, which ends the process on Node 15+. The kit now extends EventEmitter with captureRejections and reports the rejection at debug level, the way #emit already contains a listener that throws. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A migration's context gains ctx.run — the run id, direction, migration, batch, transaction attempt and actor, frozen — and ctx.logger, the kit's logger with that correlation bound into every line. A call whose fields hold userland: true is also emitted as migration:log, for the application to store and show its users; migronaut stores none of it. The runner builds a fresh context per attempt, so a transaction that the driver retries hands its body attempt 2. Hooks get the same: the run's correlation around the run, the migration's inside it. Event data is a bounded, redacted copy (redactBounded). Both fields are optional in the types, so a hand-built context still type-checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
job: { id, groupId? } names the queue job a run works for. It is bound
into ctx.run, every line ctx.logger writes, the migration:log event and
the kit's own lines for that run — nothing is stored. Validated before
the run touches anything (src/utils/job-ref.js holds the limits, for the
queue adapter to share); redo carries it into both halves.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The migration processor passes job: { id, groupId } to kit.up/down, so a
migration's ctx.run and its migration:log events name the job that ran
it. Its userland lines are also written into that job's log as '✎ …'
rows next to the processor's own — matched by job id, so a body that
outlived its timeout cannot write into the next job, and sealed before
the job settles, so nothing lands after BullMQ removed it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…m background migrations
ctx.background gains the lane's runId, its queue job and the transaction
attempt, and is frozen; ctx.logger binds it into every line, and a
userland: true call emits migration:log with kind 'background'. The
attempt is counted where the engine runs user code again in a new
transaction: a transactional batch, a transactional step and the drift
watcher's rewrite. runBackgroundSlice takes job: { id } for the lane's
lines. A dry run logs with dryRun: true and emits nothing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A lane job hands its id to runBackgroundSlice, so the slice's lines and migration:log events name it, and the lane's userland: true lines are written into that job's log. Lanes run side by side, so they are routed by the event's jobId through a map of the lanes in flight, and drained before the lane moves on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Migration Logs guide: ctx.logger and ctx.run, the userland marker, the migration:log payload, a recipe for keeping the events in a collection of your own (indexes, TTL, where to subscribe), what retries, rollbacks, timeouts and dry runs mean for them, the queue's job log rows and background migrations. The migration context, hooks, API, BullMQ and background guides, README, ARCHITECTURE, CLAUDE.md and the changelog follow. A sync test pins every declared event to the docs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The worker keeps every migration:log event in a migration_logs collection of the service's own (logs.js: indexed by run, job and migration, a TTL of 90 days, inserts drained on shutdown), and the API serves them at GET /migrations/runs/:runId/logs and GET /migrations/jobs/:id/logs. One MongoClient, injected into the kit, serves both roles. The seed migration logs what it inserted and removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- an Error in a userland call's data becomes { name, message, code?,
codeName? }: kept by reference it was stored as {} (its message is not
enumerable), and a driver error carried the offending document's values
(keyValue, errorResponse.errmsg) into the application's store and Redis
- other values go by an allowlist: Dates, RegExps and BSON values as they
are, binary data up to 4096 bytes, a Map as an object, a Set as an array,
anything else the way JSON.stringify sees it — all within the bounds
- an own __proto__ key stays a key in redactBounded and redactDeep
- the event's msg masks the values a server error quotes (redactOutbound)
- jobRefIssue checks own keys only: constructor/toString/__proto__ no
longer pass for a limit
- one plain-object predicate (canonical's isPlainObject) in redact and options
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- userlandLogRows (createMigrationQueue and both processors, default 1000, 0 mirrors none): how many userland rows one job's log takes — a lane's per slice; the rest are counted in one closing row, so a migration that logs in a loop no longer writes an entry per line into Redis. migration:log still carries every line - userlandRow writes the data's JSON only as far as the row shows it (it serialized up to ~4 MB to keep 1 KB, inside the migration's transaction), and keeps a row on one line: line breaks show as ⏎, control characters are dropped — a message can no longer forge a lifecycle row - a migration job takes a line only from its own run: the job id and the run id must both match (a job put back in the queue keeps its id) - one jobRefOf for both processors; the processor's jobFields, which collided with job-ref's, is now jobSummary Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With backgroundInline, the lanes an up drives under its lock had no job:
their userland lines named none, the migration processor dropped them
(kind 'background') and the background processor never knew them — so
they reached no job's log and could not be joined to the job.
- #runInline hands the run's job ({ id, groupId }) to drive, which names
it on the coordinator's and every lane's deps
- ctx.background, the lanes' lines and their migration:log events carry
jobId and groupId; each lane keeps a runId of its own (no parent run id)
- the migration processor takes a background line by job id and group:
the facade shares one kit between both processors, and a background
queue's lane — which has no group — may have the same id as a migration
job; the background processor leaves lines with a group alone
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- the runner's attempt count reaches beyond ctx.run: attempts on migration:success / migration:error and the kit's line when the driver retried the transaction, in a failure's context, and always as migronaut.migration.attempts on the migration span - a run that names a job puts migronaut.job.id / migronaut.job.group_id on its run span — the join for a queue of the user's own, without bullmq-otel - a ctx.logger call that is dropped (a field's getter throws) or cut to the event's bounds leaves one debug line per run (or lane slice, or dry run), with the correlation — before, both vanished without a trace Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- createRunLog (migration-logger.js) holds a run's log side — its job and actor, ctx.run made once per direction, the loggers, one counter and one set of notices — and backgroundLogs a lane slice's, a watcher's or a dry run's; migrator.js keeps one #runLog instead of five pieces, and the fallbacks that would have restarted seq mid-run are gone - migration:log is delivered without the second copy #emit made: its payload always names its own run - runner: onError gets a first attempt's context when the session never started (it got one without ctx.run/ctx.logger), and the dead session-only context is gone - types: MigrationLogger for ctx.logger — (msg, fields?), pino's (fields, msg?) and an Error, as the runtime accepts; method syntax keeps any MigronautLogger, pino or one-argument stub assignable, so a context built by hand still type-checks. A background migration:log event's runId is required: every emitted one has it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- the guide's recipe indexes and reads a queue job by { jobId, at, seq }:
a lane job's slices (and an up driving background migrations inline)
are runs of their own, each numbering from 1; it says how to buffer
writes, and that the CLI's terminal logger prints the message only
- the example service buffers its log lines and writes them with
insertMany (bounded buffer, lines over it counted and dropped), and its
README says the log routes need the auth the example leaves out
- CHANGELOG names the type change on ctx.background.generation; ARCHITECTURE
and CLAUDE.md describe the run log, the row limit, the job+run routing,
the inline lanes' group and MigrationLogger
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The guide says an event's data has credentials and the values a server
error quotes masked; only msg and Errors in data got the second. A string
copied into a field — { error: err.message } — kept an E11000's
duplicate key. redactBounded now runs every string through redactOutbound.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A ```mermaid fence becomes <MermaidDiagram>, drawn in the browser by a small theme component: the brand palette in light and dark, semantic node classes (core, store, ext, warn, muted), labels laid out under the page's own CSS, lazy loading near the viewport, and a readable minimum width that scrolls sideways on a phone. mermaid is a docs-only devDependency, loaded only by a page that has a diagram. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A new How It Works page maps the engine: its entry points, the three flows inside MigratorKit, the life of a run, the lock, where the state lives, which tool fits which change, background migrations and what a run reports. Diagrams join Core Concepts, Transactions, Declared Collections, Document Versioning, Background Migrations, the queue adapter and Migration Logs. llms.txt lists the new page and Migration Logs, which it was missing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Twelve features in a 3 x 4 grid: declared collections, Atlas Search, document versioning, background migrations, OpenTelemetry and migration logs join the core ones, some of which are merged. Below them, a diagram of how the parts fit together and the smaller features in one list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 2.1-era box drawing of the mental model becomes a diagram of the entry points, the engine and its state; the layers get a diagram of what each requires, the queue adapter its two queues, and background migrations who writes what. The snapshot line names 2.4.0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Search indexes, document versioning, background migrations, the queue adapter and observability join the reasons to choose migronaut, with a pointer to How It Works. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Migration logs for the application's users: ctx.logger bound to the run, ctx.run, and the migration:log event for the lines a migration marks userland: true — mirrored into a queue job's own log, and from background migrations too. Migronaut stores none of it. Plus architecture diagrams across the docs site, a How It Works page and a home page for what 2.1 to 2.4 added. See the CHANGELOG. Gate: lint, format:check, test:coverage (2417 tests; 97.89% lines, 93.44% branches, 98.52% functions), test:types, check:dts, size, docs:build — and the opt-in real-Redis BullMQ suite (75/75). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Release 2.4.0: migration logs for the application's users. Every migration gets
ctx.logger,bound to its run, and
ctx.run; a line markeduserland: trueis also emitted as themigration:logevent, for the application to store and show. Migronaut stores none of it — itcaptures and correlates, the application keeps. Plus architecture diagrams across the docs site, a
new How It Works page, and a home page that shows what 2.1 – 2.4 added.
Additive: a migration that never touches the new context fields, and a kit with no
migration:loglistener, behave as before. Everything new is marked experimental.
What's new
Migration logs
ctx.logger— the kit's logger withrunId,migration,direction,batch,attemptand,on a queue,
jobId/groupIdbound into every line. Pino's(fields, msg)order works too.ctx.run— the same correlation as a frozen object;attemptis 2 or more when the driverretried the transaction and the body ran again.
migration:log— emitted only for calls whose fields holduserland: true, whatever thelogger's level and with
logger: null. The payload is{ kind, runId, …correlation, level, msg, data, at, seq, truncated? };datais a bounded, redacted copy — credentials and the values aserver error quotes are masked, an
Errorbecomes{ name, message, code?, codeName? }.beforeAll/afterAllthe run's, the others the migration's.job: { id, groupId? }onup,downandredo(and{ id }onrunBackgroundSlice).attemptsonmigration:success/migration:error, thekit's log lines and the
migronaut.migrationspan;migronaut.job.id/migronaut.job.group_idon the run span of a run that names a job.
Queue adapter
userland: truelines land in the job's own log as one-line✎rows, matched by job id and run id, so a timed-out body never writes into the next job'slog.
userlandLogRowsoption (createMigrationQueueand both processors, default 1000) capsthe rows one job's log takes; the rest are counted in one closing row.
updrives inline(
backgroundInline) carry the run'sjobIdandgroupId, so their lines land in that migrationjob's log.
Background migrations
ctx.loggeris bound toctx.background— now frozen and typedBackgroundRunInfo, with thelane's
runId, itsjobIdand the transactionattempt. Auserland: truecall emitsmigration:logwithkind: 'background'; a dry run marks its linesdryRun: trueand emitsnothing.
Fixed
asyncevent listener that rejects no longer crashes the process: the kit is anEventEmitterwithcaptureRejections, and the rejection is a debug line.Docs
examples/migration-serviceapp stores thelogs and serves them over REST.
mermaidis a docs-onlydevDependency, loaded only by a page that has a diagram). A new How It Works page (the engine
and its entry points, the life of a run, the lock, where the state lives, background migrations,
what a run reports), diagrams in Core Concepts, Transactions, Declared Collections, Document
Versioning, Background Migrations, the queue adapter and Migration Logs, and four in
ARCHITECTURE.md.smaller features in one list; the Why page names the new features too.
Changed (for code typed against 2.3)
ctx.backgroundis frozen and typedBackgroundRunInfo. A type change on an experimentalAPI: its
generationisnumber | undefined(as it always was at runtime for the live driftwatcher), so strict TypeScript that reads it as a
numberneeds a check.ctx.loggeris typedMigrationLogger— everyMigronautLogger, and a pino instance, still fits.logger?andrun?are optional onMigrationContext, so a context built by hand in a teststill type-checks.
jobcarryjobId/groupId.Not in this release
kit.logs(), nomigronaut logscommand, no counters API — keeping the lines,and the UI that shows them, is the application's job (the guide has the recipe).
How to review
The commits are ordered so you can review them one at a time:
fix(core): contain rejected async event listeners…docs(examples): the migration service stores and serves migration logs;fix(core): what a migration logs is copied safely…fix(core): mask quoted server values in log data strings too;docs: render Mermaid diagrams on the docs site…docs: the Why page names what 2.1 to 2.4 added;chore(release): 2.4.0.Start with
src/core/migration-logger.js(ARCHITECTURE §5 and the "What a migration logs" part of§6.6), then
src/bullmq/processor.jsandsrc/bullmq/background-processor.js. "Things that looklike bugs but aren't" in CLAUDE.md lists the intentional behaviours:
userland: trueexactly, nostorage, rows routed by job id and run id, a job sealed before its last flush, inline lanes told
apart by their group.
Test plan
prepublishOnlygate: lint, format:check, test:coverage, test:types, check:dts —2417 tests; coverage 97.89 / 93.44 / 98.52 (gate 90/90/90).
bullmq-redis.test.js): 75/75 on redis:7-alpine.size,docs:build(no warnings); all 26 docs diagrams render in the production build, inlight and dark, and on a phone.