Repository navigation
Migronaut v2.2: Atlas Search and Vector Search indexes in declared collections - #5
Merged
Merged
Conversation
A collection definition gains `searchIndexes`: Atlas Search and Vector
Search indexes as `{ name?, type?, definition }`, validated lightly (the
shape that tells the two types apart, autoEmbed/vector mixing, repeated
fields, duplicate names) and normalized with the server's default name
and type. The new pure search-index-spec.js holds the comparison the
planner will use: documented defaults filled in on both sides, vector
fields compared as a set, type changes and immutable autoEmbed
attributes reported separately, and the readiness predicate.
`unsendable` moves to utils/canonical.js so both modules share it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
planSearchIndexes adds 'searchIndex' rows to a collection's plan: a missing index is created (one createSearchIndexes per collection), a changed definition is updated in place (modify, not destructive — the old index serves until the new one is built), and an undeclared one is kept or, under prune, dropped last. A search index is never rebuilt: a change of type or of an immutable autoEmbed attribute is a conflict naming the new-name recipe, and so is a declared index the server is still deleting. Where Search is unavailable every declared index is a conflict, or a 'skip' row when that is expected. Search submissions run before the regular index builds; a missing collection is created for its search indexes alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…unavailable converge now reads, creates, updates and drops Atlas Search / Vector Search indexes through raw commands (createSearchIndexes, updateSearchIndex, dropSearchIndex, $listSearchIndexes), so every driver in the peer range works. Nothing touches Search unless a definition declares searchIndexes. The new converge-search.js probes once per run whether the server has Search: an "unavailable" refusal (31082, 115, 6047401, 59, 40324) is the answer, a non-empty list or an empty one from 7.2.1+ means yes, and an empty list from an older server is decided by the server's search index manager setting. Checked against real 5.0, 6.0, 7.0, 8.0 and 8.2 servers. onSearchUnavailable (MIGRONAUT_ON_SEARCH_UNAVAILABLE): 'fail' (default) refuses the run before any write, with a hint; 'skip' converges everything else and reports 'skip' rows — also when the server refuses at apply time after the probe assumed Search. A vector index update is retried once with its type when a self-managed mongot asks for it. The verify phase re-reads a lagging search index list a bounded number of times before calling anything unstable, and refreshes each row's build state. result.search reports availability and the declared indexes still building, updating or failed; builds do not count against inSync. ConvergeFailedError's doc now names the replan phase too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Search indexes build in the background, so a converge returns while a
new one is still PENDING. waitForSearchIndexes (config, env
MIGRONAUT_WAIT_FOR_SEARCH_INDEXES, converge({ waitForSearchIndexes }),
CLI --wait-search / --no-wait-search) holds the run — and the lock —
after every collection's steps until each declared search index serves
its declaration: READY, queryable, nothing newer building, and for an
updated one a definition version past the one the update started from.
Polls back off from 1s to 10s and never past searchIndexWaitTimeoutMs
(default 10 minutes). A FAILED build ends the wait at once, the budget
running out fails it too: ConvergeFailedError with phase 'wait',
reason, notReady and waitedMs — everything was applied, only the builds
were not finished. An abort stops it between polls. Off by default;
never in a dry run (combining the two is refused) or in skip mode.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The converge table names search indexes ('search index' target, the
index name), adds a vector index's type and a build that is not serving
yet (PENDING/BUILDING, updating, FAILED: message) — showing an
unchanged search index that is not serving even without --verbose —
and its summary counts kept and skipped search indexes, builds under
way and failed ones. A skipped search index is no longer counted as a
change, and the drop/rebuild count comes from isDestructive.
converge --check also exits 28 when a declared search index failed to
build. migronaut audit gains a 'search' check, recorded only when the
definitions declare search indexes: pass where Search is available,
warn on a FAILED build or where it is missing with onSearchUnavailable
'skip', fail where converge would refuse.
A queue converge job's result carries `search`, and its log lines name
search indexes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tests/integration/search-atlas.test.js runs declared search indexes against a server that has Atlas Search — the only proof that the comparison rules match what a real mongot reports: create, wait and a fixed point for search and vector indexes; a matrix of definitions (analyzers, storedSource, searchAnalyzer, scalar quantization, partial hnswOptions, filter fields); in-place updates of both kinds; a refused type change; prune and leave-it-out; the audit search check. Opt-in and manual: it skips itself, with a reason, unless MIGRONAUT_TEST_ATLAS_URI is set, and CI never sets it. CONTRIBUTING.md says when and how to run it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- guide/collections: a Search indexes section — the declaration, where it works, the actions, why a search index is never rebuilt, background builds and --wait-search, servers without Search - commands/converge, commands/audit, guide/api, guide/bullmq, guide/configuration, guide/troubleshooting, reference/cli, reference/error-codes (every ConvergeFailedError phase), llms.txt - README, CHANGELOG (v2.2.0, unreleased), ARCHITECTURE (modules, §6.7, nuances, testing), CLAUDE.md (layout, nuances, taken names, the second sanctioned skip) - examples/migration-service: the Atlas local image, a search index on plans, directConnection in .env.example - package.json and the docs nav: 2.2.0 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Running the opt-in Atlas suite against mongodb-atlas-local 8.0.32 and 8.3.11 showed three things the fake could not: - mongot writes per-field defaults into latestDefinition: a string field gains indexOptions/store/norms, a number its representation, autocomplete its grams and tokenization, a document dynamic — and it reports a field indexed as several types in its own order. Static mappings were therefore modified on every run. FIELD_DEFAULTS now fills them on both sides, recursively through fields and multi, and a list of types compares as a set. - a self-managed mongot reports the definition version as latestVersion, not latestDefinitionVersion.version. - an Atlas CLI local deployment refuses every updateSearchIndex of a vector index, with or without type (IDLUnknownField on the retry). The first refusal is now the one reported, with the new-name recipe as its hint; Atlas updates in place as before. The suite no longer asserts the index stays queryable during an update (a local deployment reports it not queryable while it rebuilds), gains a nested/multi-type definition, and accepts the documented refusal for a vector update. 8/8 against both images. Docs say where updates are in place and where vector ones are refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- searchBuildState(): serving | updating | building | stale | failed | removing, shared by the planner summary, the wait, the table, --check and audit. A STALE index is reported as STALE (queryable, not replicating), not as still building. - The wait no longer holds on a FAILED or STALE index the run did not create or change (converge cannot fix it): it is named and warned about; --check still fails on a FAILED build. - A search index read that fails with a network or failover blip is retried at the next poll, up to three in a row. - The pause between polls, and the verify phase's re-reads, are cut short by an abort. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ad it
A search index list that could not be read was always reported as
phase 'plan' ("nothing was written") — also from the re-plan, the
verify phase and the wait, after writes. It now carries 'replan',
'apply' or 'wait' (reason 'unreadable'), and a run that stops for any
reason settles every row and attaches the result so far.
The probe was checked against plain mongod 7.0, 8.0 and 8.2: each
refuses $listSearchIndexes for want of Search before it looks for the
namespace, so probing a collection that does not exist yet stays sound.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ndex builds The wait for search index builds only reads, yet it held the global migration lock for as long as a build took — with convergeAfterUp, every deploy's up blocked the next pod and the queue's next job. runWithLock now hands the work a control whose release() gives the lock up early: the heartbeat stops, the lock document goes, lock:released fires with early: true, and a lost lock can no longer abort the run. converge releases it as the wait starts; the run itself (its id, span, history entry and converge:end) goes on until the wait ends. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A search index definition is compared with the documented defaults filled in on both sides. An option the server reports that the declaration does not set, and whose default migronaut does not know (a newer mongot writing a new one), made every converge update the index — and the server builds a search index again on every update. Such options are now left out of the comparison, named on the row (ignored) and in a warning. Only option objects are trimmed: a field, a mapping type or a vector field only the server has is still a difference. Also: - a modify row names what differs down to the option (mappings.fields.title.norms), the first five paths and how many more; - an unstable search index warns that every update rebuilds it; - a field named __proto__ stays a field (keys are assigned, not set). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ults Production now tolerates an option only the server reports; the opt-in suite asserts that none is left out against a real mongot, so a default missing from search-index-spec.js still fails it. 9/9 on mongodb-atlas-local 8.0 and 8.3.11. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- converge:wait events: started (with whether the lock was released),
progress every 30 s, then ready, failed, timeout, unreadable or
aborted. A queue job turns them into log lines and a 'search-wait'
progress phase, so a long wait is visible in BullMQ.
- result.search gains evidence (how Search availability was told) and
wait ({ outcome, waitedMs }); the converge history entry carries
result.search.
- migronaut.converge.search.wait.duration: one OpenTelemetry histogram
point per wait, by outcome — from telemetry.js, no new span.
- An assumed Search is logged at info when the run is about to wait.
- audit reads the declared collections' search indexes a few at a
time; a build message from mongot is kept to 500 characters.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…bout the server - converge-search-run.js: reading search indexes, the result's search summary, the wait and its reporting — out of converge.js (1440 → 1008 lines), which now reads as one flow. - server-info.js: READ_OPTIONS, READ_CONCURRENCY, readServer and the not-found codes, so audit no longer reaches into converge.js and the codes are defined once. - TARGET_LABELS and SERVED_ACTIONS live once, in converge-plan.js. No behaviour change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…BUTING Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The hot path first — comparing a declared search index with the live one: - one walk finds the differing top-level keys and the paths within them; deepEqual (canonicalize + stringify) was called at every level of the path walk, and again per top-level key — O(size × depth). Leaves now compare with === first. 1.6× faster on an unchanged definition, 4.5× on a changed one (20 to 1000 fields); - sortFields / sortMappings apply the per-item transform in the pass that computes the sort key, and reuse the array for the result; - pairing a field list with the declared one uses a Map, not a find per item; immutableChanges reads each side's fields once, not twice. Then every multi-pass chain over one array: - converge's closing report: one tally pass for changed/inSync, touched, kept, counts, history rows and notReady (it was seven, settleRest included) — summarize, touched, undeclaredKept, historyActions and counts are gone; - the plan review (prune drops, ignored options) is one pass per plan, and only looks for drops where indexes: [] / searchIndexes: [] is declared; readAndPlan, readSearch, waitTargets (Maps instead of a find per declared index), reportNotReady, refuseConflicts, replan; - the entangled-rebuild grouping no longer searches each rebuild's drops per doomed index, nor copies the lists it walks; - index-spec: the key's text/wildcard/integer-like facts and the server key in one pass; the option list is built once, not per index; - audit, the converge table and --check, the queue producer's ids and dedupe lookups; Sets for the membership checks done in loops. No behaviour change. Atlas Search suite 9/9 on atlas-local 8.0 and 8.3. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…scenario ends "should hold a failed migration on schedule ticks" ended right after its third tick, while the migration job that tick enqueued could still be running. The afterEach force-close does not wait for it, so it went on to write 0001-a.js into the next scenario's database — and markApplied unsets requestedBy/reason when a run sets none, which made "should carry who asked, and why, into the changelog" flaky on CI (one of the two harnesses, either Node version). The scenario now waits for the record to be applied (a new untilRecord helper, a findOne poll), which also replaces its hand-rolled wait for the failed one. Reproduced on the previous commit by slowing that migration by 120 ms (3/3 failures); 3/3 green with the fix; real Redis 52/52. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Declared collections can now hold Atlas Search and Vector Search indexes:
searchIndexes: [{ name?, type?, definition }]sits next toindexesandvalidator.convergekeeps them in step with the same lock, history, events, re-plan and fixed-point check as regular indexes.The change is additive (v2.2.0). A definition without
searchIndexesbehaves exactly as in 2.1, and converge never asks the server about Search for it.What's new
searchIndex.createSearchIndexesper collection.prune. The drop needs--yesin--jsonmode.$searchagainst a missing index returns nothing rather than failing. So a change no update can make is aconflictthat names the new-name recipe. That covers a change of type, and a change to anautoEmbedfield's path, model, size, quantization or modality.ignored). Otherwise a new mongot default would update and rebuild the index on every run.modifyrow names what differs down to the option, e.g.mappings.fields.title.norms.onSearchUnavailable: 'fail'is the default: the run is refused before any write, with a hint.'skip'converges everything else and reports the search indexes asskiprows. Availability is probed once per run from the server's own answer.waitForSearchIndexes,--wait-searchorMIGRONAUT_WAIT_FOR_SEARCH_INDEXES, with a budget ofsearchIndexWaitTimeoutMs(10 min by default).lock:releasedwithearly: true). The next deploy or a queue's next job does not wait out a build.--checkstill exits 28 on any FAILED build.build, and the result carriessearch: { available, evidence, notReady, wait? }. STALE is reported as STALE.converge --checkfails on a FAILED build.migronaut audithas asearchcheck.converge:waitevents:started,progressevery 30 s, then the outcome.search-waitprogress phase.migronaut.converge.search.wait.duration, recorded per outcome. It comes fromtelemetry.js, with no new span.createSearchIndexes,updateSearchIndex,dropSearchIndexand$listSearchIndexesare sent directly, so every driver in the peer range works. The driver's own helpers only start at 5.6.Internals
converge.jsis split. The search half of a run is nowconverge-search-run.js. Read options, server version and not-found codes are nowserver-info.js.auditno longer reaches intoconverge.js.deepEqualat every level, which was O(size × depth). It is now 1.6× faster on an unchanged definition and 4.5× faster on a changed one, for 20 to 1000 fields.planonly before the first write, thenreplan,applyorwait(reason: 'unreadable').Compatibility
Everything is additive, so this is a minor release (the 2.1.0 precedent). Some type unions gain a member, and an exhaustive
switchover them needs a new branch:ConvergeTargetgains'searchIndex'.ConvergeActionKindgains'skip'.'converge:wait'.MigrationJobProgress.phasegains'search-wait'.New config keys are
onSearchUnavailable,waitForSearchIndexesandsearchIndexWaitTimeoutMs. Each has aMIGRONAUT_*env var and an entry inmigronaut.schema.json. The full list is inCHANGELOG.md.Testing
pnpm test: 1352 unit and 461 integration tests pass. The integration suite runs on mongodb-memory-server 7.0.14.pnpm run test:coverage: 98.8% lines, 95.7% branches, 99.1% functions (gate 90/90/90).pnpm run test:types(tsd),pnpm run check:dts, lint and format pass.tests/integration/search-atlas.test.js): 9/9 onmongodb/mongodb-atlas-local8.0.32 and 8.3.11. Every scenario ends at a fixed point, and the lock is free during the wait. The suite stays strict about unknown server defaults. CI does not run it.mongod7.0, 8.0 and 8.2.converge.test.jsalso passes withMONGOMS_VERSION=8.0.32.Docs
converge.ARCHITECTURE.md§6.7,CLAUDE.md,CONTRIBUTING.md(when and how to run the Atlas suite), and the example service.