feat(search-pipeline)!: project inside the batch, per root type (ADR 13)#627
Open
ddeboer wants to merge 2 commits into
Open
feat(search-pipeline)!: project inside the batch, per root type (ADR 13)#627ddeboer wants to merge 2 commits into
ddeboer wants to merge 2 commits into
Conversation
Retire the whole-schema projection: projectRoots is now the only projection entry point, projecting one root type over the roots the caller (the pipeline selector) supplies rather than discovering them from rdf:type. projectGraph, frameByType and rootsByType go, and buildSubjectIndex drops its rootTypes parameter and no longer scans for type triples. TypedSearchDocument moves to @lde/search-pipeline, where a terminal routes; @lde/search now yields a bare SearchDocument and stays pipeline-free. BREAKING CHANGE: @lde/search no longer exports projectGraph or TypedSearchDocument, and buildSubjectIndex takes only the quad source (no rootTypes).
…ing writer Compose a search pipeline as one terminal and N per-type stages. searchStages builds one projecting Stage per root type: each selects its roots, extracts each root's quads, and projects the root-complete batch (projectRoots) into documents tagged with their SearchType, so memory is bounded by batchSize roots, not the dataset. selectByClass is a convenience selector for the object grain. searchIndexWriter becomes a Writer<TypedSearchDocument>: it keeps ADR 9's per-collection fan-out and run lifecycle but stops projecting and stops buffering, routing each tagged document straight to its type's engine run. TypedSearchDocument now lives here, the glue that needs it. BREAKING CHANGE: searchIndexWriter now consumes TypedSearchDocument, not Quad, and no longer projects; compose it with searchStages.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The scalability landing of the search projection: projection moves into per-type pipeline stages and the buffering writer retires, so memory is bounded by
batchSizeroots instead of the whole dataset. Design: ADR 13.What changed
@lde/search(breaking)projectRootsis now the only projection entry point – one root type over the roots the caller supplies.projectGraph,frameByType,rootsByType, andbuildSubjectIndex'srootTypesparameter (it no longer scans forrdf:type).TypedSearchDocumentmoved out to@lde/search-pipeline;@lde/searchyields a bareSearchDocumentand stays pipeline-free.@lde/search-pipeline(breaking)searchStages(options)builds one projectingStage<TypedSearchDocument>per root type: each selects its roots, extracts each root's quads, and projects the root-complete batch (projectRoots) into documents tagged with theirSearchType.selectByClass(searchType)is a convenience selector for the object grain (not a default).searchIndexWriteris now aWriter<TypedSearchDocument>: it keeps ADR 9's per-collection fan-out and run lifecycle but stops projecting and stops buffering, routing each tagged document straight to its type's engine run.new Pipeline<TypedSearchDocument>({ datasetSelector, stages: searchStages(...), writers: searchIndexWriter(...) })– one terminal, N stages.Memory is flat, asserted by counting
test/search-stages.test.tsruns a synthetic input at 10 and 10 000 roots (batchSize: 10) against a recording fake writer and asserts:projectis invokedceil(roots / batchSize)times; the maxquads.lengthhanded toprojectis identical at both sizes; and the peak concurrently-live document count is identical at both sizes. The old buffering writer's peak was every document, so this fails against it and passes here.Validation
npx nx affected -t lint typecheck test build --base=origin/mainis green (search, search-pipeline, search-typesense, search-api-graphql). The Typesense testcontainer integration test (multi-collection.integration.test.ts) runs against a live container and passes – same documents land in the same collections. Coverage is 100% on the new/changed source; no new lint warnings.Design choices not spelled out in the issue
searchStagesoptions omitwriterFor. The issue sketch listed it, but stages do not write – the terminal does.searchStagestakes only{ schema, types };writerForstays onsearchIndexWriter.searchStagesre-resolves each type from the schema byclass.assertTypeInSchemais an identity check, so the stage projects with the schema's own declaration object even if a class-equal lookalike is passed (covered by a test).queueCapacityper stage type. ADR 13 motivates it (a document is far heavier than a quad); exposed as an optional knob.selectByClass(searchType, rootVariable = 'root')takes an optional root variable so the selector's projected variable matches the stage'srootVariable.Part of #606
Implements #623