Skip to content

feat(search-pipeline)!: project inside the batch, per root type (ADR 13)#627

Open
ddeboer wants to merge 2 commits into
mainfrom
feat/project-inside-batch-per-type
Open

feat(search-pipeline)!: project inside the batch, per root type (ADR 13)#627
ddeboer wants to merge 2 commits into
mainfrom
feat/project-inside-batch-per-type

Conversation

@ddeboer

@ddeboer ddeboer commented Jul 18, 2026

Copy link
Copy Markdown
Member

The scalability landing of the search projection: projection moves into per-type pipeline stages and the buffering writer retires, so memory is bounded by batchSize roots instead of the whole dataset. Design: ADR 13.

What changed

@lde/search (breaking)

  • projectRoots is now the only projection entry point – one root type over the roots the caller supplies.
  • Removed the whole-schema path: projectGraph, frameByType, rootsByType, and buildSubjectIndex's rootTypes parameter (it no longer scans for rdf:type).
  • TypedSearchDocument moved out to @lde/search-pipeline; @lde/search yields a bare SearchDocument and stays pipeline-free.

@lde/search-pipeline (breaking)

  • searchStages(options) builds one projecting Stage<TypedSearchDocument> per root type: each selects its roots, extracts each root's quads, and projects the root-complete batch (projectRoots) into documents tagged with their SearchType. selectByClass(searchType) is a convenience selector for the object grain (not a default).
  • searchIndexWriter is now a Writer<TypedSearchDocument>: it keeps ADR 9's per-collection fan-out and run lifecycle but stops projecting and stops buffering, routing each tagged document straight to its type's engine run.
  • A search pipeline is now new Pipeline<TypedSearchDocument>({ datasetSelector, stages: searchStages(...), writers: searchIndexWriter(...) }) – one terminal, N stages.

Memory is flat, asserted by counting

test/search-stages.test.ts runs a synthetic input at 10 and 10 000 roots (batchSize: 10) against a recording fake writer and asserts: project is invoked ceil(roots / batchSize) times; the max quads.length handed to project is identical at both sizes; and the peak concurrently-live document count is identical at both sizes. The old buffering writer's peak was every document, so this fails against it and passes here.

Validation

npx nx affected -t lint typecheck test build --base=origin/main is green (search, search-pipeline, search-typesense, search-api-graphql). The Typesense testcontainer integration test (multi-collection.integration.test.ts) runs against a live container and passes – same documents land in the same collections. Coverage is 100% on the new/changed source; no new lint warnings.

Design choices not spelled out in the issue

  • searchStages options omit writerFor. The issue sketch listed it, but stages do not write – the terminal does. searchStages takes only { schema, types }; writerFor stays on searchIndexWriter.
  • searchStages re-resolves each type from the schema by class. assertTypeInSchema is an identity check, so the stage projects with the schema's own declaration object even if a class-equal lookalike is passed (covered by a test).
  • Added queueCapacity per stage type. ADR 13 motivates it (a document is far heavier than a quad); exposed as an optional knob.
  • selectByClass(searchType, rootVariable = 'root') takes an optional root variable so the selector's projected variable matches the stage's rootVariable.

Part of #606
Implements #623

ddeboer added 2 commits July 18, 2026 15:52
Retire the whole-schema projection: projectRoots is now the only projection entry point,
projecting one root type over the roots the caller (the pipeline selector) supplies rather
than discovering them from rdf:type. projectGraph, frameByType and rootsByType go, and
buildSubjectIndex drops its rootTypes parameter and no longer scans for type triples.
TypedSearchDocument moves to @lde/search-pipeline, where a terminal routes; @lde/search
now yields a bare SearchDocument and stays pipeline-free.

BREAKING CHANGE: @lde/search no longer exports projectGraph or TypedSearchDocument, and
buildSubjectIndex takes only the quad source (no rootTypes).
…ing writer

Compose a search pipeline as one terminal and N per-type stages. searchStages
builds one projecting Stage per root type: each selects its roots, extracts each
root's quads, and projects the root-complete batch (projectRoots) into documents
tagged with their SearchType, so memory is bounded by batchSize roots, not the
dataset. selectByClass is a convenience selector for the object grain.

searchIndexWriter becomes a Writer<TypedSearchDocument>: it keeps ADR 9's
per-collection fan-out and run lifecycle but stops projecting and stops
buffering, routing each tagged document straight to its type's engine run.
TypedSearchDocument now lives here, the glue that needs it.

BREAKING CHANGE: searchIndexWriter now consumes TypedSearchDocument, not Quad,
and no longer projects; compose it with searchStages.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant