Problem Statement
The BM25 shadow implementation in #207 builds a deterministic in-memory lexical index lazily for each loaded graph. It gives excellent warm-query latency on the controlled 100,000-node workload, but the first query pays the entire construction cost and each process or graph reload discards that work.
Compass also supports incremental code-graph builds. Reusing an index across graph generations is unsafe unless the index is bound to the exact graph realization, updates are computed from stable identities, corpus statistics are updated exactly, and readers can never combine one graph generation with another generation's postings. graph.json must remain the portable authority, and an unavailable or damaged accelerator must never turn a valid query into an empty or stale answer.
The final deliverable is a fully working, deterministic, bounded, disposable persistent BM25 accelerator for generic text traversal, followed by a separate evidence-gated default cutover. Correctness, coherent publication, explicit truncation, and reproducible ranking take priority over incremental speed.
Solution
Keep graph.json authoritative and store BM25 state as an optional local SQLite cache. Persist Compass-owned raw lexical facts and corpus statistics, but continue to calculate the versioned text-ranker/bm25-v1 formula in Rust; do not delegate ranking semantics to SQLite FTS. Ordinary JSON graph queries must still work when the cache directory is absent, read-only, incompatible, corrupt, or full.
Represent every indexed document and posting by stable graph node ID. For each loaded graph generation, build a validated stable-ID-to-node-position map used only by that generation. Never persist or reuse transient array positions from graph.json.
Define a canonical lexical projection for each node from the fields already used by BM25: stable ID, display label/name, qualified identifier, node kind, and source path. Canonically encode and hash each document, then hash the sorted document-fingerprint manifest to obtain a lexical-projection identity. Persist per-document field term frequencies and lengths, reverse document-to-term records, per-term document frequencies, document count, and total document length. Compute average document length and IDF from those exact persisted totals at query time.
After Compass atomically publishes a new graph realization, the index manager follows this order:
- Pin the exact graph identity and build its stable-ID map.
- Open a complete cache generation only when schema, analyzer profile, graph identity, lexical-projection identity, limits, checksums, and completion metadata all validate.
- If a compatible prior generation and trusted sorted fingerprint manifest exist, stage a new generation and apply the exact add, remove, and change sets in one transaction.
- If compatibility or delta validation fails, perform a deterministic full rebuild from the validated graph.
- Validate aggregate statistics and postings against the staged generation, mark it complete, atomically publish it, then reopen it read-only.
- Pin each query to one graph/index generation pair. Existing readers retain their immutable generation while a replacement is built and published.
An edge-only graph update may reuse lexical postings when the canonical lexical projection is identical. Structural reranking and traversal must still use the newly pinned graph, so changed edges affect results without forcing lexical reconstruction.
If a cache cannot be opened or rebuilt, use the deterministic in-memory BM25 index when resources permit; otherwise use text-ranker/full-scan-v1 with an explicit diagnostic on interfaces that expose diagnostics. Never use an index whose identity does not match, never choose an ambiguous node ID, and never interpret a limit failure as no results.
Commits
Each commit must compile and pass its narrow tests. The BM25 default changes only in the final cutover commit after all preceding evidence is reviewed.
-
Use precise ranker terminology. Rename the established profile to text-ranker/full-scan-v1 throughout the text-query code, tests, qualification reports, and documentation. Preserve it as the default and rollback profile.
-
Define the lexical document contract. Add a typed canonical lexical-document projection and versioned analyzer/profile identity in the lowest owning crate. Reject duplicate stable node IDs and bound every field before indexing.
-
Golden-test lexical fingerprints. Add canonical encoding and digest tests for field presence, Unicode, code-identifier segmentation, path normalization, ordering, and repeated serialization.
-
Decouple postings from graph positions. Change the in-memory lexical index to store stable node IDs and resolve them through the current graph generation's validated ID map. Prove reordered graph.json nodes produce identical ranked IDs.
-
Define a typed cache-generation manifest. Record format major, analyzer profile, BM25 profile, graph identity, lexical-projection identity, document count, total length, limits, checksums, build mode, and completion state. Unknown majors and impossible values fail explicitly.
-
Add bounded SQLite cache storage. Persist raw documents, reverse term records, postings, and aggregate statistics behind a private versioned schema. Reuse existing bounded SQLite, lock, temporary-build, and read-only-open conventions; SQLite ranking functions do not define Compass ranking.
-
Implement deterministic full cache construction. Stream canonical documents in stable-ID order, enforce document/term/posting/byte limits before publication, validate exact statistics, and produce a complete staged generation.
-
Make cache publication interruption-safe. Add one-writer locking, temporary generation ownership, complete-marker transition, atomic rename, read-only reopen, and safe cleanup. The graph publication transaction remains independent and succeeds even if cache construction fails.
-
Open and validate exact cache hits. Verify all manifest identities, checksums, counts, and bounds before serving queries. Corrupt, incomplete, incompatible, or mismatched generations are never queried.
-
Compute deterministic generation deltas. Merge the prior and current sorted fingerprint manifests to classify unchanged, added, removed, and changed stable IDs. Reject duplicates and inconsistent manifests instead of selecting a convenient candidate.
-
Apply additions transactionally. Insert canonical document facts and postings while updating document count, total length, and document frequencies exactly. Differentially compare the result with a full rebuild.
-
Apply changes transactionally. Remove the prior document through its reverse term records, insert the replacement, and update every affected statistic exactly. Include label, identifier, kind, source-path, and Unicode changes.
-
Apply removals transactionally. Delete documents and postings, remove zero-frequency terms, and update totals without underflow or orphan records. Include delete/re-add and stable-ID replacement cases.
-
Reuse edge-only generations safely. Detect identical lexical-projection identities, bind the existing postings to the new graph generation, and prove structural reranking/traversal observes new edges.
-
Add the cache manager and fallback policy. Select exact hit, incremental build, full rebuild, in-memory BM25, or full-scan fallback deterministically. Surface rebuild/fallback/truncation state without changing the existing query response schema.
-
Pin graph and index generations in query APIs. Prevent reload races and mixed-generation reads. Add concurrent tests where requests remain on the old immutable pair while a new pair is published.
-
Wire generic CLI and MCP text traversal. Keep interface layers thin, pass cache/reload context into the owning query service, and verify CLI/MCP parity, cancellation, limits, and no-answer behavior.
-
Add recovery, concurrency, and platform qualification. Cover killed writers at each publication phase, two concurrent builders, stale/incomplete work, read-only cache roots, disk/size-limit failures, corruption, unknown versions, graph node reordering, and macOS/Linux/Windows path behavior.
-
Publish relevance and performance evidence. Run reviewed Python, Rust, and TypeScript judgments plus representative repositories. Report full rebuild, exact-hit open, one-node add/change/remove, edge-only update, cold/warm query latency, RSS, cache size, and deterministic repeated ordering.
-
Cut over BM25 as the default in a separate reviewed commit. Change CLI/MCP/library defaults only after every correctness and promotion gate passes. Retain an explicit text-ranker/full-scan-v1 rollback selector and document the compatibility effect. If any gate fails, do not merge this commit.
Decision Document
graph.json (compass.graph/1) remains the portable source of truth. The text index is disposable and rebuildable; it is not added to graph JSON or historical realizations.- The cache is content-addressed and immutable after publication. In-place mutation of a generation is forbidden.
- A query uses one pinned graph/index pair. Matching only timestamps, file names, or node counts is insufficient.
- Persisted postings contain stable node IDs, not graph array positions. Duplicate or missing IDs are validation errors.
- The lexical projection includes only fields already consumed by the BM25 shadow profile. Arbitrary attributes remain out of scope.
- Incremental updates compare canonical per-document fingerprints. Source-file watcher events are hints, not proof of the semantic graph delta.
- Persist raw field frequencies and exact aggregates. BM25 IDF is calculated from the current generation's document frequencies and document count, because a single add/remove can change corpus-wide scores.
- Edge-only updates reuse lexical data only when the lexical-projection identity is exactly equal. Structural scoring always reads current graph topology.
- SQLite is an optional local storage mechanism, not a ranking authority or external service. Rust owns normalization, BM25 arithmetic, tie-breaking, candidate retention, and truncation semantics.
- Candidate retrieval remains bounded at the versioned profile ceiling. Exact matches and the best candidate per query term retain the current deterministic protection.
- Cache failures never block graph publication. Query fallback is deterministic and explicit; stale or partial data is never served.
- Unknown cache major versions fail validation and are rebuilt when allowed. Cache schema changes do not change the graph schema.
- Old complete generations remain immutable while readers are pinned. Garbage collection deletes only unreferenced generations under an explicit bounded retention policy.
- The default cutover is isolated so it can be reviewed or reverted without removing the persistent index implementation.
Testing Decisions
Good tests compare externally observable ranked stable IDs, source anchors, ambiguity, no-answer behavior, truncation, deterministic order, graph direction, and traversal output. Storage tests additionally inspect typed validation outcomes and exact aggregate invariants, but do not make private SQL table names a public contract.
Required correctness evidence:
- full rebuild and incremental build produce byte-equivalent canonical index exports and identical query results for add, remove, change, rename, delete/re-add, node reorder, and edge-only sequences;
- deterministic randomized edit sequences compare every incremental generation with a fresh full rebuild;
- exact graph/index identity mismatch, duplicate IDs, corruption, incomplete publication, unknown versions, and exceeded limits fail closed and never look like no-answer;
- every persisted document length, posting frequency, document frequency, total length, and document count recomputes exactly from canonical documents;
- repeated runs and process reopens return identical ranked IDs and truncation state;
- graph reload races never mix generations, and old readers remain valid until released;
- JSON, store-backed typed queries, CompassQL, graph direction, provenance, and response schemas remain unchanged;
- no network, credential, embedding, vector database, runtime parser download, or external product is needed.
Default-promotion gates:
- zero regression in reviewed exact-ID Success@1, required/must-not-return judgments, ambiguity handling, no-answer precision, source anchors, edge direction, path acceptance, and deterministic order;
- non-regressing MRR and nDCG, plus a statistically reviewable Recall@5 improvement on the expanded independently judged corpus;
- incremental and full-build differential tests pass across all supported platforms;
- exact-hit and one-file-change startup costs are materially lower than rebuilding the in-memory index;
- warm p50/p95 remains at least 30% faster than full-scan on the 100,000-node workload and representative repositories;
- cold/full-rebuild latency, peak RSS, on-disk size, and write amplification stay within thresholds approved from measured evidence rather than assumed values;
- all repository baseline, product-boundary, CLI/MCP, relevance, CompassQL, recovery, and platform gates pass.
Out of Scope
- Embeddings, vector databases, model credentials, or semantic similarity services.
- Changing
compass.graph/1, graph node identity, graph direction, or provenance semantics.
- Requiring a writable cache or SQLite database for ordinary graph JSON queries.
- Replacing typed search, CompassQL, or immutable store query behavior.
- Using filesystem watcher paths alone as the authoritative index delta.
- Persisting transient graph array indexes.
- Updating published historical realizations in place.
- Distributed cache coordination or a network index service.
- Indexing arbitrary node attributes before separately reviewed relevance evidence exists.
Further Notes
This plan follows #206 and the BM25 shadow implementation in #207. The shadow evidence currently supports continued development but not default promotion: the compact reviewed corpus is tied, while the synthetic workload shows a large warm-latency gain with higher first-use latency and memory. Persistence and exact incremental maintenance directly target those costs, but they do not by themselves prove relevance quality.
The implementation should begin only after this ownership, storage, fallback, and promotion contract is approved. Any discovered requirement that weakens graph/index identity binding, atomic publication, deterministic equivalence, or explicit failure handling requires a new review rather than an implementation shortcut.
Problem Statement
The BM25 shadow implementation in #207 builds a deterministic in-memory lexical index lazily for each loaded graph. It gives excellent warm-query latency on the controlled 100,000-node workload, but the first query pays the entire construction cost and each process or graph reload discards that work.
Compass also supports incremental code-graph builds. Reusing an index across graph generations is unsafe unless the index is bound to the exact graph realization, updates are computed from stable identities, corpus statistics are updated exactly, and readers can never combine one graph generation with another generation's postings.
graph.jsonmust remain the portable authority, and an unavailable or damaged accelerator must never turn a valid query into an empty or stale answer.The final deliverable is a fully working, deterministic, bounded, disposable persistent BM25 accelerator for generic text traversal, followed by a separate evidence-gated default cutover. Correctness, coherent publication, explicit truncation, and reproducible ranking take priority over incremental speed.
Solution
Keep
graph.jsonauthoritative and store BM25 state as an optional local SQLite cache. Persist Compass-owned raw lexical facts and corpus statistics, but continue to calculate the versionedtext-ranker/bm25-v1formula in Rust; do not delegate ranking semantics to SQLite FTS. Ordinary JSON graph queries must still work when the cache directory is absent, read-only, incompatible, corrupt, or full.Represent every indexed document and posting by stable graph node ID. For each loaded graph generation, build a validated stable-ID-to-node-position map used only by that generation. Never persist or reuse transient array positions from
graph.json.Define a canonical lexical projection for each node from the fields already used by BM25: stable ID, display label/name, qualified identifier, node kind, and source path. Canonically encode and hash each document, then hash the sorted document-fingerprint manifest to obtain a lexical-projection identity. Persist per-document field term frequencies and lengths, reverse document-to-term records, per-term document frequencies, document count, and total document length. Compute average document length and IDF from those exact persisted totals at query time.
After Compass atomically publishes a new graph realization, the index manager follows this order:
An edge-only graph update may reuse lexical postings when the canonical lexical projection is identical. Structural reranking and traversal must still use the newly pinned graph, so changed edges affect results without forcing lexical reconstruction.
If a cache cannot be opened or rebuilt, use the deterministic in-memory BM25 index when resources permit; otherwise use
text-ranker/full-scan-v1with an explicit diagnostic on interfaces that expose diagnostics. Never use an index whose identity does not match, never choose an ambiguous node ID, and never interpret a limit failure as no results.Commits
Each commit must compile and pass its narrow tests. The BM25 default changes only in the final cutover commit after all preceding evidence is reviewed.
Use precise ranker terminology. Rename the established profile to
text-ranker/full-scan-v1throughout the text-query code, tests, qualification reports, and documentation. Preserve it as the default and rollback profile.Define the lexical document contract. Add a typed canonical lexical-document projection and versioned analyzer/profile identity in the lowest owning crate. Reject duplicate stable node IDs and bound every field before indexing.
Golden-test lexical fingerprints. Add canonical encoding and digest tests for field presence, Unicode, code-identifier segmentation, path normalization, ordering, and repeated serialization.
Decouple postings from graph positions. Change the in-memory lexical index to store stable node IDs and resolve them through the current graph generation's validated ID map. Prove reordered
graph.jsonnodes produce identical ranked IDs.Define a typed cache-generation manifest. Record format major, analyzer profile, BM25 profile, graph identity, lexical-projection identity, document count, total length, limits, checksums, build mode, and completion state. Unknown majors and impossible values fail explicitly.
Add bounded SQLite cache storage. Persist raw documents, reverse term records, postings, and aggregate statistics behind a private versioned schema. Reuse existing bounded SQLite, lock, temporary-build, and read-only-open conventions; SQLite ranking functions do not define Compass ranking.
Implement deterministic full cache construction. Stream canonical documents in stable-ID order, enforce document/term/posting/byte limits before publication, validate exact statistics, and produce a complete staged generation.
Make cache publication interruption-safe. Add one-writer locking, temporary generation ownership, complete-marker transition, atomic rename, read-only reopen, and safe cleanup. The graph publication transaction remains independent and succeeds even if cache construction fails.
Open and validate exact cache hits. Verify all manifest identities, checksums, counts, and bounds before serving queries. Corrupt, incomplete, incompatible, or mismatched generations are never queried.
Compute deterministic generation deltas. Merge the prior and current sorted fingerprint manifests to classify unchanged, added, removed, and changed stable IDs. Reject duplicates and inconsistent manifests instead of selecting a convenient candidate.
Apply additions transactionally. Insert canonical document facts and postings while updating document count, total length, and document frequencies exactly. Differentially compare the result with a full rebuild.
Apply changes transactionally. Remove the prior document through its reverse term records, insert the replacement, and update every affected statistic exactly. Include label, identifier, kind, source-path, and Unicode changes.
Apply removals transactionally. Delete documents and postings, remove zero-frequency terms, and update totals without underflow or orphan records. Include delete/re-add and stable-ID replacement cases.
Reuse edge-only generations safely. Detect identical lexical-projection identities, bind the existing postings to the new graph generation, and prove structural reranking/traversal observes new edges.
Add the cache manager and fallback policy. Select exact hit, incremental build, full rebuild, in-memory BM25, or full-scan fallback deterministically. Surface rebuild/fallback/truncation state without changing the existing query response schema.
Pin graph and index generations in query APIs. Prevent reload races and mixed-generation reads. Add concurrent tests where requests remain on the old immutable pair while a new pair is published.
Wire generic CLI and MCP text traversal. Keep interface layers thin, pass cache/reload context into the owning query service, and verify CLI/MCP parity, cancellation, limits, and no-answer behavior.
Add recovery, concurrency, and platform qualification. Cover killed writers at each publication phase, two concurrent builders, stale/incomplete work, read-only cache roots, disk/size-limit failures, corruption, unknown versions, graph node reordering, and macOS/Linux/Windows path behavior.
Publish relevance and performance evidence. Run reviewed Python, Rust, and TypeScript judgments plus representative repositories. Report full rebuild, exact-hit open, one-node add/change/remove, edge-only update, cold/warm query latency, RSS, cache size, and deterministic repeated ordering.
Cut over BM25 as the default in a separate reviewed commit. Change CLI/MCP/library defaults only after every correctness and promotion gate passes. Retain an explicit
text-ranker/full-scan-v1rollback selector and document the compatibility effect. If any gate fails, do not merge this commit.Decision Document
graph.json(compass.graph/1) remains the portable source of truth. The text index is disposable and rebuildable; it is not added to graph JSON or historical realizations.Testing Decisions
Good tests compare externally observable ranked stable IDs, source anchors, ambiguity, no-answer behavior, truncation, deterministic order, graph direction, and traversal output. Storage tests additionally inspect typed validation outcomes and exact aggregate invariants, but do not make private SQL table names a public contract.
Required correctness evidence:
Default-promotion gates:
Out of Scope
compass.graph/1, graph node identity, graph direction, or provenance semantics.Further Notes
This plan follows #206 and the BM25 shadow implementation in #207. The shadow evidence currently supports continued development but not default promotion: the compact reviewed corpus is tied, while the synthetic workload shows a large warm-latency gain with higher first-use latency and memory. Persistence and exact incremental maintenance directly target those costs, but they do not by themselves prove relevance quality.
The implementation should begin only after this ownership, storage, fallback, and promotion contract is approved. Any discovered requirement that weakens graph/index identity binding, atomic publication, deterministic equivalence, or explicit failure handling requires a new review rather than an implementation shortcut.