test(memory): measure recall over a 200-entry corpus - #51
Open
arcuru-bot wants to merge 2 commits into
Open
Conversation
The memory search pipeline was covered only by hand-built three-to-five entry unit tests, which say nothing about how BM25, cosine and their fusion behave when there is enough material for ranking to matter. Generate a corpus from the project's own docs — one entry per section, two query sets — and run recall@k over it. Title queries share the document's wording and are the lexical ranker's best case; the hand-written paraphrase queries avoid it and are its worst. Tag filtering, the memory-to-embedding row join, per-model subtree isolation and dedupe-by-key are all re-checked at that scale, where a partial join or a leaking filter can actually hide. An ignored harness runs the same corpus against a live embedding endpoint. Its measurements are recorded in the memory guide, along with the correction they force: fusion weights both rankers equally, so the promise that configuring an embedder never makes recall worse holds for a missing or broken embedder but not for a working weak one.
The source filter passes Rust and Cargo files plus `.snap` snapshots, so a fixture a test reads with `include_str!` never reaches the sandbox. Cargo is green and the derivation fails to compile — the failure surfaces as a missing file at the macro, several layers away from the filter that dropped it. Let anything under a `testdata/` directory through.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The memory search pipeline was covered only by hand-built three-to-five entry unit tests, which say nothing about how BM25, cosine and their fusion behave once there is enough material for ranking to matter. This adds a corpus-scale harness and records what it measures.
What's here
dev/memory-corpus/build_corpus.pyturnsdocs/src/**/*.mdinto a 200-entry corpus (one entry per section, round-robin sampled so no single doc dominates), committed ascrates/lib/src/tools/testdata/memory_corpus.json. Two query sets ride along: 200 title queries (the section heading, sharing the document's vocabulary — BM25's best case) and 28 hand-written paraphrase queries that avoid it.crates/lib/src/tools/memory_corpus_tests.rs— 12 offline tests. Recall floors for the lexical baseline, tag filtering as a hard AND pre-filter checked against the fixture rather than against the search's own output, full memory-row-to-embedding-row join coverage, per-model subtree isolation, and dedupe-by-key over 200 revisions of one key.#[ignore]d, runs the same corpus against a live OpenAI-compatible/v1/embeddingsendpoint and prints the BM25-vs-hybrid table.What it measured
Recall@k with a locally-served
embeddinggemma:300m(768-dim, no prompt prefixes):The semantic leg earns its keep exactly where BM25 is weakest, which is also the case auto-recall lives in — a query built from the last few conversation messages is a paraphrase query by construction. BM25 alone plateaus at 0.321 on paraphrases from k=5 onward: an entry sharing no token with the query is dropped from the list entirely, so widening the result set cannot reach it.
It also turned up a doc correction.
memory.mdandconfiguration.mdboth promised that configuring an embedder "never makes recall worse than the lexical baseline". That holds for a missing or broken embedder, which falls back to BM25 — but RRF weights both legs equally, so a working weak one displaces correct lexical hits. Visible above at title/k=10, and pinned in the suite with a deliberately near-noise embedder (title recall@5 falls 0.935 → 0.850). Both sentences now say what is actually guaranteed and point at the numbers.Build change
flake.nix— the source filter passes Rust/Cargo files plus.snapsnapshots, so a fixture read withinclude_str!never reached the sandbox: cargo green, derivation red, surfacing as a missing file at the macro rather than at the filter that dropped it. Anything under atestdata/directory now comes along. No CI workflow files are touched.Gate
nix build .#checks.x86_64-linux.{lint,treefmt,test,doc,build}— green, with all 12 corpus tests passing inside thetestderivation.