second-brain-evals

Measure how good your second brain actually is at answering questions about itself.

A small, dependency-free retrieval protocol for markdown knowledge bases (Obsidian vaults, notes corpora): a fixed set of fact-based questions with golden files, a ripgrep-based retriever, an optional lightweight reranker, and an honest report — R@5, R@10, reachability, per question type.

Why

Vector stores and LLM memory get all the attention. The boring part — can your retrieval surface the right notes — is rarely measured at all. First run on a real vault (1,600+ notes): the corpus contained the fact 93% of the time, but it landed in the top-5 only 13–20% of the time. Ranking, not storage, was the bottleneck. The reranker lifted R@5 to 87% with no vector layer: term frequency + freshness + "map vs leaf" prior.

Usage

python3 eval.py --vault ~/Documents/MyVault
python3 eval.py --vault ~/Documents/MyVault --fusion   # dense + reranker (кэш эмбеддингов)

Requirements: python3 (stdlib) and ripgrep (rg); --fusion additionally needs numpy and an embeddings cache.

Modes:

• default — reranker on: term score (frequency + title bonus) × freshness (git history, mtime fallback) × map status (README/AGENTS/queue/index files get a prior); SUPERSEDED markers demote, STALE does not (staleness tracks attention, not currency); for свежесть-type questions files that are successors in the SUPERSEDED chain get a boost; plus filename-based candidates
• --no-rerank — raw rg baseline, no ranking
• --fusion — score fusion: a × cosine(dense) + (1−a) × reranker over rg ∪ dense-top-30 candidates; --fusion-a (default 0.5) sets the blend. Embeddings come from dense_eval.py --stage embed (cached under .dense-cache/); any OpenAI-compatible embeddings endpoint works (reference run: qwen3-embedding-0.6b via Vercel AI Gateway)

Output: console report + JSON in --out-dir (default ./Outputs).

Questions file

questions.json — the benchmark: each entry has id, type, question, terms (search alternation), gold (one or more files carrying the fact), optional note. Five types:

• лексика — exact terms
• парафраз — synonym/rephrasing gap (weakest spot without a dense layer)
• темпоральная — date-relative queries
• связь-сущность — entity → fact links
• свежесть — superseded facts must not outrank fresh ones

Write your own questions from facts you know live in your vault. That is the whole trick: golden answers must exist.

Results on the reference vault (15 questions)

2026-09-04 run:

Metric	Baseline (rg)	Reranker
R@5	20% (3/15)	87% (13/15)
R@10	47% (7/15)	93% (14/15)
Reachability (fact anywhere in results)	93% (14/15)	100% (15/15)

2026-09-17 run, after the dense+fusion layer and the supersession-graph fix (same 15 questions, corpus since grown):

Metric	rg	Reranker	Dense-only	Fusion
R@5	20% (3/15)	73% (11/15)	67% (10/15)	93% (14/15)
R@10	33% (5/15)	87% (13/15)	73% (11/15)	100% (15/15)

History worth knowing: the dense layer closes the paraphrase gap completely (4/4; e.g. a gold file moving from rank 28 to rank 1), but only as score fusion — unioning dense candidates into the old reranker alone changes nothing (the dense signal gets suppressed). The freshness question is not a semantic problem: dense-only still misses it (rank 11) while the SUPERSEDED-chain successor boost takes it to rank 1–2.

2026-09-19 experiment — typed-decision re-ranking (TypeSafe Jev, jev-1.13.0): the fusion shortlist (top-20) re-ranked by one batched call per question (20 per-candidate Nouls over trimmed slices; ~$0.009 per run, ~1.2 s/call). Net negative on this set: R@5 93% → 87% (v1 slices) / 80% (v2 slices); R@10 100% → 93%. It consistently lifted the known entity-link miss (rank 5 → 2) and improved four semantic questions, but broke the two relative-time questions ("which digest was triaged last", "what was received today" — that fact is not in the note texts) and, with v2 slices, one convention-file question (gold scored 0.15 while topic-adjacent notes scored 0.5–0.8). Single run per variant, trimmed slices (≤2.6k chars per candidate). Script: jev_rerank.py.

Known honest limits (recorded, not tuned away):

• Remaining fusion miss: one entity-link question at rank 5 (raw research dumps of other topics outrank the synthesis). Left for the next iteration.
• Dense embeddings cover files truncated to 20k chars; facts deeper inside very large dumps stay outside the dense representation.
• Weights are fixed on purpose: tuning them against a 15-question golden set would be overfitting, not engineering (--fusion-a was sanity-checked on a small grid, kept at 0.5).

Layout

eval.py          — protocol runner (retriever + reranker + fusion + report)
dense_eval.py    — dense layer: embeddings cache (--stage embed) + 5-mode comparison (--stage eval)
fusion_probe.py  — score-fusion weight grid (local, no API)
reader-pack.py   — reader-eval pack builder (track #2: LLM reader over top-k docs)
reader-run.py    — reader runner: LLM answers strictly from a pack (resumable)
jev_rerank.py    — experimental, not part of the protocol: typed-decision re-ranking of the fusion shortlist (TypeSafe Jev)
questions.json   — benchmark set (bring your own facts; gold_answer for reader evals)
Outputs/         — run reports (JSON)

Reader evals (track #2)

reader-pack.py builds a per-question pack: top-k files after reranking + fragments around term hits. A reader (any LLM) answers strictly from the pack; verdicts go to Outputs/reader-grades-*.json.

Reference run (2026-09-04, budget = top-5 docs): accuracy 93% (14/15), abstention 7%, reading errors 0% against the LongMemEval baseline of 15–19%. The reader abstains instead of generating from noise — that is the whole difference. At a top-2 stress budget the same run gives 80% (12/15). The one structural miss is supersession: stale "map" files outrank the fresh fact-file (see Q09). Freshness by git history plus a SUPERSEDED-marker convention moved it from rank 8 to rank 6 — still outside top-5. This was later closed: the supersession-graph successor boost for freshness-type questions puts the fact-file at rank #2 (2026-09-17 results above).

2026-09-17 rerun on both retrieval outputs (same protocol; reader = GLM-4.7-flash via Vercel AI Gateway): v0.2 packs — 7/15 correct, 3 partial, 5 abstentions (the reader itself became the weak link: it missed facts stated verbatim in the pack and flipped answers between runs); fusion packs — 13/15 correct, 0 abstentions, 2 partial. Better retrieval buys better reading; the reader model is an unpinned variable — pin it when comparing runs.

MIT licensed.