Skip to content

fix(storage): stop rebuilding every FTS index on every store open - #1014

Merged
laynepenney merged 1 commit into
devfrom
fix/fts-schema-detector-always-true
Aug 28, 2026
Merged

fix(storage): stop rebuilding every FTS index on every store open#1014
laynepenney merged 1 commit into
devfrom
fix/fts-schema-detector-always-true

Conversation

@laynepenney

Copy link
Copy Markdown
Member

Opening a store to read it took SQLite's write lock and rebuilt three FTS5
indexes. Under a concurrent writer the open did not wait — it failed.

The defect, and it is one character

All three schema-drift detectors compared the DDL in sqlite_master against the
module constant that defines it. sqlite_master.sql holds a statement without
its trailing terminator
; the constants are written as runnable script and end
in );. strip() removes the newline after the semicolon and leaves the
semicolon itself, so the two could never compare equal.

"Differs" was therefore true on every open of every store, forever. Each open
dropped three FTS5 tables and nine triggers, recreated them, and reissued three
full rebuilds over the entire corpus.

Measured directly against a real SQLite database rather than argued from the
source:

current  ends: ...tokenize="porter unicode61 tokenchars '._+'" )
expected ends: ...tokenize="porter unicode61 tokenchars '._+'" );
EQUAL? False        _needs_fts_migration() -> True
length current 206, expected 207, difference 1 char
CONTROL: strip the trailing ';' from expected -> EQUAL? True

The control is the part that makes this a diagnosis rather than a guess: removing
exactly the semicolon makes the comparison succeed, so nothing else about the two
strings differs.

A note on how this defect is easy to describe wrongly, because I received it
described that way.
It is not that the comparison is unnormalized — it already
lowercases and collapses whitespace on both sides. What it fails to normalize is
the statement terminator. Anyone "fixing the normalization" by adding case or
whitespace handling would add what is already there and change nothing.

The fix

Both sides of every comparison now go through one _normalize_ddl helper that
puts a statement into the form sqlite_master stores it in. All three detectors
route through it today, which evidence.txt enumerates by line number.

What this does not guarantee, stated because v1 claimed it did. Nothing in
the code compels a fourth FTS table's detector to use the helper — a new table
can still ship its own comparison and reintroduce the defect. v1's body, commit
message, and the helper's own docstring all said a fourth table "cannot
reintroduce it". That was a universal the code does not enforce, Atlas blocked it
at r2, and it is removed from all three surfaces rather than softened.

The thing that would make it true by construction is a table→DDL registry plus a
guard test enumerating every CREATE VIRTUAL TABLE … fts5 constant in the module
and asserting each has a routed detector. That is a real mechanism and it is
filed as a follow-up rather than folded in here: a data-driven loop on its own
would still not earn the sentence, since a fourth table can bypass the registry
too, so it would change code without making the claim true.

The branch this unblocks has never executed in production

While the detector always reported drift, control never reached the else arm
that verifies the three sync triggers survived a crash during save_chunks. That
arm starts running on every open, everywhere, at once. So it gets witnesses
before it meets production rather than through it — one pair per table rather
than one standing in for three.

Witnesses: 17, and mutation-proven

tests/recall/test_fts_schema_detector.py.

mutation result
remove rstrip(";") — restore the defect 14 of 17 red
bypass the helper on chunks_fts only 4 red, and they are the chunks-parameterized witnesses plus the cross-table reopen

The second row is the one worth reading. The suite does not merely notice that
something regressed; it localizes which table did, because the drift and
trigger witnesses are parameterized per table. A single-table regression cannot
hide behind two healthy siblings.

Recorded because it nearly shipped as a passing witness that proved nothing:
the obvious else-branch test — drop a trigger, reopen, assert it came back —
passes under the mutation. The migration arm reinstalls triggers too, so that
test cannot tell which branch ran. Only the orphan-entry witness distinguishes
them: an index entry written with no backing content row is erased by a rebuild
and preserved by a no-op.

Verification at this head

  • pytest tests/recall/test_fts_schema_detector.py: 17 passed.
  • Full suite: 3120 passed, 25 skipped, 5 xfailed, 73 subtests passed.
  • Import path pinned and printed, so a green cannot come from a different
    checkout: synapt.recall.storage -> .../synapt-dev/synapt/src/synapt/recall/storage.py,
    interpreter .venv/bin/python, Python 3.13.2.
  • Rebased onto live origin/dev; git patch-id --stable is identical before
    and after the rebase, so this is the same change replayed, not a new one.
  • Mutation restores verified by re-hashing the source against a baseline
    fingerprint taken before the first mutation, under a trap … EXIT INT TERM.

Why this is being frozen now

The change was authored two weeks ago and its gate request went unserviced while
both reviewers were on another chain. It is re-frozen rather than re-submitted
because the intervening 49 commits on dev required a rebase, and a verdict must
bind bytes that exist. Everything above was re-measured at this head; nothing is
carried from the original freeze.

Premium boundary

recall is OSS. This is index maintenance only: no identity resolution, no org
context, no workspace-identity derivation.

Opening a store to READ it took SQLite's write lock and rebuilt three
FTS5 indexes. Under a concurrent writer the open did not wait, it
failed.

All three schema-drift detectors compared the DDL stored in
sqlite_master against the module constant that defines it. sqlite_master
holds a statement without its trailing terminator; the constants are
written as runnable script and end in `);`. strip() removes the newline
after the semicolon and leaves the semicolon, so "differs" was true on
every open of every store, forever. Each open dropped three FTS5 tables
and nine triggers, recreated them, and reissued three full rebuilds over
the entire corpus.

Both sides of every comparison now go through one _normalize_ddl helper
that puts a statement into the form sqlite_master stores it in. All
three detectors route through it. Nothing in the code compels a fourth
one to, so adding an FTS table means routing its detector there
deliberately.

The branch this unblocks has never executed in production. When the
detector always reported drift, control never reached the else arm that
verifies the three sync triggers survived a crash during save_chunks.
That arm starts running on every open, everywhere, at once -- so it gets
witnesses before it meets production rather than through it, one pair
per table rather than one standing in for three.

Mutation-verified: restoring the pre-fix comparison turns 14 of the 17
witnesses red, with the failure asserting on drift rather than erroring
from another layer. Bypassing the helper on chunks_fts alone turns 4
red, and they are the chunks-parameterized witnesses plus the
cross-table reopen, so the suite localizes which table regressed.

Worth recording, because it nearly shipped as a passing witness that
proved nothing: the obvious else-branch witness -- drop a trigger,
reopen, assert it came back -- PASSES under the mutation. The migration
arm reinstalls triggers too, so that test cannot tell which branch ran.
Only the orphan-entry witness distinguishes them: an index entry written
with no backing content row is erased by a rebuild and preserved by a
no-op.

Premium boundary: OSS. Index maintenance only, no identity or org
semantics.
@laynepenney
laynepenney merged commit 7ee663d into dev Aug 28, 2026
12 checks passed
@laynepenney
laynepenney deleted the fix/fts-schema-detector-always-true branch August 28, 2026 19:33
@github-actions github-actions Bot locked and limited conversation to collaborators Aug 28, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant