Skip to content

libsql-sqlite3: Clear vector index on DELETE without WHERE - #2302

Open
r3wretrhy wants to merge 1 commit into
tursodatabase:mainfrom
r3wretrhy:fix/delete-all-vector-index-2289
Open

r3wretrhy wants to merge 1 commit into
tursodatabase:mainfrom
r3wretrhy:fix/delete-all-vector-index-2289

Conversation

@r3wretrhy

Copy link
Copy Markdown

Fixes #2289.

DELETE FROM t with no WHERE clause takes SQLite's truncate optimization in sqlite3DeleteFrom: it emits OP_Clear for the table and for each index's b-tree instead of deleting rows one by one. A vector index stores its entries in <index>_shadow, not in its b-tree, so that path left the shadow rows in place. The next insert that reused one of the old rowids then failed with vector index(insert): failed to insert shadow row. DELETE ... WHERE 1 = 1 goes through the row-by-row path, which calls vectorIndexDelete, so it wasn't affected.

On the truncate path, this opens each vector index with OP_OpenVectorIdx and OPFLAG_FORDELETE, which clears the index through vectorIndexClear, then closes the cursor. That's the same thing sqlite3RefillIndex does when REINDEX has to clear a vector index. The OP_Clear on the (empty) b-tree is unchanged.

The bundled amalgamations in libsql-ffi/bundled (plain and SQLite3MultipleCiphers) are regenerated from the patched source, so the diff there is the same hunk.

Testing

Built libsql-sqlite3 on Linux x86_64 and ran:

CREATE TABLE t(id INTEGER PRIMARY KEY, v F32_BLOB(2));
CREATE INDEX t_idx ON t(libsql_vector_idx(v));
INSERT INTO t VALUES (1, vector('[1,2]')), (2, vector('[3,4]'));
DELETE FROM t;
SELECT count(*) FROM t_idx_shadow;
INSERT INTO t VALUES (1, vector('[5,6]'));

Before: the shadow table still has 2 rows and the insert fails with the error above. After: 0 rows, the insert succeeds, and vector_top_k returns the new row. A DELETE FROM t inside a transaction that is rolled back leaves both the table and the shadow table with their rows, and changes() still reports the number of deleted rows.

Added vector-delete-all to test/libsql_vector_index.test; it fails without the change and passes with it. With the change, testfixture passes libsql_vector_index.test, libsql_vector.test, delete.test, delete2.test, delete3.test, delete4.test and trigger2.test. I didn't run the Rust crates' tests or the full all.test run.

A DELETE without a WHERE clause takes the truncate optimization, which
runs OP_Clear on the table and each index b-tree. A vector index keeps
its entries in its shadow table rather than in its b-tree, so the shadow
rows were left behind and the next insert that reused a rowid failed
with "vector index(insert): failed to insert shadow row".

Open each vector index with OPFLAG_FORDELETE on that path, which clears
it the same way REINDEX does.

Fixes tursodatabase#2289

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Unqualified DELETE leaves libsql_vector_idx shadow rows and breaks the next insert

1 participant