An embedded storage engine for Rust that consistently outperforms SQLite on concurrent read/write workloads — typically 1.5-2.5x faster in mixed benchmarks, scaling to 100K+ operations/second with secondary indexes.
Built from scratch around a B+ tree with per-table concurrency, a redo-log WAL, lazy row decoding, and zero-copy filter evaluation. Supports ACID transactions, per-table AES-256-GCM encryption, overflow pages for large blobs, and an optional async API.
- Features
- Quick Start
- API
- Benchmarks
- Encryption
- Async API
- ACID Transactions
- Vector Search
- Skills & Guides
- Architecture
- License
Storage
- Integer-keyed B+ tree with auto-increment row IDs
- Secondary indexes with O(log n) lookup and insert
- Overflow pages for large rows — blobs up to 10MB (configurable), zero overhead on normal rows
- Redo-log WAL with configurable durability (immediate fsync, deferred, or none)
- Crash recovery via forward WAL replay
Performance
- Concurrent readers never block each other or writers
- Per-table write locks — different tables are fully concurrent
- Lazy row decoding —
row.get("column")decodes only the requested column - Zero-copy filter evaluation on raw page bytes
- Batch bulk operations via single-pass leaf-chain walks
Vector Search (opt-in vector feature)
- HNSW approximate nearest-neighbor search with mmap'd storage and dedicated WAL
- AVX2 SIMD distance computation (cosine, euclidean, dot product) — runtime-detected
- Pre-filtered search — metadata predicates evaluated inline during graph traversal
- Diversity-based neighbor selection for high-quality graph construction
- Per-collection AES-256-GCM encryption and per-collection RwLock concurrency
Security & Reliability
- Per-table AES-256-GCM encryption (opt-in, zero overhead when off)
- ACID transactions with rollback (opt-in, zero overhead when off)
Integration
- Async API via optional
tokiofeature
[dependencies]
boogy-db = { path = "." }use boogy_db::*;
let db = BoogyDb::open("my.boogy")?;
db.create_table("users", &[
ColumnDef::new("name", Type::Text),
ColumnDef::new("email", Type::Text),
])?;
// Insert (auto-increment ID)
let id = db.insert("users", &[
("name", Value::Text("Alice".into())),
("email", Value::Text("alice@example.com".into())),
])?;
// Get by ID
let row = db.get("users", id)?.unwrap();
println!("{}", row.get("name").unwrap()); // Text("Alice")
// Query with filters, sort, pagination
let result = db.find("users", FindOptions {
filters: vec![Filter::eq("name", "Alice")],
limit: Some(10),
..Default::default()
})?;
// Secondary indexes
db.create_index("users", "idx_email", "email")?;
// Count (uses index when available)
let n = db.count("users", &[Filter::eq("email", "alice@example.com")])?;
// Transactions (guard-based)
let tx = db.begin()?;
tx.insert("users", &[("name", Value::Text("Bob".into()))])?;
tx.commit()?;| Method | Description |
|---|---|
insert(table, data) |
Insert a row, returns auto-increment u64 ID |
insert_with_id(table, id, data) |
Insert with caller-supplied ID |
get(table, id) |
Get a row by ID |
update(table, id, fields) |
Update specific columns |
delete(table, id) |
Delete a row |
find(table, opts) |
Query with filters, sort, limit/offset |
count(table, filters) |
Count matching rows |
count_with(table, filters, or_groups) |
Count matching rows with an OR-of-AND clause |
insert_many(table, rows) |
Batch insert |
update_where(table, filters, fields) |
Bulk update |
delete_where(table, filters) |
Bulk delete |
create_index(table, name, column) |
Create a single-column secondary index |
create_index_ex(table, name, cols, unique) |
Create a composite (multi-column) index, optionally unique |
scan_batch(table, filters, or_groups, order, after, limit) |
Ordered range-scan cursor — returns one bounded page + a resume token |
upsert_increment(table, key, counter, delta, set) |
Atomic keyed find-or-insert-then-add; integer or real delta |
drop_index(table, name) |
Drop an index |
create_table(table, columns) |
Create a table |
create_table_encrypted(table, columns, key) |
Create an encrypted table (AES-256-GCM) |
unlock_table(table, key) |
Provide the key for an encrypted table after reopen |
drop_table(table) |
Drop a table |
transaction(fn) |
Multi-table transaction (callback-based) |
begin() |
Begin a guard-based transaction (alternative to transaction(fn)) |
set_acid(enabled) |
Enable/disable ACID transaction mode |
set_max_row_size(bytes) |
Set maximum row size in bytes (default 10MB) |
Queries (find, count, delete_where, update_where) accept a list of Filter values that are ANDed together. Each filter targets a single column. For OR conditions, FindOptions also carries an or_groups field — see OR-of-AND groups below.
| Constructor | SQL Equivalent | Description |
|---|---|---|
Filter::eq(col, val) |
col = val |
Equality |
Filter::ne(col, val) |
col != val |
Inequality |
Filter::lt(col, val) |
col < val |
Less than |
Filter::le(col, val) |
col <= val |
Less than or equal |
Filter::gt(col, val) |
col > val |
Greater than |
Filter::ge(col, val) |
col >= val |
Greater than or equal |
Filter::in_list(col, values) |
col IN (...) |
Match any value in list |
Filter::is_null(col) |
col IS NULL |
Column is null or absent |
Filter::is_not_null(col) |
col IS NOT NULL |
Column has a non-null value |
// IN: match rows where status is one of several values
let result = db.find("orders", FindOptions {
filters: vec![Filter::in_list("status", vec![
Value::Text("pending".into()),
Value::Text("processing".into()),
])],
..Default::default()
})?;
// IS NULL: find rows with missing email
let result = db.find("users", FindOptions {
filters: vec![Filter::is_null("email")],
..Default::default()
})?;
// IS NOT NULL: find rows that have a verified_at timestamp
let result = db.find("users", FindOptions {
filters: vec![Filter::is_not_null("verified_at")],
..Default::default()
})?;
// Combine filters (AND): active users with no email
let result = db.find("users", FindOptions {
filters: vec![
Filter::eq("status", "active"),
Filter::is_null("email"),
],
..Default::default()
})?;FindOptions.or_groups: Vec<Vec<Filter>> adds an OR clause on top of filters. The full predicate a row must satisfy is:
ALL(filters) AND ( or_groups.is_empty() OR ANY(group in or_groups: ALL(group)) )
filters stays a mandatory AND-prefix; each group in or_groups is itself an AND of its filters, and the groups are ORed together. An empty or_groups (the default) is exactly the historical filters-only behavior — fully backward compatible.
This is enough to express composite keyset pagination, e.g. "the page after (score, id) = (c, cursor) ordered by score DESC, id DESC":
// WHERE deleted_at = '' AND ( score < c OR (score = c AND id < cursor) )
let result = db.find("posts", FindOptions {
filters: vec![Filter::eq("deleted_at", "")], // AND-prefix
or_groups: vec![
vec![Filter::lt("score", c)], // score < c
vec![Filter::eq("score", c), Filter::lt("id", cursor)], // OR (score = c AND id < cursor)
],
sort: vec![Sort::desc("score"), Sort::desc("id")],
limit: Some(page_size),
..Default::default()
})?;count ignores or_groups (it takes a plain &[Filter]); use count_with(table, filters, or_groups) to count with an OR clause.
Performance note: when
or_groupsis non-empty the query takes the scan + in-memory predicate path — the index/single-filter fast paths only understand the ANDfilters, so they're bypassed. ThefiltersAND-prefix still narrows the set; arbitrary OR has no single index to accelerate it.
db.create_index("users", "idx_email", "email")?;create_index builds a single-column secondary index. Inserts, updates, and deletes automatically maintain the index. Queries with an equality filter on an indexed column use the index for O(log n) lookup.
// Composite (multi-column) index — sorts by (user_a, user_b, rowid)
db.create_index_ex("edges", "by_pair", &["user_a", "user_b"], false)?;
// Enforced unique index — rejects any insert/upsert whose key tuple already exists
db.create_index_ex("edges", "by_pair", &["user_a", "user_b"], true)?;create_index_ex(table, name, columns, unique) generalizes create_index:
- Composite: columns are concatenated in key order using sortable byte encodings (integer big-endian, text length-prefixed). The index sorts rows lexicographically by the column tuple, then by rowid as a tiebreaker.
- Unique enforcement: when
unique = true, anyinsertorupsert_incrementwhose key tuple matches an existing row returnsBoogyError::UniqueViolation { index }. The check happens before any mutation — a rejected write leaves no partial state. - A single-column non-unique index is identical to
create_index(table, name, columns[0]).
scan_batch pages through a table in bounded memory by walking its primary-key tree or a named index tree in order, one batch at a time. It is the primitive for streaming jobs that can't afford to load a full table into memory.
use boogy_db::{ScanOrder, ScanKey, SortDir};
// Page through the whole table in PK order, 500 rows at a time.
let mut after: Option<ScanKey> = None;
loop {
let batch = db.scan_batch(
"events",
&[], // AND filters (optional)
&[], // OR-of-AND groups (optional)
ScanOrder::primary_key(SortDir::Asc),
after.clone(),
500,
)?;
for row in &batch.rows {
// process row
}
after = batch.last_key;
if after.is_none() { break; } // exhausted
}// Walk in descending order of a secondary index.
let batch = db.scan_batch(
"posts",
&[Filter::eq("deleted_at", "")],
&[],
ScanOrder::index("by_score", SortDir::Desc),
None, // start from the beginning
100,
)?;ScanOrder constructors:
| Constructor | Walks |
|---|---|
ScanOrder::primary_key(dir) |
Base table in rowid (insertion) order |
ScanOrder::index(name, dir) |
Named secondary index in its sorted order |
ScanBatch fields:
| Field | Type | Description |
|---|---|---|
rows |
Vec<Row> |
Rows in scan order for this page |
last_key |
Option<ScanKey> |
Opaque resume token — pass as after to get the next page; None when the scan is exhausted |
Filters and OR groups work identically to find. last_key = None when fewer than limit rows were returned — the scan is exhausted and there is nothing more to fetch.
upsert_increment finds a row by a key tuple, adds a numeric delta to a counter column, and writes optional additional columns — or inserts the row if it doesn't exist yet. Pair it with a composite unique index on the key columns so the lookup uses the index and uniqueness is enforced.
// Accumulate per-(user_a, user_b) interaction counts.
db.create_table("edges", &[
ColumnDef::new("user_a", Type::Text),
ColumnDef::new("user_b", Type::Text),
ColumnDef::new("n", Type::Integer),
ColumnDef::new("updated_at", Type::Integer),
])?;
db.create_index_ex("edges", "by_pair", &["user_a", "user_b"], true)?;
// First call: row does not exist → inserts (user_a="a", user_b="b", n=1, updated_at=now).
db.upsert_increment(
"edges",
&[("user_a", Value::Text("a".into())), ("user_b", Value::Text("b".into()))],
"n",
Value::Integer(1),
&[("updated_at", Value::Integer(now))],
)?;
// Second call: row exists → n becomes 1 + 2 = 3; updated_at is replaced.
db.upsert_increment(
"edges",
&[("user_a", Value::Text("a".into())), ("user_b", Value::Text("b".into()))],
"n",
Value::Integer(2),
&[("updated_at", Value::Integer(now))],
)?;Signature: upsert_increment(table, key, counter, delta, set) -> Result<u64>
| Parameter | Type | Description |
|---|---|---|
key |
&[(&str, Value)] |
Column-value pairs that identify the row |
counter |
&str |
Column to accumulate |
delta |
Value |
Integer or Real amount to add (first insert uses this value directly) |
set |
&[(&str, Value)] |
Additional columns to write/overwrite on every call |
| return | u64 |
Rowid of the inserted or updated row |
Type rules: Integer + Integer → Integer; Real + Real → Real; mixed → Real. A null or absent counter is treated as 0 of the delta's type. A non-numeric delta returns BoogyError::SchemaMismatch.
AcidTransaction also exposes upsert_increment so it composes with multi-table atomic transactions.
All benchmarks run on:
| CPU | Intel Core i7-10875H @ 2.30 GHz (8 cores / 16 threads, boost to 5.1 GHz) |
| RAM | 32 GB DDR4 |
| OS | Manjaro Linux 6.18.4 |
| Rust | Edition 2024, compiled with --release |
| SQLite | 3.x via rusqlite with WAL mode, synchronous=NORMAL |
Each benchmark shows boogy-db at two durability settings:
- None — No WAL writes. Fastest mode. Data survives process crashes (OS page cache) but not power loss.
- Normal — Redo-log WAL: after-images written on every commit, not fsynced. Data file updated on checkpoint (clean shutdown). Survives process crashes; may lose the last few milliseconds of writes on power loss. This is the production-realistic setting, comparable to SQLite's
synchronous=NORMAL.
SQLite runs at PRAGMA synchronous=NORMAL (WAL mode) in all tests.
Isolated insert and get operations at various table sizes. Each row has 3 columns (two short text fields and an integer).
| Table Size | Insert (None) | Insert (Normal) | Get |
|---|---|---|---|
| 100 rows | 428K/s (2.3 us) | 203K/s (4.9 us) | 5.2M/s (0.2 us) |
| 1,000 rows | 440K/s (2.3 us) | 199K/s (5.0 us) | 5.2M/s (0.2 us) |
| 5,000 rows | 441K/s (2.3 us) | 207K/s (4.8 us) | 4.8M/s (0.2 us) |
| 10,000 rows | 433K/s (2.3 us) | 207K/s (4.8 us) | 3.8M/s (0.3 us) |
Get performance is identical across durability modes (reads don't touch the WAL).
A realistic API workload: 30% inserts, 30% point reads, 25% filtered queries (equality filter with LIMIT 20), and 15% filtered counts. The table starts with 1,000 rows (3 text/integer columns) and grows over 5 seconds. Filtered operations search on one of 10 distinct values.
Without secondary index (both engines do full table scans for filtered queries):
| boogy (None) | boogy (Normal) | SQLite | ratio (Normal vs SQLite) | |
|---|---|---|---|---|
| Total ops/sec | 21,118 | 21,112 | 11,798 | 1.79x |
| p50 latency | 2 us | 5 us | 8 us | |
| p99 latency | 530 us | 521 us | 913 us | |
| Insert | 6,292/s | 6,290/s | 3,551/s | |
| Get | 6,346/s | 6,344/s | 3,527/s | |
| Find (limit 20) | 5,270/s | 5,269/s | 2,936/s | |
| Count | 3,210/s | 3,210/s | 1,784/s |
Without indexes, reads dominate the workload and WAL overhead is negligible. boogy-db is ~1.79x faster at both durability levels.
With secondary index on the filter column:
| boogy (None) | boogy (Normal) | SQLite | ratio | |
|---|---|---|---|---|
| Total ops/sec | 90,640 | 82,228 | 39,359 | 2.09x |
| p50 latency | 6 us | 6 us | 11 us | |
| p99 latency | 86 us | 81 us | 201 us | |
| Insert | 27,049/s | 24,544/s | 11,712/s | |
| Get | 27,291/s | 24,715/s | 11,815/s | |
| Find (limit 20) | 22,604/s | 20,507/s | 9,841/s | |
| Count | 13,696/s | 12,462/s | 5,991/s |
With indexes, Normal durability is 2.09x faster than SQLite. The redo-log WAL keeps the None-to-Normal gap small (~10%) by writing only to the WAL, never to the data file during commits.
Same workload distributed across multiple threads hitting the same table simultaneously.
Without secondary index:
| Threads | boogy (None) | boogy (Normal) | SQLite | ratio |
|---|---|---|---|---|
| 1 | 21,206 | 21,072 | 11,580 | 1.82x |
| 2 | 22,029 | 21,855 | 15,301 | 1.43x |
| 4 | 23,630 | 23,749 | 19,198 | 1.24x |
| 8 | 26,152 | 25,918 | 21,217 | 1.22x |
With secondary index:
| Threads | boogy (None) | boogy (Normal) | SQLite | ratio |
|---|---|---|---|---|
| 1 | 93,815 | 85,013 | 39,157 | 2.17x |
| 2 | 94,612 | 84,870 | 42,288 | 2.01x |
| 4 | 101,765 | 88,480 | 46,639 | 1.90x |
| 8 | 109,207 | 89,601 | 51,073 | 1.75x |
Readers operate on a shared page cache without blocking each other or writers. The ratio column compares Normal (the production-realistic mode) against SQLite.
Simulates a social media app fetching a user profile and their latest 5 posts. The database has 500 users with 50 posts each (25,000 posts total). Each query picks a random user, fetches their profile row, then fetches their 5 most recent posts sorted by timestamp.
boogy-db performs this as two separate calls (get for the user + find with filter/sort/limit for the posts). SQLite performs it as a single JOIN query with ORDER BY. Both engines have an index on the posts' author column where noted.
| Configuration | boogy (None) | boogy (Normal) | SQLite | ratio |
|---|---|---|---|---|
| No index, 1 thread | 2,419 q/s | 2,397 q/s | 743 q/s | 3.23x |
| With index, 1 thread | 66,317 q/s | 66,682 q/s | 44,619 q/s | 1.49x |
| With index, no sort | 467,255 q/s | 462,829 q/s | 161,346 q/s | 2.87x |
| With index, 4 threads | 175,424 q/s | 160,466 q/s | 120,091 q/s | 1.34x |
| With index, 8 threads | 210,254 q/s | 185,018 q/s | 120,673 q/s | 1.53x |
The "no sort" row shows performance when fetching any 5 posts without ordering, isolating the cost of application-side sorting. The sorted rows are the realistic case — boogy-db is 1.49x faster despite doing two separate queries and application-side sorting while SQLite uses its native query planner with a single JOIN.
Batch insert, update, and delete operations. Bulk insert uses insert_many (boogy-db) vs a single BEGIN/INSERT.../COMMIT transaction (SQLite). Bulk update and delete operate on a 10,000-row table.
Bulk Insert (single batch):
| Batch Size | boogy (None) | boogy (Normal) | SQLite | ratio |
|---|---|---|---|---|
| 100 | 848K r/s | 888K r/s | 493K r/s | 1.80x |
| 1,000 | 859K r/s | 839K r/s | 544K r/s | 1.54x |
| 10,000 | 781K r/s | 809K r/s | 569K r/s | 1.42x |
| 50,000 | 729K r/s | 753K r/s | 547K r/s | 1.38x |
Bulk Insert with Index on one column:
| Batch Size | boogy (None) | boogy (Normal) | SQLite | ratio |
|---|---|---|---|---|
| 100 | 485K r/s | 485K r/s | 444K r/s | 1.09x |
| 1,000 | 386K r/s | 354K r/s | 435K r/s | 0.81x |
| 10,000 | 319K r/s | 309K r/s | 404K r/s | 0.77x |
| 50,000 | 260K r/s | 261K r/s | 388K r/s | 0.67x |
Bulk Update (update_where vs UPDATE ... WHERE, 10K-row table):
| Rows Affected | boogy (None) | boogy (Normal) | SQLite | ratio |
|---|---|---|---|---|
| ~1,000 | 1.02M r/s | 654K r/s | 1.02M r/s | 0.64x |
| ~2,000 | 1.12M r/s | 620K r/s | 836K r/s | 0.74x |
Bulk Delete (delete_where vs DELETE ... WHERE, 10K-row table):
| Rows Deleted | boogy (None) | boogy (Normal) | SQLite | ratio |
|---|---|---|---|---|
| ~1,000 | 2.9M r/s | 2.7M r/s | 2.2M r/s | 1.22x |
| ~5,000 | 7.1M r/s | 5.3M r/s | 6.1M r/s | 0.87x |
| ~9,000 | 7.9M r/s | 5.7M r/s | 7.7M r/s | 0.75x |
boogy-db now beats SQLite on bulk inserts at ALL batch sizes (without index). Bulk delete wins at small batches. Bulk update and indexed bulk insert remain areas where SQLite leads.
Compares boogy-db's ACID transactions (set_acid(true)) against SQLite's BEGIN/COMMIT. boogy-db uses Durability::None; SQLite uses WAL + synchronous=NORMAL. "fast" column shows boogy-db with ACID off (current non-atomic begin()/commit()).
Transaction Insert (N rows per transaction, 10K total rows):
| Rows/Tx | boogy (ACID) | boogy (fast) | SQLite | ratio |
|---|---|---|---|---|
| 1 | 384K r/s | 434K r/s | 130K r/s | 2.95x |
| 10 | 616K r/s | 442K r/s | 436K r/s | 1.41x |
| 50 | 651K r/s | 447K r/s | 609K r/s | 1.07x |
| 100 | 649K r/s | 441K r/s | 635K r/s | 1.02x |
| 500 | 565K r/s | 448K r/s | 653K r/s | 0.86x |
ACID transactions amortize commit overhead across rows. At 10+ rows/tx, boogy-db ACID is faster than the non-ACID fast path because dirty pages accumulate and are published in one batch.
Mixed Transaction (1 insert + 2 gets + 1 update per tx, 1K-row table):
| boogy (ACID) | boogy (fast) | SQLite | ratio |
|---|---|---|---|
| 110K tx/s | 166K tx/s | 59K tx/s | 1.87x |
Single-Insert Transaction Throughput:
| boogy (ACID) | boogy (fast) | SQLite | ratio |
|---|---|---|---|
| 391K tx/s | 411K tx/s | 143K tx/s | 2.73x |
Rollback Cost (begin + 10 inserts + drop, no commit):
| boogy (ACID) | boogy (fast) |
|---|---|
| 110K rb/s | 42K rb/s |
ACID rollback is faster than the fast path's "rollback" because ACID discards the private page buffer (cheap), while the fast path's individual commits can't be undone.
boogy-db supports opt-in AES-256-GCM encryption at the table level. Encrypted tables store ciphertext on disk and in the WAL, while the in-memory page cache always holds plaintext. Unencrypted tables have zero encryption overhead.
let key: [u8; 32] = /* your 256-bit key */;
db.create_table_encrypted("secrets", &[
ColumnDef::new("token", Type::Text),
ColumnDef::new("data", Type::Blob),
], &key)?;
// All operations work identically — encryption is transparent
let id = db.insert("secrets", &[
("token", Value::Text("sk_live_abc123".into())),
("data", Value::Blob(sensitive_bytes)),
])?;
let row = db.get("secrets", id)?.unwrap();Keys are never stored on disk. On reopen, call unlock_table before accessing encrypted tables:
let db = BoogyDb::open("my.boogy")?;
// Unencrypted tables work immediately
let _ = db.get("public_table", 1)?;
// Encrypted tables require the key
db.unlock_table("secrets", &key)?;
let _ = db.get("secrets", 1)?;
// Without unlocking, operations return BoogyError::TableLocked- Algorithm: AES-256-GCM with random 12-byte nonces per page write. The GCM auth tag provides both confidentiality and integrity (stronger than CRC32).
- Encrypted page layout:
[nonce:12][ciphertext:4068][auth_tag:16]= 4096 bytes (same page size). - Key management: Caller provides a raw
[u8; 32]key. Key derivation (Argon2, HKDF, etc.) is the caller's responsibility. Different tables can use different keys. - Encryption points: Pages are encrypted before writing to the WAL and data file, decrypted on cache miss when reading from disk. The page cache always holds plaintext for fast access.
- Wrong key detection:
unlock_tableverifies the key by attempting to decrypt the table's root page. If the GCM auth tag doesn't match, it returnsBoogyError::InvalidKey. - Index encryption: Secondary indexes on encrypted tables are encrypted with the same key.
- Performance impact: ~1.5µs per page for AES-256-GCM with AES-NI. Only affects cache misses and WAL writes — cached reads have zero overhead.
Enable the tokio feature for async support:
[dependencies]
boogy-db = { path = ".", features = ["tokio"] }AsyncBoogyDb wraps the synchronous core with zero overhead — methods call the sync implementation directly without spawn_blocking or thread dispatch. This works because boogy-db operations are fast (microsecond-scale for cached reads) and rarely block on disk I/O in steady state.
use boogy_db::AsyncBoogyDb;
#[tokio::main]
async fn main() -> boogy_db::Result<()> {
let db = AsyncBoogyDb::open("my.boogy").await?;
db.create_table("users", &[
ColumnDef::new("name", Type::Text),
]).await?;
let id = db.insert("users", &[
("name", Value::Text("Alice".into())),
]).await?;
let row = db.get("users", id).await?.unwrap();
println!("{:?}", row.get("name"));
Ok(())
}AsyncBoogyDb is Clone (backed by Arc<BoogyDb>), so it can be shared across tasks cheaply. All methods from the sync API are available. The full sync BoogyDb is also accessible via db.inner().
boogy-db supports opt-in ACID-compliant transactions. When enabled, multi-operation transactions are truly atomic (all-or-nothing) with full rollback on failure or drop. When disabled (the default), operations commit individually for maximum throughput.
let db = BoogyDb::open("my.boogy")?;
db.set_acid(true);
db.set_durability(Durability::Immediate); // fsync for full durabilityFor full ACID compliance, combine set_acid(true) (atomicity + consistency) with Durability::Immediate (durability). Durability::Normal provides durability against process crashes but not power loss.
// Multi-operation transaction — all-or-nothing
let mut tx = db.begin()?;
tx.insert("users", &[("name", Value::Text("Alice".into()))])?;
tx.insert("posts", &[("title", Value::Text("Hello".into()))])?;
tx.commit()?; // atomic publish — both rows visible at once
// Drop without commit = full rollback
{
let mut tx = db.begin()?;
tx.insert("users", &[("name", Value::Text("Bob".into()))])?;
// dropped here — nothing is written, database unchanged
}
// Reads within a transaction see uncommitted writes
let mut tx = db.begin()?;
let id = tx.insert("users", &[("name", Value::Text("Carol".into()))])?;
let row = tx.get("users", id)?.unwrap(); // sees Carol
tx.commit()?;When ACID mode is on, standalone operations outside begin() are automatically wrapped in mini-transactions. When off, operations commit individually with zero overhead — one AtomicBool check per operation.
ACID transactions hold a private dirty page buffer. Each operation briefly acquires the global write lock (microseconds), does the B+ tree mutation in the private buffer, and releases. Tables not touched by the transaction are completely unblocked. commit() publishes all pages atomically in one batch. Drop without commit discards the buffer — a zero-cost rollback.
HNSW-based approximate nearest-neighbor search for dense vector embeddings. Custom mmap'd storage with its own WAL, AVX2 SIMD distance computation, pre-filtered search, diversity-based graph construction, per-collection encryption, and per-collection concurrency. Enable with the vector feature flag.
[dependencies]
boogy-db = { path = ".", features = ["vector"] }use boogy_db::*;
let db = BoogyDb::open("my.boogy")?;
db.create_table("documents", &[
ColumnDef::new("title", Type::Text),
ColumnDef::new("category", Type::Text),
])?;
// Create a vector collection linked to the table
db.create_vector_collection("documents", "embeddings",
&VectorCollectionOptions::new(768, DistanceMetric::Cosine),
)?;
// Insert a row and its embedding
let id = db.insert("documents", &[
("title", Value::Text("Hello world".into())),
("category", Value::Text("general".into())),
])?;
db.vector_insert("documents", "embeddings", id, &embedding)?;
// Batch insert
db.vector_insert_batch("documents", "embeddings", &[
(id1, embedding1),
(id2, embedding2),
])?;
// k-NN search
let results = db.vector_search("documents", "embeddings", &query,
&VectorSearchOptions::new(10),
)?;
for r in &results {
let row = db.get("documents", r.rowid)?.unwrap();
println!("{}: {}", row.get("title").unwrap(), r.distance);
}
// Filtered search (pre-filtered — inline during HNSW traversal)
let results = db.vector_search("documents", "embeddings", &query,
&VectorSearchOptions {
k: 10,
ef_search: 50,
filter: Some(Filter::eq("category", "general")),
},
)?;
// Update / delete
db.vector_update("documents", "embeddings", id, &new_embedding)?;
db.vector_delete("documents", "embeddings", id)?;let key: [u8; 32] = /* your 256-bit key */;
db.create_vector_collection("documents", "embeddings",
&VectorCollectionOptions {
dimensions: 768,
metric: DistanceMetric::Cosine,
key: Some(key),
..Default::default()
},
)?;
// After reopening the database, unlock the encrypted collection
db.unlock_vector_collection("documents", "embeddings", &key)?;Plaintext lives in memory (mmap) for zero-copy HNSW traversal. Ciphertext on disk (AES-256-GCM). The WAL is also encrypted. Wrong key on unlock returns DecryptionFailed.
| Metric | Constant | Best For |
|---|---|---|
| Cosine similarity | DistanceMetric::Cosine |
Text embeddings (magnitude-independent) |
| Euclidean distance | DistanceMetric::Euclidean |
Image features, spatial data |
| Dot product | DistanceMetric::DotProduct |
Pre-normalized vectors where magnitude encodes relevance |
All three are AVX2-accelerated on x86-64 (runtime-detected, scalar fallback on other architectures).
| Parameter | Default | Description |
|---|---|---|
m |
16 | Bi-directional links per node. Higher = better recall, more memory. |
ef_construction |
200 | Candidate list size during index build. Higher = better graph quality. |
ef_search |
10 | Candidate list size during search. Higher = better recall, slower search. Per-query via VectorSearchOptions::ef_search. |
Each vector collection is a separate mmap'd file ({db}.{table}.{collection}.vec) with its own WAL. The HNSW graph uses diversity-based neighbor selection (Malkov & Yashunin heuristic) for high-quality graph construction. Filtered search evaluates metadata predicates inline during graph traversal — no post-filter inflation.
Concurrency: per-collection RwLock. Searches on different collections never block each other. Writes to one collection don't block searches on another. Rowid-to-node mappings are persisted in internal boogy-db tables and automatically restored on reopen.
Compared against usearch (the fastest single-file vector search library) at identical HNSW parameters (128 dims, M=16, ef_construction=200, ef_search=50, k=10):
| 10K vectors | boogy-db | usearch |
|---|---|---|
| Search | 192 µs | 131 µs |
| Insert | 1,974 v/s | 2,619 v/s |
| Recall | 66.9% | 65.9% |
| 50K vectors | boogy-db | usearch |
|---|---|---|
| Search | 296 µs | 294 µs |
| Insert | 1,132 v/s | 1,186 v/s |
| Recall | 39.3% | 35.9% |
At 50K vectors, boogy-db matches usearch on search latency while providing higher recall, ACID durability, WAL-based crash recovery, and per-collection encryption — features usearch doesn't have. The recall advantage comes from diversity-based neighbor selection.
The vector feature adds zero overhead to existing CRUD operations when no vector collections are in use.
The skills/ directory contains step-by-step guides for working with boogy-db:
For application developers (skills/consumer/):
- Schema Design — table layout, column types, indexes, migration patterns
- Configuration — durability, ACID mode, encryption setup
- Query Patterns — filters, pagination, joins, bulk ops, transactions
- Async Usage — tokio integration, sharing across tasks, axum/actix patterns
- Error Handling — error variants, matching, recovery patterns
For boogy-db contributors (skills/internal/):
- Adding a Table Method — locking protocol, WriteGuard, index maintenance
- Adding a Benchmark — benchmark structure, SQLite comparison
- Optimizing Hot Paths — zero-copy, peek_dirty, leaf-chain walks
- Modifying Page Format — page layout, header, compatibility
Single file per database, 4 KB page-aligned. Each table is a separate B+ tree with u64 integer keys and fixed 12-byte branch entries. Rows are stored inline on leaf pages with an offset directory for O(1) column access. Rows exceeding page capacity (~4KB) spill into linked overflow pages transparently. The system page (table registry) is limited to 4KB, which constrains the total metadata size (table names, column definitions, index names). This accommodates approximately 20-30 tables with typical schemas.
Readers and writers never block each other. The page cache (Arc<Page> pointers behind an RwLock) allows concurrent reads with a brief shared lock. Writers get exclusive access to a dirty-page overlay via WriteGuard, holding it only for the duration of a single B+ tree mutation. Per-table RwLocks ensure different tables are fully independent.
Redo-log design. Commits write after-images to the WAL — the data file is never modified during normal operation. On clean shutdown, cached pages flush to the data file and the WAL is truncated. On crash, the WAL is replayed forward to restore committed state. Three durability levels: Immediate (fsync per commit), Normal (WAL write, no fsync), None (no WAL).
Each secondary index is a separate B+ tree keyed by composite (encoded_value, rowid) bytes, sorted for correct byte-order comparison. Lookups do a prefix scan on the index tree, then batch-fetch matching rows from the data tree in a single leaf-chain walk.
Per-table AES-256-GCM. Plaintext lives in the page cache; encryption happens only at I/O boundaries (WAL writes, disk flushes). Keys are never stored on disk. The system page (schema metadata) is always plaintext.
When enabled, transactions hold a private dirty-page buffer. Each operation briefly acquires the global write lock, mutates pages in the buffer, and releases. commit() publishes all pages atomically. Drop without commit discards the buffer — zero-cost rollback. Tables not touched by a transaction are unblocked.
Apache-2.0