bookie

An embeddable, strongly-consistent distributed key-value store: SlateDB for storage, openraft for consensus.

One SlateDB per raft group, at one cluster-wide object-store prefix, and the group's leader is its only writer — SlateDB fences the rest through the manifest. Every other replica reads that same database through slatedb::DbReader. Raft replicates commands, not bytes, so replicas cost availability rather than storage.

Durable state lives in object storage (S3, GCS, Azure, a local directory, or an in-process store). The only per-node state is the raft log, which is regenerable from the group's database plus the quorum's logs.

use bookie::net::mem::MemRouter;
use bookie::{Bookie, BookieConfig, Cmd, ReadConsistency};

# async fn example(config: BookieConfig) -> bookie::Result<()> {
let router = MemRouter::new();
let node = Bookie::open(config, router.factory(1)).await?;
router.register(1, node.raft().clone());
node.initialize().await?;

node.write(Cmd::put("greeting", "hello")).await?;
let value = node.get(b"greeting", ReadConsistency::Linearizable).await?;
# Ok(())
# }

The library is usable without the server feature; that feature adds the gRPC transport, the bookie binary, and tracing setup. cargo build --no-default-features needs no protobuf toolchain.

Architecture

       client (gRPC KvSvc / AdminSvc, or the library API)
                        │
                     Bookie ──── ReadConsistency: Linearizable | LeaderLocal | Stale
                    ╱      ╲
        openraft::Raft      StateMachine ──── Storage: one of
              │                   │             Db       (leader: the group's writer)
        DiskLog (local disk)      │             DbReader (replica: same database)
              │                   └─ snapshots: metadata only, zero bytes
        RaftSvc over gRPC (or net::mem in process)

                    the group's one SlateDB, at one prefix
                          s3://bucket/bookie/…

• src/log/ — the raft log: segment files with CRC-framed records, a metadata file replaced atomically, group commit (many pipelined appends, one fsync), and an advisory lock so two processes cannot share one log directory — advisory being why the binary refuses to start on a network filesystem it recognises.
• src/sm/ — the state machine: user keys under u, internal state under \xFFmeta/, and every apply writing its data and its applied-log-id in one atomic SlateDB batch. sm/store.rs owns which handle this node holds onto the group's database — writer, read-only view, or nothing — and how a handoff moves it.
• src/net/ — gRPC services (RaftSvc, KvSvc, AdminSvc) plus an in-process transport used by tests and by embedders who want one replica.
• src/node.rs — bringing those together, and the read and write paths.
• src/client.rs — leader cache, redirect following, and a retry policy that is a claim about effects rather than about error codes.
• src/shard/ — key-range sharding: a routing table replicated by a meta group, a host running one raft group per shard, the projected-clone copy a split or merge is made of, and the advice a cluster's reported shape argues for. See Sharding.

docs/design.md explains why each of those is the way it is. The short version:

1. apply() is deterministic. No clocks, no randomness, no environment. A command that cannot be applied is refused identically on every replica rather than failing the apply.
2. Applied state is atomic with the data it describes, which is what makes acknowledging a write at raft-quorum latency safe — and what makes the database a prefix of the command stream, so a new writer knows where to resume.
3. One place builds stored keys (sm::keys), so no user key can collide with internal state.
4. A snapshot carries no data: it names how far the shared database is durable, so a learner or a purged-behind follower catches up in one message.
5. One writer per group, and the fence is the handoff. A new writer replays the tail its predecessor never wrote and skips what the database already holds — while still answering a client waiting on a skipped entry, because the database holding it is that write having succeeded. Losing the fence demotes a node to a reader instead of failing it.
6. Purge is gated on shared durability — an entry is only dropped once the group's database holds it, or a handoff would have nothing to replay.
7. Reads are a per-request choice of three consistency levels.

Running a cluster

# terminal 1..3. Add --features tls for a cluster with certificates; the shipped
# examples run on loopback and say `allow_insecure = true` instead — plus
# `allow_insecure_admin` / `allow_insecure_peers`, which is what lets three separate
# processes be formed over `init` and talk raft without certificates.
cargo run --bin bookie -- serve --config examples/n1.toml   # ...n2.toml, n3.toml

# form the cluster out of node 1's configured peers, then use it
cargo run --bin bookie -- init --addr 127.0.0.1:9001
cargo run --bin bookie -- kv --addr 127.0.0.1:9002 put foo bar   # a follower redirects
cargo run --bin bookie -- kv --addr 127.0.0.1:9002 get foo       # linearizable by default
cargo run --bin bookie -- kv --addr 127.0.0.1:9003 get foo --consistency stale
cargo run --bin bookie -- kv --addr 127.0.0.1:9001 cas foo --expect bar --new baz
cargo run --bin bookie -- metrics --addr 127.0.0.1:9003
cargo run --bin bookie -- status --config examples/n3.toml

A node that was in nobody's peer list joins with bookie join --leader <addr> --config <its config>, which adds it as a learner and then promotes it. Re-running join finishes an interrupted one.

Keys and values are hex:/str:-prefixed both ways, so anything the CLI prints can be passed straight back in. CLI writes attach a fresh request id so a retried shell command cannot undo a later write; library callers that wrap Client::put in their own retry middleware should use IdempotentClient (or put_idempotent) instead — see docs/ops.md.

Configuration

node_id = 1
listen_addr = "127.0.0.1:9001"
log_dir = "/var/lib/bookie/n1/raft-log"
# allow_shared_log_dir = false        # `true` runs log_dir on NFS/SMB, where the
                                      # one-node-per-directory lock is not reliable

[[peers]]                 # every member of the initial cluster, including this node
id = 1
addr = "127.0.0.1:9001"

[store]
uri = "s3://my-bucket"    # or file:///var/lib/bookie, memory://, memory://name, env://
prefix = "bookie"         # the group's database; identical on every node

[raft]                    # all defaults shown
cluster_name = "bookie"
heartbeat_interval_ms = 250
election_timeout_min_ms = 750
election_timeout_max_ms = 1500
max_payload_entries = 32
snapshot_logs_since_last = 5000     # 0 disables snapshots, and needs the flag below
allow_disabled_snapshots = false    # 0 above also stops the log ever being purged
max_in_snapshot_log_to_keep = 1000
purge_batch_size = 256

[maintenance]
compaction = true         # SlateDB's compactor, run by whichever node is the writer
garbage_collection = true # SlateDB's garbage collector, same

[reader]                  # how a node reads the database it does not write
manifest_poll_interval_ms = 5000    # a stale read's freshness, and manifest traffic
checkpoint_lifetime_ms = 60000      # how long a dead replica keeps objects alive

[backlog]                 # what this node risks before it stops taking writes
max_log_bytes = 0                   # 0 is off; bounds the log above the durable watermark
max_undurable_ms = 0                # 0 is off; how long the store may stand still
allow_unbounded = false             # `true` runs a remote store with neither bound

[server]                  # what the listener admits; all defaults shown, none may be 0
principal_concurrency = 32          # in-flight RPCs per authenticated identity
principal_rate = 1000               # RPCs started per identity per second; with no TLS
                                    # there is one identity, so this is the node's RPS
client_request_bytes = 134217728    # decoded client/admin payloads held at once
peer_request_bytes = 134217728      # decoded raft payloads, on a separate budget
request_body_bytes = 268435456      # undecoded body frames in flight, all requests
max_body_bytes = 134217728          # and what any one body may have in flight
max_concurrent_scans = 128          # database views a scan may hold open at once
max_scan_buffer_bytes = 67108864    # what those scans may buffer, together

# Optional Prometheus scrape listen address. Unset means metrics are still
# recorded and still appear on Admin `Metrics`, but nothing listens. No auth.
# Non-loopback binds need metrics_allow_remote = true (and NetworkPolicy / firewall).
# metrics_listen_addr = "127.0.0.1:9100"
# metrics_allow_remote = false

[tls]                     # transport security, and the authorization built on it
cert = "/etc/bookie/pki/node-1.pem"   # this node's certificate; CN must be `node-1`
key = "/etc/bookie/pki/node-1.key"
ca = "/etc/bookie/pki/ca.pem"         # the one authority that defines this cluster
allow_insecure = false                # `true` runs with no TLS, and has to be said
# Without TLS, unauthenticated callers get `read` + `write` only. These two grant the
# rest, to anything that can reach the port, and are refused when TLS is configured.
# allow_insecure_admin = false          # the control plane: membership, split, move
# allow_insecure_peers = false          # raft traffic, as any node id it likes
# Addresses or CIDRs an admin request may make this node dial beyond its own peer
# table — for adding a learner or moving a shard to a node not configured here.
# outbound_allow = ["10.0.0.0/8", "203.0.113.7:9001"]

[tls.clients]             # what each client certificate's common name may do
reporting = ["read"]
app = ["write"]
operator = ["admin"]

BOOKIE_NODE_ID, BOOKIE_LISTEN_ADDR, BOOKIE_LOG_DIR, BOOKIE_PEERS, BOOKIE_STORE_URI, BOOKIE_STORE_PREFIX, and BOOKIE_METRICS_LISTEN_ADDR override the file. bookie check --config <path> resolves everything and prints the result.

Three sections decide how storage behaves over time:

• maintenance. Both tasks live inside the database the writer holds, so they run in exactly one process per group and a handoff moves them with the fence. Both are needed for bounded storage: garbage collection reclaims superseded manifests and orphaned SSTs, and compaction rewrites the LSM so a read does not have to merge an unbounded number of overlapping runs. Switch them off only if something else runs them.
• reader. Each replica's read-only view holds a checkpoint in the shared manifest and refreshes it by compare-and-swap, so a short poll interval buys freshness with manifest traffic, and the checkpoint lifetime is how long a replica that died keeps objects alive against garbage collection. The lifetime must be at least 1 s and at least twice the poll interval; a bad value fails at startup rather than at a replica's first read.
• backlog. A write is acknowledged once raft has a quorum, not once the store has the data, and purge waits for the store — so an outage is a local log that only grows, and what is above the durable watermark is what a correlated loss of disks would cost. max_log_bytes bounds the part of the log object storage has not got — not the log's whole footprint, which snapshot retention keeps large whatever the store has, so a footprint bound no purge could satisfy would make a healthy group read-only; max_undurable_ms bounds how long the exposure may last. Both default to off, and reaching either refuses writes while leaving voting, replication, and reads alone. The library keeps that default, because an embedded node's risk is its embedder's to state — but the bookie binary refuses to serve a store it does not share a failure domain with (s3://, gs://, az://, env://) until one of the two limits is set or allow_unbounded = true says the risk is accepted.

Cluster formation and later membership changes also verify the storage itself. The first node conditionally creates an identity object beside the database; every initial peer, learner, and shard-move target must report that same identity before it can enter membership. This catches identical memory:// names in different processes and identical file:// paths on different hosts, which configuration-string comparison cannot distinguish.

Formation is also serialized cluster-wide. init conditionally creates a genesis record beside the group's database — a cluster id, the database's path, the protocol version, and a digest of the initial membership — and exactly one peer list can claim it. Running init twice, or against every node, agrees with the record and is a no-op; running it with a different peer list over the same database is refused, because both succeeding would be two raft groups writing one database, each fencing the other. The record is verified again at every membership change and before a writer is taken, so storage that is not this group's cannot be fenced by way of the apply path either. init is refused outright over a database a cluster is already running against that carries no record — a cluster formed by a build that wrote none, or restored without it — because forming a second cluster over live data is the split this exists to prevent; bookie genesis adopt --config <path>, run from a node that is a voter in that cluster, writes the record it never had.

Security

Every service — RaftSvc, KvSvc, and AdminSvc — is behind one mutually authenticated TLS listener, and a caller's identity is the common name of the client certificate rustls verified against tls.ca. Nothing else is trusted: not the address a caller dialed from, not a header, not a field in the request.

• Peers carry the common name node-<id>, which authorizes raft traffic for that id and no other. A cluster's peer certificates sit on every machine in it, so one compromised host holding a valid peer certificate would otherwise be able to vote as every node at once.
• Clients are the names listed under [tls.clients]. A name that is not listed is refused, so there is no default authority.
• read / write / admin widen in that order, and peer is disjoint from all three: a peer certificate cannot read a key, and no client certificate — not even admin — can send raft traffic.
• Administration is denied by default. A leaked write credential, which is the one every application holds, cannot form a cluster, change membership, or move a shard.

allow_insecure = true runs a node with no transport security, for an isolated network or a single-machine test. It has to be stated: a node with neither certificates nor that flag refuses to bind rather than serving plaintext. An embedded node over net::mem opens no socket and needs neither. It is a claim about routing, so it also limits where such a node may bind — loopback or a private address, never a wildcard or a globally routable one.

What it grants unauthenticated callers is read and write, and nothing more. The two authorities that would let anything able to reach the port take the cluster rather than use it are withheld and named separately: allow_insecure_admin = true for the control plane (Init, ChangeMembership, Promote, Split, Merge, MoveShard, …), and allow_insecure_peers = true for raft traffic — which, with no certificate to bind a message to a node id, is authority to vote as any node. An insecure cluster of several processes needs both; an embed that forms itself in process needs neither. Both are refused when certificates are configured, where every authority comes from a name.

The listener also bounds retained request bytes separately for peers and clients, limits each authenticated identity's concurrency and request rate, and caps HTTP/2 streams per connection. Raft has its own byte reserve, so client pressure cannot consume the capacity needed for replication. Atomic batches, memberships, shard member sets, and bootstrap split points have explicit count ceilings. Administrative addresses may only name the node id and address already present in the local peer configuration — plus whatever tls.outbound_allow names, as exact addresses or CIDRs — and that is checked before the server performs any outbound probe.

Build with --features tls; without it, a configuration that names certificates is refused at startup rather than quietly ignored. tests/tls_cluster.rs is the evidence: a cluster forming and serving over mTLS, and each way in closed to a caller that should not have it.

Against a secured cluster the CLI must present an operator client certificate listed under [tls.clients] (typically admin), not a node's node-<id> peer leaf:

cargo run --features tls --bin bookie -- kv \
  --addr 127.0.0.1:9001 \
  --tls-cert /etc/bookie/pki/operator.pem \
  --tls-key  /etc/bookie/pki/operator.key \
  --tls-ca   /etc/bookie/pki/ca.pem \
  get foo

--config may still supply seed addresses; without --tls-* a TLS-enabled config is refused rather than silently presenting the node's peer certificate (which cannot authorize KV or Admin). BOOKIE_TLS_CERT / BOOKIE_TLS_KEY / BOOKIE_TLS_CA work the same way.

Consistency

Level	What it costs	What it promises
Linearizable (default)	a read-index round trip to a quorum, no log write; served by the writer	sees every write acknowledged before the read started
LeaderLocal	a leadership check on the local node; served by the writer	stale only if this node has been deposed and has not noticed — not linearizable; use only when that window is acceptable
Stale	any node answers; a replica pays object-store GETs on a cache miss	behind by the WAL flush interval plus one manifest poll — behind, never divergent

An unset consistency on the wire means Linearizable, and an unrecognized one is refused rather than downgraded: a field added later cannot quietly weaken a read. gRPC and the typed clients never silently fall back from Linearizable to LeaderLocal.

Client::scan is a collecting convenience and defaults to a 64 MiB resident-memory ceiling. Use Client::scan_stream to consume a large prefix incrementally with bounded read-ahead; the sharded client exposes the same streaming API.

A scan's stream is not bounded by request_timeout, which is the deadline on a unary request and on opening the scan. A stream open for as long as the caller keeps reading would be cancelled by any deadline short enough to be useful on a get, which would make the client's timeout the thing deciding how much a scan may return. scan_idle_timeout bounds silence instead — how long the stream may go without producing the next entry — and the server bounds the stream from its side too, with an idle guard and a total duration.

Numbers

Measured on an Apple M5 Max (18 cores, macOS 25.5, local-filesystem object store, in-process transport) with cargo bench, at reduced sample counts (30 samples for the read benches, 10 for the write ones), so differences under ~15% are noise. They are latencies for one operation, not a throughput ceiling, and the absolute values are dominated by this machine's fsync and by SlateDB's 100 ms flush interval — the ratios are the point.

These numbers are not production defaults. The benches tighten raft and reader timers so elections and stale freshness do not dominate the sample; production config is slower by design:

Knob	Bench	Default (BookieConfig)
raft.heartbeat_interval_ms	50	250
raft.election_timeout_*_ms	150–400	750–1500
reader.manifest_poll_interval_ms	50	5_000

Failover and Stale freshness SLOs must be sized from the defaults (and from the remote store's RTT), not from this table. There is no published remote-object-store bench series in-repo yet.

Benchmark	Median
write/raft/1-node	4.5 ms
write/raft/3-node	11.8 ms
apply/file_store/deferred_flush (batch applied, not waiting for object storage)	9.9 µs
apply/file_store/await_durable (same batch, await_durable: true, local file:// store)	102 ms
log_append/serial	3.6 ms
log_append/pipelined/16	7.6 ms for 16 appends (0.48 ms each)
read/3-node/get/linearizable (leader, writer)	17 µs
read/3-node/get/leader_local (leader, writer)	1.2 µs
read/3-node/get/stale (follower, read-only view)	19 µs

Four things worth reading off that table:

• await_durable: false is what buys that. Waiting for object storage costs four orders of magnitude more than applying a batch. Writes are safe without it because the applied-log-id lands in the same atomic batch as the data, so a crash that loses the memtable loses the applied-log-id too and raft replays the entries. The durable number is a local file:// store; a remote bucket adds network RTT and will dominate further — there is no published S3/GCS apply series in-repo yet.
• Group commit works. Sixteen pipelined appends cost 2.4 ms of fsync between them instead of 16 × 3.6 ms; the benchmark also prints the fsync count (about four appends per fsync under this load).
• The write path did not change with shared storage. Same log, same quorum ack, and only the leader ever wrote the database it applies to.
• A stale read on a replica is what shared storage costs. It reads the group's database through a read-only view rather than a local memtable — 19 µs here against 0.85 µs when every node had its own copy, and object-store GETs rather than a cache hit against a remote store. Read levels still differ by more than an order of magnitude, which is why the level is a per-request choice.

Sharding

shard::ShardedBookie runs the meta group plus one raft group per shard in one process. Each group gets its own subdirectory of log_dir and its own subprefix of store.prefix (bookie/meta, bookie/shard-1, …) — one database per group, written by that group's leader, so groups fail independently and losing one shard's leader does not move any other shard's. The subprefix derives from the cluster's prefix, so every replica of a shard names the same database. Sharding is also how write throughput is scaled, since one group is bounded by its single writer.

use bookie::net::mem::MemRouter;
use bookie::{Cmd, ReadConsistency, ShardedBookie};

let router = MemRouter::new();
let host = ShardedBookie::open(config, router.clone()).await?;
host.initialize().await?;                                  // form the meta group
host.bootstrap([bytes::Bytes::from_static(b"m")]).await?;   // two shards: [-inf, m), [m, +inf)

host.write(Cmd::put("apple", "1")).await?;                  // routed to shard-1
host.get(b"zebra", ReadConsistency::Linearizable).await?;   // routed to shard-2

The routing table lives under one key in the meta group and is changed by compare-and-swap (MetaCmd), so it always partitions the keyspace exactly and a retried proposal cannot apply twice. Each node reconciles against its own meta replica (ShardedBookie::reconcile), opening the groups it is a member of and closing the ones it is not.

Over gRPC it is the same cluster driven by commands. Every raft message names its group (0 is the meta group, so a single-group cluster is unchanged on the wire), and each change asks for a state the cluster then reconciles towards:

cargo run --bin bookie -- serve --config examples/n1.toml --sharded   # ...n2, n3
cargo run --bin bookie -- init --addr 127.0.0.1:9001                  # form the meta group
cargo run --bin bookie -- shard bootstrap --addr 127.0.0.1:9001 m     # [-inf, m), [m, +inf)

cargo run --bin bookie -- kv --sharded --addr 127.0.0.1:9002 put apple 1
cargo run --bin bookie -- shard split --addr 127.0.0.1:9001 1 c       # divide shard-1 at `c`
cargo run --bin bookie -- shard merge --addr 127.0.0.1:9001 3 4       # put two halves back
cargo run --bin bookie -- shard move --addr 127.0.0.1:9001 2 \
    --node 1=127.0.0.1:9001 --node 2=127.0.0.1:9002                   # re-replicate, no keys move

cargo run --bin bookie -- shard status --addr 127.0.0.1:9001          # sizes, placement, progress
cargo run --bin bookie -- shard suggest --addr 127.0.0.1:9001         # what the shape argues for

Seven things are worth knowing before using it:

• Operations never cross shards. A batch or a prefix scan spanning shards is refused (Error::CrossShard) rather than served without atomicity; a key belonging to a shard this node does not host is Error::WrongShard, naming the nodes that do and the epoch it answered from — which is what a ShardedClient's routing cache refreshes on.
• Splitting or merging a populated shard moves no bytes and fences only for cutover. A source records a replicated log boundary, takes a projected SlateDB checkpoint while reads and writes continue, and builds each target from that live copy. It then seals briefly, replays the retained ordered log suffix into the unopened targets with per-target progress markers, and swaps the routing table. A missing suffix is refused rather than producing an incomplete target.
• A shard's size is a node's answer, not the cluster's. Only a node holding a handle on a group's database can measure it, so shard status asks every node and merges: a node that does not answer leaves a size unknown rather than zero. The numbers are estimates off each manifest, they exclude what is still in a memtable or the WAL, and straight after a split both halves report the objects they share — so the sum over a cluster's shards is an upper bound on what the store holds. shard status also names the storage retired shards left behind, which is space to reclaim rather than a fault.
• Nothing rebalances on its own. shard suggest reports imbalance and prints the command for each change it argues for; no controller runs it. Every routing change is operator-issued.
• A migration waits rather than fails, so time is the signal. Every step is idempotent and taken by whichever node can take it, which means a change that cannot proceed sits in its stage instead of erroring. bookie_migrations_in_stage{stage=...} and bookie_migration_stage_age_seconds are what an alert watches, and bookie shard status names the stage each change is in.
• A sharded node's readiness is every group's. A key reaches the group that owns it, so a node whose meta group is healthy while a shard group it hosts cannot serve — or while the table has given it a shard it has not opened yet — reports not-ready and is taken out of a load balancer. The reasons name the condition and never the shard, so a metric labelled with them does not grow a series per split; bookie shard status is where an operator finds out which shard. See docs/ops.md.
• --sharded is checked against the store, not trusted. The two kinds of cluster keep their data in different places under one prefix — a database per group beneath it when sharded, one database at it when not — so the binary reads the layout before binding its port and refuses a mode that disagrees with it. Without that, a node started in the wrong mode opens an empty database beside the cluster's real data and waits to be initialized, which is how one cluster becomes two.

Testing

make check                      # fmt-check + clippy -D warnings + test + doc + fuzz-build + loom
cargo test --all-features       # unit, cluster, gRPC, snapshot, fault, simulation, linearizability
cargo test --test process_cluster -- --ignored   # three real `bookie serve` processes
cargo build --no-default-features               # the library stays embeddable
make loom                                       # every interleaving of the shared cells
make fuzz-build                                 # every fuzz target still compiles
make fuzz TARGET=log_directory                  # one target, until it is stopped (nightly)
make coverage                                   # line coverage, needs cargo-llvm-cov
cargo bench                                     # writes and reads

Beyond per-module unit tests, the suite includes:

• tests/cluster.rs, tests/grpc_cluster.rs — a real raft cluster in process and over loopback sockets: elections, redirects, membership changes, restarts, a client answered for an entry the database already held, and a store whose layout refuses a node started in the other mode.
• tests/openraft_conformance.rs — openraft's own storage conformance suite run against DiskLog plus StateMachine.
• tests/linearizability.rs — concurrent compare-and-swap writers and linearizable readers under repeated leader loss, with a history check afterwards: no acknowledged write lost, none applied twice, no read behind an acknowledged write.
• tests/shared_storage.rs — the writer handoff driven by hand, with no raft core above it: the tail a new writer must replay, the entries it must skip, a replayed compare-and-swap seeing the state it originally saw, and a fenced writer becoming a reader instead of stopping the node.
• tests/simulation.rs — the same cluster on a turmoil simulated network: every node in one thread on a paused clock, with seeded latencies and packet loss. Cut links rather than stopped hosts, so a deposed leader keeps running and keeps believing it leads; a node crashed and bounced back; and concurrent compare-and-swap writers plus linearizable readers through repeated partitions, checked for lost, duplicated, and stale-read outcomes over several seeds. Only the socket is simulated — BookieServer::serve_with_incoming and GrpcNetworkFactory::with_dialer are the two seams that make the transport a parameter.
• tests/faults.rs — an object store that fails on command (the writer keeps serving, a replica's view freezes rather than failing), a writer evicted by SlateDB's fence, and a raft log whose disk refuses a write.
• tests/purge.rs — the log is never purged past what the group's database holds, and a replica whose view is behind purges less rather than more.
• tests/backlog.rs — a node past its [backlog] budget refuses writes and nothing else: it keeps voting, keeps serving reads, names the limit it is over, and reports what it holds rather than latching.
• tests/snapshot.rs, tests/grpc_snapshot.rs — a learner and a purged-behind follower catching up without moving any bytes, and the state surviving a restart.
• tests/amplification.rs (--ignored) — the measurement the whole layout is for: the same volume written through a 1-node and a 3-node cluster leaves the same bytes in the store, and one manifest rather than three.
• tests/sharded_cluster.rs — several raft groups per node: keys routed to the group that owns them, cross-shard operations refused, one shard losing its leader while the other keeps serving, a node dropped from a shard closing that group, a populated shard split and merged with every key still readable, a split interrupted at every stage finishing on restart, a change in flight timed in the stage it is in and forgotten once it finishes, a node reporting itself not-ready while a group it hosts cannot serve or while the table has given it a shard it has not opened, and a node measuring the shards it hosts while only naming the rest.
• tests/grpc_sharded_cluster.rs — the same cluster over sockets: every raft message addressed to its group, a stale routing cache refreshing on a redirect, a readiness probe answering for the shard groups a node serves, and a cluster's reported shape turned into advice and that advice carried out.

Three kinds of test generate their own inputs rather than being given them, which is where the cases nobody writes down come from:

• Properties (proptest, in the modules they cover) — a routing table still partitions the keyspace after any sequence of splits, merges, re-memberings and migration stages; every key routes to exactly one shard; intersecting two key ranges keeps exactly the keys both hold; a segment scan finds every record that was written and no single-byte change anywhere in a record goes unnoticed; a command's size estimate never understates its encoding.
• A model (tests/log_model.rs) — the raft log driven through arbitrary sequences of appends, truncations, purges, votes and restarts, compared after every step against a BTreeMap that says what it should hold. What this finds needs several operations in one order, which is what a leader losing an election mid-replication produces.
• Fuzzing (fuzz/, bodies in src/fuzz.rs) — every place bytes this process did not write become values it acts on: a command or response off the wire, a routing table or genesis record out of the object store, a request id and the dedup keys built from it, a configuration file, a peer certificate's subject name, and a raft log directory that is written properly and then damaged at a chosen offset. The bodies live in the library so tests/fuzz_targets.rs runs all of them on an ordinary cargo test — over generated bytes, over mutations of valid encodings, and over every input committed under fuzz/corpus or fuzz/artifacts — and make fuzz runs one of them properly. This is not decorative: it is what found unhex panicking on a dedup key holding a multi-byte character, in the apply path, where a panic is every replica's at once.
• Interleavings (loom, models in src/sync.rs) — the cells a node shares outside a lock, checked against every legal ordering rather than the ones this machine produces: a shutdown flag publishes the refusal that preceded it, a shutdown phase cannot go backwards under two callers, a snapshot sequence hands no number out twice, and a recorded routing epoch publishes the groups the pass that recorded it opened. loom replaces those atomics under --cfg bookie_loom, so the code it checks is the code that ships.

Operating notes

• Storage does not multiply with replicas: one database per group, one copy of the data, one compactor and one garbage collector — all in the process holding the writer. tests/amplification.rs is the measurement.
• Write throughput per group is bounded by one writer. More shards is how that is scaled; shared storage changes what each shard costs, not what one shard can do.
• An unavailable object store blocks durability, not the cluster. SlateDB retries a store that is refusing writes rather than giving up, so a flush waits instead of failing — while writes keep being acknowledged and the writer keeps serving reads from memory. It costs more than it used to, though: a replica's read-only view stops advancing and its Stale reads keep answering from its last view, and a node restarting against a store that will not answer cannot learn the group's watermark, which is fatal at startup rather than retried.
• That outage is unbounded unless you bound it. Purge is gated on the shared database being durable, so a store that has stopped answering is a local raft log that only grows, and everything above the durable watermark is held by local disks alone. [backlog] is where a deployment says how much of that it will carry: max_log_bytes bounds the disk and max_undurable_ms bounds the exposure. Reaching either refuses writes and nothing else — the node keeps voting, replicating, and serving reads, so a group whose store has failed degrades to read-only rather than out of the cluster. Both default to off, because the honest default for "how much unreplicated data may this deployment risk" is the one the deployment states — and the bookie binary refuses to start against a remote store until the deployment has stated it (backlog.allow_unbounded = true states that the risk is accepted).
• The object store is a tier-0 dependency, and its availability is the ceiling. There is one copy of the data and it lives there, so the store is a shared fate domain: every group under one bucket shares it, and multi-AZ compute buys no storage independence. What a store outage costs is bounded in the ways above — writes keep being acknowledged until [backlog] says stop, and the writer keeps serving reads out of memory — but three things are not available while it lasts: a replica's Stale reads stop advancing, a node restarting cannot learn the group's watermark, and a new writer cannot open the database to take over. So a group survives losing its leader or its store, not both, and a workload that needs reads to keep working from local disks through a store outage needs a storage model where each replica has its own copy — which is the trade this one is the other side of. Run the bucket multi-AZ, keep the backlog limits tight enough that the disks outlast the outage you plan for, and see docs/ops.md for the runbook.
• Shutdown has one deadline, not one per step. Bookie::stop gives the whole sequence SHUTDOWN_DEADLINE, and a sharded node shares that one budget across every group it hosts rather than spending it once each. What is abandoned when it passes is the object-store flush and close; the local raft log is made durable first, so what did not reach the store is replayed from the log on the next start.
• A replica's read-only view holds a checkpoint in the shared manifest. N replicas add N checkpoints the writer's garbage collection must respect, a dead replica leaves one until reader.checkpoint_lifetime_ms expires, and refreshing them is manifest traffic proportional to replicas × 1/manifest_poll_interval_ms. A view that closes under a replica — an expired checkpoint, a failed poller — is reopened rather than waited out: the background follower checks whether the handle still works instead of treating the role the node holds as evidence that it does.
• A raft log that cannot finish a destructive operation stops answering. A truncation or a purge that deleted some of its files and then failed leaves an index describing files that are not there, and a purge fails that way even when it deleted nothing, because its point reached the disk first. Neither is retryable in place — the retry would find the watermark already advanced and report success over files it never touched — so the log refuses every later request and says why. Reopening the directory is the repair: recovery rebuilds the index from what the files hold and finishes the purge the metadata still owes.
• memory:// is single-process. The store lives in a process-global registry, so memory:// (or memory://name) shares one set of bytes between nodes in one process and nothing at all between processes.
• Certificate management is the deployment's. There is no issuance, rotation, or revocation here: a node reads three PEM files at startup and does not re-read them, so rotating a certificate means restarting the node. Revocation is not checked at all — a compromised certificate is valid until its authority is replaced. See docs/ops.md for the rotation procedure.
• Health is not the same as listening. grpc.health.v1.Health on the data-plane port answers liveness on the empty service name (process up) and readiness on ready (initialized, storage open, within durability budget, not shutting down). Admin Metrics reports the same readiness tokens, durability lag, retry-window occupancy, admission headroom, and shutdown phase. Set metrics_listen_addr for a Prometheus scrape port; it has no authentication.
• Logs are structured when asked. --log-format json / BOOKIE_LOG_FORMAT=json writes one JSON object per line on stderr; each RPC span carries a request_id. User key and value bytes are never logged.
• A request id is remembered for a window, not forever. put_idempotent, delete_idempotent, batch_idempotent, and cas_idempotent attach a RequestId that the group remembers, so a retry after an ambiguous failure is answered with what the first attempt did rather than applied a second time. IdempotentClient does the same with a caller-supplied (or UUID) factory on every write. The table holding those answers is replicated and therefore bounded — 16,384 requests or 64 MiB, whichever comes first, oldest dropped — so a retry from further back than that applies again as though it carried no id. There is no separate "window expired" error: once the recording is gone the group cannot tell a late retry from a first use, so re-apply is intentional. The bound is in requests rather than in seconds so that a slow cluster does not silently shorten the window; a retry inside one client's attempts is nowhere near it, and one delayed by anything else — a queue redelivering, an operator replaying by hand — should be bounded against how full the window is: bookie metrics reports dedup_entries and dedup_bytes against their limits, and bookie_dedup_entries{,_limit} / bookie_dedup_bytes{,_limit} are the gauges — read only on a node holding a writer handle, since the table is one per group and it is what fills it, and on a sharded node reported as the fullest window of any group the node hosts rather than the meta group's, which carries the routing table and no request ids. A write sent without an id is unchanged: it is not retried after an ambiguous failure, because a retry commits wherever it lands and can put back a value another writer had overwritten. Do not put a generic gRPC retry interceptor in front of plain put/delete/cas — that is the anti-pattern this client exists to prevent.
• An id is per group, not per cluster. A shard's group remembers the requests it answered, so a retry that a routing change sends to a different group is a fresh request there.

Operating procedures — RPO/RTO, backup/restore, quorum-loss recovery, rolling upgrade, certificate rotation, and DR drills — live in docs/ops.md. Starter Kubernetes manifests are under deploy/kubernetes/.

License

MIT.