sqld: after an unclean shutdown the fsynced replication log (wallog) is truncated and rebuilt from the database file; with the default durability_mode acknowledged tables and rows are lost after a power loss #2297

Summary

With the default configuration, sqld acknowledges a write after appending its frames to the replication log dbs/<ns>/wallog and fsyncing it (LogFile::commit/write_header, logger.rs#L179-L201, added in #2200). The SQLite WAL of the database (dbs/<ns>/data-wal) is not synced, because the default namespace durability_mode is relaxed, i.e. PRAGMA synchronous = NORMAL (connection/config.rs#L115-L136).

After a power loss the wallog still holds every acknowledged commit, but on start-up sqld sees the leftover .sentinel file (namespace/configurator/helpers.rs#L61-L73), logs Replication log is dirty, recovering from database file. (logger.rs#L588-L606) and in recover() checkpoints the database, truncates the wallog (log_file.reset() → set_len(0)) and rebuilds it from the database file (logger.rs#L685-L696, #L431-L438). The database file and its WAL hold what SQLite synced (the state as of the last checkpoint) plus whatever part of the unsynced WAL the kernel happened to write back before the power loss. The one durable copy of the recent commits is the log that recovery throws away. Acknowledged tables and rows are gone, and no error is reported.

Reproduction

As root, with Docker and device-mapper (no special hardware): the data volume is a dm device that is switched to the error target without a flush, which drops everything not yet written to it, as a power cut does. commit=3600 keeps ext4's periodic journal commit out of the way; the result is the same when the journal is committed just before the cut.

IMG=ghcr.io/tursodatabase/libsql-server:latest; W=/tmp/sqld-pf; P=18080
q() { curl -s "http://127.0.0.1:$P/" -H 'Content-Type: application/json' -d "{\"statements\": [\"$1\"]}"; echo; }
mkdir -p $W/data
truncate -s 1G $W/disk.img; LOOP=$(losetup -f --show $W/disk.img); SZ=$(blockdev --getsz $LOOP)
dmsetup create sqldpf --table "0 $SZ linear $LOOP 0"
mkfs.ext4 -q /dev/mapper/sqldpf; mount -o commit=3600 /dev/mapper/sqldpf $W/data
docker run -d --name sqld --user 0:0 -v $W/data:/var/lib/sqld -p $P:8080 $IMG
until q "SELECT 1" | grep -q results; do sleep 0.5; done
sleep 1; sync -f $W/data                     # start from a durable installation

q "CREATE TABLE t1 (id INTEGER PRIMARY KEY, v TEXT)"
q "INSERT INTO t1 VALUES (1, 'one'), (2, 'two'), (3, 'three')"
q "SELECT count(*) FROM t1"                  # 3
ls -l $W/data/iku.db/dbs/default             # data-wal 12392, wallog 12424

# power cut: kill sqld, then fail every further I/O to the volume without flushing
docker kill sqld
dmsetup suspend --noflush --nolockfs sqldpf; dmsetup load sqldpf --table "0 $SZ error"; dmsetup resume sqldpf
umount -l $W/data; dmsetup remove --retry sqldpf; losetup -d $LOOP

# "reboot"
LOOP=$(losetup -f --show $W/disk.img); mount $LOOP $W/data
ls -l $W/data/iku.db/dbs/default             # data-wal 4096 (not synced), wallog 12424 (synced: all 3 commits)
docker start sqld; until q "SELECT 1" | grep -q results; do sleep 0.5; done
q "SELECT count(*) FROM t1"                  # {"error":"no such table: t1"}
docker logs sqld 2>&1 | grep dirty           # Replication log is dirty, recovering from database file.
ls -l $W/data/iku.db/dbs/default             # wallog 4184: truncated and rebuilt from the database file

Output on a build of main (d8bc862); :latest (0.24.33) prints the same:

[{"results":{"columns":["count(*)"],"rows":[[3]],...}}]
-rw-r--r-- 1 666 666 12392 ... data-wal
-rw-r--r-- 1 666 666 12424 ... wallog
--- after the power cut:
-rw-r--r-- 1 666 666  4096 ... data-wal
-rw-r--r-- 1 666 666 12424 ... wallog
[{"error":"no such table: t1"}]
INFO ... libsql_server::replication::primary::logger: Replication log is dirty, recovering from database file.
-rw-r--r-- 1 666 666     0 ... data-wal
-rw-r--r-- 1 666 666  4184 ... wallog

The exact outcome depends on how much of the unsynced data-wal the kernel wrote back before the cut. In 10 runs of this script (7 on main, 3 on :latest), 3 printed no such table: t1, 5 printed count(*) = 0 (data-wal 12288 bytes after the cut: the last frame is torn, so the INSERT commit is dropped and only the CREATE TABLE survives), and 2 kept the 3 rows. In every run the wallog was 12424 bytes (all 3 commits) after the cut.

A longer workload over the HTTP API (create t1 with 1,000 rows, an index, UPDATE/DELETE, ALTER TABLE, create/drop t2, then 20,000 more rows, a range delete of 1,000 rows, 1,000 more rows), each statement batch acknowledged, then a power cut:

(17,900 rows is the state at SQLite's last automatic checkpoint.)

Setting the namespace to durability_mode: strong through the admin API (sqld started with --admin-listen-addr, then POST /v1/namespaces/default/config with {"block_reads":false,"block_writes":false,"durability_mode":"strong"}) before the same script: t1 has its 3 rows after the cut (3 of 3 on main). With LIBSQL_IGNORE_DIRTY_LOG=1 the wallog is kept, but the acknowledged rows are still missing from the database whenever the data-wal lost them, so the log and the database disagree.

Expected

A write that sqld acknowledged, after making it durable in the wallog, should survive a power loss. On dirty start-up, the frames of the wallog that are newer than the database should be applied to it (the log has the page images and commit boundaries for that), instead of truncating the log and rebuilding it from a database file that may be older.

If losing recent commits on power failure is intended with the default relaxed mode, it would help to document durability_mode (it currently appears only in the admin API and in #1533) and to say that the default does not survive a power loss even though every commit is fsynced to the wallog. In that case the per-commit wallog fsync does not give the client any durability.

Environment

• sqld main d8bc862 (2026-10-13; built from source with the repository's toolchain 1.85.0 in a rust:1.85-slim-bookworm container, sqld 0.24.33 (d8bc862d 2026-10-18)); ghcr.io/tursodatabase/libsql-server:latest = 0.24.33, revision d6c75af (2026-08-23). The commits between the two do not touch libsql-server.
• Docker on Ubuntu 22.04.2 LTS, kernel 5.15.0-187-generic, ext4 data volume, default sqld configuration (SQLD_NODE=primary).

Related

• Database getting corrupted on power loss #2254 (open): after a power loss sqld fails to start on an empty JSON file (stats file). Different symptom (a startup failure); libsql-server: Don't fail namespace load on corrupt stats file #2255 addresses it.
• server/replication: fsync frame data in wallog before writing header #2200 (merged Dec 2025): fsync of wallog frames before the header, so that the log is consistent after a crash. That makes the wallog durable per commit; the dirty-recovery path above then discards it.
• Sqld will recover databases multiple times if namespace is unloaded #1107 (closed): sqld recovers a namespace from the database file repeatedly when it is unloaded; same recovery path, not about lost data.