A checkpoint drops a page image committed while it drains the WAL index: the commit is lost and an id can return another document #311

A checkpoint that runs while commits continue can throw away a page image that was committed after the checkpoint wrote that page. The page file keeps the older image and the WAL index no longer has the newer one. Reads then return the older page: a committed document disappears, or an id returns another document. Once the WAL index is empty, the WAL is truncated, so the lost commit is not recovered on reopen either.

This explains the failure of ConcurrentConsistencyTests.HighContention_ManySmallTransactions_ConsistentFinalState in CI run 38749678260 on main (519b20f, 2026-10-15). Line 244 of that file is the "d" assertion. So FindByIdAsync(id) returned a document from the same writer and the same transaction (t and tx matched) but with another document index, and CountAsync had just returned the expected 250.

Mechanism (0450b97)

CheckpointCoreAsync (src/BLite.Core/Storage/StorageEngine.Recovery.cs:146-151) drains the entries it flushed like this:

if (_walIndex.TryGetValue(kvp.Key, out var current) && ReferenceEquals(current, kvp.Value))
{
    _walIndex.TryRemove(kvp.Key, out _);

In single-process mode the checkpoint takes no lock that commits wait for. In multi-process mode it releases _commitLock before the flush and the drain. The group-commit writer publishes images with _walIndex[pageId] = image (StorageEngine.GroupCommit.cs:212). So:

1. The checkpoint snapshots _walIndex while page P holds image v1, and writes v1 to the page file.
2. The drain reads _walIndex[P], finds v1, and the check passes.
3. A commit publishes v2: _walIndex[P] = v2.
4. The drain calls TryRemove(P) and removes v2.

The page file now has v1, and nothing holds v2 (its WAL record stays only until the next truncation). What happens next depends on the page type:

• Data page: the next insert into P reads v1's slot table and gets the slot of the document v2 added. The primary index then has two ids at the same (page, slot), and the earlier id returns the later document. This is the CI failure. CountAsync is unaffected because both index entries still resolve.
• Index page: the next insert rebuilds the leaf from v1, and the key v2 added is gone: FindByIdAsync returns null.

Reproduction

Without changing any code (storage level). One writer per page commits an image with an increasing counter in a loop. After each commit it reads the page back with no transaction. A second task calls CheckpointAsync() in a loop.

var config = PageFileConfig.Server(dbPath) with { LockTimeout = new LockTimeout { ReadTimeoutMs = 60_000, WriteTimeoutMs = 60_000 } };
using var storage = new StorageEngine(dbPath, config);
var pages = Enumerable.Range(0, 8).Select(_ => storage.AllocateCollectionPage("pages")).ToArray();
using var stop = new CancellationTokenSource(TimeSpan.FromSeconds(3));
var checkpointer = Task.Run(async () => { while (!stop.IsCancellationRequested) await storage.CheckpointAsync(); });
await Task.WhenAll(pages.Select(pageId => Task.Run(async () =>
{
    var image = new byte[storage.PageSize]; var readBack = new byte[storage.PageSize];
    for (int n = 1; !stop.IsCancellationRequested; n++)
    {
        BitConverter.TryWriteBytes(image.AsSpan(64), n);
        var txn = storage.BeginTransaction();
        storage.WritePage(pageId, txn.TransactionId, image);
        await txn.CommitAsync();
        storage.ReadPage(pageId, null, readBack);
        Assert.Equal(n, BitConverter.ToInt32(readBack, 64)); // committed image n must read back as n
    }
})));

With 32 CPU-burning threads running alongside (so the checkpoint thread gets preempted between the two calls), 2 of 16 three-second runs failed. One failure read page C0000043: committed image 9061, read back image 9060. On an idle machine, 0 of 20 runs failed: the gap is two adjacent dictionary calls.

With the gap widened (fault injection). Put a Thread.Sleep(1) between TryGetValue and TryRemove. Then run 8 collections × 300 auto-commit DynamicCollection.InsertAsync calls with a CheckpointAsync() loop running, and read every id back:

The first example has the CI failure's exact shape: the id returns the document inserted right after it into the same page.

The test itself under load. I ran the test class 60 times with DOTNET_PROCESSOR_COUNT=2 and two full-suite copies running alongside. The run also included a variant that records every document read back (5 rounds per iteration). The silent mismatch did not come back in those 60 iterations. The failures were the 500 ms lock timeouts (#303, #302), plus once an unrelated routing error (#312).

Fix

Remove the entry only if it still holds the flushed image, in one atomic step:

if (((ICollection<KeyValuePair<uint, byte[]>>)_walIndex).Remove(kvp))

ConcurrentDictionary compares the value with EqualityComparer<byte[]>.Default, which is reference equality for arrays, under the bucket lock. This works on net10.0 and netstandard2.1.

Remaining window (multi-process only, not fixed by the above). The following _walOffsets.TryRemove(kvp.Key, …) can still remove the offset of an image published after the atomic remove. A later checkpoint then treats that image as outside every reader's window. This is a smaller issue; it could be fixed by capturing the offset in the snapshot and removing it conditionally the same way.

Affected versions: since non-blocking checkpoints (455d28a, "fix: Non-blocking checkpoints; defer metadata writes", first released in v3.8.0) through v5.2.3 and main at 0450b97.

Related: #312 (the routing error above), #288 (reliability epic), #302 / #303 (the other failure mode of the same test class), #304 (a different lost-write bug: two open transactions on the same page).