Skip to content

Free-list save: stop drawing after 8 rounds, derive the round budget (ADR-0044) - #175

Merged
qdequele merged 3 commits into
mainfrom
adr-0044-freelist-save-bound
Nov 12, 2026
Merged

qdequele merged 3 commits into
mainfrom
adr-0044-freelist-save-bound

Conversation

@qdequele

Copy link
Copy Markdown
Owner

Problem

freelist_save (commit step C1) loops until the free-page records stop changing. Release builds limited it to 64 rounds with an assert!. The termination proof (SPEC 05 GC-13) bounds the rounds by the pool sizes and the number of committed GC pages, which can be thousands, so 64 did not follow from it. A valid state could in principle panic the commit. No data would be lost (nothing is written before C2), but a retry of the same batch would likely panic again: for Meilisearch, an index that cannot be written.

This has never been observed. With a temporary counter, cargo test --workspace, the crash harness (200 and 50 multi-process cycles) and the reader stress tests needed at most 3 rounds. The fuzz targets were not measured.

Change

ADR-0044, Option C with Option B's budget:

  • After 8 rounds the save stops drawing from the drain pool and the annex pool. It allocates from loose pages or extends the file. Draws are the only way one round makes the previous round's records stale, so the save then ends within G + 3 rounds (G = committed GC pages). The argument is the new rule GC-13a. The ids left in the pools stay listed and are reusable by later transactions; the cost is a file a few pages larger.
  • The last guard is derived from the proof: 3·(P + D + A) + G + 64 rounds, computed at entry (GC-13). It is never below the old limit. It is still a panic.
  • No format change.

Tests

No test reaches round 9 at the default threshold, so the new path gets its own run: just save-no-draw, also a CI job. It builds with --cfg zerodb_save_draw_rounds="0" and ="1" (no round draws; only the first round draws) and runs the tests of zerodb-core and zerodb and 200 crash cycles. Those tests check contents and page accounting (every page reachable or listed free, never both).

Two tests are skipped in that run, by name in the recipe, because they assert a file size and page numbers that hold only when the save reuses pool pages: over_cap_freed_set_spills_whole_to_gc_tree and delete_rebalance_logical_images_match_golden. They run unchanged in the normal suite.

Checks run locally (macOS arm64)

  • cargo fmt --all -- --check, cargo clippy --workspace --all-targets -- -D warnings: clean
  • cargo test --workspace: all pass
  • just crash-test-quick, just loom, just save-no-draw: pass
  • Not run locally: miri, the fuzz gate, the Linux clippy run, just stress, just bench-ab. The allocator's hot path gains one empty match arm; I make no performance claim.

For the reviewer

  • The ADR is marked accepted on the maintainer's instruction to proceed with its recommendation. If that is not what was meant, the status line is the thing to change.
  • The GC-13a argument and the budget derivation in GC-13 are new text in the spec and deserve a careful read: the code is small, the argument is the substance.
  • Open questions left in the ADR: the threshold (8), panic or error for the last guard, and whether to keep a permanent round counter.
  • The second commit is the one-line clippy 1.100 fix that is also in kv-compare: cold-cache read benchmark, and a Graviton4 + gp3 run #173, so that CI can pass here before kv-compare: cold-cache read benchmark, and a Graviton4 + gp3 run #173 merges.

🤖 Generated with Claude Code

qdequele and others added 2 commits November 12, 2026 20:25
…(ADR-0044)

The save loop was limited to 64 rounds. The termination proof (SPEC 05
GC-13) bounds the rounds by the pool sizes and the committed GC pages, not
by a constant, so a valid state could in principle panic the commit, and a
retry of the same batch would likely panic again.

After 8 rounds the save now allocates from loose pages or by extending the
file, and no longer from the drain pool or the annex pool. Draws are the
only way one round makes the previous round's records stale, so the save
then ends within a bounded number of rounds (GC-13a), with a file a few
pages larger. The remaining guard is computed at entry from the proof's
quantities and is never below the old limit.

`just save-no-draw`, also a CI job, builds with the threshold at 0 and at 1
and runs the engine's tests and the crash smoke on that path.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`run` returns `ExitCode`, which is itself `#[must_use]`. Clippy in Rust
1.100 reports the attribute as `double_must_use`, and CI runs clippy with
`-D warnings` on the current stable toolchain.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 4 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 9872253d-04e0-4519-aef6-77b2ab7ec689
Commits

Reviewing files that changed from the base of the PR and between 9115969 and b1f76f6.

Files selected for processing: 13

  • .github/workflows/ci.yml
  • CHANGELOG.md
  • CONTRIBUTING.md
  • crates/zerodb-core/Cargo.toml
  • crates/zerodb-core/src/rwtxn/alloc.rs
  • crates/zerodb-core/src/rwtxn/freelist.rs
  • crates/zerodb-core/src/rwtxn/mod.rs
  • crates/zerodb-tools/src/lib.rs
  • crates/zerodb/tests/save_at_map_limit.rs
  • docs/DECISIONS.md
  • docs/SPEC/05-gc.md
  • docs/adr/0044-freelist-save-round-bound.md
  • justfile
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Once the save has stopped drawing (GC-13a), an allocation the pools could
serve extended the file instead. With the file at the map's end that
returned `MapFull` and failed a commit that succeeded before. In that case
the save now draws from the pools as it does in its first rounds; it then
ends by GC-13 and the round budget.

`save_at_map_limit.rs` covers it: under `just save-no-draw` it fails
without the fallback. Also corrects comments that still described the
save's allocation mode as fixed for the whole procedure.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@qdequele

Copy link
Copy Markdown
Owner Author

Follow-up commit, after two independent checks of the first one.

Independent review. A second agent, given the code and the three claims but not my reasoning, tried to break them (termination after the switch, the budget as a true bound, no leak or double listing across the switch). It could not, and verified each step against the code. It found one real gap and several stale comments:

  • MapFull in no-draw mode (fixed here). From round 9 on, an allocation the pools could serve extended the file instead. With the file at the map's end the commit failed with MapFull, where it succeeded before. The save now draws from the pools when the file cannot be extended. In that case GC-13a's argument does not apply and the save ends by GC-13 and the budget, as it did before this PR.
  • Test. crates/zerodb/tests/save_at_map_limit.rs: the file a few pages from the map's end, with more free pages than the annex lists. Under just save-no-draw it fails at the second commit without the fallback and passes with it. At the default threshold it passes either way.
  • Comments and spec that still said the save's allocation mode is fixed for the whole procedure are corrected (freelist.rs, mod.rs, SPEC 05 GC-12 and the pseudo-code).

Fuzz. diff_ops (ZeroDB against LMDB) ran 10 minutes, 65,429 runs, built with --cfg zerodb_save_draw_rounds="0": no finding. That run was on the first commit, before the MapFull fallback.

An agent's review is not a substitute for a human who knows LMDB's free list. The parts that most need one are the GC-13 budget paragraph and GC-13a in docs/SPEC/05-gc.md.

Re-run locally after this commit: fmt, clippy, cargo test --workspace, just save-no-draw, just crash-test-quick, just loom: all pass.

@qdequele
qdequele merged commit c7e1abf into main Nov 12, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant