Skip to content

fix(rekey): a reader leaving mid-deferral must not strand the retirement - #30

Open
presempathy-awb wants to merge 4 commits into
NodeDB-Lab:mainfrom
presempathy-awb:upstream-fix/rekey-retirement-race
Open

fix(rekey): a reader leaving mid-deferral must not strand the retirement#30
presempathy-awb wants to merge 4 commits into
NodeDB-Lab:mainfrom
presempathy-awb:upstream-fix/rekey-retirement-race

Conversation

@presempathy-awb

Copy link
Copy Markdown
Contributor

Fixes #29.

The defect

retire_rekey_source_when_safe took the decision to defer under
tracked_readers and recorded the obligation under pending_key_retirements
— two locks with a gap between them, and nothing holding the reader set still
across it.

So the last reader could leave inside that gap. Its unregister_read is the only
caller of drain_pending_key_retirements, and it only drains when the reader vec
becomes empty, so it ran, read a still-empty obligation list, and returned. The
push landed a moment later in a list whose only reader had gone, and nothing
visits that list again unless another reader happens to register and drop.

The superseded master key then stayed leasable for the life of the handle while
rekey_db returned Ok(()) — no error, no log line, no completion state to
query. EpochKeyring::remove's own doc names what breaks: "without it the
superseded master key stays leasable for the life of the handle, and the store
keeps answering under the key the rotation was performed to stop honouring."

The fix

Hold pending_key_retirements across the reader check, so deciding to defer and
recording what was deferred are one step. drain_pending_key_retirements takes
the same two locks in the same order.

No call path nests them the other way: unregister_read releases
tracked_readers before it drains, and nothing else in the tree touches
pending_key_retirements. Whichever side gets there first, the outcome is the
same — if the reader is still registered the obligation is recorded before its
drain can run; if it has already left, the retirement happens inline.

Also stopped the drain discarding work it never attempted: it mem::take-d the
whole list up front, so an error mid-loop dropped every remaining obligation, and
unregister_read only logs the error. Entries now leave the list once their
retirement has succeeded. Latent today — the one reachable error is the
active-epoch refusal, which strictly increasing epochs already prevent — but it
is the same shape as the bug above.

How to test it

a_reader_leaving_mid_deferral_still_retires_the_source_epoch drives a reader
out at exactly the instant the old code pushed into, then requires the epoch to
retire anyway. The interleaving is made deterministic by a #[cfg(test)] hook
invoked inside the critical section, in the style of the existing
RekeyTestFault / VisibilityTestHook seams, rather than by a sleep or a race.

It discriminates rather than merely passing: revert only the lock order in
retire_rekey_source_when_safe and it fails with "an obligation no reader is
waiting on must not survive the deferral"
.

The runtime is current_thread on purpose — the interleaving comes from a real
OS thread via std::thread::scope, not from the executor, so it needs no
rt-multi-thread.

Against c6dbba8:

  • cargo test --lib — 515 passed
  • cargo clippy --lib --all-features — clean
  • cargo fmt --check — clean

Tradeoffs

The test seam adds one #[cfg(test)] field to Db plus its two constructor
sites. That is the cost of pinning this deterministically; the alternative is a
stress loop that passes both before and after the fix.

No format change, no public API change, no durability ordering change — the two
locks are in-process only, so there is nothing for an existing store to do on
upgrade.

Scope

Crash-recovery, resume and abort paths were already sound: open/recovery.rs
always reaches resume_rekey_intent → retirement, refuses non-standalone opens
with an intent in flight, and a crash before retirement achieves it implicitly
because a reopen installs only the header's epoch. This deferral gap was the only
leak I found.

Worth stating since it bounds what the fix buys: retirement drops the key from
the in-memory EpochKeyring; it does not make the source epoch underivable,
since derive_mk(kek, kek_salt, epoch) reproduces it from the KEK at will.

presempathy-awb and others added 4 commits August 11, 2026 06:03
retire_recorded previously stopped draining at the first failed
retirement, leaving every obligation behind it stuck queued as well.
Retirements are independent epochs, so it now retains only the
entries that actually fail and keeps retiring the rest, reporting the
first error once the pass is done.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Rekey source-epoch retirement is silently dropped if the last reader leaves mid-deferral

2 participants