[jbrown9513]

Stacked on #98.

Problem

On a ramp upgrade, the process being replaced released every table when it stopped. The replacement then opened each table from nothing, and every write to that table waited for the open. Per table on staging this took about 5 s after the release. Adding the original's own ~1.5–2 s release gave a worst write of 7–9.5 s per upgrade.

Staging logs broke that cost into four parts:

• Loading the dataset.
• Reading the next epoch. next_writer_epoch probed 64 shard-manifest versions past the hint.
• Seeding the arrival number. highest_seq ran a max(_seq) scan over a cold table, about 2 s.
• Warming generations. This held the table's writer, so a write that arrived meanwhile waited behind it.

The upgrade's pre-swap flush also held each table's writer until its generation was written.

Change

Ownership and fencing are unchanged: the successor claims and writes nothing before a release.

• Prepared tables. Beside its sweep, a process that sees a live session of its own node (the one it is replacing) loads each table that session holds, read-only:
  • the dataset, its indexes, and every flushed generation;
  • it reloads every 10 s so late generations are warm too.
  • The claim that follows a release refreshes that dataset to its latest version (Prepared::refresh) and opens only the writer on it (Table::open_prepared).
  • Every claim now opens the dataset once, not twice.
• The release carries the next arrival number. OwnerRecord.next_seq is written by the releasing owner while it still holds the writer. A claim from a release starts from it (seq=handed in the open log), so there is no table scan. Only a claim from a release inherits it, and only once.
• Warming no longer holds the writer. Table::warmer captures what to open under the writer; Warmer::run opens it without the writer.
• The epoch read probes 2 versions past the hint, as the writer's own manifest reads do. The hint is written after every manifest write, so 64 parallel HEADs found nothing more.
• Flushing no longer holds the writer. Engine::flush (/flush_lsm, which an upgrade calls on every table first) freezes the memtable under the writer and waits for the flush without it (Table::seal).
• Diagnostics:
  • each release logs flush_ms and elapsed_ms;
  • each claim logs load_ms, epoch_ms, record_ms, open_ms and prepared=;
  • a routing step that holds a request over 1 s logs which step it was and what it answered (walleye.route slow).

I also tried releasing one key at a time to the successor (the local-only commit from the previous investigation). With prepared tables it was worse: the slowest takeover was 2.4 s against 1.6 s concurrent, in the same local setup. It is not in this PR.

Tests

• tests/successor.rs: an original and its replacement (same node id).
  • Half the writers go through the replacement; half go to the original until it stops answering.
  • Each table is timed from its release, as the bucket shows it, to the replacement's first acknowledged write. Every table must move once and read back exactly once.
  • Local disk (CI): slowest takeover 0.6 s.
  • MinIO behind a 5 ms one-way relay (--ignored, needs minio): 0.9–1.3 s in a debug build. Before this change it was 1.4–1.9 s, and the 64-HEAD probe alone was 0.5–1.1 s at a 30 ms round trip.
  • The 10 ms round trip is close to staging, where a Tigris LIST takes 8–19 ms.
  • A release build takes 0.5–0.7 s from claim to open writer.
• ownership::a_successor_knows_what_its_predecessor_holds_and_where_it_stopped: the predecessor's keys are seen, the handed number is inherited once, and a claim from anything but a release inherits nothing.
• cargo fmt --check and clippy (-D warnings) are clean. walleye-lance and walleye-node tests pass: 36 test binaries.

Staging

• Setup: three chains of 10 counted ramp upgrades each, alternating two images built from the same engine source. Each image was built by lakeday #218's guarded build from Serve only what the edge forwarded when WALLEYE_EDGE_KEY is set #90 + Embed the digest of the source tree the binary was compiled from #96 + this branch.
• Digests: every node's GET /internal/version digest was checked before its upgrade counted, and all 30 matched.
• Writers: six, three through the instance router and three direct to Fly.

chain	engine	worst write per counted upgrade (s)	over 5 s
before this PR	#98	8.9, 7.1, 8.4, 7.6, 8.4, 8.5, 7.5, 8.4, 9.6, 25.4	10 of 10
1	prepared, handed seq, unheld warm, 2-version probe	3.4, 2.5, 4.8, 3.1, 3.5, 5.4, 4.3, 3.6, 3.0, 8.5	2
2	+ unheld flush, release timing	3.5, 3.5, 2.4, 3.5, 3.1, 9.4, 2.7, 3.0, 3.1, 3.4	1
3	+ slow-route logging	3.2, 3.3, 4.0, 7.7, 3.5, 4.5, 3.4, 2.8, 3.3, 2.5	1

• Reads and hangs: every chain read back exactly once: no row missing, none twice, and nothing present that was not acknowledged. No write hit the 30 s proxy hang in any chain.
• Successor side: from claim to writer it takes 0.7–1.0 s, and each claim logs prepared=true.

The outliers. Each one coincides with object-store latency, not with the successor's open:

• Chain 1 upgrade 6: the pre-swap flush held writers under a slow period, with engine-side writes at 3.5 s. The unheld flush in chain 2 removes it; no stall at an upgrade's start since.
• Chain 1 upgrade 10: one table's release checkpoint took 6 s. At the same moment a manifest read on the successor took 2.8 s.
• Chain 2 upgrade 6: the original's release checkpoints took 0.8–2.4 s. Writes for three tables that the successor had forwarded to the original came back only when the original exited. The route logging was added to see this.
• Chain 3 upgrade 4: steady writes already took 1.1–1.5 s just before the stop. After that, each step on the handover path was slow:
  1. release checkpoint: 0.8–2.7 s, with forwarded writes waiting 3–3.6 s on it;
  2. claim and open on the successor: 1.4–2.7 s;
  3. first write on the new writer: 1.2–2.5 s.

What is left is round trips in three places:

• the original's release checkpoint, which holds the writer;
• the MemWAL writer open: claim epoch, fence sentinel, replay;
• the first append.

With normal object-store latency the whole handover stays under 5 s. Under a latency spike it does not.