[jbrown9513]

On staging, test:ownership refused writes during a graceful restart and right after a roll, with 503 owner-unreachable ... does not accept connections and its lease has not lapsed yet and occasionally 409 stale-owner. After this PR neither refuses anything. Based on staging/consolidated (6efe7e7).

Causes

1. A starting node joined the cluster before it could answer. Service::open → Engine::open published the process's lease and started the ownership sweep. Only after that did main bind the listener.
   • From the moment peers sampled that lease, they placed tables on the new node, handed tables back to it and forwarded requests to its address. Nothing was listening yet.
   • The forward's connects failed, so it answered owner-unreachable, which promises nothing was applied.
   • A graceful restart and every node of a roll go through this start. That is the staging symptom.
2. A request that found its table moving gave up after one retry. route_to_owner re-read ownership once on stale-owner, no-owner, or a local refusal from a process that had just lost the table. During a hand-back followed at once by a hand-over, the second attempt could still find the table moving, and it answered 409 stale-owner to the client.

Fix

• Join only once listening. Service::open is now Service::prepare followed by Service::start:
  • prepare opens storage, the engine and the cache. Nothing is visible to peers yet.
  • start publishes the lease (Engine::start), then starts the renew/sample loops, the ownership sweep, alarms, views, readiness and the other background tasks.
  • main calls prepare, binds the listener, builds the server, then calls start. A peer that connects in between is queued in the backlog, not refused.
  • Service::open remains as prepare + start, for hosts and tests that listen beforehand or never do.
• Patience for a moving table.
  • When an answer says nothing was done because the table was not where it was looked for (stale-owner, no-owner, or a local refusal), the request is sent again, re-reading ownership each time, for up to 8 s, 250 ms apart, before the caller sees that answer. This is safe because nothing was delivered.
  • owner-unreachable is not retried with that patience. The forward already retries a refused connect at once, and an owner that joins only once listening refuses connections only when it is gone. Its lease says so, and the existing quick 503 with Retry-After for a dead owner is kept (ownership.rs checks it).
• Stopping side unchanged, and now tested. A stopping node keeps answering and forwarding through its release plus the 3 s grace. The test harness's graceful stop now does the same as the binary.
• Docs and perch:
  • docs/self-hosting/cluster.md gets a paragraph on this.
  • New perch rule a-lease-names-an-address-that-answers.

Tests

New file crates/walleye-node/tests/graceful.rs. It runs a 3-node launch cluster:

• 20 ms archive latency and 2 ms between replicas;
• 6 tables with writers sending each write to the next live node, so most writes are forwarded;
• a node's lease-to-listening gap (LAUNCH_OPENING, 1.5 s) emulated in the harness, which now binds after opening, as the binary does.

Test	What it does	On 6efe7e7	On this branch
a_graceful_restart_refuses_nothing	node-1 restarted gracefully twice	fails: 503 owner-unreachable owner node-1… does not accept connections and its lease has not lapsed yet (the staging error)	passes
a_roll_refuses_nothing	all three nodes restarted in turn	fails with 7 refusals: the same owner-unreachable, plus 409 stale-owner	passes

Both require every write answered 200 and every acknowledged row present exactly once. On this branch both passed 5 times out of 5.

Gates

• cargo fmt --all -- --check: clean.
• cargo clippy --workspace --all-targets: no warnings in touched crates.
• cargo test -p walleye-node: 184 passed, 0 failed, 2 ignored. Run one test binary at a time because of disk space.

Deploy

• No configuration changes.
• A roll does not need the control plane to wait for tables=served after each node. The suite fix that skips an upgrade to the image an instance already runs is lakeday #223.

[jbrown9513]

Staging rerun on 0e5e59f (image sha256:84fbd43c…, source 998be0f7…, every node verified): failover 3/3 passed, but the restart leg refused 1 write: 503 owner-unreachable owner node-2.f975e4c0f52a does not accept connections ... (http://8e314df772d328.vm.<app>.internal:8080/...), 4 s after that node's new lease. The lease named the node by its .vm.<app>.internal hostname: with WALLEYE_MEMBERS set, the engine took its own endpoint from the member list and ignored WALLEYE_ADVERTISE_URL (which cell-boot sets to http://[FLY_PRIVATE_IP]:8080). A restarted Machine's hostname resolves for its peers only some time after it starts, so binding before publishing was not enough. 49b0023 makes WALLEYE_ADVERTISE_URL replace the node's own member entry, so the lease carries the 6PN address, reachable the moment the node listens. Unit test an_advertised_address_is_where_a_listed_node_answers; fmt/clippy clean; lib tests 71 passed.