Fix: faster failover after an unplanned leader loss

[VerifiedOrganic]

Draft. The latest commit, 38c448d71f1fd32e762b57026a76109098691e30, repairs only the t25 election-retry fixture. Production code is unchanged by that commit. Full remote qualification is running before independent review and merge.

Faster and less disruptive failover after an unplanned leader loss, compatible with a rolling upgrade from a release without Pre-Vote. Each change is preceded by a test that fails on the previous commit.

1. Overlap the follower lease with the election timeout

A follower with a committed vote campaigned only after its leader lease (election_timeout_max) and then a full sampled election timeout. Both windows start at the last leader contact. An unplanned leader loss therefore went undetected for at least election_timeout_max + election_timeout_min, plus one tick.

• A follower now campaigns after the longer of the two.
• The follower lease is election_timeout_min, the minimum election timeout of the Raft dissertation (section 4.2.3). Every sampled timeout covers it, so campaigns stay randomized. While the lease is valid, a follower still rejects every other candidate.
• RED (elect/t20): with elections in [1500, 1600) ms and a 50 ms heartbeat, the first campaign started 3.18 s after the loss against a 2.275 s bound. It now starts after about 1.6 s.

2. Give AppendEntries its own deadline

Every AppendEntries, including heartbeats and the heartbeats that confirm leadership for a linearizable read, was bounded by heartbeat_interval. A short heartbeat therefore forced a short deadline on every replication RPC.

• New Config::append_entries_timeout: Option<u64>. Unset, it remains heartbeat_interval.
• RED (append_entries/t62): with a 50 ms heartbeat and a 1,000 ms deadline, every AppendEntries carried 50 ms. A follower that held one AppendEntries for 300 ms received it four times. It now receives it once.

3. Pre-Vote, and the leader's quorum-acknowledged lease

A voter that is cut off from the others campaigned on every election timeout and persisted each new term. When it could be reached again, or after a restart, the leader saw the higher term in an AppendEntries response and stepped down, although a quorum had served it throughout.

• New Config::enable_pre_vote: Option<bool> (unset means disabled), RaftNetwork::pre_vote, PreVoteReply and Raft::pre_vote. The default pre_vote reports the target as unable to answer without contacting it, so a network without Pre-Vote runs the classic election (section 5). An error, including an unreachable peer, is never a grant.
• Answering a Pre-Vote persists nothing and changes no term, lease or server state. A quorum of grants starts the real election. A rejection with a greater log delays the next attempt, and one with a strictly higher vote catches the voter up to it in non-committed form.
• Any vote the voter accepts afterwards, including a heartbeat, ends an in-flight round. The tick-driven re-evaluation of the server state does not.
• A leader never renews the lease on its own vote. It now rejects Vote and Pre-Vote while a quorum acknowledged it within the lease. A planned leadership transfer releases that lease, as before.
• RED (elect/t21): a voter cut off for four election timeouts in a 3- or 5-voter cluster went from term 1 to term 5. After a reconnect or a restart from its stores, the leader was deposed. It now keeps term 1, and the leader keeps leading.
• RED (elect/t22): a follower that can send but not receive got the leader's grant, and the leader stepped down after about 0.6 s. The leader now keeps leading.

The design follows the upstream Openraft Pre-Vote work on main, adapted to this branch: the Pre-Vote round, adopting a higher vote from a rejection, cancelling a round once its election is won, and gating votes by the quorum-acknowledged lease.

4. Require the minimum election timeout to outlast the heartbeat tick

A leader sends heartbeats on its engine tick, every three halves of heartbeat_interval. With rule 1, a minimum election timeout at or below that tick lets every follower campaign between two heartbeats of a healthy leader. Config::validate now rejects that configuration with the existing ElectionTimeoutLTHeartBeat error, whose message names the tick.

• Evidence: hosted CI ran the raft-kv-rocksdb example cluster test with a 250 ms heartbeat and [299, 300) ms elections, and it failed when a write reached a follower with no known leader. The example now uses [800, 1200) ms. Two test configurations that never elect are adjusted to stay valid.
• Two example chores make the single-threaded example pass current nightly clippy (items_after_test_module) and its own rustfmt check.

5. Use Pre-Vote only while every reachable voter can answer it

During a rolling upgrade some voters run a release without Pre-Vote, and some networks do not implement it. Such a voter can grant a vote but never a Pre-Vote. Counting it as rejecting could deadlock an election: in one case the survivors with the up-to-date log needed its grant while its own log was behind; in another the voter still believed it led, because nothing it sent arrived, and never campaigned.

• RaftNetwork::pre_vote returns PreVoteReply. A network sends the request and returns Answered only for a target positively known to answer Pre-Vote: the capability was negotiated on the connection, or the peer is an in-process peer that passes it to Raft::pre_vote. For any other reachable target it sends nothing and returns Unsupported, and the voter runs the classic election for that campaign at once. An unreachable target is an error: it grants neither request, so it neither counts as a grant nor forces the classic election.
• The default pre_vote returns Unsupported, so a network that does not implement it always runs the classic election.
• Each campaign decides again, so a voter set without such voters uses Pre-Vote again.
• A Pre-Vote response never makes the requester adopt a vote for itself. No voter holds such a vote for a term the requester never campaigned in.
• RED (elect/t24): a voter that cannot answer Pre-Vote leads, is cut off, and the others elect a successor, which is then lost too. The old leader is reachable again but nothing it sends arrives, so it never learns of the later term. The remaining voter was never elected within ten election timeouts. It is now elected with the old leader's classic vote.
• elect/t23: voters that cannot answer Pre-Vote and miss the last entries no longer block an election with 3 or 5 voters.
• An earlier revision of this PR added a stale-candidate rule for the second case. With this gating, elect/t23 passes without it, so it is removed and the vote rules are again those of the base release.

6. Tell Pre-Vote rounds apart by an identifier

Rounds in one term propose the same hypothetical vote, and a response was matched to the in-flight round by that vote. A grant delayed in the notification queue could reach a later round in the same term. If its sender had been removed from the membership in between, counting it panicked the Raft core.

• Each round carries an identifier through Command::SendPreVote and the response notification, and the engine ignores a response to any round but the one in flight.
• A grant from a node outside a round's quorum set is ignored instead of panicking.
• RED (engine unit test): a same-term AppendEntries removes a voter and ends a round, a new round starts with the same vote, and the removed voter's grant for the first round panicked with target not in quorum set. It is now ignored.

7. Retry a rejected Pre-Vote round after the election-timeout window

A round that had not reached a quorum was held for a whole sampled election timeout. After an unplanned leader loss, a survivor whose last leader contact came shortly before another survivor's meets that survivor's running lease in its first round, and the retry came up to about twice the maximum election timeout after the loss.

• A round rejected by a running lease is retried after the width of the election-timeout window, election_timeout_max - election_timeout_min, on a later tick. A round awaiting replies remains open until its Vote deadline so a slow answer is still considered.
• The first-successful-campaign bound of election_timeout_max plus one tick assumes surviving voters answer Pre-Vote within that retry window and ticks run on time. Slower replies can extend election time.
• RED (elect/t25): with elections in [1000, 1150) ms, a 20 ms heartbeat, an up-to-date voter that stopped hearing from the leader 400 ms before a lagging one, the up-to-date voter was elected 1.68 to 1.87 s after the loss against a 1.18 s bound. It is now elected after 1.0 to 1.1 s.

8. Give up an in-flight AppendEntries when replication closes

A leader that grants a vote stops leading and joins every replication stream before it answers, so no stream still reads a log range that a later truncation removes. A stream waiting for an AppendEntries answer did not notice that it was closed. With the deadline of section 2 longer than the candidate's vote deadline, a learner that never answers blocked the handoff and the Raft core for that long.

• Closing a stream drops a watch sender that an in-flight AppendEntries races, and the request is given up at once. The entries are read before the request is sent, and a snapshot child is still cancelled and joined, so every storage operation a stream owns is still awaited.
• RED (append_entries/t63): with a 10 s AppendEntries deadline and a learner that never answers, the vote that deposes the leader was answered after 9.04 s. It is now answered in milliseconds. A replication unit test covers the same close.

t25 fixture repair

The delayed-network job exposed a fixture race: a request admitted before the old leader was isolated could finish its send delay after the append quota was restored, catch up the lagging voter, and let that voter win legitimately. The test then waited for the wrong expected winner without reaching its timing assertion.

Keep the AppendEntries quota at zero through the election, elapsed-time assertion and term assertion, then restore it. The expected winner, contact skew, stale-log premise, 1,180 ms bound and 120 ms scheduling allowance remain intact. Vote and Pre-Vote requests do not consume the append quota.

Compatibility

• New Config fields with defaults that keep the old behavior, except the follower lease (rule 1). That affects every deployment: the first campaign after a leader loss comes up to election_timeout_max sooner.
• RaftNetwork::pre_vote is new in this PR; its default reports Unsupported, which keeps classic elections when Pre-Vote is enabled on a network that does not implement it.
• Rule 4 rejects configurations whose minimum election timeout does not exceed three halves of the heartbeat interval; they were accepted before.
• Two existing integration cases relied on the old ordering and now control automatic elections explicitly (append_entries/t61, elect/t11). Their assertions are unchanged.

Validation of 38c448d7

• Current-nightly t25 with OPENRAFT_NETWORK_SEND_DELAY=30, RUST_TEST_THREADS=2, debug logs and full backtraces: 1 passed. The trace retains the stale log, records rejected Pre-Votes, and elects node 1 after 1.159 s, within the unchanged allowance.
• Pinned-toolchain cargo fmt --all -- --check and focused cargo clippy --no-deps --manifest-path tests/Cargo.toml --test elect -- -D warnings pass. Local current-nightly formatting could not run because that toolchain lacks rustfmt; hosted lint remains required.
• Full push and pull-request CI, including openraft-test (nightly, 30), must pass on this head. Earlier green results at 47ba29a6 are historical and do not qualify this candidate.