Repository navigation
test(partitions): expose bugs when a slow replica rejoins the cluster - #4653
Merged
Merged
Conversation
A replica that lags or runs slow can disturb its partition after the fault ends. A state transfer it installs can leave the partition in an endless view change. Add a test for that failure and for two related ones. Each test is ignored as a known bug, so CI stays green until the fixes land. After a replica adopts a `StartView` and then installs a transfer below the view head, the install rewinds the sequencer but keeps the view. Its next `DoViewChange` carries the transfer op in the header but starts the suffix at the old head, so peers drop it as unreadable. If the transfer is more than 128 ops below the head, the vote also leaves uncommitted ops outside its window. Two shard unit tests replay the consensus steps of the install, build the vote, and decode it as a peer does. Under sustained writes, a backup that fell past the evicted ring never installs a transfer. The first request pins the offer at the primary's commit point, and the pull runs one retry interval later. By then the backup journaled a newer op and refuses the offer as stale. An integration spec cuts node 2 off, writes one message per second after the heal, and waits for an install. A replica whose messages arrive later than the view-change status timeout keeps the two healthy replicas leaving their working view. A simulator test shows this with latency alone. For this, the packet simulator gains a directed extra delay per link. The delay does not draw from the PRNG, so a run with zero extra delay replays the same trace.
thepatrykk
approved these changes
Oct 26, 2026
Codecov Report❌ Patch coverage is
❌ Your patch check has failed because the patch coverage (22.22%) is below the target coverage (50.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## master #4653 +/- ##
=============================================
- Coverage 90.77% 81.09% -9.68%
Complexity 2989 2989
=============================================
Files 1459 1452 -7
Lines 348165 241903 -106262
Branches 292747 186485 -106262
=============================================
- Hits 316042 196176 -119866
- Misses 25485 39323 +13838
+ Partials 6638 6404 -234
🚀 New features to boost your workflow:
|
hubcio
approved these changes
Oct 26, 2026
hubcio
added a commit
that referenced
this pull request
Oct 27, 2026
A replica that fell behind and rejoined through state transfer hit three bugs. The tests that #4653 added as ignored now run. A transfer installed below the head of an adopted StartView rewound the sequencer to the transfer op. Peers then dropped its DoViewChange as unreadable, and a transfer past the 128-entry suffix window left a gap between the vote's commit point and its suffix. The install now raises the commit floor and never lowers the head, like the metadata install and a restart. It also counts its journal wipe and checkpoint restore as journal mutations, so the next vote rebuilds its cached suffix. Under steady writes, journal repair fetched committed ops past the op the offer pinned. The stale guard protected every held op up to the higher of the WAL head and commit_max, so it refused every offer. It now protects held ops only up to the accepted WAL head, or commit_max without a WAL: every partition ack waits for that head, so this replica never acknowledged an op above it. Such a refusal now has its own error. A replica in a view change moved to the next view on each status timeout. When its messages took longer than that to arrive, each StartViewChange that landed pulled the healthy pair out of the view they had just settled in. It now holds its view while it lacks the votes for its DoViewChange and an older view's primary still sends it heartbeats. Without such heartbeats it still escalates, because Recovering peers never send a StartViewChange. A newer view ends the hold, because the cluster already elected past it. Each hold logs at INFO.
hubcio
added a commit
that referenced
this pull request
Oct 27, 2026
A replica that fell behind and rejoined through state transfer hit three bugs. The tests that #4653 added as ignored now run. A transfer installed below the head of an adopted StartView rewound the sequencer to the transfer op. Peers then dropped its DoViewChange as unreadable, and a transfer past the 128-entry suffix window left a gap between the vote's commit point and its suffix. The install now raises the commit floor and never lowers the head, like the metadata install and a restart. It also counts its journal wipe and checkpoint restore as journal mutations, so the next vote rebuilds its cached suffix. A restart before repair still lost that head. The install and a fenced view window both certify the adopted view for the prefix that the WAL holds. Recovery then counted the view as proven and skipped the adopted head. The replica voted for the view with its head at the install op. That can stall a view change, and with persisted durability it can cut a committed op. A restart now also resumes the adopted view when the recovered head is below the adopted head. Under steady writes, journal repair fetched committed ops past the op the offer pinned. The stale guard protected every held op up to the higher of the WAL head and commit_max, so it refused every offer. It now protects held ops only up to the accepted WAL head, or commit_max without a WAL: every partition ack waits for that head, so this replica never acknowledged an op above it. Such a refusal now has its own error. A replica in a view change moved to the next view on each status timeout. When its messages took longer than that to arrive, each StartViewChange that landed pulled the healthy pair out of the view they had just settled in. It now holds its view while it lacks the votes for its DoViewChange and an older view's primary still sends it heartbeats. Without such heartbeats it still escalates, because Recovering peers never send a StartViewChange. A newer view ends the hold, because the cluster already elected past it. Each hold logs at INFO. Metadata consensus runs the same code, so the hold covers its view changes too.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A replica that lags or runs slow can disturb its partition after the
fault ends. A state transfer it installs can leave the partition in an
endless view change. Add a test for that failure and for two related
ones. Each test is ignored as a known bug, so CI stays green until the
fixes land.
After a replica adopts a
StartViewand then installs a transferbelow the view head, the install rewinds the sequencer but keeps the
view. Its next
DoViewChangecarries the transfer op in the headerbut starts the suffix at the old head, so peers drop it as
unreadable. If the transfer is more than 128 ops below the head, the
vote also leaves uncommitted ops outside its window. Two shard unit
tests replay the consensus steps of the install, build the vote, and
decode it as a peer does.
Under sustained writes, a backup that fell past the evicted ring
never installs a transfer. The first request pins the offer at the
primary's commit point, and the pull runs one retry interval later.
By then the backup journaled a newer op and refuses the offer as
stale. An integration spec cuts node 2 off, writes one message per
second after the heal, and waits for an install.
A replica whose messages arrive later than the view-change status
timeout keeps the two healthy replicas leaving their working view. A
simulator test shows this with latency alone. For this, the packet
simulator gains a directed extra delay per link. The delay does not
draw from the PRNG, so a run with zero extra delay replays the same
trace.