Repository navigation
fix(cluster)!: let slow replicas catch up under load - #4658
Merged
Merged
Conversation
hubcio
force-pushed
the
fix/slow-replica-transfer
branch
from
October 27, 2026 07:59
f563662 to
1823eef
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #4658 +/- ##
============================================
+ Coverage 90.71% 90.91% +0.20%
+ Complexity 3857 3854 -3
============================================
Files 1548 1551 +3
Lines 386919 395228 +8309
Branches 321059 329369 +8310
============================================
+ Hits 350998 359341 +8343
+ Misses 28739 28661 -78
- Partials 7182 7226 +44
🚀 New features to boost your workflow:
|
hubcio
force-pushed
the
fix/slow-replica-transfer
branch
2 times, most recently
from
October 28, 2026 11:13
a8fc4be to
29297d4
Compare
hubcio
changed the title
hubcio
force-pushed
the
fix/slow-replica-transfer
branch
2 times, most recently
from
October 29, 2026 09:21
1f948d8 to
9120482
Compare
hubcio
changed the title
4 tasks
hubcio
force-pushed
the
fix/slow-replica-transfer
branch
2 times, most recently
from
October 30, 2026 07:39
625971f to
05b1522
Compare
numinnex
previously approved these changes
Oct 30, 2026
spetz
previously approved these changes
Oct 30, 2026
A slow replica could throttle healthy peers, churn views and repeat snapshots after losing their required repair tails. Repair windows could also fill the WAL before a durable prefix was applied. Fan out prepares independently and replay peers in bounded batches with fenced progress replies. Bound replay work per shard tick and charge timeout retries to the same peer window. Apply durable repair prefixes and wait for WAL headroom before further trusted fetches. Recheck busy peers promptly without releasing their batch credit and drain full replay windows before backing off unavailable history. Reuse verified segments and fetch missing suffixes. Move hashing, index reconstruction and spills outside the shard pump with owned cancellation and installation guards. Pin admitted snapshot tails under the shared configurable cap, charge full retained allocations, and keep recovering nodes out of new primary placements. Require PreVote quorum before changing views and step down when fresh heartbeat replies no longer establish quorum. Preserve acknowledged operations without a WAL and reject superseded owner installs before sequencing. BREAKING CHANGE: CommitReply and PrepareProgress require a coordinated server upgrade. The client protocol is unchanged.
The primary stops sending prepares to a backup when its send window fills or while a progress probe is out, and resumes only on the next 10 ms consensus tick. The default simulated network reorders packets, so gap repair hides this stall. Add an ordered network mode, off by default, that delivers each link in send order like a TCP link. Two tests send one produce per tick and require each answer within its four hops plus one probe round trip. Both fail here and pass on c92d2bd.
The window to each backup was stop-and-wait: sends paused while a progress probe was out and resumed on the next 10 ms tick. Both backup windows filled at the same op, so fault-free produce p99 rose from 2.25 to 11.34 ms. The flow now pipelines credit. A checkpoint probe follows the frame that fills half the window on the same FIFO link. Its reply credits only the bytes sent before it, so no send waits. Replies carry a new held_op, taken from reserved header bytes, so an unflushed persisted WAL no longer reads as a gap. A retransmit no longer starts a replay. Counters for refused and replayed prepares stay at zero while backups keep up, and the simulator tests now require that.
It runs for about 125 s locally and 127 to 131 s in CI, the only cluster test over 60 s. Run it with --run-ignored only.
A ready progress reply lowered the replay cursor but never raised it. Busy replies during a state transfer parked the cursor on an op that the primary later dropped. After the backup caught up, the replay from that op failed on every try, and the cursor refused every live prepare. Under load, the backup stayed on pull repair and never got its primaries back. If the backup holds the op at the cursor, a ready reply now moves the cursor to the op after the held prefix. A gap already moved it there. The seeded flow model now drops old ops and repairs the backup. A live replay also read up to 8 MiB of bodies before the window admitted the first frame. It now asks the window before each read. A prefix read that outlives its bound is now asked to stop and kept until it lands, so a transfer never holds two artifact buffers.
A lagging backup can hold hundreds of thousands of unapplied entries. On every tick, committed_headers_from looped over all of them to pick at most 64 ops. One shard then spent most of its CPU in this walk and applied entries slower than new prepares arrived. Its journal grew without limit, and the replica never caught up. Look up each op in the first-occurrence index instead. A walk now costs one lookup per returned header.
A lagging backup holds hundreds of thousands of journal entries and flushes after almost every commit walk. Each eviction drained the journal and appended every retained entry again, which decoded each batch and rebuilt the op, offset and timestamp indexes. A shard with three catching-up partitions stayed at 100% CPU and the backup did not catch up while producers kept writing. The entries and headers now live in deques with a front offset, so an eviction pops the prefix and removes its index keys in O(evicted). The journal keeps running message and byte totals for journal.info. A journal out of op order, or a prefix op with a copy in the retained tail, still rebuilds the tail. Both shapes are rare.
The rebase onto #4687 kept a second shutdown of the fast consumer after master already shuts it down and drops it, so the integration tests did not compile.
A serving shard read released prepare bodies and transfer chunks inline on its message pump. Each read that went to disk stalled the frames, ticks and replies of every group on that shard. A replica that catches up after a long outage can need gigabytes of such reads from its peers. The pump now decides what to serve, and a detached task reads the bytes. When the bytes land, the pump serves the request again against the current state. Reads run one at a time per shard, and the next starts only after the pump took the last result.
hubcio
force-pushed
the
fix/slow-replica-transfer
branch
from
October 31, 2026 07:43
cf88620 to
f608e66
Compare
The callout cap test held its slow callouts for 30 s, which is also the operation deadline of the overflow login. When a late retry found a free slot, it reached the mock service and the test counted 3 requests, not 2. Hold the callouts for twice that deadline. The Swift backlog test allowed four reads for the message on the other partition, but TLS runs needed up to seven. Each fetch in flight takes a data connection of its own, so the backlog can answer while the other fetch opens one. Read until the message arrives, and fail if half of the backlog came first.
spetz
approved these changes
Oct 31, 2026
mmodzelewski
approved these changes
Oct 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A slow replica could throttle healthy peers, churn views and repeat
snapshots after losing their required repair tails. Repair windows
could also fill the WAL before a durable prefix was applied. Builds on
the regressions in #4653.
Fan out prepares independently so healthy replicas can keep
progressing. Replay peers in bounded batches with fenced progress
replies. Bound replay work per shard tick and charge timeout retries
to the same peer window. Apply durable repair prefixes and wait for
WAL headroom before further fetches. Recheck busy peers without
releasing their batch credit and drain full replay windows before
backing off unavailable history.
Reuse verified segments and fetch missing suffixes. Move hashing,
index reconstruction and spills outside the shard pump with owned
cancellation and installation guards. Pin admitted snapshot tails
under the shared configurable cap and charge full retained
allocations. The cap defaults to 1 GiB and bounds extra retained
repair memory, not total process RSS. Keep recovering nodes out of new
primary placements.
Require PreVote quorum before changing views and step down when fresh
heartbeat replies no longer establish quorum. Preserve acknowledged
operations without a WAL and reject superseded owner installs before
sequencing.
Local validation passed 3,252 affected tests, 1,047 non-connector
integration tests and 46 doctests. Workspace build, Clippy,
formatting, manifest order, documentation and pre-commit checks
passed. The final rebase preserves the feature patch unchanged. Codex
implemented the changes and tests, ran validation and performed the
four-role review.
The pre-review AWS delay and restart test at 150,000 messages/s
recovered all 12 topics under continued load with zero publish errors.
Later review fixes have local validation and CI; that AWS run predates
them.
BREAKING CHANGE: CommitReply and PrepareProgress require a coordinated
server upgrade. The client protocol is unchanged.