Skip to content

Add proportional jitter to the reconnect delays - #1255

Merged
hiroshihorie merged 6 commits into
mainfrom
port/js-2124-reconnect-jitter
Oct 27, 2026
Merged

hiroshihorie merged 6 commits into
mainfrom
port/js-2124-reconnect-jitter

Conversation

@xianshijing-lk

Copy link
Copy Markdown
Contributor

Port of livekit/client-sdk-js#2124.

The bug

Engine.reconnect computed its delay as:

var delay = defaultRetryDelaysInMs[_reconnectAttempts];
// Add random jitter to prevent thundering herd on reconnect
if (_reconnectAttempts > 1) {
  delay += math.Random().nextInt(1000);
}

Two problems, the same pair the JS SDK had before #2124:

  1. The first two attempts get no jitter at all. defaultRetryDelaysInMs[0] is 0, so every client disconnected by the same event reconnects at the same instant, then again in lockstep 300ms later. The herd only begins to spread on the third attempt — by which point the server has already taken two synchronised waves.
  2. The jitter that is applied is additive, not proportional. A flat 0–1000ms window swamps a 300ms delay (up to 4.3x) and barely perturbs a 7000ms one (14%).

Repro

Added test/core/reconnect_delay_test.dart. Against the current logic (first commit extracts the delay into reconnectDelayInMs with behaviour unchanged, so the test exercises the real code):

reconnect delay spreads out clients that disconnect at the same moment FAILED
  Expected: a value greater than <1>  Actual: <1>
reconnect delay later retries also vary between callers FAILED
  Expected: a value greater than <1>  Actual: <1>
reconnect delay scales jitter with the base delay FAILED
  Expected: a value less than or equal to <1800>  Actual: <1831>
  retry 2 exceeded 1.5x of 1200

100 callers asking for their first — and second — retry delay all got the identical value.

The fix

Scale each delay by a random factor in [0.5, 1.5). The first retry has a base delay of 0 with nothing to scale, so it draws from a fixed 0–500ms window instead; a zero at any later index is an explicit request to retry immediately and still passes through as zero.

Verification

flutter test — 471/471 pass (the three assertions above fail before the change). flutter analyze lib/src/core/engine.dart clean.

Note

reconnectDelayInMs is marked @internal; defaultRetryDelaysInMs was already public and is unchanged.

🤖 Generated with Claude Code

The first entry of defaultRetryDelaysInMs is 0 and jitter only applied
from the third attempt onwards, so every client disconnected by the same
event reconnected at the same instant, then again in lockstep 300ms
later. Only from the third attempt did they start to spread.

The jitter that was applied was also additive (a flat 0-1000ms), which
is disproportionate at both ends: it swamps a 300ms delay and barely
moves a 7000ms one.

Extract the delay into reconnectDelayInMs and scale each delay by a
random factor in [0.5, 1.5). The first retry has no base delay to scale,
so it draws from a fixed 0-500ms window; a zero at any later index still
passes through as an immediate retry.

Port of livekit/client-sdk-js#2124.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
devin-ai-integration[bot]

This comment was marked as resolved.

Comment thread lib/src/core/engine.dart
xianshijing-lk and others added 3 commits October 23, 2026 16:52
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The delay is rounded to whole milliseconds, so a draw near the top of
the window rounds up to the bound itself and the exclusive assertion
failed roughly one run in a thousand. Assert the inclusive bound, and
raise the iteration count so the edge is actually exercised.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
500ms is longer than the 300ms that follows it in this schedule, which
delays recovery from a brief blip for no added benefit. The window only
has to pull a herd of clients apart, so 150ms is enough.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@xianshijing-lk

Copy link
Copy Markdown
Contributor Author

Agreed — 500ms is longer than the 300ms that immediately follows it in this schedule, so it was slowing down the common case (a brief blip) without buying anything. The window only needs to be wide enough to pull a herd of clients apart, not to back off.

Dropped it to 150ms in 0e1a0f5. The first retry still fires fast, and 150ms of spread is plenty to decorrelate clients hitting the SFU at once.

This is now a deliberate divergence from the JS SDK, which uses 500ms — worth noting in case you'd rather keep strict parity across SDKs. The constant is maxFirstRetryJitterInMs, and the test asserts against it rather than a literal, so changing it is a one-line move either way.

defaultRetryDelaysInMs is a fixed const with no policy hook, so the
"explicit immediate retry at a later index" case carried over from the
JS SDK cannot happen here. Key the fixed window on the zero base delay
alone, which is the actual reason it is needed.
The other SDK's constant can change independently and the comment would
go stale. The comparison belongs in the PR discussion, not the source.
@hiroshihorie
hiroshihorie merged commit c13eb81 into main Oct 27, 2026
15 checks passed
@hiroshihorie
hiroshihorie deleted the port/js-2124-reconnect-jitter branch October 27, 2026 10:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants