Skip to content

Fix waiting for the data channel send buffer to drain - #1267

Draft
hiroshihorie wants to merge 3 commits into
hiroshi/close-engine-on-terminal-pathsfrom
hiroshi/fix-buffer-status-wait
Draft

hiroshihorie wants to merge 3 commits into
hiroshi/close-engine-on-terminal-pathsfrom
hiroshi/fix-buffer-status-wait

Conversation

@hiroshihorie

@hiroshihorie hiroshihorie commented Oct 29, 2026 •

Copy link
Copy Markdown
Member

Stacked on #1268. Review only the last commit, bb1c5b8. The merge commit before it brings in #1268.

Engine.waitForBufferStatusLow is the backpressure wait for data stream writes. It had three bugs:

  1. Could hang after a full reconnect. It polled only a cached "buffer is low" flag that a full reconnect never reset, so a waiter on a full buffer could keep waiting on the replaced channel. The flag is now reset when the channels are created, and the wait also checks the live channel. Flagged by Devin on Apply backpressure to native data stream sends #1266. Didn't reproduce on macOS (see E2E), but the fix no longer relies on that.
  2. Resolved normally when the engine closed. The caller then went on to send on a closing engine. It now fails with 'Engine disconnected'.
  3. Leaked a listener per blocking wait. The EngineClosingEvent listener was never cancelled. The wait no longer uses a listener at all.

Bugs 2 and 3 are fixed by checking the engine's closed state in the wait loop instead of listening for EngineClosingEvent. That only works because #1268 marks the engine closed synchronously on every terminal path, including room disposal and a reconnect that gives up. Hence the stack.

This matters more with #1166, where every native data stream packet goes through this wait.

Tests

test/core/buffer_status_wait_test.dart has one test per bug. All three fail on #1268 and on main, and pass here. MockDataChannel gains fakeBufferedAmount, identical to #1266's, so the two merge cleanly. analyze, format, import_sorter, version check and flutter test all pass.

E2E (macOS, local livekit-server --dev)

A local server drains the channel faster than the 2 MB low-water mark, so these runs used test-only builds with the mark lowered to 32 KB (not in this PR):

main this PR
Full reconnect while the buffer is past the mark (3 runs) the in-flight write fails cleanly, no hang same, and a send after the reconnect goes out
Engine listeners after 16 MiB of backpressured writes +4 +0

On macOS the old channel's low-buffer event during teardown releases the waiter, so bug 1 doesn't show there. Other platforms are unchecked.

Merge order

#1264, then #1268, then this PR. Once #1268 merges, this PR retargets to main and I'll merge main back in.

waitForBufferStatusLow had three problems:

- It polled only the cached buffer status, which changes on a send or on
  the channel's low-buffer event. When a full reconnect replaced a backed
  up channel, the new one fired neither, so a waiter could hang until
  disconnect. The cache is now reset when the channels are created, and
  the wait also checks the live channel.
- When the engine closed during a wait, the wait resolved normally and
  the caller went on to send. It now fails.
- Every wait that blocked left an EngineClosingEvent listener behind
  until the engine closed. The listener is now cancelled.

MockDataChannel gains fakeBufferedAmount to simulate a channel that has
not drained.
…-paths' into hiroshi/fix-buffer-status-wait

# Conflicts:
#	lib/src/core/engine.dart
Every terminal path now marks the engine closed synchronously, so the wait
can check that state directly. This drops the listener the wait had to
remove again, and a wait started after closure fails the same way.
@hiroshihorie
hiroshihorie changed the base branch from main to hiroshi/close-engine-on-terminal-paths October 29, 2026 20:11

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant