Skip to content

fix: keep the track processor across mute and restartTrack - #1249

Merged
hiroshihorie merged 4 commits into
mainfrom
hiroshi/keep-processor-across-restart
Oct 26, 2026
Merged

hiroshihorie merged 4 commits into
mainfrom
hiroshi/keep-processor-across-restart

Conversation

@hiroshihorie

@hiroshihorie hiroshihorie commented Oct 21, 2026 •

Copy link
Copy Markdown
Member

Problem

A TrackProcessor set on a local track is silently dropped in two cases:

  • restartTrack(): it calls stop(), which destroys the processor and clears _processor, before it reads _processor to attach it to the new track. So the processor is always lost. setCameraPosition() and switchCamera() go through restartTrack(), so a camera switch drops a video processor.
  • Mute and unmute with the default options: stopAudioCaptureOnMute and stopCameraCaptureOnMute both default to true. Muting then calls stop(), and unmuting calls restartTrack(), so a microphone or camera processor is gone after the first mute and unmute. Apps that set these options to false only toggle enabled and are not affected.

Fix

Three commits:

  1. Keep the processor across mute and restartTrack(). mute() and restartTrack() now stop and dispose the media track without releasing the processor. stop() still releases it, also when a stop-on-mute already stopped the media track. Before, that case skipped stopProcessor(). stop() keeps calling super.stop() itself because Track.stop is @mustCallSuper, so the shared teardown takes its result as a parameter.
  2. Restart the processor instead of re-creating it. restartTrack() used to call destroy() and then init() on the same processor. That emitted a processor removed event followed by an added one, and lost what onPublish() had set up, since onPublish() is not called again. It now calls TrackProcessor.restart() with the new media track, as the JS SDK does. Krisp implements restart() on web, Android and iOS. If restart() throws, the processor stays attached and the new track is still used.
  3. Send the processed track after restartTrack(). The sender got the new raw track before the processor was attached to it, and setProcessedTrack() only swaps the local track. A processor with a processed track, such as Krisp on web, was then no longer applied to what was sent after an unmute or a camera switch. The processor is now restarted first, and the sender and the simulcast senders get mediaStreamTrack, which is the processed track when there is one.

On native platforms this re-binding is needed because flutter_webrtc creates a new MediaStreamTrack for every unmute with stop-on-mute and for every camera switch. Swift and Android keep the processor on the capturer, source or audio processing module, so their tracks are never replaced.

Testing

  • test/track/local_track_processor_test.dart:
    • mute and unmute, and restartTrack(): the processor stays, is restarted on the new track, and is not destroyed
    • a processor with a processed track: the sender gets processed-mic-2 after unmute and processed-mic-3 after restartTrack(). Against the previous revision it got mic-2 and mic-3.
    • a processor whose restart() throws: the sender still gets the new track
    • stop() after a mute releases the processor
  • flutter test: 471 passed. flutter analyze, dart format and import_sorter are clean.
  • macOS e2e through a local livekit-server (new e2e-flutter scenario, microphone and camera): a recording processor across publish, mute, unmute, restartTrack() and unpublish. It is initialised once, restarted on each new media track, destroyed only on unpublish, onPublish runs once, and the subscriber keeps receiving packets. Both the microphone and the camera variants pass. main fails the microphone variant with "processor dropped by mute". The camera variant was not run against main.
  • Desktop e2e on Linux arm64 in Docker, against the first commit: a screen share with a native video processor attached through a TrackProcessor (flutter_webrtc's desktop AddVideoProcessor, feat(desktop): let native plugins attach video processors to local tracks flutter-webrtc/flutter-webrtc#2233). After restartTrack() the processor moved to the new MediaStreamTrack and the subscriber kept decoding processed frames. Passed 3 runs in a row. Its test processor implements restart() as detach and attach, so it makes the same native calls with the later commits, but it was not re-run.

Found while working on CLTCU-14 (desktop video processors for Proton).

restartTrack called stop(), which destroys the processor and clears
_processor, before it read _processor to attach it to the new track, so
the processor was always dropped.

Mute went through stop() too. With the default stopAudioCaptureOnMute
and stopCameraCaptureOnMute (both true), muting stops the media track and
unmuting restarts it, so a microphone or camera processor was lost after
the first mute and unmute, and on every camera switch.

mute and restartTrack now stop the media track and keep the processor,
and restartTrack moves it to the new track as before. stop() still
releases it, also when a mute already stopped the media track.
@hiroshihorie
hiroshihorie marked this pull request as ready for review October 26, 2026 04:52
hiroshihorie added a commit that referenced this pull request Oct 26, 2026
)

## Why

flutter_webrtc 1.6.3 carries two things livekit_client needs:

- **Desktop video processors (CLTCU-14).** Windows and Linux plugins can
attach native video processors to local tracks
(flutter-webrtc/flutter-webrtc#2233), on par with Android, iOS and
macOS. 1.6.3 also fixes a Linux deadlock in
`desktopCapturer.getSources()` that hung every Linux screen share on
flutter-webrtc `main` (flutter-webrtc/flutter-webrtc#2244).
- **eMudhra CA compromise.** WebRTC-SDK 150.7871.04 and libwebrtc
m150.7871.07 no longer bundle `AAA Certificate Services`, which
cross-signs the compromised eMudhra intermediates, in the TURN/TLS
roots.

## What changed

Two commits:

1. `flutter_webrtc: 1.6.3`, and WebRTC-SDK `150.7871.04` in both
podspecs and `android/build.gradle`. The podspecs pin WebRTC-SDK exactly
like flutter_webrtc's own, so they have to move together. `pubspec.lock`
only changes the `flutter_webrtc` entry.
2. `livekit_ffi: ^0.13.2`. Its libwebrtc also drops the AAA root. The
SDK only calls `buildVersion()` today, so this changes no behavior.

Each commit has its own changeset.

## Verification

- `flutter analyze`, `flutter test` (466 passed) and `check_version`
pass locally.
- Linux arm64 e2e in Docker through a local livekit-server, using
flutter_webrtc with these fixes plus #1249. A screen share with a native
video processor attached through a `TrackProcessor` reaches the
subscriber processed, on both simulcast layers and across
`restartTrack()`. That run used the flutter-webrtc commits before the
release tag, not the published 1.6.3.

Not verified yet:

- iOS, Android and macOS example builds with WebRTC-SDK 150.7871.04. CI
covers the builds.
- Windows at runtime.
- That the shipped binaries no longer contain `AAA Certificate
Services`, which is checked with `grep` on the resolved WebRTC binaries.

## Related

- #1249 keeps a `TrackProcessor` across
`restartTrack()` and default mute and unmute. Without it, a desktop
processor is lost on every camera switch.
restartTrack() destroyed the processor and then called init() on the
same instance for the new media track. That emitted a processor removed
event followed by an added one, and lost what onPublish() had set up,
since onPublish() is not called again.

TrackProcessor already has restart() for this. restartTrack() now calls
it with the new media track and keeps the processor attached. A
processor whose restart() throws stays attached and the new track is
still used.
restartTrack() replaced the sender track with the new raw track before
the processor was attached to it. setProcessedTrack() only swaps the
local track, so a processor with a processed track, such as Krisp on
web, was no longer applied to what the sender carried after an unmute
or a camera switch.

The processor is now restarted on the new track first, and the sender
and the simulcast senders get mediaStreamTrack, which is the processed
track when the processor has one.
@hiroshihorie
hiroshihorie merged commit 75f77ba into main Oct 26, 2026
15 checks passed
@hiroshihorie
hiroshihorie deleted the hiroshi/keep-processor-across-restart branch October 26, 2026 18:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants