feat: add encoded frame transforms and a media recorder - #300
Merged
Merged
Conversation
Encoded frame transforms (insertable streams): RTCRtpSender and RTCRtpReceiver get setTransform(RTCEncodedFrameTransformer), which sees every encoded frame between encoder and packetizer, or depacketizer and decoder, and may read, replace or drop its payload. The transform runs on a thread of its own per sender or receiver, never on a media thread, and frames reach Java without a copy unless the payload is read. Senders gain generateKeyFrame(), receivers requestKeyFrame(). One native transformer is installed per sender or receiver, found through a weak registry, so that a Java transform and native observers share it and a video sender is not restarted on every change. The native extension API gains encoded frame observers, through which the media module's new MediaRecorder writes the frames of senders and receivers into Matroska, WebM or MP4 files without re-encoding. FFmpeg is built with the matroska, webm and mp4 muxers, extract_extradata and the AV1 parser for it.
macOS prefers H.264 through VideoToolbox by default. A transform that changes the whole payload breaks H.264, whose packetizer splits frames at start codes in it, so nothing reached the receiver there; the recorder tests check for VP8 as well. VideoToolbox on CI runners also encodes far fewer frames, so the tests wait for fewer.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two features that build on each other:
Encoded frame transforms (insertable streams).
RTCRtpSenderandRTCRtpReceivergetsetTransform(RTCEncodedFrameTransformer), which sees every encoded frame between encoder and packetizer, or depacketizer and decoder, and may read, replace or drop its payload. Frames expose their metadata (RTCEncodedVideoFrame: key frame, size, RID, layers;RTCEncodedAudioFrame: sequence number, audio level, CSRCs). Senders gaingenerateKeyFrame(), receiversrequestKeyFrame().MediaRecorder(media module). Records senders and receivers into.mkv,.webmor.mp4without re-encoding: encoded frames go into the file as they are.Design
getData()is called, into a per-thread direct buffer reused across frames. A frame used after its transform returned, or from another thread, throwsIllegalStateException; nothing can reach a native frame that has moved on.webrtc_java_api.h) gains encoded frame observers (appended members, same version). The recorder copies frames on the WebRTC thread into a bounded queue and muxes on its own thread. It starts video at a key frame and requests one (throttled), writes the header once all tracks are described or after 3 s, maps RTP timestamps onto a common timeline, and uses fragmented MP4 so a cut-short file still plays. A codec the container cannot hold (e.g. VP8 in MP4) leaves that track out with a warning instead of failing the file.extract_extradataand the AV1 parser. The CI FFmpeg cache key already hashes the component list, so no workflow change is needed.Also in this PR
EncryptedRecordingExample: AES-GCM end-to-end encryption through transforms (codec header kept in the clear and authenticated), with the decrypted side recorded.ApiCheckhelper used by the player and the recorder.Testing
webrtc: full suite 184/184, including 9 new transform tests (encrypted round trip, metadata, stale-frame access, drop and clear, exceptions, audio, key frame requests, shared transforms). Clean under-Pjni-check(noFATAL ERROR in native method).webrtc-java-media: full suite 47/47, including 7 new recorder tests that read the recording back (codec, size, duration), cover the left-out-codec path, empty-file deletion and stopping from the listener.EncryptedRecordingExampleend to end: 808 frames encrypted/decrypted, 0 rejected, 10 s recording written.