diff --git a/README.md b/README.md index 8a3a0e1e..827525c1 100644 --- a/README.md +++ b/README.md @@ -17,6 +17,8 @@ The library provides a comprehensive set of Java classes that map to the WebRTC - **Media capabilities** - Audio and video capture from cameras and microphones - **Desktop capture** - Screen and application window sharing - **Media file playback** - Send video and audio files over a peer connection in place of a camera and microphone, with the optional FFmpeg-based `webrtc-java-media` module +- **Call recording** - Record what a peer connection sends or receives into MKV, WebM or MP4 files, without re-encoding, with the `webrtc-java-media` module +- **Encoded transforms** - Read, change or drop encoded audio and video frames on their way through a sender or receiver, e.g. for end-to-end encryption, like insertable streams in the browser - **Data channels** - Bidirectional peer-to-peer data exchange - **Statistics API** - Detailed metrics for monitoring connection quality - **Simple integration** - Available as a Maven dependency @@ -30,6 +32,8 @@ For more detailed information, check out the documentation: - [Guides](https://jrtc.dev/guide/) - Comprehensive documentation on using the library - [Examples](https://jrtc.dev/guide/examples) - Sample code demonstrating various features - [Media Files](https://jrtc.dev/guide/media/media-files) - Sending video and audio files with the media module +- [Media Recording](https://jrtc.dev/guide/media/media-recording) - Recording calls into media files with the media module +- [Encoded Transforms](https://jrtc.dev/guide/advanced/encoded-transforms) - Reading and changing encoded frames, e.g. for end-to-end encryption - [Build Notes](https://jrtc.dev/guide/build) - Instructions for building the library from source ## License diff --git a/docs/.vitepress/sidebar.ts b/docs/.vitepress/sidebar.ts index 07228b5c..55a77a74 100644 --- a/docs/.vitepress/sidebar.ts +++ b/docs/.vitepress/sidebar.ts @@ -23,6 +23,7 @@ function sidebarGuide(): DefaultTheme.SidebarItem[] { { text: 'Media Constraints', link: '/media/constraints' }, { text: 'Media Directionality', link: '/media/directionality' }, { text: 'Media Files', link: '/media/media-files' }, + { text: 'Media Recording', link: '/media/media-recording' }, ], }, { @@ -72,6 +73,7 @@ function sidebarGuide(): DefaultTheme.SidebarItem[] { collapsed: false, items: [ { text: 'Field Trials', link: '/advanced/field-trials' }, + { text: 'Encoded Transforms', link: '/advanced/encoded-transforms' }, ], }, ] diff --git a/docs/guide/advanced/encoded-transforms.md b/docs/guide/advanced/encoded-transforms.md new file mode 100644 index 00000000..507cae98 --- /dev/null +++ b/docs/guide/advanced/encoded-transforms.md @@ -0,0 +1,106 @@ +# Encoded Transforms + +This guide explains how to read and change the encoded frames of a call as they pass through an `RTCRtpSender` or an `RTCRtpReceiver`, the way [WebRTC Encoded Transforms](https://www.w3.org/TR/webrtc-encoded-transform/) (also known as insertable streams) do in a browser. It covers: + +- Setting a transform on a sender or a receiver +- Reading and replacing a frame's payload +- Dropping frames +- What each frame tells about itself +- Asking for key frames +- End-to-end encryption as a worked example +- Threading and performance + +A sender's transform sees every frame after the encoder and before the packetizer; a receiver's sees every frame after the depacketizer and before the decoder. Everything between the two, whether that is the network, a TURN server or an SFU, only ever carries what the transform produced. That is what makes end-to-end encryption possible, and also frame metadata, watermarks, or analysis of the encoded stream. + +## Setting a Transform + +An `RTCEncodedFrameTransformer` is a functional interface with a single method, `transform(RTCEncodedFrame frame)`. It changes the frame in place, and the frame is sent on when the method returns: + +```java +RTCRtpSender sender = peerConnection.addTrack(videoTrack, List.of("stream")); + +sender.setTransform(frame -> { + System.out.println(frame); // mime type, size, key frame, timestamp, SSRC +}); +``` + +On the receiving side, the receivers are there once the remote description is set: + +```java +for (RTCRtpReceiver receiver : peerConnection.getReceivers()) { + receiver.setTransform(frame -> inspect(frame)); +} +``` + +`setTransform(null)` removes the transform again, after which frames pass unchanged. + +::: tip +Set the transform of a video sender before negotiating when you can. The first transform set on a running video sender restarts its send stream, which costs a key frame. Later changes, including removing it, cost nothing. +::: + +The transform belongs to the native sender or receiver, not to the Java object: every `RTCRtpSender` instance for the same sender shares it, setting it through one replaces what was set through another, and disposing of an instance leaves it in place. + +## Reading and Changing the Payload + +`getData()` returns the payload in a `ByteBuffer`, from position zero to its size. The buffer holds a copy, which may be changed freely; `setData(...)` then writes it back: + +```java +receiver.setTransform(frame -> { + ByteBuffer data = frame.getData(); + + for (int i = 0; i < data.limit(); i++) { + data.put(i, (byte) (data.get(i) ^ 0x5A)); + } + + frame.setData(data); +}); +``` + +`setData` takes a `ByteBuffer` (its remaining bytes) or a `byte[]`, and the new payload may be larger or smaller than the old one. The payload is only copied into Java when `getData()` is called, so a transform that looks at the metadata alone costs no copy at all. + +A frame, and the buffer `getData()` returned, is valid only while `transform` runs and only on its thread. The buffer's memory is reused for the next frame, so copy what you need to keep. Reading or changing a frame after the transform returned throws an `IllegalStateException`; its metadata stays readable. + +## Dropping Frames + +`frame.drop()` releases the frame instead of sending it on. A transform that throws drops the frame as well, and the exception goes to the thread's uncaught exception handler. It is dropped rather than sent on unchanged because an encryption transform that failed must never send the frame in the clear. + +A receiver cannot decode a video frame that depends on one it never got, so dropping video frames usually means waiting for the next key frame. + +## Frame Metadata + +Every `RTCEncodedFrame` tells its `getMimeType()` (e.g. `video/VP8`, `audio/opus`), `getSize()`, RTP `getTimestamp()`, `getSsrc()`, `getPayloadType()` and `getCaptureTimeUs()`. The subclasses add what is particular to their kind: + +| `RTCEncodedVideoFrame` | `RTCEncodedAudioFrame` | +| --- | --- | +| `isKeyFrame()` | `getSequenceNumber()` (received frames) | +| `getWidth()`, `getHeight()` (key frames) | `getAudioLevel()` in -dBov | +| `getRid()` (simulcast layer) | `getContributingSources()` | +| `getFrameId()`, `getSpatialIndex()`, `getTemporalIndex()` | | + +## Key Frames + +A transform that needs a key frame, e.g. to start over after a key change, can ask for one: + +```java +videoSender.generateKeyFrame(); // the local encoder makes the next frame a key frame +videoReceiver.requestKeyFrame(); // asks the remote sender for one +``` + +Both do nothing for audio. + +## End-to-End Encryption + +A complete example with AES-GCM is in [`EncryptedRecordingExample`](https://github.com/devopvoid/webrtc-java/blob/master/webrtc-examples/src/main/java/dev/onvoid/webrtc/examples/EncryptedRecordingExample.java). Two points matter for any encrypting transform: + +- **Keep the codec header in the clear.** The packetizer and depacketizer read the start of each frame: a VP8 receiver finds key frames and the frame size in the first bytes. Browsers leave 10 bytes of a VP8 key frame, 3 of a VP8 delta frame and 1 of an Opus frame unencrypted, and authenticate them as associated data instead. H.264 is split at its NAL unit start codes before it is sent, so encrypting it whole breaks the packetizer; prefer VP8, VP9 or AV1 for encrypted calls, or leave the NAL unit headers in the clear. +- **Make each frame self-contained.** Frames are lost, so each needs what it takes to be decrypted, such as its IV, and a receiver must drop a frame that fails to authenticate rather than pass it on. + +## Threading and Performance + +The transform never runs on a thread that carries media. Each sender and receiver with a transform gets a thread of its own. Frames are queued to that thread and transformed there one at a time, in order, and are handed back to WebRTC from it. This means that: + +- a slow transform delays only its own frames, never the connection; +- a transform may call back into the peer connection, even close it, without deadlocking; +- a transform that falls several seconds behind has frames dropped until it catches up, rather than having them queue up in memory. + +Per frame, the cost is one small Java object and, only if the payload is read, one copy into a buffer that is reused from frame to frame. Senders and receivers without a transform are not touched at all. diff --git a/docs/guide/examples.md b/docs/guide/examples.md index a6636a84..5e9a4e9a 100644 --- a/docs/guide/examples.md +++ b/docs/guide/examples.md @@ -95,6 +95,24 @@ mvn exec:java -D"exec.mainClass=dev.onvoid.webrtc.examples.MediaFilePlayerExampl ``` ::: +## Encrypted Recording + +The [`EncryptedRecordingExample`](https://github.com/devopvoid/webrtc-java/blob/master/webrtc-examples/src/main/java/dev/onvoid/webrtc/examples/EncryptedRecordingExample.java) encrypts a call end to end with encoded frame transforms, and records what the receiving side decrypted into a media file. See the [Encoded Transforms](/guide/advanced/encoded-transforms) and [Media Recording](/guide/media/media-recording) guides for the APIs it uses. + +**Key features demonstrated:** +- Encrypting every encoded audio and video frame with AES-GCM in an `RTCEncodedFrameTransformer` on the senders, and decrypting it on the receivers +- Keeping the codec header of each frame in the clear for the packetizer, and authenticating it instead +- Dropping frames that fail to encrypt or to authenticate, rather than passing them on +- Recording the receivers with a `MediaRecorder`, without decoding or re-encoding + +::: info +Like the [Media File](#media-file) example, this one needs the `webrtc-java-media` module. The output file is optional; its extension picks the container. + +```bash +mvn exec:java -D"exec.mainClass=dev.onvoid.webrtc.examples.EncryptedRecordingExample" -D"exec.args=call.mkv" +``` +::: + ## Web Client The [`WebClientExample`](https://github.com/devopvoid/webrtc-java/blob/master/webrtc-examples/src/main/java/dev/onvoid/webrtc/examples/web/WebClientExample.java) demonstrates how to combine WebSocket signaling with WebRTC peer connections for real-time communication between web and Java clients. diff --git a/docs/guide/index.md b/docs/guide/index.md index b851497e..c478919a 100644 --- a/docs/guide/index.md +++ b/docs/guide/index.md @@ -8,6 +8,7 @@ This section provides detailed guides for various features of the webrtc-java li - [Bitrate and Framerate Constraints](/guide/media/constraints) - Controlling media quality - [Send-only and Receive-only](/guide/media/directionality) - Configure transceiver directions (send-only, receive-only or inactive) - [Media Files](/guide/media/media-files) - Sending video and audio files instead of a camera and microphone +- [Media Recording](/guide/media/media-recording) - Recording what a call sends or receives into a media file, without re-encoding ## Audio @@ -36,6 +37,11 @@ This section provides detailed guides for various features of the webrtc-java li - [RTC Stats](/guide/monitoring/rtc-stats) - Monitoring connection quality and performance - [Logging](/guide/monitoring/logging) - Configuring and using the logging system +## Advanced + +- [Field Trials](/guide/advanced/field-trials) - Enabling experimental features and tuning WebRTC internals +- [Encoded Transforms](/guide/advanced/encoded-transforms) - Reading and changing encoded frames, e.g. for end-to-end encryption + ## Additional Resources For a complete API reference, check the [JavaDoc](https://javadoc.io/doc/dev.onvoid.webrtc/webrtc-java/latest/index.html). \ No newline at end of file diff --git a/docs/guide/media/media-recording.md b/docs/guide/media/media-recording.md new file mode 100644 index 00000000..db3c23c9 --- /dev/null +++ b/docs/guide/media/media-recording.md @@ -0,0 +1,87 @@ +# Media Recording + +This guide explains how to record what a peer connection sends or receives into a media file with `MediaRecorder`, from the `webrtc-java-media` module. It covers: + +- Recording senders and receivers +- Choosing the container +- Following a recording with a listener +- How tracks are started, timed and left out +- Recording an end-to-end encrypted call + +`MediaRecorder` writes the encoded frames of each sender or receiver into the file as they are, without decoding or re-encoding them. A recording therefore costs next to no CPU, keeps exactly the quality that went over the network, and needs no codec at all, only the container. The frames are taken in native code, straight from WebRTC, and written by a thread of the recorder's own, so neither Java nor a slow disk ever holds up the call. + +See [Media Files](/guide/media/media-files) for how to add the module to a project. + +## Recording a Call + +Add the senders and receivers to record, then start: + +```java +try (MediaRecorder recorder = new MediaRecorder(Paths.get("call.mkv"))) { + recorder.addTrack(videoReceiver); + recorder.addTrack(audioReceiver); + recorder.start(); + + // ... the call goes on ... + + recorder.stop(); +} +``` + +A sender's frames are recorded as they leave the encoder, a receiver's as they go to the decoder. Senders and receivers of any number of peer connections can go into one file, e.g. both sides of a call. + +`stop()` waits until everything received so far is written and the file is finished, and returns whether the file holds a recording. A recorder that never got any media to write deletes its file rather than leave an unplayable one behind. `close()` stops a recording that still runs. + +::: warning +Keep the senders and receivers undisposed while recording. The recorder asks them for key frames, and a disposed one cannot be asked. +::: + +## Containers + +The container follows the file name: + +| Extension | Video | Audio | Notes | +| --- | --- | --- | --- | +| `.mkv` | VP8, VP9, AV1, H.264, H.265 | Opus, G.711 | Holds every codec WebRTC sends; the safe choice. | +| `.webm` | VP8, VP9, AV1 | Opus | Plays in browsers. | +| `.mp4` | VP9, AV1, H.264, H.265 | Opus | Written fragmented, so it plays up to where a recording was cut short. | + +A name FFmpeg does not know gets Matroska. A track whose codec the file cannot hold, VP8 in MP4 for example, is left out and reported as a warning, and the rest is recorded. + +## Following a Recording + +A `MediaRecorderListener` hears when the file begins, what was left out, and whether writing failed: + +```java +recorder.setListener(new MediaRecorderListener() { + + @Override + public void onStarted() { + System.out.println("Recording"); + } + + @Override + public void onWarning(String message) { + System.out.println("Warning: " + message); + } + + @Override + public void onError(String message) { + System.out.println("Failed: " + message); + } +}); +``` + +Calls arrive in order on a thread the recorder keeps for them, so a listener may take its time and may even stop the recorder. + +## How Tracks Are Recorded + +- **Video starts at a key frame.** A decoder can do nothing with the frames before one, so they are skipped. The recorder asks the sender or receiver for a key frame, so recording starts within moments instead of waiting until WebRTC sends one of its own accord, which it rarely does. +- **The file begins once the tracks are known.** The file header describes every stream, and a stream is only known from its first frames. The file begins once every track has sent some, or once the tracks that have waited three seconds for the rest. A track that sends nothing by then is left out, with a warning. +- **Tracks share one timeline.** Each track is placed by when its first frame arrived, and follows its own RTP timestamps from there, so its timing is exact. The tracks of one sender line up within the jitter of the network. +- **One simulcast layer.** A sender with simulcast is recorded at the layer whose frames reach the recorder first. +- **Nothing is lost to a slow disk, up to a point.** Frames queue up in memory for the writer, up to 64 MB. Beyond that, frames are dropped, and video starts over at the next key frame. + +## Recording an Encrypted Call + +A sender's frames are recorded before an [encoded transform](/guide/advanced/encoded-transforms) runs on them, and a receiver's after it. An end-to-end encrypted call is therefore recorded in the clear, on either side. [`EncryptedRecordingExample`](https://github.com/devopvoid/webrtc-java/blob/master/webrtc-examples/src/main/java/dev/onvoid/webrtc/examples/EncryptedRecordingExample.java) shows both together. diff --git a/docs/index.md b/docs/index.md index 3e3d6dc6..1dd4739c 100644 --- a/docs/index.md +++ b/docs/index.md @@ -28,6 +28,10 @@ features: details: Audio and video capture from cameras and microphones devices, with support for custom media sources for flexible streaming solutions. - title: Media File Playback details: Send video and audio files over a peer connection. The optional media module decodes with FFmpeg in native code and paces playback in real time, keeping audio and video in sync. + - title: Call Recording + details: Record what a peer connection sends or receives into MKV, WebM or MP4 files. Encoded frames go into the file as they are, without re-encoding, so recording costs next to no CPU and keeps the exact quality of the call. + - title: End-to-End Encryption + details: Encoded transforms let Java code read, change or drop every encoded frame between encoder and network, like insertable streams in the browser; the building block for end-to-end encryption, frame metadata and stream analysis. - title: Screen Sharing details: Share application windows or the full desktop with minimal setup; integrate screen capture streams like any other media stream. - title: Data Channels diff --git a/webrtc-examples/src/main/java/dev/onvoid/webrtc/examples/EncryptedRecordingExample.java b/webrtc-examples/src/main/java/dev/onvoid/webrtc/examples/EncryptedRecordingExample.java new file mode 100644 index 00000000..ec894559 --- /dev/null +++ b/webrtc-examples/src/main/java/dev/onvoid/webrtc/examples/EncryptedRecordingExample.java @@ -0,0 +1,500 @@ +/* + * Copyright 2026 Alex Andres + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package dev.onvoid.webrtc.examples; + +import java.nio.ByteBuffer; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.security.GeneralSecurityException; +import java.security.SecureRandom; +import java.util.List; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicLong; +import java.util.logging.Level; +import java.util.logging.Logger; + +import javax.crypto.Cipher; +import javax.crypto.SecretKey; +import javax.crypto.spec.GCMParameterSpec; +import javax.crypto.spec.SecretKeySpec; + +import dev.onvoid.webrtc.CreateSessionDescriptionObserver; +import dev.onvoid.webrtc.PeerConnectionFactory; +import dev.onvoid.webrtc.PeerConnectionObserver; +import dev.onvoid.webrtc.RTCAnswerOptions; +import dev.onvoid.webrtc.RTCConfiguration; +import dev.onvoid.webrtc.RTCEncodedFrame; +import dev.onvoid.webrtc.RTCEncodedFrameTransformer; +import dev.onvoid.webrtc.RTCEncodedVideoFrame; +import dev.onvoid.webrtc.RTCIceCandidate; +import dev.onvoid.webrtc.RTCOfferOptions; +import dev.onvoid.webrtc.RTCPeerConnection; +import dev.onvoid.webrtc.RTCPeerConnectionState; +import dev.onvoid.webrtc.RTCRtpReceiver; +import dev.onvoid.webrtc.RTCRtpSender; +import dev.onvoid.webrtc.RTCRtpTransceiver; +import dev.onvoid.webrtc.RTCSessionDescription; +import dev.onvoid.webrtc.SetSessionDescriptionObserver; +import dev.onvoid.webrtc.media.audio.AudioDeviceModule; +import dev.onvoid.webrtc.media.audio.AudioLayer; +import dev.onvoid.webrtc.media.audio.AudioTrack; +import dev.onvoid.webrtc.media.audio.CustomAudioSource; +import dev.onvoid.webrtc.media.recorder.MediaRecorder; +import dev.onvoid.webrtc.media.recorder.MediaRecorderListener; +import dev.onvoid.webrtc.media.video.CustomVideoSource; +import dev.onvoid.webrtc.media.video.NativeI420Buffer; +import dev.onvoid.webrtc.media.video.VideoFrame; +import dev.onvoid.webrtc.media.video.VideoTrack; + +/** + * Encrypts a call end to end with encoded frame transforms, and records what + * the receiving side decrypted into a media file. + *
+ * This example shows how to: + *
+ * Both peers live in this application and the key is simply shared between + * them; a real application exchanges it over its signaling channel, or better + * derives it with a key agreement such as MLS. + *
+ * Run it with an optional output file, whose extension picks the container: + *
+ * java dev.onvoid.webrtc.examples.EncryptedRecordingExample call.mkv + *+ * + * @author Alex Andres + */ +public class EncryptedRecordingExample { + + private static final Logger LOG = Logger.getLogger(EncryptedRecordingExample.class.getName()); + + private static final int RECORD_SECONDS = 10; + + private static final int WIDTH = 640; + private static final int HEIGHT = 480; + + + public static void main(String[] args) throws Exception { + Path file = Paths.get(args.length > 0 ? args[0] : "encrypted-call.mkv"); + + // Pushed audio needs a factory whose audio layer does not capture. + AudioDeviceModule audioModule = new AudioDeviceModule(AudioLayer.kDummyAudio); + PeerConnectionFactory factory = new PeerConnectionFactory(audioModule); + + CustomVideoSource videoSource = new CustomVideoSource(); + CustomAudioSource audioSource = new CustomAudioSource(); + VideoTrack videoTrack = factory.createVideoTrack("video", videoSource); + AudioTrack audioTrack = factory.createAudioTrack("audio", audioSource); + + Peer caller = new Peer(factory); + Peer callee = new Peer(factory); + caller.remote = callee; + callee.remote = caller; + + FrameCipher cipher = new FrameCipher(newKey()); + + RTCRtpSender videoSender = caller.connection.addTrack(videoTrack, List.of("stream")); + RTCRtpSender audioSender = caller.connection.addTrack(audioTrack, List.of("stream")); + + // Set before negotiating, so the encoder needs no restart. + videoSender.setTransform(cipher::encrypt); + audioSender.setTransform(cipher::encrypt); + + callee.setRemoteDescription(caller.createOffer()); + caller.setRemoteDescription(callee.createAnswer()); + + // The callee's transceivers follow the order of the offer. + RTCRtpTransceiver[] transceivers = callee.connection.getTransceivers(); + RTCRtpReceiver videoReceiver = transceivers[0].getReceiver(); + RTCRtpReceiver audioReceiver = transceivers[1].getReceiver(); + + videoReceiver.setTransform(cipher::decrypt); + audioReceiver.setTransform(cipher::decrypt); + + caller.connected.await(10, TimeUnit.SECONDS); + callee.connected.await(10, TimeUnit.SECONDS); + + MediaGenerator generator = new MediaGenerator(videoSource, audioSource); + generator.start(); + + try (MediaRecorder recorder = new MediaRecorder(file)) { + recorder.setListener(new MediaRecorderListener() { + + @Override + public void onStarted() { + LOG.info("Recording into " + recorder.getFile()); + } + + @Override + public void onWarning(String message) { + LOG.warning(message); + } + + @Override + public void onError(String message) { + LOG.severe(message); + } + }); + + // The receivers hand the recorder what they decrypted. + recorder.addTrack(videoReceiver); + recorder.addTrack(audioReceiver); + recorder.start(); + + Thread.sleep(TimeUnit.SECONDS.toMillis(RECORD_SECONDS)); + + if (recorder.stop()) { + LOG.info("Recorded " + RECORD_SECONDS + " seconds into " + recorder.getFile()); + } + } + finally { + generator.stop(); + + LOG.info(String.format("Encrypted %d frames, decrypted %d, rejected %d", + cipher.encrypted.get(), cipher.decrypted.get(), cipher.rejected.get())); + + videoReceiver.dispose(); + audioReceiver.dispose(); + for (RTCRtpTransceiver transceiver : transceivers) { + transceiver.dispose(); + } + videoSender.dispose(); + audioSender.dispose(); + + caller.connection.close(); + callee.connection.close(); + + videoTrack.dispose(); + audioTrack.dispose(); + videoSource.dispose(); + audioSource.dispose(); + + factory.dispose(); + audioModule.dispose(); + } + } + + private static SecretKey newKey() { + byte[] key = new byte[16]; + new SecureRandom().nextBytes(key); + + return new SecretKeySpec(key, "AES"); + } + + + /** + * AES-GCM over the payload of each frame. An encrypted frame is laid out + * as the clear header, the ciphertext with its tag, then the 12-byte IV. + * Each transform runs on a thread of its own, one per sender or receiver, + * so each call makes its own Cipher rather than share one. + */ + private static class FrameCipher { + + private static final int IV_SIZE = 12; + private static final int TAG_BITS = 128; + + private final SecretKey key; + private final SecureRandom random = new SecureRandom(); + + // The audio and the video transform each run on a thread of their own. + final AtomicLong encrypted = new AtomicLong(); + final AtomicLong decrypted = new AtomicLong(); + final AtomicLong rejected = new AtomicLong(); + + + FrameCipher(SecretKey key) { + this.key = key; + } + + void encrypt(RTCEncodedFrame frame) { + ByteBuffer data = frame.getData(); + int clear = clearBytes(frame, data.remaining()); + + byte[] iv = new byte[IV_SIZE]; + random.nextBytes(iv); + + try { + Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding"); + cipher.init(Cipher.ENCRYPT_MODE, key, new GCMParameterSpec(TAG_BITS, iv)); + + ByteBuffer header = data.duplicate(); + header.limit(clear); + cipher.updateAAD(header); + + ByteBuffer out = ByteBuffer.allocate(clear + cipher.getOutputSize(data.remaining() - clear) + IV_SIZE); + ByteBuffer headerCopy = data.duplicate(); + headerCopy.limit(clear); + out.put(headerCopy); + + data.position(clear); + cipher.doFinal(data, out); + out.put(iv); + out.flip(); + + frame.setData(out); + encrypted.incrementAndGet(); + } + catch (GeneralSecurityException e) { + // Never send a frame that failed to encrypt. + frame.drop(); + LOG.log(Level.WARNING, "Encrypting a frame failed", e); + } + } + + void decrypt(RTCEncodedFrame frame) { + ByteBuffer data = frame.getData(); + int size = data.remaining(); + int clear = clearBytes(frame, size); + + if (size < clear + TAG_BITS / 8 + IV_SIZE) { + frame.drop(); + rejected.incrementAndGet(); + return; + } + + byte[] iv = new byte[IV_SIZE]; + data.position(size - IV_SIZE); + data.get(iv); + + try { + Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding"); + cipher.init(Cipher.DECRYPT_MODE, key, new GCMParameterSpec(TAG_BITS, iv)); + + ByteBuffer header = data.duplicate(); + header.position(0); + header.limit(clear); + cipher.updateAAD(header); + + ByteBuffer out = ByteBuffer.allocate(size); + ByteBuffer headerCopy = data.duplicate(); + headerCopy.position(0); + headerCopy.limit(clear); + out.put(headerCopy); + + ByteBuffer body = data.duplicate(); + body.position(clear); + body.limit(size - IV_SIZE); + cipher.doFinal(body, out); + out.flip(); + + frame.setData(out); + decrypted.incrementAndGet(); + } + catch (GeneralSecurityException e) { + // A frame that does not authenticate was tampered with, or + // encrypted with another key: it must not reach the decoder. + frame.drop(); + rejected.incrementAndGet(); + } + } + + /** + * The bytes left in the clear: the VP8 payload header, which the + * receiver's depacketizer reads to find key frames and the frame + * size (10 bytes on a key frame, 3 otherwise), and the Opus TOC byte. + */ + private static int clearBytes(RTCEncodedFrame frame, int size) { + int clear = 1; + + if (frame instanceof RTCEncodedVideoFrame video) { + clear = video.isKeyFrame() ? 10 : 3; + } + + return Math.min(clear, size); + } + } + + + /** + * Pushes a moving test picture at 30 frames and a tone in 10 ms chunks + * per second, in real time. + */ + private static class MediaGenerator { + + private final CustomVideoSource videoSource; + private final CustomAudioSource audioSource; + + private volatile boolean running; + private Thread thread; + + + MediaGenerator(CustomVideoSource videoSource, CustomAudioSource audioSource) { + this.videoSource = videoSource; + this.audioSource = audioSource; + } + + void start() { + running = true; + thread = new Thread(this::run, "media-generator"); + thread.setDaemon(true); + thread.start(); + } + + void stop() throws InterruptedException { + running = false; + thread.join(); + } + + private void run() { + byte[] audio = new byte[480 * 2]; + long startNs = System.nanoTime(); + long chunks = 0; + long frames = 0; + + while (running) { + long elapsedMs = (System.nanoTime() - startNs) / 1_000_000; + + while (chunks * 10 <= elapsedMs) { + for (int i = 0; i < 480; i++) { + double t = (chunks * 480 + i) / 48000.0; + short sample = (short) (Math.sin(2 * Math.PI * 440 * t) * 6000); + audio[2 * i] = (byte) sample; + audio[2 * i + 1] = (byte) (sample >> 8); + } + + audioSource.pushAudio(audio, 16, 48000, 1, 480); + chunks++; + } + + if (frames * 1000 / 30 <= elapsedMs) { + NativeI420Buffer buffer = NativeI420Buffer.allocate(WIDTH, HEIGHT); + ByteBuffer y = buffer.getDataY(); + int stride = buffer.getStrideY(); + int offset = (int) (frames * 4); + + // Diagonal stripes that move, so every frame differs. + for (int row = 0; row < HEIGHT; row++) { + for (int col = 0; col < WIDTH; col++) { + y.put(row * stride + col, (byte) (((row + col + offset) / 16 % 2) * 180 + 40)); + } + } + + VideoFrame frame = new VideoFrame(buffer, 0); + videoSource.pushFrame(frame); + frame.release(); + frames++; + } + + try { + Thread.sleep(2); + } + catch (InterruptedException e) { + return; + } + } + } + } + + + /** + * One end of the call, exchanging candidates with the other directly. + */ + private static class Peer implements PeerConnectionObserver { + + final RTCPeerConnection connection; + final CountDownLatch connected = new CountDownLatch(1); + + volatile Peer remote; + + + Peer(PeerConnectionFactory factory) { + connection = factory.createPeerConnection(new RTCConfiguration(), this); + } + + @Override + public void onIceCandidate(RTCIceCandidate candidate) { + remote.connection.addIceCandidate(candidate); + } + + @Override + public void onConnectionChange(RTCPeerConnectionState state) { + if (state == RTCPeerConnectionState.CONNECTED) { + connected.countDown(); + } + } + + RTCSessionDescription createOffer() throws Exception { + CompletableFuture
+ * The container follows the file name: {@code .mkv} (Matroska) holds every + * codec WebRTC sends and is the safe choice; {@code .webm} holds VP8, VP9, + * AV1 and Opus; {@code .mp4} holds H.264, H.265, AV1, VP9 and Opus, and is + * written fragmented, so that it plays up to where a recording was cut short. + * A track whose codec the file cannot hold is left out and reported to the + * {@link MediaRecorderListener#onWarning(String) listener}. + *
+ * Example: + *
{@code
+ * try (MediaRecorder recorder = new MediaRecorder(Paths.get("call.mkv"))) {
+ * recorder.addTrack(videoReceiver);
+ * recorder.addTrack(audioReceiver);
+ * recorder.start();
+ *
+ * // ... the call goes on ...
+ *
+ * recorder.stop();
+ * }
+ * }
+ * + * How it behaves: + *
+ * Stopping a recorder that is stopped does nothing. Stopping one that was + * never started deletes its file. + * + * @return True if the file holds a recording, false if it was deleted. + */ + public boolean stop() { + boolean recorded; + + synchronized (lock) { + if (handle == 0) { + return false; + } + if (state == MediaRecorderState.STOPPED) { + return Files.exists(file); + } + + recorded = stop(handle); + state = MediaRecorderState.STOPPED; + + if (events != null) { + // Lets the reports already queued run, then ends the thread. + events.shutdown(); + } + } + + if (!recorded) { + try { + Files.deleteIfExists(file); + } + catch (IOException e) { + // The file stays behind, empty; there is nothing else to do. + } + } + + return recorded; + } + + /** + * Returns what the recorder is currently doing. + * + * @return The recorder state. + */ + public MediaRecorderState getState() { + synchronized (lock) { + return state; + } + } + + /** + * @return The file this recorder writes to. + */ + public Path getFile() { + return file; + } + + /** + * Stops the recording if it still runs and releases the native recorder. + * Closing a recorder that is closed does nothing. + */ + @Override + public void close() { + stop(); + + synchronized (lock) { + if (handle != 0) { + dispose(handle); + handle = 0; + } + } + } + + /** Called with the lock held, in the idle state. */ + private void addTrack(long frames, Runnable keyFrameRequest) { + if (frames == 0) { + throw new IllegalArgumentException("The sender or receiver was disposed"); + } + + // The native recorder takes over the reference the handle carries. + addTrack(handle, frames); + + keyFrameRequests.add(keyFrameRequest); + } + + private void checkState(MediaRecorderState expected) { + if (handle == 0) { + throw new IllegalStateException("The recorder is closed"); + } + if (state != expected) { + throw new IllegalStateException("The recorder is " + state.name().toLowerCase() + + ", not " + expected.name().toLowerCase()); + } + } + + private void post(Runnable event) { + ExecutorService executor = events; + + if (executor == null) { + return; + } + + try { + executor.execute(() -> { + try { + event.run(); + } + catch (Throwable e) { + Thread thread = Thread.currentThread(); + thread.getUncaughtExceptionHandler().uncaughtException(thread, e); + } + }); + } + catch (RejectedExecutionException e) { + // Stopped in the meantime; nobody is waiting for this any more. + } + } + + /** Called by native code on the writer thread. */ + private void onNativeStarted() { + post(() -> { + MediaRecorderListener current = listener; + + if (current != null) { + current.onStarted(); + } + }); + } + + /** Called by native code on the writer thread. */ + private void onNativeWarning(String message) { + post(() -> { + MediaRecorderListener current = listener; + + if (current != null) { + current.onWarning(message); + } + }); + } + + /** Called by native code on the writer thread. */ + private void onNativeError(String message) { + post(() -> { + MediaRecorderListener current = listener; + + if (current != null) { + current.onError(message); + } + }); + } + + /** Called by native code on the writer thread. */ + private void onNativeKeyFrameNeeded(int track) { + if (track < 0 || track >= keyFrameRequests.size()) { + return; + } + + Runnable request = keyFrameRequests.get(track); + + // Run away from the writer thread: asking WebRTC waits for its + // signaling thread, which may be busy stopping this very recorder. + post(() -> { + try { + request.run(); + } + catch (RuntimeException e) { + // Not negotiated yet, or disposed of by the application. The + // recording waits for the next key frame instead. + } + }); + } + + private native long create(String path, long tableAddress) throws IOException; + + private static native int addTrack(long handle, long frames); + + private static native void start(long handle); + + private static native boolean stop(long handle); + + private static native void dispose(long handle); + +} diff --git a/webrtc-java-media/src/main/java/dev/onvoid/webrtc/media/recorder/MediaRecorderListener.java b/webrtc-java-media/src/main/java/dev/onvoid/webrtc/media/recorder/MediaRecorderListener.java new file mode 100644 index 00000000..56ebaba6 --- /dev/null +++ b/webrtc-java-media/src/main/java/dev/onvoid/webrtc/media/recorder/MediaRecorderListener.java @@ -0,0 +1,58 @@ +/* + * Copyright 2026 Alex Andres + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package dev.onvoid.webrtc.media.recorder; + +/** + * What a {@link MediaRecorder} reports while it records. + *
+ * Calls arrive in order on a thread the recorder keeps for them, never on a
+ * thread that carries media, so an implementation may take its time and may
+ * call back into the recorder, including to stop it.
+ *
+ * @author Alex Andres
+ */
+public interface MediaRecorderListener {
+
+ /**
+ * The file header was written and media now goes into the file. This
+ * happens once every track has sent enough to describe its stream (for
+ * video, its first key frame), or once the tracks that have waited a few
+ * seconds for the others.
+ */
+ default void onStarted() {
+ }
+
+ /**
+ * Something went wrong that the recording carries on without, e.g. a
+ * track whose codec the file cannot hold, which is then left out.
+ *
+ * @param message What went wrong.
+ */
+ default void onWarning(String message) {
+ }
+
+ /**
+ * Writing the file failed, e.g. because the disk is full. Nothing more
+ * goes into the file; what is in it stays, and {@link MediaRecorder#stop()}
+ * still finishes it as far as it can.
+ *
+ * @param message What went wrong.
+ */
+ default void onError(String message) {
+ }
+
+}
diff --git a/webrtc-java-media/src/main/java/dev/onvoid/webrtc/media/recorder/MediaRecorderState.java b/webrtc-java-media/src/main/java/dev/onvoid/webrtc/media/recorder/MediaRecorderState.java
new file mode 100644
index 00000000..5d00663c
--- /dev/null
+++ b/webrtc-java-media/src/main/java/dev/onvoid/webrtc/media/recorder/MediaRecorderState.java
@@ -0,0 +1,35 @@
+/*
+ * Copyright 2026 Alex Andres
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package dev.onvoid.webrtc.media.recorder;
+
+/**
+ * The states a {@link MediaRecorder} goes through, in this order.
+ *
+ * @author Alex Andres
+ */
+public enum MediaRecorderState {
+
+ /** Created, taking tracks, not recording yet. */
+ IDLE,
+
+ /** Started; frames go into the file. */
+ RECORDING,
+
+ /** Stopped, with the file finished. A recorder cannot start again. */
+ STOPPED
+
+}
diff --git a/webrtc-java-media/src/main/java/module-info.java b/webrtc-java-media/src/main/java/module-info.java
index 7ea0e974..b55c7029 100644
--- a/webrtc-java-media/src/main/java/module-info.java
+++ b/webrtc-java-media/src/main/java/module-info.java
@@ -1,11 +1,13 @@
/**
* Media extension for webrtc-java: reads media files and network streams with
- * FFmpeg and feeds them into a peer connection.
+ * FFmpeg and feeds them into a peer connection, and records what a peer
+ * connection sends or receives into media files.
*/
module webrtc.java.media {
requires webrtc.java;
exports dev.onvoid.webrtc.media.player;
+ exports dev.onvoid.webrtc.media.recorder;
}
diff --git a/webrtc-java-media/src/test/java/dev/onvoid/webrtc/media/recorder/MediaRecorderTest.java b/webrtc-java-media/src/test/java/dev/onvoid/webrtc/media/recorder/MediaRecorderTest.java
new file mode 100644
index 00000000..36fab892
--- /dev/null
+++ b/webrtc-java-media/src/test/java/dev/onvoid/webrtc/media/recorder/MediaRecorderTest.java
@@ -0,0 +1,294 @@
+/*
+ * Copyright 2026 Alex Andres
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package dev.onvoid.webrtc.media.recorder;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+import dev.onvoid.webrtc.PeerConnectionFactory;
+import dev.onvoid.webrtc.RTCRtpReceiver;
+import dev.onvoid.webrtc.media.audio.AudioDeviceModule;
+import dev.onvoid.webrtc.media.audio.AudioLayer;
+import dev.onvoid.webrtc.media.player.MediaInfo;
+import dev.onvoid.webrtc.media.player.MediaReader;
+
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInstance;
+import org.junit.jupiter.api.io.TempDir;
+import org.junit.jupiter.api.parallel.Execution;
+import org.junit.jupiter.api.parallel.ExecutionMode;
+
+/**
+ * Tests recording a call between two local peer connections, checked by
+ * reading the recorded file back.
+ */
+@TestInstance(TestInstance.Lifecycle.PER_CLASS)
+@Execution(ExecutionMode.SAME_THREAD)
+class MediaRecorderTest {
+
+ private static final long RECORD_MS = 3000;
+
+ private AudioDeviceModule audioModule;
+ private PeerConnectionFactory factory;
+
+ @TempDir
+ Path tempDir;
+
+
+ @BeforeAll
+ void initFactory() {
+ // A dummy audio layer, because this factory sends pushed audio.
+ audioModule = new AudioDeviceModule(AudioLayer.kDummyAudio);
+ factory = new PeerConnectionFactory(audioModule);
+ }
+
+ @AfterAll
+ void disposeFactory() {
+ factory.dispose();
+ audioModule.dispose();
+ }
+
+ @Test
+ void recordsReceivedCallIntoMatroska() throws Exception {
+ Path file = tempDir.resolve("received.mkv");
+ Listener listener = new Listener();
+
+ try (TestCall call = new TestCall(factory)) {
+ // Well into the call, so that the receiver has had its key frame
+ // long ago and the recorder has to ask for one.
+ Thread.sleep(1500);
+
+ RTCRtpReceiver video = call.getReceiver(0);
+ RTCRtpReceiver audio = call.getReceiver(1);
+
+ try (MediaRecorder recorder = new MediaRecorder(file)) {
+ recorder.setListener(listener);
+ recorder.addTrack(video);
+ recorder.addTrack(audio);
+ recorder.start();
+
+ assertEquals(MediaRecorderState.RECORDING, recorder.getState());
+ assertTrue(listener.started.await(10, TimeUnit.SECONDS), "not started");
+
+ Thread.sleep(RECORD_MS);
+
+ assertTrue(recorder.stop());
+ assertEquals(MediaRecorderState.STOPPED, recorder.getState());
+ }
+ finally {
+ video.dispose();
+ audio.dispose();
+ }
+ }
+
+ assertNull(listener.error.get());
+ assertTrue(listener.warnings.isEmpty(), listener.warnings.toString());
+
+ MediaInfo info = readInfo(file);
+
+ assertTrue(info.hasVideo());
+ assertTrue(info.hasAudio());
+ assertEquals("vp8", info.getVideoCodec());
+ assertEquals(TestCall.WIDTH, info.getVideoWidth());
+ assertEquals(TestCall.HEIGHT, info.getVideoHeight());
+ assertEquals("opus", info.getAudioCodec());
+ assertEquals(48000, info.getSampleRate());
+ assertDuration(info);
+ }
+
+ @Test
+ void recordsSentCallIntoWebm() throws Exception {
+ Path file = tempDir.resolve("sent.webm");
+ Listener listener = new Listener();
+
+ try (TestCall call = new TestCall(factory);
+ MediaRecorder recorder = new MediaRecorder(file)) {
+ recorder.setListener(listener);
+ recorder.addTrack(call.getVideoSender());
+ recorder.addTrack(call.getAudioSender());
+ recorder.start();
+
+ assertTrue(listener.started.await(10, TimeUnit.SECONDS), "not started");
+
+ Thread.sleep(RECORD_MS);
+
+ assertTrue(recorder.stop());
+ }
+
+ assertNull(listener.error.get());
+
+ MediaInfo info = readInfo(file);
+
+ assertEquals("vp8", info.getVideoCodec());
+ assertEquals("opus", info.getAudioCodec());
+ assertDuration(info);
+ }
+
+ @Test
+ void leavesOutCodecFileCannotHold() throws Exception {
+ // MP4 has no place for VP8, which the call sends.
+ Path file = tempDir.resolve("audio-only.mp4");
+ Listener listener = new Listener();
+
+ try (TestCall call = new TestCall(factory);
+ MediaRecorder recorder = new MediaRecorder(file)) {
+ recorder.setListener(listener);
+ recorder.addTrack(call.getVideoSender());
+ recorder.addTrack(call.getAudioSender());
+ recorder.start();
+
+ assertTrue(listener.started.await(10, TimeUnit.SECONDS), "not started");
+
+ Thread.sleep(RECORD_MS);
+
+ assertTrue(recorder.stop());
+ }
+
+ assertFalse(listener.warnings.isEmpty());
+ assertTrue(listener.warnings.get(0).contains("video/VP8"), listener.warnings.get(0));
+
+ MediaInfo info = readInfo(file);
+
+ assertFalse(info.hasVideo());
+ assertEquals("opus", info.getAudioCodec());
+ }
+
+ @Test
+ void deletesFileWithoutMedia() throws Exception {
+ Path file = tempDir.resolve("nothing.mkv");
+
+ try (TestCall call = new TestCall(factory);
+ MediaRecorder recorder = new MediaRecorder(file)) {
+ assertTrue(Files.exists(file));
+
+ recorder.addTrack(call.getVideoSender());
+ recorder.start();
+
+ // Too soon for any frame to have made it into the file.
+ assertFalse(recorder.stop());
+ }
+
+ assertFalse(Files.exists(file));
+ }
+
+ @Test
+ void unstartedRecorderDeletesFile() throws Exception {
+ Path file = tempDir.resolve("unstarted.mkv");
+
+ try (MediaRecorder recorder = new MediaRecorder(file)) {
+ assertEquals(MediaRecorderState.IDLE, recorder.getState());
+ assertThrows(IllegalStateException.class, recorder::start);
+ }
+
+ assertFalse(Files.exists(file));
+ }
+
+ @Test
+ void rejectsTracksOnceStarted() throws Exception {
+ Path file = tempDir.resolve("rejects.mkv");
+
+ try (TestCall call = new TestCall(factory);
+ MediaRecorder recorder = new MediaRecorder(file)) {
+ recorder.addTrack(call.getVideoSender());
+ recorder.start();
+
+ assertThrows(IllegalStateException.class, () -> recorder.addTrack(call.getAudioSender()));
+ assertThrows(IllegalStateException.class, recorder::start);
+ }
+ }
+
+ @Test
+ void stopFromListener() throws Exception {
+ Path file = tempDir.resolve("listener-stop.mkv");
+ AtomicReference