feat: encode AV1 on the GPU with Media Foundation and NVENC

docs/guide/advanced/video-codecs.md

@@ -24,17 +24,19 @@ PeerConnectionFactory factory = PeerConnectionFactory.builder()
         .build();
 ```
 
-| Platform | Hardware encoding |
-|---|---|
-| Windows | H.264 with NVENC on NVIDIA GPUs, otherwise through the Media Foundation encoder of the GPU driver (AMD, Intel) |
-| Linux | H.264 with NVENC on NVIDIA GPUs, otherwise through the VA-API encoder of the GPU driver (Intel, AMD) |
-| macOS | H.264 through VideoToolbox, as with `DefaultVideoEncoderFactory` |
+| Platform | H.264 | AV1 |
+|---|---|---|
+| Windows | NVENC on NVIDIA GPUs, otherwise the Media Foundation encoder of the GPU driver (AMD, Intel) | NVENC on NVIDIA RTX 40 and newer, otherwise Media Foundation on GPUs that encode AV1 (e.g. AMD RDNA3 and newer, Intel Arc) |
+| Linux | NVENC on NVIDIA GPUs, otherwise the VA-API encoder of the GPU driver (Intel, AMD) | NVENC on NVIDIA RTX 40 and newer |
+| macOS | VideoToolbox, as with `DefaultVideoEncoderFactory` | Software |
+
+Hardware encoders produce a single layer. A stream that asks for SVC, such as an AV1 stream with a scalability mode like `L1T3`, is encoded in software.
 
 NVENC needs an NVIDIA driver of version 522 or newer on Windows, 520 or newer on Linux. VA-API needs libva 2 and a driver that encodes H.264, such as Intel's `intel-media-va-driver` (iHD) or Mesa's `mesa-va-drivers` for AMD, and access to a render node in `/dev/dri`. All of them are loaded at run time, so nothing needs to be installed on machines without them.
 
-The hardware encoders take over H.264 Constrained Baseline and Baseline with packetization mode 1, formats the software encoder offers too, so encoding in hardware never changes what is negotiated. When a hardware encoder fails to start, for example because the GPU has no encoder sessions left, or fails while encoding, the stream switches to the next encoder in line (e.g. from NVENC to Media Foundation on Windows), and finally to the software encoder, and continues with a key frame. On a machine without a hardware encoder, the factory encodes like `DefaultVideoEncoderFactory`.
+The hardware encoders take over H.264 Constrained Baseline and Baseline with packetization mode 1, and AV1 profile 0, formats the software encoders offer too, so encoding in hardware never changes what is negotiated. When a hardware encoder fails to start, for example because the GPU has no encoder sessions left, or fails while encoding, the stream switches to the next encoder in line (e.g. from NVENC to Media Foundation on Windows), and finally to the software encoder, and continues with a key frame. On a machine without a hardware encoder, the factory encodes like `DefaultVideoEncoderFactory`.
 
-Which encoder a stream uses shows in the `encoderImplementation` statistic of its `outbound-rtp` stats, e.g. `NVENC (NVIDIA GeForce RTX 4070)`, `MediaFoundation (AMDh264Encoder)`, `VA-API (Intel iHD driver ...)` or `OpenH264`.
+Which encoder a stream uses shows in the `encoderImplementation` statistic of its `outbound-rtp` stats, e.g. `NVENC (NVIDIA GeForce RTX 4070)`, `MediaFoundation (AMDav1Encoder)`, `VA-API (Intel iHD driver ...)`, `OpenH264` or `libaom`.
 
 ### Native Codecs
 

webrtc-jni/src/main/cpp/include/media/video/codec/EncoderOutputProcessor.h

@@ -0,0 +1,87 @@
+/*
+ * Copyright 2026 Alex Andres
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef JNI_WEBRTC_MEDIA_VIDEO_CODEC_ENCODER_OUTPUT_PROCESSOR_H_
+#define JNI_WEBRTC_MEDIA_VIDEO_CODEC_ENCODER_OUTPUT_PROCESSOR_H_
+
+#include "api/video/encoded_image.h"
+#include "api/video/video_codec_type.h"
+#include "api/video_codecs/sdp_video_format.h"
+#include "api/video_codecs/video_encoder.h"
+#include "common_video/h264/h264_bitstream_parser.h"
+#include "modules/video_coding/codecs/h264/include/h264_globals.h"
+#include "modules/video_coding/include/video_codec_interface.h"
+#include "modules/video_coding/svc/scalable_video_controller_no_layering.h"
+
+#include <cstdint>
+#include <span>
+#include <vector>
+
+namespace jni
+{
+	// Turns what a hardware encoder produces for a frame into what WebRTC
+	// sends, the same way for every hardware encoder: H.264 as Annex B NAL
+	// units, AV1 as OBUs in the low overhead bitstream format.
+	//
+	// A receiver can start decoding only from a key frame that carries the
+	// parameter sets, the SPS and PPS of H.264 or the sequence header of AV1.
+	// Encoders put them in front of the first key frame, not necessarily in
+	// front of every one, so they are kept and put back where missing. The
+	// frame dependencies WebRTC needs for AV1 are those of a stream without
+	// layers, which is all hardware encoders produce here.
+	class EncoderOutputProcessor
+	{
+		public:
+			EncoderOutputProcessor(webrtc::VideoCodecType codec, const webrtc::SdpVideoFormat & format);
+
+			// Forgets the parameter sets, for a stream that starts over.
+			void Reset();
+
+			// Fills the image with the bitstream of one frame, and the codec
+			// specific information WebRTC needs to send it. The caller sets
+			// the size of the image before, which the AV1 frame dependencies
+			// refer to, and its timestamps. The frame is a key
+			// frame if the encoder says so or, for H.264, if it holds an IDR
+			// slice. Returns false if the bitstream cannot be parsed, or a key
+			// frame lacks parameter sets and there are none to put back.
+			bool Process(std::span<const uint8_t> bitstream, bool keyFrame, webrtc::EncodedImage & image,
+				webrtc::CodecSpecificInfo & info);
+
+			// The quality scaling settings that suit the codec: the thresholds
+			// of WebRTC's H.264 encoder, and none for AV1, whose QP is not
+			// parsed from the bitstream.
+			webrtc::VideoEncoder::ScalingSettings GetScalingSettings() const;
+
+		private:
+			bool ProcessH264(std::span<const uint8_t> bitstream, bool & keyFrame, std::vector<uint8_t> & output);
+			bool ProcessAv1(std::span<const uint8_t> bitstream, bool keyFrame, std::vector<uint8_t> & output);
+
+			void FillCodecSpecificInfo(const webrtc::EncodedImage & image, bool keyFrame,
+				webrtc::CodecSpecificInfo & info);
+
+		private:
+			const webrtc::VideoCodecType codec;
+			webrtc::H264PacketizationMode packetizationMode;
+
+			// The SPS and PPS, or the sequence header OBU, last seen.
+			std::vector<uint8_t> parameterSets;
+
+			webrtc::H264BitstreamParser h264Parser;
+			webrtc::ScalableVideoControllerNoLayering svcController;
+	};
+}
+
+#endif

webrtc-jni/src/main/cpp/include/media/video/codec/linux/VaapiH264Encoder.h

@@ -17,14 +17,13 @@
 #ifndef JNI_WEBRTC_MEDIA_VIDEO_CODEC_VAAPI_H264_ENCODER_H_
 #define JNI_WEBRTC_MEDIA_VIDEO_CODEC_VAAPI_H264_ENCODER_H_
 
+#include "media/video/codec/EncoderOutputProcessor.h"
 #include "media/video/codec/linux/VaapiLibrary.h"
 
 #include "api/video/video_frame.h"
 #include "api/video_codecs/sdp_video_format.h"
 #include "api/video_codecs/video_codec.h"
 #include "api/video_codecs/video_encoder.h"
-#include "common_video/h264/h264_bitstream_parser.h"
-#include "modules/video_coding/codecs/h264/include/h264_globals.h"
 
 #include <va/va.h>
 #include <va/va_enc_h264.h>
@@ -103,8 +102,7 @@ namespace jni
 
 			webrtc::EncodedImageCallback * callback;
 
-			webrtc::H264BitstreamParser bitstreamParser;
-			webrtc::H264PacketizationMode packetizationMode;
+			EncoderOutputProcessor outputProcessor;
 	};
 }
 

webrtc-jni/src/main/cpp/include/media/video/codec/nvenc/NvencLibrary.h

@@ -60,6 +60,11 @@ namespace jni
 			// The name of the device, e.g. "NVIDIA GeForce RTX 4070".
 			const std::string & DeviceName() const;
 
+			// Whether the device encodes H.264, and AV1, which only newer
+			// GPUs do.
+			bool SupportsH264() const;
+			bool SupportsAv1() const;
+
 			// Retains the primary context of the device. Every successful call
 			// has to be matched by ReleaseContext().
 			bool RetainContext(CUcontext * context) const;
@@ -73,6 +78,9 @@ namespace jni
 
 			bool Load();
 
+			// Asks an encode session which codecs the device supports.
+			void QueryCodecs();
+
 		private:
 			using CuInit = CUresult (JNI_CUDAAPI *)(unsigned int flags);
 			using CuDeviceGetCount = CUresult (JNI_CUDAAPI *)(int * count);
@@ -101,6 +109,8 @@ namespace jni
 			NV_ENCODE_API_FUNCTION_LIST api = {};
 			CUdevice device = 0;
 			std::string deviceName;
+			bool h264 = false;
+			bool av1 = false;
 	};
 }
 

webrtc-jni/src/main/cpp/include/media/video/codec/nvenc/NvencH264Encoder.h → webrtc-jni/src/main/cpp/include/media/video/codec/nvenc/NvencVideoEncoder.h

@@ -14,17 +14,17 @@
  * limitations under the License.
  */
 
-#ifndef JNI_WEBRTC_MEDIA_VIDEO_CODEC_NVENC_H264_ENCODER_H_
-#define JNI_WEBRTC_MEDIA_VIDEO_CODEC_NVENC_H264_ENCODER_H_
+#ifndef JNI_WEBRTC_MEDIA_VIDEO_CODEC_NVENC_VIDEO_ENCODER_H_
+#define JNI_WEBRTC_MEDIA_VIDEO_CODEC_NVENC_VIDEO_ENCODER_H_
 
+#include "media/video/codec/EncoderOutputProcessor.h"
 #include "media/video/codec/nvenc/NvencLibrary.h"
 
+#include "api/video/video_codec_type.h"
 #include "api/video/video_frame.h"
 #include "api/video_codecs/sdp_video_format.h"
 #include "api/video_codecs/video_codec.h"
 #include "api/video_codecs/video_encoder.h"
-#include "common_video/h264/h264_bitstream_parser.h"
-#include "modules/video_coding/codecs/h264/include/h264_globals.h"
 
 #include <nvEncodeAPI.h>
 
@@ -34,19 +34,20 @@
 
 namespace jni
 {
-	// Encodes H.264 with NVENC, the encoder of NVIDIA GPUs, on the primary
-	// CUDA context of the device.
+	// Encodes H.264 or AV1 with NVENC, the encoder of NVIDIA GPUs, on the
+	// primary CUDA context of the device.
 	//
 	// Encoding is synchronous: with no B-frames and a low-latency preset,
 	// NVENC returns each frame as soon as it is encoded, so frames go in and
 	// out on the encoder thread. Frames are passed in system memory as NV12,
 	// into an input buffer NVENC allocates. Anything that fails makes the
 	// encoder give up, so that the next encoder in line takes over.
-	class NvencH264Encoder : public webrtc::VideoEncoder
+	class NvencVideoEncoder : public webrtc::VideoEncoder
 	{
 		public:
-			NvencH264Encoder(NvencLibrary & library, const webrtc::SdpVideoFormat & format);
-			~NvencH264Encoder() override;
+			// The codec is H.264 or AV1.
+			NvencVideoEncoder(NvencLibrary & library, webrtc::VideoCodecType codec, const webrtc::SdpVideoFormat & format);
+			~NvencVideoEncoder() override;
 
 			int32_t InitEncode(const webrtc::VideoCodec * codecSettings, const Settings & settings) override;
 			int32_t RegisterEncodeCompleteCallback(webrtc::EncodedImageCallback * callback) override;
@@ -58,13 +59,16 @@ namespace jni
 		private:
 			bool OpenSession();
 			bool Configure();
+			void ConfigureCodec();
 			void ApplyRates();
 			bool CopyToInput(const webrtc::VideoFrame & frame, uint32_t * pitch);
 			void DestroySession();
 
 		private:
 			NvencLibrary & library;
 			const NV_ENCODE_API_FUNCTION_LIST & api;
+			const webrtc::VideoCodecType codec;
+			const GUID codecGuid;
 			const std::string implementationName;
 
 			CUcontext context;
@@ -82,8 +86,7 @@ namespace jni
 
 			webrtc::EncodedImageCallback * callback;
 
-			webrtc::H264BitstreamParser bitstreamParser;
-			webrtc::H264PacketizationMode packetizationMode;
+			EncoderOutputProcessor outputProcessor;
 	};
 }
 

webrtc-jni/src/main/cpp/include/media/video/codec/nvenc/NvencVideoEncoderFactory.h

@@ -29,9 +29,10 @@
 
 namespace jni
 {
-	// Creates NVENC encoders on an NVIDIA GPU. It offers H.264 in the
-	// profiles WebRTC's software encoder offers too, and only with
-	// packetization mode 1: mode 0 needs each NAL unit to fit a packet.
+	// Creates NVENC encoders on an NVIDIA GPU, for H.264 and, on GPUs that
+	// have it, AV1. It offers them in the formats WebRTC's software encoders
+	// offer too: H.264 only with packetization mode 1, since mode 0 needs each
+	// NAL unit to fit a packet, and AV1 in profile 0.
 	class NvencVideoEncoderFactory : public webrtc::VideoEncoderFactory
 	{
 		public:

webrtc-jni/src/main/cpp/include/media/video/codec/windows/MFEncoderUtils.h

@@ -26,9 +26,10 @@
 
 namespace jni
 {
-	// Lists the hardware H.264 encoder transforms that take NV12, best
-	// first. Media Foundation has to be started.
-	HRESULT EnumerateHardwareH264Encoders(std::vector<Microsoft::WRL::ComPtr<IMFActivate>> & encoders);
+	// Lists the hardware encoder transforms that take NV12 and produce the
+	// given video format, such as MFVideoFormat_H264, best first. Media
+	// Foundation has to be started.
+	HRESULT EnumerateHardwareEncoders(const GUID & format, std::vector<Microsoft::WRL::ComPtr<IMFActivate>> & encoders);
 
 	// Returns the name the driver gives a transform, e.g. "AMDh264Encoder".
 	std::string GetTransformName(IMFActivate * activate);

webrtc-jni/src/main/cpp/include/media/video/codec/windows/MFH264Encoder.h → webrtc-jni/src/main/cpp/include/media/video/codec/windows/MFVideoEncoder.h

@@ -14,18 +14,18 @@
  * limitations under the License.
  */
 
-#ifndef JNI_WEBRTC_MEDIA_VIDEO_CODEC_MF_H264_ENCODER_H_
-#define JNI_WEBRTC_MEDIA_VIDEO_CODEC_MF_H264_ENCODER_H_
+#ifndef JNI_WEBRTC_MEDIA_VIDEO_CODEC_MF_VIDEO_ENCODER_H_
+#define JNI_WEBRTC_MEDIA_VIDEO_CODEC_MF_VIDEO_ENCODER_H_
 
+#include "media/video/codec/EncoderOutputProcessor.h"
 #include "media/video/codec/windows/MFTransformEvents.h"
 #include "platform/windows/MFInitializer.h"
 
+#include "api/video/video_codec_type.h"
 #include "api/video/video_frame.h"
 #include "api/video_codecs/sdp_video_format.h"
 #include "api/video_codecs/video_codec.h"
 #include "api/video_codecs/video_encoder.h"
-#include "common_video/h264/h264_bitstream_parser.h"
-#include "modules/video_coding/codecs/h264/include/h264_globals.h"
 
 #include <mftransform.h>
 #include <strmif.h>
@@ -42,20 +42,21 @@
 
 namespace jni
 {
-	// Encodes H.264 with the hardware encoder of the GPU, through the Media
-	// Foundation transform its driver provides. These transforms are
+	// Encodes H.264 or AV1 with the hardware encoder of the GPU, through the
+	// Media Foundation transform its driver provides. These transforms are
 	// asynchronous: they ask for input and announce output through events,
 	// which arrive on a Media Foundation thread, so encoded frames are
 	// handed to WebRTC from there.
 	//
 	// Frames are passed in system memory as NV12; the transform uploads them.
 	// Anything that fails makes the encoder give up, so that WebRTC switches
 	// to the software encoder.
-	class MFH264Encoder : public webrtc::VideoEncoder, public MFTransformEventListener
+	class MFVideoEncoder : public webrtc::VideoEncoder, public MFTransformEventListener
 	{
 		public:
-			explicit MFH264Encoder(const webrtc::SdpVideoFormat & format);
-			~MFH264Encoder() override;
+			// The codec is H.264 or AV1.
+			MFVideoEncoder(webrtc::VideoCodecType codec, const webrtc::SdpVideoFormat & format);
+			~MFVideoEncoder() override;
 
 			int32_t InitEncode(const webrtc::VideoCodec * codecSettings, const Settings & settings) override;
 			int32_t RegisterEncodeCompleteCallback(webrtc::EncodedImageCallback * callback) override;
@@ -79,14 +80,13 @@ namespace jni
 			HRESULT CreateTransform();
 			HRESULT ConfigureTypes();
 			void ConfigureCodec();
-			void SetBitrate(uint32_t bitrateBps);
 			HRESULT CreateInputSample(const webrtc::VideoFrame & frame, IMFSample ** sample);
 			void ProcessOutput();
 			void DeliverOutput(IMFSample * sample);
 			void ShutdownTransform();
 
 		private:
-			const webrtc::SdpVideoFormat format;
+			const webrtc::VideoCodecType codec;
 			std::string implementationName;
 
 			std::unique_ptr<MFInitializer> mfInitializer;
@@ -123,10 +123,8 @@ namespace jni
 			LONGLONG lastSampleTime;
 			bool keyFrameRequested;
 
-			// Used on the event thread only.
-			std::vector<uint8_t> parameterSets;
-			webrtc::H264BitstreamParser bitstreamParser;
-			webrtc::H264PacketizationMode packetizationMode;
+			// Used on the event thread only, and reset while no events arrive.
+			EncoderOutputProcessor outputProcessor;
 	};
 }
 

webrtc-jni/src/main/cpp/include/media/video/codec/windows/MFVideoEncoderFactory.h

@@ -27,14 +27,15 @@
 
 namespace jni
 {
-	// Creates the Media Foundation hardware encoders of the GPU. It offers
-	// H.264 in the profiles WebRTC's software encoder offers too, and only
-	// with packetization mode 1: mode 0 needs each NAL unit to fit a packet,
-	// which hardware encoders cannot be relied on to keep to.
+	// Creates the Media Foundation hardware encoders of the GPU, for H.264 and
+	// AV1, whichever the GPU has. It offers them in the formats WebRTC's
+	// software encoders offer too: H.264 only with packetization mode 1, since
+	// mode 0 needs each NAL unit to fit a packet, which hardware encoders
+	// cannot be relied on to keep to, and AV1 in profile 0.
 	class MFVideoEncoderFactory : public webrtc::VideoEncoderFactory
 	{
 		public:
-			// Returns a factory, or null if there is no hardware H.264
+			// Returns a factory, or null if there is no hardware H.264 or AV1
 			// encoder on this system.
 			static std::unique_ptr<MFVideoEncoderFactory> Create();
 
@@ -45,7 +46,11 @@ namespace jni
 				const webrtc::SdpVideoFormat & format) override;
 
 		private:
-			MFVideoEncoderFactory() = default;
+			MFVideoEncoderFactory(bool h264, bool av1);
+
+		private:
+			const bool h264;
+			const bool av1;
 	};
 }