feat: add a hardware video decoder factory, with Media Foundation on Windows

docs/guide/advanced/video-codecs.md

@@ -38,6 +38,27 @@ The hardware encoders take over H.264 Constrained Baseline and Baseline with pac
 
 Which encoder a stream uses shows in the `encoderImplementation` statistic of its `outbound-rtp` stats, e.g. `NVENC (NVIDIA GeForce RTX 4070)`, `MediaFoundation (AMDav1Encoder)`, `VA-API (Intel iHD driver ...)`, `OpenH264` or `libaom`.
 
+### Hardware Decoding
+
+`HardwareVideoDecoderFactory` does the same for decoding:
+
+```java
+PeerConnectionFactory factory = PeerConnectionFactory.builder()
+        .setVideoEncoderFactory(new HardwareVideoEncoderFactory())
+        .setVideoDecoderFactory(new HardwareVideoDecoderFactory())
+        .build();
+```
+
+| Platform | Hardware decoding |
+|---|---|
+| Windows | H.264 and AV1, on GPUs that decode them, through the Media Foundation decoders of Windows on Direct3D 11 (DXVA) |
+| Linux | Not yet; decoding is in software |
+| macOS | VideoToolbox, as with `DefaultVideoDecoderFactory` |
+
+AV1 on Windows needs the *AV1 Video Extension*, which Windows 11 includes. Decoded frames are copied from GPU memory back to system memory, where WebRTC's frames are, so hardware decoding pays off mostly at high resolutions and with many streams; at low resolutions WebRTC's software decoders are about as cheap. A hardware decoder that fails, or turns out to decode in software, is replaced by the software decoder, which starts with the next key frame.
+
+Which decoder a stream uses shows in the `decoderImplementation` statistic of its `inbound-rtp` stats, e.g. `MediaFoundation (Microsoft H264 Video Decoder MFT)` or `MediaFoundation (AV1VideoExtension)`.
+
 ### Native Codecs
 
 The encoders and decoders these factories create are `NativeVideoEncoder`s and `NativeVideoDecoder`s. They are placeholders that make WebRTC create the built-in codec, which then runs entirely inside WebRTC, so their methods are not to be called from Java.
@@ -189,7 +210,7 @@ Whatever an encoder, decoder or factory method throws is logged and treated as `
 - `VideoEncoderFactory`, `VideoDecoderFactory` — create the codecs of a factory.
 - `VideoEncoder`, `VideoDecoder`, `EncodedImage` — codecs implemented in Java.
 - `DefaultVideoEncoderFactory`, `DefaultVideoDecoderFactory` — the built-in codecs.
-- `HardwareVideoEncoderFactory` — the built-in encoders, on the GPU where possible.
+- `HardwareVideoEncoderFactory`, `HardwareVideoDecoderFactory` — the built-in codecs, on the GPU where possible.
 - `RTCRtpTransceiver.setCodecPreferences()` — chooses among the negotiated codecs.
 
 For the full API, see the JavaDoc of the `dev.onvoid.webrtc.media.video.codec` package.

webrtc-jni/src/main/cpp/CMakeLists.txt

@@ -135,7 +135,7 @@ elseif(LINUX)
     target_link_libraries(${PROJECT_NAME} ${CXX_LIBS})
     target_link_libraries(${PROJECT_NAME} dl)
 elseif(WIN32)
-    target_link_libraries(${PROJECT_NAME} dwmapi.lib mf.lib mfreadwrite.lib mfplat.lib mfuuid.lib shcore.lib)
+    target_link_libraries(${PROJECT_NAME} d3d11.lib dwmapi.lib dxguid.lib mf.lib mfreadwrite.lib mfplat.lib mfuuid.lib shcore.lib)
 endif()
 
 install(TARGETS ${PROJECT_NAME}

webrtc-jni/src/main/cpp/include/JNI_HardwareVideoDecoderFactory.h

@@ -0,0 +1,36 @@
+/*
+ * Copyright 2026 Alex Andres
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include <jni.h>
+/* Header for class dev_onvoid_webrtc_media_video_codec_HardwareVideoDecoderFactory */
+
+#ifndef _Included_dev_onvoid_webrtc_media_video_codec_HardwareVideoDecoderFactory
+#define _Included_dev_onvoid_webrtc_media_video_codec_HardwareVideoDecoderFactory
+#ifdef __cplusplus
+extern "C" {
+#endif
+	/*
+	 * Class:     dev_onvoid_webrtc_media_video_codec_HardwareVideoDecoderFactory
+	 * Method:    getSupportedCodecsInternal
+	 * Signature: ()[Ldev/onvoid/webrtc/media/video/codec/VideoCodecInfo;
+	 */
+	JNIEXPORT jobjectArray JNICALL Java_dev_onvoid_webrtc_media_video_codec_HardwareVideoDecoderFactory_getSupportedCodecsInternal
+	(JNIEnv *, jclass);
+
+#ifdef __cplusplus
+}
+#endif
+#endif

webrtc-jni/src/main/cpp/include/media/video/codec/DefaultVideoCodecFactories.h

@@ -34,6 +34,11 @@ namespace jni
 	// macOS the default encoders already use VideoToolbox.
 	std::unique_ptr<webrtc::VideoEncoderFactory> CreateHardwareVideoEncoderFactory();
 	std::unique_ptr<webrtc::VideoDecoderFactory> CreateDefaultVideoDecoderFactory();
+
+	// The default decoders, with the decoders of the GPU in front of them
+	// where the platform has them, falling back to the software ones. On
+	// macOS the default decoders already use VideoToolbox.
+	std::unique_ptr<webrtc::VideoDecoderFactory> CreateHardwareVideoDecoderFactory();
 }
 
 #endif

webrtc-jni/src/main/cpp/include/media/video/codec/FallbackVideoDecoder.h

@@ -0,0 +1,71 @@
+/*
+ * Copyright 2026 Alex Andres
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef JNI_WEBRTC_MEDIA_VIDEO_CODEC_FALLBACK_VIDEO_DECODER_H_
+#define JNI_WEBRTC_MEDIA_VIDEO_CODEC_FALLBACK_VIDEO_DECODER_H_
+
+#include "api/video/encoded_image.h"
+#include "api/video_codecs/video_decoder.h"
+
+#include <memory>
+#include <optional>
+
+namespace jni
+{
+	// Decodes with a hardware decoder, and switches to a software decoder of
+	// the same codec when the hardware one fails to configure, or gives up
+	// while decoding by returning WEBRTC_VIDEO_CODEC_FALLBACK_SOFTWARE. The
+	// switch is for good. The software decoder starts with the frame the
+	// hardware one gave up on; if that is no key frame, it fails to decode
+	// it, and the receiver asks the sender for a key frame.
+	//
+	// WebRTC has such a wrapper too, but it is not part of the WebRTC
+	// library this library links.
+	class FallbackVideoDecoder : public webrtc::VideoDecoder
+	{
+		public:
+			FallbackVideoDecoder(std::unique_ptr<webrtc::VideoDecoder> hardware,
+				std::unique_ptr<webrtc::VideoDecoder> software);
+			~FallbackVideoDecoder() override = default;
+
+			using webrtc::VideoDecoder::Decode;
+
+			bool Configure(const Settings & settings) override;
+			int32_t Decode(const webrtc::EncodedImage & image, int64_t renderTimeMs) override;
+			int32_t RegisterDecodeCompleteCallback(webrtc::DecodedImageCallback * callback) override;
+			int32_t Release() override;
+			DecoderInfo GetDecoderInfo() const override;
+			const char * ImplementationName() const override;
+
+		private:
+			// Configures the software decoder with the settings the hardware
+			// one was given. Returns whether it is ready.
+			bool StartSoftware();
+
+			webrtc::VideoDecoder * Active() const;
+
+		private:
+			const std::unique_ptr<webrtc::VideoDecoder> hardware;
+			const std::unique_ptr<webrtc::VideoDecoder> software;
+
+			bool useSoftware;
+
+			std::optional<Settings> settings;
+			webrtc::DecodedImageCallback * callback;
+	};
+}
+
+#endif

webrtc-jni/src/main/cpp/include/media/video/codec/HardwareVideoDecoderFactory.h

@@ -0,0 +1,57 @@
+/*
+ * Copyright 2026 Alex Andres
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef JNI_WEBRTC_MEDIA_VIDEO_CODEC_HARDWARE_VIDEO_DECODER_FACTORY_H_
+#define JNI_WEBRTC_MEDIA_VIDEO_CODEC_HARDWARE_VIDEO_DECODER_FACTORY_H_
+
+#include "api/environment/environment.h"
+#include "api/video_codecs/sdp_video_format.h"
+#include "api/video_codecs/video_decoder.h"
+#include "api/video_codecs/video_decoder_factory.h"
+
+#include <memory>
+#include <vector>
+
+namespace jni
+{
+	// Puts hardware decoders in front of software ones, the way
+	// HardwareVideoEncoderFactory does with encoders: a codec is decoded by
+	// the first hardware factory that has it, falling back to the next and
+	// finally to the software decoder.
+	class HardwareVideoDecoderFactory : public webrtc::VideoDecoderFactory
+	{
+		public:
+			// The hardware factories are in order of preference.
+			HardwareVideoDecoderFactory(std::vector<std::unique_ptr<webrtc::VideoDecoderFactory>> hardware,
+				std::unique_ptr<webrtc::VideoDecoderFactory> software);
+			~HardwareVideoDecoderFactory() override = default;
+
+			std::vector<webrtc::SdpVideoFormat> GetSupportedFormats() const override;
+			std::unique_ptr<webrtc::VideoDecoder> Create(const webrtc::Environment & env,
+				const webrtc::SdpVideoFormat & format) override;
+
+		private:
+			const std::vector<std::unique_ptr<webrtc::VideoDecoderFactory>> hardware;
+			const std::unique_ptr<webrtc::VideoDecoderFactory> software;
+	};
+
+	// Returns the factories for the hardware decoders of the platform that are
+	// available, in order of preference; none if the platform has none this
+	// library supports, or no device that can decode. Defined per platform.
+	std::vector<std::unique_ptr<webrtc::VideoDecoderFactory>> CreatePlatformHardwareVideoDecoderFactories();
+}
+
+#endif

webrtc-jni/src/main/cpp/include/media/video/codec/VideoDecoderFactoryWrapper.h

@@ -59,12 +59,15 @@ namespace jni
 
 					jclass nativeDecoderClass;
 					jfieldID nativeDecoderCodecInfo;
+					jfieldID nativeDecoderHardwareAcceleration;
 			};
 
 		private:
 			const JavaGlobalRef<jobject> factory;
 			const std::shared_ptr<JavaVideoDecoderFactoryClass> javaClass;
+			// The built-in decoders, and those with the GPU's in front.
 			const std::unique_ptr<webrtc::VideoDecoderFactory> defaultFactory;
+			const std::unique_ptr<webrtc::VideoDecoderFactory> hardwareFactory;
 
 			std::vector<webrtc::SdpVideoFormat> supportedFormats;
 	};

webrtc-jni/src/main/cpp/include/media/video/codec/windows/MFDecoderUtils.h

@@ -0,0 +1,55 @@
+/*
+ * Copyright 2026 Alex Andres
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef JNI_WEBRTC_MEDIA_VIDEO_CODEC_MF_DECODER_UTILS_H_
+#define JNI_WEBRTC_MEDIA_VIDEO_CODEC_MF_DECODER_UTILS_H_
+
+#include <d3d11.h>
+#include <mfapi.h>
+#include <mfidl.h>
+#include <mftransform.h>
+#include <wrl/client.h>
+
+#include <string>
+#include <vector>
+
+namespace jni
+{
+	// Creates a Direct3D 11 device on the default adapter that video can be
+	// decoded on, protected for use from several threads, as a decoder
+	// transform and the thread reading its output both use it.
+	HRESULT CreateVideoDevice(Microsoft::WRL::ComPtr<ID3D11Device> & device,
+		Microsoft::WRL::ComPtr<ID3D11DeviceContext> & context);
+
+	// Whether the GPU of the device decodes the given DXVA profile, such as
+	// D3D11_DECODER_PROFILE_H264_VLD_NOFGT, in hardware.
+	bool SupportsDecoderProfile(ID3D11Device * device, const GUID & profile);
+
+	// Lists the decoder transforms for the given video format, such as
+	// MFVideoFormat_H264, best first. These are the synchronous transforms of
+	// Windows, which decode on the GPU through DXVA when given a Direct3D
+	// device. Media Foundation has to be started.
+	HRESULT EnumerateDecoders(const GUID & format, std::vector<Microsoft::WRL::ComPtr<IMFActivate>> & decoders);
+
+	// Activates the first of the decoders that can decode on a Direct3D 11
+	// device, and gives it the device manager. The activation object of the
+	// transform is returned with it, to shut it down with.
+	HRESULT ActivateDirect3DDecoder(const std::vector<Microsoft::WRL::ComPtr<IMFActivate>> & decoders,
+		IMFDXGIDeviceManager * manager, Microsoft::WRL::ComPtr<IMFActivate> & activate,
+		Microsoft::WRL::ComPtr<IMFTransform> & transform);
+}
+
+#endif