feat: decode VP9 on the GPU with Media Foundation on Windows

docs/guide/advanced/video-codecs.md

@@ -51,13 +51,15 @@ PeerConnectionFactory factory = PeerConnectionFactory.builder()
 
 | Platform | Hardware decoding |
 |---|---|
-| Windows | H.264 and AV1, on GPUs that decode them, through the Media Foundation decoders of Windows on Direct3D 11 (DXVA) |
+| Windows | H.264, AV1 and VP9 (profile 0), on GPUs that decode them, through the Media Foundation decoders of Windows on Direct3D 11 (DXVA) |
 | Linux | Not yet; decoding is in software |
-| macOS | VideoToolbox, as with `DefaultVideoDecoderFactory` |
+| macOS | H.264 through VideoToolbox, as with `DefaultVideoDecoderFactory`, and VP9 (profile 0) through VideoToolbox's VP9 decoder, on Macs that have one |
 
-AV1 on Windows needs the *AV1 Video Extension*, which Windows 11 includes. Decoded frames are copied from GPU memory back to system memory, where WebRTC's frames are, so hardware decoding pays off mostly at high resolutions and with many streams; at low resolutions WebRTC's software decoders are about as cheap. A hardware decoder that fails, or turns out to decode in software, is replaced by the software decoder, which starts with the next key frame.
+AV1 on Windows needs the *AV1 Video Extension*, which Windows 11 includes, and VP9 the *VP9 Video Extensions*, a free component of the Microsoft Store that a system may not have; without it, VP9 is decoded in software. Streams with spatial layers (SVC) are decoded in software too, since their layers reach a decoder without the index that says where each ends. Decoded frames are copied from GPU memory back to system memory, where WebRTC's frames are, so hardware decoding pays off mostly at high resolutions and with many streams; at low resolutions WebRTC's software decoders are about as cheap. A hardware decoder that fails, or turns out to decode in software, is replaced by the software decoder, which starts with the next key frame.
 
-Which decoder a stream uses shows in the `decoderImplementation` statistic of its `inbound-rtp` stats, e.g. `MediaFoundation (Microsoft H264 Video Decoder MFT)` or `MediaFoundation (AV1VideoExtension)`.
+On macOS, VP9 decoding in hardware saves CPU: in a measurement on an Apple M2 it took a fraction of the processor time libvpx needs, but multi-threaded libvpx was about as fast on the clock. VideoToolbox decodes VP9 in hardware only where the Mac has a decoder for it, which Apple silicon does; a Mac without one decodes VP9 with libvpx as before. Profile 2 (10 bit), sizes below 64x64 or above 4096x4096, and streams with spatial layers (SVC) are decoded with libvpx too.
+
+Which decoder a stream uses shows in the `decoderImplementation` statistic of its `inbound-rtp` stats, e.g. `MediaFoundation (Microsoft H264 Video Decoder MFT)`, `MediaFoundation (AV1VideoExtension)` or `VideoToolbox (VP9)`.
 
 ### Native Codecs
 

webrtc-jni/src/main/cpp/include/media/video/codec/macos/VTVideoDecoderFactory.h

@@ -0,0 +1,53 @@
+/*
+ * Copyright 2026 Alex Andres
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef JNI_WEBRTC_MEDIA_VIDEO_CODEC_VT_VIDEO_DECODER_FACTORY_H_
+#define JNI_WEBRTC_MEDIA_VIDEO_CODEC_VT_VIDEO_DECODER_FACTORY_H_
+
+#include "api/environment/environment.h"
+#include "api/video_codecs/sdp_video_format.h"
+#include "api/video_codecs/video_decoder.h"
+#include "api/video_codecs/video_decoder_factory.h"
+
+#include <memory>
+#include <vector>
+
+namespace jni
+{
+	// Creates the VideoToolbox decoders that WebRTC's own decoders for macOS
+	// do not offer: VP9, in profile 0, the format the software factory
+	// offers first. H.264 is decoded through VideoToolbox by the default
+	// decoders already.
+	class VTVideoDecoderFactory : public webrtc::VideoDecoderFactory
+	{
+		public:
+			// Returns a factory, or null if this Mac has no hardware decoder
+			// for VP9. The decoder VideoToolbox has for VP9 has to be
+			// registered first; this does it, once for the process.
+			static std::unique_ptr<VTVideoDecoderFactory> Create();
+
+			~VTVideoDecoderFactory() override = default;
+
+			std::vector<webrtc::SdpVideoFormat> GetSupportedFormats() const override;
+			std::unique_ptr<webrtc::VideoDecoder> Create(const webrtc::Environment & env,
+				const webrtc::SdpVideoFormat & format) override;
+
+		private:
+			VTVideoDecoderFactory() = default;
+	};
+}
+
+#endif

webrtc-jni/src/main/cpp/include/media/video/codec/macos/VTVp9Decoder.h

@@ -0,0 +1,117 @@
+/*
+ * Copyright 2026 Alex Andres
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef JNI_WEBRTC_MEDIA_VIDEO_CODEC_VT_VP9_DECODER_H_
+#define JNI_WEBRTC_MEDIA_VIDEO_CODEC_VT_VP9_DECODER_H_
+
+#include "api/video/encoded_image.h"
+#include "api/video_codecs/video_decoder.h"
+#include "modules/video_coding/utility/vp9_uncompressed_header_parser.h"
+
+#include <CoreMedia/CoreMedia.h>
+#include <CoreVideo/CoreVideo.h>
+#include <VideoToolbox/VideoToolbox.h>
+
+#include <cstdint>
+
+namespace jni
+{
+	// Decodes VP9 profile 0 on the media engine or GPU of a Mac, through the
+	// VP9 decoder VideoToolbox offers once it is registered.
+	//
+	// The decompression session needs the properties of the stream, so it is
+	// created on the first key frame, from the header the key frame carries,
+	// and again whenever a key frame changes the size or the range. Decoding
+	// is synchronous: each frame comes out of VideoToolbox before Decode
+	// returns. A frame is handed on as the CVPixelBuffer VideoToolbox made,
+	// without a copy.
+	//
+	// Whatever VideoToolbox cannot decode in place goes to the software
+	// decoder, by returning WEBRTC_VIDEO_CODEC_FALLBACK_SOFTWARE: a profile
+	// other than 0, a size outside what the hardware decodes, frames with
+	// spatial layers, and a decoder that keeps failing.
+	class VTVp9Decoder : public webrtc::VideoDecoder
+	{
+		public:
+			VTVp9Decoder();
+			~VTVp9Decoder() override;
+
+			using webrtc::VideoDecoder::Decode;
+
+			bool Configure(const Settings & settings) override;
+			int32_t Decode(const webrtc::EncodedImage & image, int64_t renderTimeMs) override;
+			int32_t RegisterDecodeCompleteCallback(webrtc::DecodedImageCallback * callback) override;
+			int32_t Release() override;
+			DecoderInfo GetDecoderInfo() const override;
+			const char * ImplementationName() const override;
+
+		private:
+			// What a session is created for. A key frame that differs from it
+			// needs a new format description, and often a new session.
+			struct StreamConfig
+			{
+				int width = 0;
+				int height = 0;
+				bool fullRange = false;
+				webrtc::Vp9ColorSpace colorSpace = webrtc::Vp9ColorSpace::CS_UNKNOWN;
+
+				bool operator==(const StreamConfig & other) const;
+			};
+
+			// The result of decoding one frame, filled in by the callback of
+			// the session.
+			struct Output
+			{
+				OSStatus status = noErr;
+				CVImageBufferRef image = nullptr;
+			};
+
+			static void OnOutput(void * decoder, void * frame, OSStatus status, VTDecodeInfoFlags flags,
+				CVImageBufferRef image, CMTime presentationTime, CMTime duration);
+
+			// Reads the stream properties from a key frame header. Returns
+			// false for a stream the hardware decoder does not take.
+			static bool ReadConfig(const webrtc::Vp9UncompressedHeader & header, StreamConfig & config);
+
+			// Makes sure there is a session for the stream, creating or
+			// replacing it where the stream changed.
+			bool EnsureSession(const StreamConfig & config);
+			bool CreateSession(const StreamConfig & config, CMVideoFormatDescriptionRef format);
+			void DestroySession();
+
+			// Decodes one encoded frame as a single sample.
+			OSStatus DecodeSample(const webrtc::EncodedImage & image, Output & output);
+
+			// Counts a failure and tells WebRTC what to do about it.
+			int32_t Fail(OSStatus status);
+
+			void Deliver(const webrtc::EncodedImage & image, int64_t renderTimeMs, CVImageBufferRef pixels);
+
+		private:
+			webrtc::DecodedImageCallback * callback;
+
+			VTDecompressionSessionRef session;
+			CMVideoFormatDescriptionRef format;
+			StreamConfig active;
+
+			// Set after a failure, until the next key frame.
+			bool requireKeyFrame;
+			int consecutiveErrors;
+			int64_t sampleCount;
+	};
+}
+
+#endif

webrtc-jni/src/main/cpp/include/media/video/codec/windows/MFVideoDecoder.h

@@ -38,7 +38,7 @@
 
 namespace jni
 {
-	// Decodes H.264 or AV1 on the GPU, with the decoder transform of Windows
+	// Decodes H.264, AV1 or VP9 on the GPU, with the decoder transform of Windows
 	// for the codec, which decodes through DXVA on the Direct3D 11 device it
 	// is given.
 	//
@@ -52,7 +52,7 @@ namespace jni
 	class MFVideoDecoder : public webrtc::VideoDecoder
 	{
 		public:
-			// The codec is H.264 or AV1.
+			// The codec is H.264, AV1 or VP9.
 			explicit MFVideoDecoder(webrtc::VideoCodecType codec);
 			~MFVideoDecoder() override;
 

webrtc-jni/src/main/cpp/include/media/video/codec/windows/MFVideoDecoderFactory.h

@@ -27,14 +27,15 @@
 
 namespace jni
 {
-	// Creates the Media Foundation decoders that decode on the GPU, for H.264
-	// and AV1, whichever the GPU decodes and Windows has a decoder for that
+	// Creates the Media Foundation decoders that decode on the GPU, for H.264,
+	// AV1 and VP9, whichever the GPU decodes and Windows has a decoder for that
 	// uses Direct3D 11. It offers them in the formats WebRTC's software
-	// decoders offer too: H.264 in all its profiles, and AV1 in profile 0.
+	// decoders offer too: H.264 in all its profiles, and AV1 and VP9 in
+	// profile 0.
 	class MFVideoDecoderFactory : public webrtc::VideoDecoderFactory
 	{
 		public:
-			// Returns a factory, or null if the GPU decodes neither codec.
+			// Returns a factory, or null if the GPU decodes none of the codecs.
 			static std::unique_ptr<MFVideoDecoderFactory> Create();
 
 			~MFVideoDecoderFactory() override = default;
@@ -44,11 +45,12 @@ namespace jni
 				const webrtc::SdpVideoFormat & format) override;
 
 		private:
-			MFVideoDecoderFactory(bool h264, bool av1);
+			MFVideoDecoderFactory(bool h264, bool av1, bool vp9);
 
 		private:
 			const bool h264;
 			const bool av1;
+			const bool vp9;
 	};
 }
 

webrtc-jni/src/main/cpp/src/media/video/codec/DefaultVideoCodecFactories.cpp

@@ -54,17 +54,13 @@ namespace jni
 
 	std::unique_ptr<webrtc::VideoEncoderFactory> CreateHardwareVideoEncoderFactory()
 	{
-#ifdef __APPLE__
-		return CreateDefaultVideoEncoderFactory();
-#else
 		std::vector<std::unique_ptr<webrtc::VideoEncoderFactory>> hardware = CreatePlatformHardwareVideoEncoderFactories();
 
 		if (hardware.empty()) {
 			return CreateDefaultVideoEncoderFactory();
 		}
 
 		return std::make_unique<HardwareVideoEncoderFactory>(std::move(hardware), CreateDefaultVideoEncoderFactory());
-#endif
 	}
 
 	std::unique_ptr<webrtc::VideoDecoderFactory> CreateDefaultVideoDecoderFactory()
@@ -82,16 +78,12 @@ namespace jni
 
 	std::unique_ptr<webrtc::VideoDecoderFactory> CreateHardwareVideoDecoderFactory()
 	{
-#ifdef __APPLE__
-		return CreateDefaultVideoDecoderFactory();
-#else
 		std::vector<std::unique_ptr<webrtc::VideoDecoderFactory>> hardware = CreatePlatformHardwareVideoDecoderFactories();
 
 		if (hardware.empty()) {
 			return CreateDefaultVideoDecoderFactory();
 		}
 
 		return std::make_unique<HardwareVideoDecoderFactory>(std::move(hardware), CreateDefaultVideoDecoderFactory());
-#endif
 	}
 }

webrtc-jni/src/main/cpp/src/media/video/codec/macos/MacHardwareVideoCodecFactories.cpp

@@ -0,0 +1,42 @@
+/*
+ * Copyright 2026 Alex Andres
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "media/video/codec/HardwareVideoDecoderFactory.h"
+#include "media/video/codec/HardwareVideoEncoderFactory.h"
+#include "media/video/codec/macos/VTVideoDecoderFactory.h"
+
+namespace jni
+{
+	std::vector<std::unique_ptr<webrtc::VideoEncoderFactory>> CreatePlatformHardwareVideoEncoderFactories()
+	{
+		// H.264 is encoded through VideoToolbox by the default encoders, and
+		// VideoToolbox has no encoder for the other codecs.
+		return {};
+	}
+
+	std::vector<std::unique_ptr<webrtc::VideoDecoderFactory>> CreatePlatformHardwareVideoDecoderFactories()
+	{
+		std::vector<std::unique_ptr<webrtc::VideoDecoderFactory>> factories;
+
+		// H.264 is decoded through VideoToolbox by the default decoders;
+		// VP9 is the codec they decode in software.
+		if (auto videoToolbox = VTVideoDecoderFactory::Create()) {
+			factories.push_back(std::move(videoToolbox));
+		}
+
+		return factories;
+	}
+}

webrtc-jni/src/main/cpp/src/media/video/codec/macos/VTVideoDecoderFactory.cpp

@@ -0,0 +1,66 @@
+/*
+ * Copyright 2026 Alex Andres
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "media/video/codec/macos/VTVideoDecoderFactory.h"
+#include "media/video/codec/macos/VTVp9Decoder.h"
+
+#include "rtc_base/logging.h"
+
+#include <VideoToolbox/VideoToolbox.h>
+
+namespace jni
+{
+	namespace
+	{
+		// Whether this Mac decodes VP9 in hardware. VideoToolbox has the VP9
+		// decoder only after it is registered, so that comes first. The
+		// answer does not change while the process runs, and is asked once.
+		bool HasHardwareVp9Decoder()
+		{
+			static const bool available = [] {
+				VTRegisterSupplementalVideoDecoderIfAvailable(kCMVideoCodecType_VP9);
+
+				const bool supported = VTIsHardwareDecodeSupported(kCMVideoCodecType_VP9);
+
+				RTC_LOG(LS_INFO) << "VideoToolbox hardware decoder for VP9: " << supported;
+
+				return supported;
+			}();
+
+			return available;
+		}
+	}
+
+	std::unique_ptr<VTVideoDecoderFactory> VTVideoDecoderFactory::Create()
+	{
+		if (!HasHardwareVp9Decoder()) {
+			return nullptr;
+		}
+
+		return std::unique_ptr<VTVideoDecoderFactory>(new VTVideoDecoderFactory());
+	}
+
+	std::vector<webrtc::SdpVideoFormat> VTVideoDecoderFactory::GetSupportedFormats() const
+	{
+		return { webrtc::SdpVideoFormat::VP9Profile0() };
+	}
+
+	std::unique_ptr<webrtc::VideoDecoder> VTVideoDecoderFactory::Create(const webrtc::Environment & env,
+		const webrtc::SdpVideoFormat & format)
+	{
+		return std::make_unique<VTVp9Decoder>();
+	}
+}