feat: add a hardware video decoder factory, with Media Foundation on Windows - #315
Merged
Merged
Conversation
…Windows HardwareVideoDecoderFactory puts the decoders of the GPU in front of the built-in ones where the platform has them, the way HardwareVideoEncoderFactory does for encoders. DefaultVideoDecoderFactory, and a PeerConnectionFactory without a decoder factory, decode as before. On Windows, H.264 and AV1 are decoded by the Media Foundation decoders of Windows on a Direct3D 11 device, so through DXVA on the GPU. A codec is offered only where the GPU has its DXVA decode profile and Windows has a decoder that uses Direct3D 11 (the AV1 Video Extension for AV1). The decoders run synchronously in low-latency mode; each decoded frame is copied from its texture into a staging texture, read back and converted from NV12 to I420, the size taken from the display aperture. A decoder whose output is not in GPU memory decodes in software, which WebRTC's own decoders do better, so it gives up then, as on any failure. FallbackVideoDecoder switches to the software decoder when the hardware one fails to configure or gives up while decoding; the software decoder starts with the frame at hand, and the receiver asks for a key frame if it cannot decode it. The hardware decoders offer the formats the software decoders offer, so negotiation does not change. Linux has no hardware decoders yet; on macOS both factories use VideoToolbox.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #314 (then #313, #312, #310, #303): this PR targets its branch and should be merged after it.
Adds
HardwareVideoDecoderFactory, which decodes on the GPU where the platform supports it. It is the decoder counterpart ofHardwareVideoEncoderFactory, and opt-in in the same way:DefaultVideoDecoderFactory, and aPeerConnectionFactorywithout a decoder factory, decode as before.HardwareVideoDecoderFactoryDefaultVideoDecoderFactory, all software; VA-API and NVDEC are to comeDefaultVideoDecoderFactory: VideoToolboxBehavior
D3D11_DECODER_PROFILE_H264_VLD_NOFGT,D3D11_DECODER_PROFILE_AV1_VLD_PROFILE0, with NV12 output) and Windows has a decoder transform that uses Direct3D 11. For AV1 that's the AV1 Video Extension, which Windows 11 includes.FallbackVideoDecoderswitches to the software decoder when the hardware one fails to configure or fails while decoding. The software decoder starts with the frame at hand; if that isn't a key frame, it fails and the receiver asks the sender for a key frame.decoderImplementationstat ofinbound-rtp, e.g.MediaFoundation (Microsoft H264 Video Decoder MFT)orMediaFoundation (AV1VideoExtension).Native side
MFVideoDecodercreates a Direct3D 11 device with video support and multithread protection, and hands it to a synchronous decoder transform through anIMFDXGIDeviceManager. It decodes in low-latency mode, so each frame comes out right after it goes in....CodecFactories.cpp.d3d11.libanddxguid.libon Windows.Testing
HardwareVideoDecoderIntegrationTeston Windows with an AMD Radeon RX 9070 XT, with-Dwebrtc.test.hardwareDecoder=true -Dwebrtc.test.hardwareAv1Decoder=true:MediaFoundation (Microsoft H264 Video Decoder MFT), AV1 asMediaFoundation (AV1VideoExtension)mvn -pl webrtc test: 201 tests pass.-Pjni-check: 201 tests pass, noFATAL ERROR in native method.g++ -std=c++20 -Wall -Wextra -fsyntax-onlyin WSL.Not covered:
Docs
The video codecs guide (
docs/guide/advanced/video-codecs.md) has a new "Hardware Decoding" section.