Skip to content

feat: add a hardware video decoder factory, with Media Foundation on Windows - #315

Merged
devopvoid merged 2 commits into
feat/av1-hardware-encoderfrom
feat/hardware-decoders
Oct 1, 2026
Merged

devopvoid merged 2 commits into
feat/av1-hardware-encoderfrom
feat/hardware-decoders

Conversation

@devopvoid

Copy link
Copy Markdown
Owner

Stacked on #314 (then #313, #312, #310, #303): this PR targets its branch and should be merged after it.

Adds HardwareVideoDecoderFactory, which decodes on the GPU where the platform supports it. It is the decoder counterpart of HardwareVideoEncoderFactory, and opt-in in the same way:

PeerConnectionFactory factory = PeerConnectionFactory.builder()
        .setVideoEncoderFactory(new HardwareVideoEncoderFactory())
        .setVideoDecoderFactory(new HardwareVideoDecoderFactory())
        .build();

DefaultVideoDecoderFactory, and a PeerConnectionFactory without a decoder factory, decode as before.

Platform HardwareVideoDecoderFactory
Windows H.264 and AV1 on the GPU, through the Media Foundation decoders of Windows on Direct3D 11 (DXVA), falling back to FFmpeg and dav1d
Linux Same as DefaultVideoDecoderFactory, all software; VA-API and NVDEC are to come
macOS Same as DefaultVideoDecoderFactory: VideoToolbox

Behavior

  • Codec availability: a codec is offered only where the GPU has its DXVA decode profile (D3D11_DECODER_PROFILE_H264_VLD_NOFGT, D3D11_DECODER_PROFILE_AV1_VLD_PROFILE0, with NV12 output) and Windows has a decoder transform that uses Direct3D 11. For AV1 that's the AV1 Video Extension, which Windows 11 includes.
  • Negotiation: the hardware decoders offer the formats the software decoders offer (H.264 in all profiles, AV1 profile 0), so negotiation doesn't change. AV1 profile 1 stays with dav1d.
  • Fallback: FallbackVideoDecoder switches to the software decoder when the hardware one fails to configure or fails while decoding. The software decoder starts with the frame at hand; if that isn't a key frame, it fails and the receiver asks the sender for a key frame.
  • Diagnostics: the decoder in use shows in the decoderImplementation stat of inbound-rtp, e.g. MediaFoundation (Microsoft H264 Video Decoder MFT) or MediaFoundation (AV1VideoExtension).

Native side

  • MFVideoDecoder creates a Direct3D 11 device with video support and multithread protection, and hands it to a synchronous decoder transform through an IMFDXGIDeviceManager. It decodes in low-latency mode, so each frame comes out right after it goes in.
  • Readback: each decoded frame is a texture. It's copied into a staging texture, mapped, and converted from NV12 to I420, cropped to the display aperture (the texture may be taller, e.g. 1088 rows for 1080).
  • GPU check: if the output isn't a DXGI buffer, the transform decoded in software, which FFmpeg and dav1d do better. The decoder then gives up and the software decoder takes over.
  • Refactor: the platform files now define both the encoder and the decoder factories, so they're renamed ...CodecFactories.cpp.
  • Build: links d3d11.lib and dxguid.lib on Windows.

Testing

  • HardwareVideoDecoderIntegrationTest on Windows with an AMD Radeon RX 9070 XT, with -Dwebrtc.test.hardwareDecoder=true -Dwebrtc.test.hardwareAv1Decoder=true:
    • H.264 decodes as MediaFoundation (Microsoft H264 Video Decoder MFT), AV1 as MediaFoundation (AV1VideoExtension)
    • both output GPU textures, and every frame arrives at 320x240
    • the default and hardware factories list the same codecs
  • mvn -pl webrtc test: 201 tests pass. -Pjni-check: 201 tests pass, no FATAL ERROR in native method.
  • The platform-independent and Linux sources pass g++ -std=c++20 -Wall -Wextra -fsyntax-only in WSL.

Not covered:

  • Padded frames: the test call is 320x240, a multiple of 16, so it doesn't exercise a decoded texture larger than the picture. The crop to the display aperture is untested.
  • Cost at low resolutions: the copy back from the GPU costs something, so hardware decoding pays off mainly at high resolutions and with many streams. The guide says so.

Docs

The video codecs guide (docs/guide/advanced/video-codecs.md) has a new "Hardware Decoding" section.

…Windows

HardwareVideoDecoderFactory puts the decoders of the GPU in front of the
built-in ones where the platform has them, the way
HardwareVideoEncoderFactory does for encoders. DefaultVideoDecoderFactory,
and a PeerConnectionFactory without a decoder factory, decode as before.

On Windows, H.264 and AV1 are decoded by the Media Foundation decoders of
Windows on a Direct3D 11 device, so through DXVA on the GPU. A codec is
offered only where the GPU has its DXVA decode profile and Windows has a
decoder that uses Direct3D 11 (the AV1 Video Extension for AV1). The
decoders run synchronously in low-latency mode; each decoded frame is
copied from its texture into a staging texture, read back and converted
from NV12 to I420, the size taken from the display aperture. A decoder
whose output is not in GPU memory decodes in software, which WebRTC's own
decoders do better, so it gives up then, as on any failure.

FallbackVideoDecoder switches to the software decoder when the hardware
one fails to configure or gives up while decoding; the software decoder
starts with the frame at hand, and the receiver asks for a key frame if it
cannot decode it. The hardware decoders offer the formats the software
decoders offer, so negotiation does not change. Linux has no hardware
decoders yet; on macOS both factories use VideoToolbox.
@devopvoid
devopvoid added this pull request to stack #311 October 1, 2026 22:04
@devopvoid
devopvoid merged commit 3b4fbf0 into main Oct 1, 2026
17 checks passed
@devopvoid
devopvoid deleted the feat/hardware-decoders branch October 1, 2026 22:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant