Skip to content

feat: add a hardware video encoder factory, with Media Foundation H.264 - #310

Merged
devopvoid merged 1 commit into
feat/custom-video-codec-factoriesfrom
feat/mf-hardware-encoder
Oct 1, 2026
Merged

devopvoid merged 1 commit into
feat/custom-video-codec-factoriesfrom
feat/mf-hardware-encoder

Conversation

@devopvoid

@devopvoid devopvoid commented Sep 30, 2026 •

Copy link
Copy Markdown
Owner

Stacked on #303: this PR targets its branch and should be merged after it.

Adds HardwareVideoEncoderFactory, which encodes on the GPU where the platform supports it. On Windows it encodes H.264 through the Media Foundation encoder of the GPU driver. Hardware encoding is opt-in:

PeerConnectionFactory factory = PeerConnectionFactory.builder()
        .setVideoEncoderFactory(new HardwareVideoEncoderFactory())
        .build();

DefaultVideoEncoderFactory is unchanged, and so is a PeerConnectionFactory without an encoder factory.

Behavior

Platform HardwareVideoEncoderFactory
Windows H.264 through the GPU driver's Media Foundation encoder (NVIDIA, AMD, Intel), falling back to OpenH264
macOS Same as DefaultVideoEncoderFactory: H.264 through VideoToolbox
Linux Same as DefaultVideoEncoderFactory, all software; VA-API and NVENC are planned
  • Negotiation: the hardware encoder only takes over formats the software encoder also offers (H.264 Constrained Baseline and Baseline, packetization mode 1). The factory therefore lists the same codecs as DefaultVideoEncoderFactory, and what gets negotiated never depends on the GPU.
  • Fallback: if the hardware encoder fails to start (e.g. no encoder sessions left) or fails while encoding, the stream switches to the software encoder and restarts with a key frame. On a machine without a hardware encoder, the factory encodes like the default one.
  • Diagnostics: the encoder in use shows in the encoderImplementation stat of outbound-rtp, e.g. MediaFoundation (AMDh264Encoder).

Native side

  • MFH264Encoder drives the asynchronous hardware transform through its event model. MFTransformEvents pumps the events on a Media Foundation thread, which is also where encoded frames are handed to WebRTC.
    • Frames go in as NV12 in system memory (converted with libyuv).
    • Settings through ICodecAPI: low-latency mode, no B-frames, CBR, a forced IDR frame when WebRTC requests a key frame, and bitrate changes while encoding.
    • The SPS/PPS are put in front of every key frame; encoders don't reliably repeat them.
    • A frame is dropped (and reported as dropped) if the encoder hasn't asked for input within 20 ms.
  • FallbackVideoEncoder handles the switch to software. WebRTC's own software fallback wrapper isn't part of the prebuilt webrtc.lib. Adding it would mean a new WebRTC build target, and with it a full WebRTC rebuild on every platform, for one small wrapper.
  • HardwareVideoEncoderFactory (native) puts a platform's hardware encoders in front of the software ones. Platforms plug in through CreatePlatformHardwareVideoEncoderFactory(); Linux has a stub for now.

Testing

  • HardwareVideoEncoderIntegrationTest:

    • the default factory encodes H.264 with OpenH264
    • the hardware factory encodes H.264 on the GPU
    • both factories list the same codecs

    Without a hardware encoder the GPU test is skipped, as on CI runners. -Dwebrtc.test.hardwareEncoder=true makes it fail instead.

  • On Windows with an AMD Radeon RX 9070 XT, with that property set, frames go through MediaFoundation (AMDh264Encoder) and decode on the receiving side.

  • mvn -pl webrtc test: 197 tests pass. mvn -pl webrtc test -Pjni-check: 197 tests pass, with no FATAL ERROR in native method.

Not verified yet:

  • the switch to software in the middle of a stream (it can't be forced from a test)
  • NVIDIA and Intel GPUs
  • the Linux and macOS builds, which CI will be the first to compile

Docs

The video codecs guide (docs/guide/advanced/video-codecs.md) has a new "Hardware Encoding" section.

HardwareVideoEncoderFactory puts the encoders of the GPU in front of the
built-in ones where the platform has them. On Windows, H.264 is encoded by
the Media Foundation encoder the GPU driver provides (NVIDIA, AMD, Intel),
driven through its asynchronous event model, with frames passed in system
memory as NV12. Low-latency mode, no B-frames, CBR, forced IDR frames on
request and bitrate changes while encoding are set through ICodecAPI; the
parameter sets are put in front of every key frame.

The hardware encoder takes over only formats the software encoder has too
(H.264 Constrained Baseline and Baseline, packetization mode 1), so the
factory offers the same codecs as DefaultVideoEncoderFactory and hardware
never changes what is negotiated. A FallbackVideoEncoder switches to the
software encoder when the hardware one fails to start or gives up while
encoding, and restarts the stream with a key frame. WebRTC's own fallback
wrapper is not part of the linked WebRTC library.

Hardware encoding is opt-in: DefaultVideoEncoderFactory, and a
PeerConnectionFactory without an encoder factory, encode as before. Linux
has no hardware encoder yet; on macOS both factories use VideoToolbox.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant