Skip to content

feat: encode AV1 on the GPU with Media Foundation and NVENC - #314

Merged
devopvoid merged 1 commit into
feat/vaapi-encoderfrom
feat/av1-hardware-encoder
Oct 1, 2026
Merged

devopvoid merged 1 commit into
feat/vaapi-encoderfrom
feat/av1-hardware-encoder

Conversation

@devopvoid

Copy link
Copy Markdown
Owner

Stacked on #313 (then #312, #310, #303): this PR targets its branch and should be merged after it.

HardwareVideoEncoderFactory now encodes AV1 in hardware where the GPU has an AV1 encoder. The Java API is unchanged.

Platform AV1 in hardware
Windows NVENC on NVIDIA RTX 40 and newer, otherwise the Media Foundation encoder of the GPU driver (e.g. AMD RDNA3 and newer, Intel Arc)
Linux NVENC on NVIDIA RTX 40 and newer; not through VA-API yet (see below)
macOS Software (libaom), as before

Behavior

  • Negotiation: the hardware encoders offer AV1 profile 0, the format libaom offers. Negotiation doesn't change, and a GPU without an AV1 encoder simply leaves AV1 to libaom.
  • Scalability modes: hardware encoders produce a single layer (L1T1). A stream that asks for another scalability mode fails to initialize the hardware encoder, and FallbackVideoEncoder hands it to libaom.
  • Frame dependencies: WebRTC sends AV1 frame dependencies in the dependency descriptor, taken from the encoder's generic frame info. For hardware encoders this is filled like a stream without layers (ScalableVideoControllerNoLayering), with the template structure on key frames.
  • Sequence header: the AV1 sequence header OBU is kept and put back in front of a key frame that lacks one, after the temporal delimiter, just as the H.264 SPS/PPS are.

Changes

  • EncoderOutputProcessor is new. It holds the bitstream handling all hardware encoders share: parameter sets (H.264 SPS/PPS, AV1 sequence header), key frame detection, QP (parsed for H.264; none for AV1, so quality scaling is off there) and codec-specific info. It parses AV1 OBUs in the low overhead format, and a key frame with no parameter sets to fall back on fails, so the stream goes to software. The Media Foundation, NVENC and VA-API encoders now all use it, which removes three copies of the H.264 handling.
  • Media Foundation: MFH264Encoder becomes MFVideoEncoder and serves H.264 and AV1 (MFVideoFormat_AV1, eAVEncAV1VProfile_Main_420_8). The factory offers each codec a hardware MFT exists for.
  • NVENC: NvencH264Encoder becomes NvencVideoEncoder and serves H.264 and AV1 (Main profile, sequence header repeated on every key frame, low overhead output). At load, NvencLibrary opens one encode session to ask which codecs the GPU supports, since NVENC only answers that inside a session.
  • VA-API: the H.264 encoder uses the shared processor. There's no AV1 here yet: VA-API drivers generally expect the application to write the AV1 sequence and frame header OBUs itself, a hand-written bitstream that should be done with hardware to test against.

Testing

  • Real hardware, AV1 and H.264: on Windows with an AMD Radeon RX 9070 XT, HardwareVideoEncoderIntegrationTest with -Dwebrtc.test.hardwareEncoder=true -Dwebrtc.test.hardwareAv1Encoder=true encodes AV1 as MediaFoundation (AMDav1Encoder) and H.264 as MediaFoundation (AMDh264Encoder). The receiving side decodes both (AV1 with dav1d).
  • Test suites: mvn -pl webrtc test: 198 tests pass. -Pjni-check: 198 tests pass, with no FATAL ERROR in native method.
  • Linux sources: the NVENC, VA-API and shared sources pass g++ -std=c++20 -Wall -Wextra -fsyntax-only in WSL against the vendored headers. CI (clang) will be the first full Linux build.
  • New test: hardwareEncodesAv1 is skipped where the GPU has no AV1 encoder, unless -Dwebrtc.test.hardwareAv1Encoder=true is set.

Needs testing

  • NVENC AV1 on an RTX 40 or newer, on Windows and Linux. The encoderImplementation stat should read NVENC (...) for AV1.
  • The fallback to libaom when SVC is requested. It can't be triggered from Java yet, because RTCRtpEncodingParameters has no scalability mode.

HardwareVideoEncoderFactory now encodes AV1 in hardware where the GPU has
an AV1 encoder: with NVENC on NVIDIA RTX 40 and newer, on Windows and
Linux, and on Windows otherwise with the Media Foundation encoder of the
driver, e.g. on AMD RDNA3 and newer and Intel Arc. The encoders offer AV1
profile 0, the format libaom offers, so negotiation does not change.

Hardware encoders produce a single layer. A stream that asks for another
scalability mode fails to initialize the hardware encoder, which hands it
to libaom. WebRTC sends the frame dependencies of AV1 in the dependency
descriptor, taken from the generic frame info, which is filled like that
of a stream without layers.

The Media Foundation and NVENC encoders now serve both codecs, and the
bitstream handling all hardware encoders shared is one class,
EncoderOutputProcessor: it keeps the parameter sets, the SPS and PPS of
H.264 or the sequence header OBU of AV1, and puts them back in front of a
key frame that lacks them, and it fills the QP and the codec specific
information. NVENC asks an encode session once which codecs the GPU
supports.
@devopvoid
devopvoid added this pull request to stack #311 October 1, 2026 22:04
@devopvoid
devopvoid merged commit e7fcc74 into main Oct 1, 2026
17 checks passed
@devopvoid
devopvoid deleted the feat/av1-hardware-encoder branch October 1, 2026 22:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant