feat: encode AV1 on the GPU with Media Foundation and NVENC - #314
Merged
Merged
Conversation
HardwareVideoEncoderFactory now encodes AV1 in hardware where the GPU has an AV1 encoder: with NVENC on NVIDIA RTX 40 and newer, on Windows and Linux, and on Windows otherwise with the Media Foundation encoder of the driver, e.g. on AMD RDNA3 and newer and Intel Arc. The encoders offer AV1 profile 0, the format libaom offers, so negotiation does not change. Hardware encoders produce a single layer. A stream that asks for another scalability mode fails to initialize the hardware encoder, which hands it to libaom. WebRTC sends the frame dependencies of AV1 in the dependency descriptor, taken from the generic frame info, which is filled like that of a stream without layers. The Media Foundation and NVENC encoders now serve both codecs, and the bitstream handling all hardware encoders shared is one class, EncoderOutputProcessor: it keeps the parameter sets, the SPS and PPS of H.264 or the sequence header OBU of AV1, and puts them back in front of a key frame that lacks them, and it fills the QP and the codec specific information. NVENC asks an encode session once which codecs the GPU supports.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #313 (then #312, #310, #303): this PR targets its branch and should be merged after it.
HardwareVideoEncoderFactorynow encodes AV1 in hardware where the GPU has an AV1 encoder. The Java API is unchanged.Behavior
FallbackVideoEncoderhands it to libaom.ScalableVideoControllerNoLayering), with the template structure on key frames.Changes
EncoderOutputProcessoris new. It holds the bitstream handling all hardware encoders share: parameter sets (H.264 SPS/PPS, AV1 sequence header), key frame detection, QP (parsed for H.264; none for AV1, so quality scaling is off there) and codec-specific info. It parses AV1 OBUs in the low overhead format, and a key frame with no parameter sets to fall back on fails, so the stream goes to software. The Media Foundation, NVENC and VA-API encoders now all use it, which removes three copies of the H.264 handling.MFH264EncoderbecomesMFVideoEncoderand serves H.264 and AV1 (MFVideoFormat_AV1,eAVEncAV1VProfile_Main_420_8). The factory offers each codec a hardware MFT exists for.NvencH264EncoderbecomesNvencVideoEncoderand serves H.264 and AV1 (Main profile, sequence header repeated on every key frame, low overhead output). At load,NvencLibraryopens one encode session to ask which codecs the GPU supports, since NVENC only answers that inside a session.Testing
HardwareVideoEncoderIntegrationTestwith-Dwebrtc.test.hardwareEncoder=true -Dwebrtc.test.hardwareAv1Encoder=trueencodes AV1 asMediaFoundation (AMDav1Encoder)and H.264 asMediaFoundation (AMDh264Encoder). The receiving side decodes both (AV1 with dav1d).mvn -pl webrtc test: 198 tests pass.-Pjni-check: 198 tests pass, with noFATAL ERROR in native method.g++ -std=c++20 -Wall -Wextra -fsyntax-onlyin WSL against the vendored headers. CI (clang) will be the first full Linux build.hardwareEncodesAv1is skipped where the GPU has no AV1 encoder, unless-Dwebrtc.test.hardwareAv1Encoder=trueis set.Needs testing
encoderImplementationstat should readNVENC (...)for AV1.RTCRtpEncodingParametershas no scalability mode.