From 91ea20522b1ece48a574fb958dcfd46fb91df101 Mon Sep 17 00:00:00 2001 From: Alex Andres Date: Wed, 30 Sep 2026 03:09:09 +0200 Subject: [PATCH] feat: encode AV1 on the GPU with Media Foundation and NVENC HardwareVideoEncoderFactory now encodes AV1 in hardware where the GPU has an AV1 encoder: with NVENC on NVIDIA RTX 40 and newer, on Windows and Linux, and on Windows otherwise with the Media Foundation encoder of the driver, e.g. on AMD RDNA3 and newer and Intel Arc. The encoders offer AV1 profile 0, the format libaom offers, so negotiation does not change. Hardware encoders produce a single layer. A stream that asks for another scalability mode fails to initialize the hardware encoder, which hands it to libaom. WebRTC sends the frame dependencies of AV1 in the dependency descriptor, taken from the generic frame info, which is filled like that of a stream without layers. The Media Foundation and NVENC encoders now serve both codecs, and the bitstream handling all hardware encoders shared is one class, EncoderOutputProcessor: it keeps the parameter sets, the SPS and PPS of H.264 or the sequence header OBU of AV1, and puts them back in front of a key frame that lacks them, and it fills the QP and the codec specific information. NVENC asks an encode session once which codecs the GPU supports. --- docs/guide/advanced/video-codecs.md | 16 +- .../video/codec/EncoderOutputProcessor.h | 87 ++++++ .../video/codec/linux/VaapiH264Encoder.h | 6 +- .../media/video/codec/nvenc/NvencLibrary.h | 10 + ...NvencH264Encoder.h => NvencVideoEncoder.h} | 25 +- .../codec/nvenc/NvencVideoEncoderFactory.h | 7 +- .../video/codec/windows/MFEncoderUtils.h | 7 +- .../{MFH264Encoder.h => MFVideoEncoder.h} | 28 +- .../codec/windows/MFVideoEncoderFactory.h | 17 +- .../video/codec/EncoderOutputProcessor.cpp | 285 ++++++++++++++++++ .../video/codec/linux/VaapiH264Encoder.cpp | 71 +---- .../media/video/codec/nvenc/NvencLibrary.cpp | 67 +++- ...cH264Encoder.cpp => NvencVideoEncoder.cpp} | 140 +++++---- .../codec/nvenc/NvencVideoEncoderFactory.cpp | 22 +- .../video/codec/windows/MFEncoderUtils.cpp | 4 +- .../{MFH264Encoder.cpp => MFVideoEncoder.cpp} | 151 ++++------ .../codec/windows/MFVideoEncoderFactory.cpp | 62 +++- .../codec/HardwareVideoEncoderFactory.java | 4 +- .../HardwareVideoEncoderIntegrationTest.java | 60 +++- 19 files changed, 767 insertions(+), 302 deletions(-) create mode 100644 webrtc-jni/src/main/cpp/include/media/video/codec/EncoderOutputProcessor.h rename webrtc-jni/src/main/cpp/include/media/video/codec/nvenc/{NvencH264Encoder.h => NvencVideoEncoder.h} (77%) rename webrtc-jni/src/main/cpp/include/media/video/codec/windows/{MFH264Encoder.h => MFVideoEncoder.h} (81%) create mode 100644 webrtc-jni/src/main/cpp/src/media/video/codec/EncoderOutputProcessor.cpp rename webrtc-jni/src/main/cpp/src/media/video/codec/nvenc/{NvencH264Encoder.cpp => NvencVideoEncoder.cpp} (81%) rename webrtc-jni/src/main/cpp/src/media/video/codec/windows/{MFH264Encoder.cpp => MFVideoEncoder.cpp} (80%) diff --git a/docs/guide/advanced/video-codecs.md b/docs/guide/advanced/video-codecs.md index 22e1f2d7..8c72898d 100644 --- a/docs/guide/advanced/video-codecs.md +++ b/docs/guide/advanced/video-codecs.md @@ -24,17 +24,19 @@ PeerConnectionFactory factory = PeerConnectionFactory.builder() .build(); ``` -| Platform | Hardware encoding | -|---|---| -| Windows | H.264 with NVENC on NVIDIA GPUs, otherwise through the Media Foundation encoder of the GPU driver (AMD, Intel) | -| Linux | H.264 with NVENC on NVIDIA GPUs, otherwise through the VA-API encoder of the GPU driver (Intel, AMD) | -| macOS | H.264 through VideoToolbox, as with `DefaultVideoEncoderFactory` | +| Platform | H.264 | AV1 | +|---|---|---| +| Windows | NVENC on NVIDIA GPUs, otherwise the Media Foundation encoder of the GPU driver (AMD, Intel) | NVENC on NVIDIA RTX 40 and newer, otherwise Media Foundation on GPUs that encode AV1 (e.g. AMD RDNA3 and newer, Intel Arc) | +| Linux | NVENC on NVIDIA GPUs, otherwise the VA-API encoder of the GPU driver (Intel, AMD) | NVENC on NVIDIA RTX 40 and newer | +| macOS | VideoToolbox, as with `DefaultVideoEncoderFactory` | Software | + +Hardware encoders produce a single layer. A stream that asks for SVC, such as an AV1 stream with a scalability mode like `L1T3`, is encoded in software. NVENC needs an NVIDIA driver of version 522 or newer on Windows, 520 or newer on Linux. VA-API needs libva 2 and a driver that encodes H.264, such as Intel's `intel-media-va-driver` (iHD) or Mesa's `mesa-va-drivers` for AMD, and access to a render node in `/dev/dri`. All of them are loaded at run time, so nothing needs to be installed on machines without them. -The hardware encoders take over H.264 Constrained Baseline and Baseline with packetization mode 1, formats the software encoder offers too, so encoding in hardware never changes what is negotiated. When a hardware encoder fails to start, for example because the GPU has no encoder sessions left, or fails while encoding, the stream switches to the next encoder in line (e.g. from NVENC to Media Foundation on Windows), and finally to the software encoder, and continues with a key frame. On a machine without a hardware encoder, the factory encodes like `DefaultVideoEncoderFactory`. +The hardware encoders take over H.264 Constrained Baseline and Baseline with packetization mode 1, and AV1 profile 0, formats the software encoders offer too, so encoding in hardware never changes what is negotiated. When a hardware encoder fails to start, for example because the GPU has no encoder sessions left, or fails while encoding, the stream switches to the next encoder in line (e.g. from NVENC to Media Foundation on Windows), and finally to the software encoder, and continues with a key frame. On a machine without a hardware encoder, the factory encodes like `DefaultVideoEncoderFactory`. -Which encoder a stream uses shows in the `encoderImplementation` statistic of its `outbound-rtp` stats, e.g. `NVENC (NVIDIA GeForce RTX 4070)`, `MediaFoundation (AMDh264Encoder)`, `VA-API (Intel iHD driver ...)` or `OpenH264`. +Which encoder a stream uses shows in the `encoderImplementation` statistic of its `outbound-rtp` stats, e.g. `NVENC (NVIDIA GeForce RTX 4070)`, `MediaFoundation (AMDav1Encoder)`, `VA-API (Intel iHD driver ...)`, `OpenH264` or `libaom`. ### Native Codecs diff --git a/webrtc-jni/src/main/cpp/include/media/video/codec/EncoderOutputProcessor.h b/webrtc-jni/src/main/cpp/include/media/video/codec/EncoderOutputProcessor.h new file mode 100644 index 00000000..8a4d3141 --- /dev/null +++ b/webrtc-jni/src/main/cpp/include/media/video/codec/EncoderOutputProcessor.h @@ -0,0 +1,87 @@ +/* + * Copyright 2026 Alex Andres + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef JNI_WEBRTC_MEDIA_VIDEO_CODEC_ENCODER_OUTPUT_PROCESSOR_H_ +#define JNI_WEBRTC_MEDIA_VIDEO_CODEC_ENCODER_OUTPUT_PROCESSOR_H_ + +#include "api/video/encoded_image.h" +#include "api/video/video_codec_type.h" +#include "api/video_codecs/sdp_video_format.h" +#include "api/video_codecs/video_encoder.h" +#include "common_video/h264/h264_bitstream_parser.h" +#include "modules/video_coding/codecs/h264/include/h264_globals.h" +#include "modules/video_coding/include/video_codec_interface.h" +#include "modules/video_coding/svc/scalable_video_controller_no_layering.h" + +#include +#include +#include + +namespace jni +{ + // Turns what a hardware encoder produces for a frame into what WebRTC + // sends, the same way for every hardware encoder: H.264 as Annex B NAL + // units, AV1 as OBUs in the low overhead bitstream format. + // + // A receiver can start decoding only from a key frame that carries the + // parameter sets, the SPS and PPS of H.264 or the sequence header of AV1. + // Encoders put them in front of the first key frame, not necessarily in + // front of every one, so they are kept and put back where missing. The + // frame dependencies WebRTC needs for AV1 are those of a stream without + // layers, which is all hardware encoders produce here. + class EncoderOutputProcessor + { + public: + EncoderOutputProcessor(webrtc::VideoCodecType codec, const webrtc::SdpVideoFormat & format); + + // Forgets the parameter sets, for a stream that starts over. + void Reset(); + + // Fills the image with the bitstream of one frame, and the codec + // specific information WebRTC needs to send it. The caller sets + // the size of the image before, which the AV1 frame dependencies + // refer to, and its timestamps. The frame is a key + // frame if the encoder says so or, for H.264, if it holds an IDR + // slice. Returns false if the bitstream cannot be parsed, or a key + // frame lacks parameter sets and there are none to put back. + bool Process(std::span bitstream, bool keyFrame, webrtc::EncodedImage & image, + webrtc::CodecSpecificInfo & info); + + // The quality scaling settings that suit the codec: the thresholds + // of WebRTC's H.264 encoder, and none for AV1, whose QP is not + // parsed from the bitstream. + webrtc::VideoEncoder::ScalingSettings GetScalingSettings() const; + + private: + bool ProcessH264(std::span bitstream, bool & keyFrame, std::vector & output); + bool ProcessAv1(std::span bitstream, bool keyFrame, std::vector & output); + + void FillCodecSpecificInfo(const webrtc::EncodedImage & image, bool keyFrame, + webrtc::CodecSpecificInfo & info); + + private: + const webrtc::VideoCodecType codec; + webrtc::H264PacketizationMode packetizationMode; + + // The SPS and PPS, or the sequence header OBU, last seen. + std::vector parameterSets; + + webrtc::H264BitstreamParser h264Parser; + webrtc::ScalableVideoControllerNoLayering svcController; + }; +} + +#endif diff --git a/webrtc-jni/src/main/cpp/include/media/video/codec/linux/VaapiH264Encoder.h b/webrtc-jni/src/main/cpp/include/media/video/codec/linux/VaapiH264Encoder.h index b7b08ac2..e2d5cc03 100644 --- a/webrtc-jni/src/main/cpp/include/media/video/codec/linux/VaapiH264Encoder.h +++ b/webrtc-jni/src/main/cpp/include/media/video/codec/linux/VaapiH264Encoder.h @@ -17,14 +17,13 @@ #ifndef JNI_WEBRTC_MEDIA_VIDEO_CODEC_VAAPI_H264_ENCODER_H_ #define JNI_WEBRTC_MEDIA_VIDEO_CODEC_VAAPI_H264_ENCODER_H_ +#include "media/video/codec/EncoderOutputProcessor.h" #include "media/video/codec/linux/VaapiLibrary.h" #include "api/video/video_frame.h" #include "api/video_codecs/sdp_video_format.h" #include "api/video_codecs/video_codec.h" #include "api/video_codecs/video_encoder.h" -#include "common_video/h264/h264_bitstream_parser.h" -#include "modules/video_coding/codecs/h264/include/h264_globals.h" #include #include @@ -103,8 +102,7 @@ namespace jni webrtc::EncodedImageCallback * callback; - webrtc::H264BitstreamParser bitstreamParser; - webrtc::H264PacketizationMode packetizationMode; + EncoderOutputProcessor outputProcessor; }; } diff --git a/webrtc-jni/src/main/cpp/include/media/video/codec/nvenc/NvencLibrary.h b/webrtc-jni/src/main/cpp/include/media/video/codec/nvenc/NvencLibrary.h index e59cee25..e95b6e92 100644 --- a/webrtc-jni/src/main/cpp/include/media/video/codec/nvenc/NvencLibrary.h +++ b/webrtc-jni/src/main/cpp/include/media/video/codec/nvenc/NvencLibrary.h @@ -60,6 +60,11 @@ namespace jni // The name of the device, e.g. "NVIDIA GeForce RTX 4070". const std::string & DeviceName() const; + // Whether the device encodes H.264, and AV1, which only newer + // GPUs do. + bool SupportsH264() const; + bool SupportsAv1() const; + // Retains the primary context of the device. Every successful call // has to be matched by ReleaseContext(). bool RetainContext(CUcontext * context) const; @@ -73,6 +78,9 @@ namespace jni bool Load(); + // Asks an encode session which codecs the device supports. + void QueryCodecs(); + private: using CuInit = CUresult (JNI_CUDAAPI *)(unsigned int flags); using CuDeviceGetCount = CUresult (JNI_CUDAAPI *)(int * count); @@ -101,6 +109,8 @@ namespace jni NV_ENCODE_API_FUNCTION_LIST api = {}; CUdevice device = 0; std::string deviceName; + bool h264 = false; + bool av1 = false; }; } diff --git a/webrtc-jni/src/main/cpp/include/media/video/codec/nvenc/NvencH264Encoder.h b/webrtc-jni/src/main/cpp/include/media/video/codec/nvenc/NvencVideoEncoder.h similarity index 77% rename from webrtc-jni/src/main/cpp/include/media/video/codec/nvenc/NvencH264Encoder.h rename to webrtc-jni/src/main/cpp/include/media/video/codec/nvenc/NvencVideoEncoder.h index 4c33d8f7..03db69e2 100644 --- a/webrtc-jni/src/main/cpp/include/media/video/codec/nvenc/NvencH264Encoder.h +++ b/webrtc-jni/src/main/cpp/include/media/video/codec/nvenc/NvencVideoEncoder.h @@ -14,17 +14,17 @@ * limitations under the License. */ -#ifndef JNI_WEBRTC_MEDIA_VIDEO_CODEC_NVENC_H264_ENCODER_H_ -#define JNI_WEBRTC_MEDIA_VIDEO_CODEC_NVENC_H264_ENCODER_H_ +#ifndef JNI_WEBRTC_MEDIA_VIDEO_CODEC_NVENC_VIDEO_ENCODER_H_ +#define JNI_WEBRTC_MEDIA_VIDEO_CODEC_NVENC_VIDEO_ENCODER_H_ +#include "media/video/codec/EncoderOutputProcessor.h" #include "media/video/codec/nvenc/NvencLibrary.h" +#include "api/video/video_codec_type.h" #include "api/video/video_frame.h" #include "api/video_codecs/sdp_video_format.h" #include "api/video_codecs/video_codec.h" #include "api/video_codecs/video_encoder.h" -#include "common_video/h264/h264_bitstream_parser.h" -#include "modules/video_coding/codecs/h264/include/h264_globals.h" #include @@ -34,19 +34,20 @@ namespace jni { - // Encodes H.264 with NVENC, the encoder of NVIDIA GPUs, on the primary - // CUDA context of the device. + // Encodes H.264 or AV1 with NVENC, the encoder of NVIDIA GPUs, on the + // primary CUDA context of the device. // // Encoding is synchronous: with no B-frames and a low-latency preset, // NVENC returns each frame as soon as it is encoded, so frames go in and // out on the encoder thread. Frames are passed in system memory as NV12, // into an input buffer NVENC allocates. Anything that fails makes the // encoder give up, so that the next encoder in line takes over. - class NvencH264Encoder : public webrtc::VideoEncoder + class NvencVideoEncoder : public webrtc::VideoEncoder { public: - NvencH264Encoder(NvencLibrary & library, const webrtc::SdpVideoFormat & format); - ~NvencH264Encoder() override; + // The codec is H.264 or AV1. + NvencVideoEncoder(NvencLibrary & library, webrtc::VideoCodecType codec, const webrtc::SdpVideoFormat & format); + ~NvencVideoEncoder() override; int32_t InitEncode(const webrtc::VideoCodec * codecSettings, const Settings & settings) override; int32_t RegisterEncodeCompleteCallback(webrtc::EncodedImageCallback * callback) override; @@ -58,6 +59,7 @@ namespace jni private: bool OpenSession(); bool Configure(); + void ConfigureCodec(); void ApplyRates(); bool CopyToInput(const webrtc::VideoFrame & frame, uint32_t * pitch); void DestroySession(); @@ -65,6 +67,8 @@ namespace jni private: NvencLibrary & library; const NV_ENCODE_API_FUNCTION_LIST & api; + const webrtc::VideoCodecType codec; + const GUID codecGuid; const std::string implementationName; CUcontext context; @@ -82,8 +86,7 @@ namespace jni webrtc::EncodedImageCallback * callback; - webrtc::H264BitstreamParser bitstreamParser; - webrtc::H264PacketizationMode packetizationMode; + EncoderOutputProcessor outputProcessor; }; } diff --git a/webrtc-jni/src/main/cpp/include/media/video/codec/nvenc/NvencVideoEncoderFactory.h b/webrtc-jni/src/main/cpp/include/media/video/codec/nvenc/NvencVideoEncoderFactory.h index 188f5698..646131a1 100644 --- a/webrtc-jni/src/main/cpp/include/media/video/codec/nvenc/NvencVideoEncoderFactory.h +++ b/webrtc-jni/src/main/cpp/include/media/video/codec/nvenc/NvencVideoEncoderFactory.h @@ -29,9 +29,10 @@ namespace jni { - // Creates NVENC encoders on an NVIDIA GPU. It offers H.264 in the - // profiles WebRTC's software encoder offers too, and only with - // packetization mode 1: mode 0 needs each NAL unit to fit a packet. + // Creates NVENC encoders on an NVIDIA GPU, for H.264 and, on GPUs that + // have it, AV1. It offers them in the formats WebRTC's software encoders + // offer too: H.264 only with packetization mode 1, since mode 0 needs each + // NAL unit to fit a packet, and AV1 in profile 0. class NvencVideoEncoderFactory : public webrtc::VideoEncoderFactory { public: diff --git a/webrtc-jni/src/main/cpp/include/media/video/codec/windows/MFEncoderUtils.h b/webrtc-jni/src/main/cpp/include/media/video/codec/windows/MFEncoderUtils.h index 85cf8e37..f49e30d8 100644 --- a/webrtc-jni/src/main/cpp/include/media/video/codec/windows/MFEncoderUtils.h +++ b/webrtc-jni/src/main/cpp/include/media/video/codec/windows/MFEncoderUtils.h @@ -26,9 +26,10 @@ namespace jni { - // Lists the hardware H.264 encoder transforms that take NV12, best - // first. Media Foundation has to be started. - HRESULT EnumerateHardwareH264Encoders(std::vector> & encoders); + // Lists the hardware encoder transforms that take NV12 and produce the + // given video format, such as MFVideoFormat_H264, best first. Media + // Foundation has to be started. + HRESULT EnumerateHardwareEncoders(const GUID & format, std::vector> & encoders); // Returns the name the driver gives a transform, e.g. "AMDh264Encoder". std::string GetTransformName(IMFActivate * activate); diff --git a/webrtc-jni/src/main/cpp/include/media/video/codec/windows/MFH264Encoder.h b/webrtc-jni/src/main/cpp/include/media/video/codec/windows/MFVideoEncoder.h similarity index 81% rename from webrtc-jni/src/main/cpp/include/media/video/codec/windows/MFH264Encoder.h rename to webrtc-jni/src/main/cpp/include/media/video/codec/windows/MFVideoEncoder.h index 42958e79..c0b21f3f 100644 --- a/webrtc-jni/src/main/cpp/include/media/video/codec/windows/MFH264Encoder.h +++ b/webrtc-jni/src/main/cpp/include/media/video/codec/windows/MFVideoEncoder.h @@ -14,18 +14,18 @@ * limitations under the License. */ -#ifndef JNI_WEBRTC_MEDIA_VIDEO_CODEC_MF_H264_ENCODER_H_ -#define JNI_WEBRTC_MEDIA_VIDEO_CODEC_MF_H264_ENCODER_H_ +#ifndef JNI_WEBRTC_MEDIA_VIDEO_CODEC_MF_VIDEO_ENCODER_H_ +#define JNI_WEBRTC_MEDIA_VIDEO_CODEC_MF_VIDEO_ENCODER_H_ +#include "media/video/codec/EncoderOutputProcessor.h" #include "media/video/codec/windows/MFTransformEvents.h" #include "platform/windows/MFInitializer.h" +#include "api/video/video_codec_type.h" #include "api/video/video_frame.h" #include "api/video_codecs/sdp_video_format.h" #include "api/video_codecs/video_codec.h" #include "api/video_codecs/video_encoder.h" -#include "common_video/h264/h264_bitstream_parser.h" -#include "modules/video_coding/codecs/h264/include/h264_globals.h" #include #include @@ -42,8 +42,8 @@ namespace jni { - // Encodes H.264 with the hardware encoder of the GPU, through the Media - // Foundation transform its driver provides. These transforms are + // Encodes H.264 or AV1 with the hardware encoder of the GPU, through the + // Media Foundation transform its driver provides. These transforms are // asynchronous: they ask for input and announce output through events, // which arrive on a Media Foundation thread, so encoded frames are // handed to WebRTC from there. @@ -51,11 +51,12 @@ namespace jni // Frames are passed in system memory as NV12; the transform uploads them. // Anything that fails makes the encoder give up, so that WebRTC switches // to the software encoder. - class MFH264Encoder : public webrtc::VideoEncoder, public MFTransformEventListener + class MFVideoEncoder : public webrtc::VideoEncoder, public MFTransformEventListener { public: - explicit MFH264Encoder(const webrtc::SdpVideoFormat & format); - ~MFH264Encoder() override; + // The codec is H.264 or AV1. + MFVideoEncoder(webrtc::VideoCodecType codec, const webrtc::SdpVideoFormat & format); + ~MFVideoEncoder() override; int32_t InitEncode(const webrtc::VideoCodec * codecSettings, const Settings & settings) override; int32_t RegisterEncodeCompleteCallback(webrtc::EncodedImageCallback * callback) override; @@ -79,14 +80,13 @@ namespace jni HRESULT CreateTransform(); HRESULT ConfigureTypes(); void ConfigureCodec(); - void SetBitrate(uint32_t bitrateBps); HRESULT CreateInputSample(const webrtc::VideoFrame & frame, IMFSample ** sample); void ProcessOutput(); void DeliverOutput(IMFSample * sample); void ShutdownTransform(); private: - const webrtc::SdpVideoFormat format; + const webrtc::VideoCodecType codec; std::string implementationName; std::unique_ptr mfInitializer; @@ -123,10 +123,8 @@ namespace jni LONGLONG lastSampleTime; bool keyFrameRequested; - // Used on the event thread only. - std::vector parameterSets; - webrtc::H264BitstreamParser bitstreamParser; - webrtc::H264PacketizationMode packetizationMode; + // Used on the event thread only, and reset while no events arrive. + EncoderOutputProcessor outputProcessor; }; } diff --git a/webrtc-jni/src/main/cpp/include/media/video/codec/windows/MFVideoEncoderFactory.h b/webrtc-jni/src/main/cpp/include/media/video/codec/windows/MFVideoEncoderFactory.h index 0e729d33..0719429a 100644 --- a/webrtc-jni/src/main/cpp/include/media/video/codec/windows/MFVideoEncoderFactory.h +++ b/webrtc-jni/src/main/cpp/include/media/video/codec/windows/MFVideoEncoderFactory.h @@ -27,14 +27,15 @@ namespace jni { - // Creates the Media Foundation hardware encoders of the GPU. It offers - // H.264 in the profiles WebRTC's software encoder offers too, and only - // with packetization mode 1: mode 0 needs each NAL unit to fit a packet, - // which hardware encoders cannot be relied on to keep to. + // Creates the Media Foundation hardware encoders of the GPU, for H.264 and + // AV1, whichever the GPU has. It offers them in the formats WebRTC's + // software encoders offer too: H.264 only with packetization mode 1, since + // mode 0 needs each NAL unit to fit a packet, which hardware encoders + // cannot be relied on to keep to, and AV1 in profile 0. class MFVideoEncoderFactory : public webrtc::VideoEncoderFactory { public: - // Returns a factory, or null if there is no hardware H.264 + // Returns a factory, or null if there is no hardware H.264 or AV1 // encoder on this system. static std::unique_ptr Create(); @@ -45,7 +46,11 @@ namespace jni const webrtc::SdpVideoFormat & format) override; private: - MFVideoEncoderFactory() = default; + MFVideoEncoderFactory(bool h264, bool av1); + + private: + const bool h264; + const bool av1; }; } diff --git a/webrtc-jni/src/main/cpp/src/media/video/codec/EncoderOutputProcessor.cpp b/webrtc-jni/src/main/cpp/src/media/video/codec/EncoderOutputProcessor.cpp new file mode 100644 index 00000000..ae94a39d --- /dev/null +++ b/webrtc-jni/src/main/cpp/src/media/video/codec/EncoderOutputProcessor.cpp @@ -0,0 +1,285 @@ +/* + * Copyright 2026 Alex Andres + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "media/video/codec/EncoderOutputProcessor.h" + +#include "api/video/render_resolution.h" +#include "api/video_codecs/scalability_mode.h" +#include "common_video/h264/h264_common.h" +#include "modules/video_coding/codecs/interface/common_constants.h" +#include "rtc_base/logging.h" + +namespace jni +{ + namespace + { + // The QP thresholds of WebRTC's own H.264 encoder. + constexpr int kLowH264QpThreshold = 24; + constexpr int kHighH264QpThreshold = 37; + + // AV1 OBU types, section 6.2.2 of the AV1 specification. + constexpr uint8_t kObuSequenceHeader = 1; + constexpr uint8_t kObuTemporalDelimiter = 2; + + struct Obu + { + uint8_t type; + // The whole OBU, header included. + size_t offset; + size_t size; + }; + + // Reads a leb128 value, section 4.10.5 of the AV1 specification. + bool ReadLeb128(std::span data, size_t & position, uint64_t & value) + { + value = 0; + + for (int i = 0; i < 8; i++) { + if (position >= data.size()) { + return false; + } + + const uint8_t byte = data[position++]; + + value |= static_cast(byte & 0x7F) << (i * 7); + + if (!(byte & 0x80)) { + return true; + } + } + + return false; + } + + // Splits a temporal unit into its OBUs, section 5.3 of the AV1 + // specification. An OBU without a size field runs to the end. + bool ParseObus(std::span data, std::vector & obus) + { + size_t position = 0; + + while (position < data.size()) { + const size_t offset = position; + const uint8_t header = data[position++]; + + // The forbidden bit. + if (header & 0x80) { + return false; + } + + const uint8_t type = (header >> 3) & 0x0F; + const bool extension = header & 0x04; + const bool hasSize = header & 0x02; + + if (extension) { + position++; + } + + uint64_t payloadSize = 0; + + if (hasSize) { + if (!ReadLeb128(data, position, payloadSize)) { + return false; + } + } + else { + payloadSize = position <= data.size() ? data.size() - position : 0; + } + + if (position > data.size() || payloadSize > data.size() - position) { + return false; + } + + position += static_cast(payloadSize); + + obus.push_back(Obu { type, offset, position - offset }); + } + + return true; + } + } + + EncoderOutputProcessor::EncoderOutputProcessor(webrtc::VideoCodecType codec, const webrtc::SdpVideoFormat & format) : + codec(codec), + packetizationMode(webrtc::H264PacketizationMode::NonInterleaved) + { + auto mode = format.parameters.find("packetization-mode"); + + if (mode == format.parameters.end() || mode->second != "1") { + packetizationMode = webrtc::H264PacketizationMode::SingleNalUnit; + } + } + + void EncoderOutputProcessor::Reset() + { + parameterSets.clear(); + } + + bool EncoderOutputProcessor::Process(std::span bitstream, bool keyFrame, + webrtc::EncodedImage & image, webrtc::CodecSpecificInfo & info) + { + std::vector output; + + const bool processed = codec == webrtc::kVideoCodecAV1 + ? ProcessAv1(bitstream, keyFrame, output) + : ProcessH264(bitstream, keyFrame, output); + + if (!processed) { + return false; + } + + image.SetEncodedData(webrtc::EncodedImageBuffer::Create(output.data(), output.size())); + image.set_frame_type(keyFrame + ? webrtc::VideoFrameType::kVideoFrameKey + : webrtc::VideoFrameType::kVideoFrameDelta); + + if (codec == webrtc::kVideoCodecH264) { + h264Parser.ParseBitstream(std::span(output.data(), output.size())); + image.qp_ = h264Parser.GetLastSliceQp().value_or(-1); + } + else { + image.qp_ = -1; + } + + FillCodecSpecificInfo(image, keyFrame, info); + + return true; + } + + bool EncoderOutputProcessor::ProcessH264(std::span bitstream, bool & keyFrame, + std::vector & output) + { + std::vector parameterSetsFound; + bool hasSps = false; + bool hasPps = false; + + const std::vector nalus = webrtc::H264::FindNaluIndices(bitstream); + + if (nalus.empty()) { + return false; + } + + for (const webrtc::H264::NaluIndex & nalu : nalus) { + const size_t end = nalu.payload_start_offset + nalu.payload_size; + const webrtc::H264::NaluType type = webrtc::H264::ParseNaluType(bitstream[nalu.payload_start_offset]); + + if (type == webrtc::H264::NaluType::kSps || type == webrtc::H264::NaluType::kPps) { + hasSps |= type == webrtc::H264::NaluType::kSps; + hasPps |= type == webrtc::H264::NaluType::kPps; + + parameterSetsFound.insert(parameterSetsFound.end(), + bitstream.begin() + nalu.start_offset, bitstream.begin() + end); + } + else if (type == webrtc::H264::NaluType::kIdr) { + keyFrame = true; + } + } + + if (hasSps && hasPps) { + parameterSets = std::move(parameterSetsFound); + } + else if (keyFrame) { + if (parameterSets.empty()) { + RTC_LOG(LS_WARNING) << "H.264 encoder produced a key frame without SPS and PPS"; + return false; + } + + output.insert(output.end(), parameterSets.begin(), parameterSets.end()); + } + + output.insert(output.end(), bitstream.begin(), bitstream.end()); + + return true; + } + + bool EncoderOutputProcessor::ProcessAv1(std::span bitstream, bool keyFrame, + std::vector & output) + { + std::vector obus; + + if (!ParseObus(bitstream, obus) || obus.empty()) { + RTC_LOG(LS_WARNING) << "AV1 encoder produced a bitstream that is not in the low overhead format"; + return false; + } + + bool hasSequenceHeader = false; + + for (const Obu & obu : obus) { + if (obu.type == kObuSequenceHeader) { + hasSequenceHeader = true; + parameterSets.assign(bitstream.begin() + obu.offset, bitstream.begin() + obu.offset + obu.size); + } + } + + if (!keyFrame || hasSequenceHeader) { + output.assign(bitstream.begin(), bitstream.end()); + return true; + } + + if (parameterSets.empty()) { + RTC_LOG(LS_WARNING) << "AV1 encoder produced a key frame without a sequence header"; + return false; + } + + // A temporal delimiter has to stay first in its temporal unit. + size_t start = 0; + + if (obus.front().type == kObuTemporalDelimiter) { + start = obus.front().size; + output.insert(output.end(), bitstream.begin(), bitstream.begin() + start); + } + + output.insert(output.end(), parameterSets.begin(), parameterSets.end()); + output.insert(output.end(), bitstream.begin() + start, bitstream.end()); + + return true; + } + + void EncoderOutputProcessor::FillCodecSpecificInfo(const webrtc::EncodedImage & image, bool keyFrame, + webrtc::CodecSpecificInfo & info) + { + info.codecType = codec; + + if (codec == webrtc::kVideoCodecH264) { + info.codecSpecific.H264.packetization_mode = packetizationMode; + info.codecSpecific.H264.temporal_idx = webrtc::kNoTemporalIdx; + info.codecSpecific.H264.base_layer_sync = false; + info.codecSpecific.H264.idr_frame = keyFrame; + return; + } + + // AV1 has no codec specific information; WebRTC sends its frame + // dependencies from these, in the dependency descriptor. + auto layerFrames = svcController.NextFrameConfig(/*restart=*/keyFrame); + info.generic_frame_info = svcController.OnEncodeDone(layerFrames[0]); + info.scalability_mode = webrtc::ScalabilityMode::kL1T1; + + if (keyFrame) { + info.template_structure = svcController.DependencyStructure(); + info.template_structure->resolutions = { + webrtc::RenderResolution(image._encodedWidth, image._encodedHeight) + }; + } + } + + webrtc::VideoEncoder::ScalingSettings EncoderOutputProcessor::GetScalingSettings() const + { + if (codec == webrtc::kVideoCodecH264) { + return webrtc::VideoEncoder::ScalingSettings(kLowH264QpThreshold, kHighH264QpThreshold); + } + + return webrtc::VideoEncoder::ScalingSettings::kOff; + } +} diff --git a/webrtc-jni/src/main/cpp/src/media/video/codec/linux/VaapiH264Encoder.cpp b/webrtc-jni/src/main/cpp/src/media/video/codec/linux/VaapiH264Encoder.cpp index 0c0fe440..efb66d9a 100644 --- a/webrtc-jni/src/main/cpp/src/media/video/codec/linux/VaapiH264Encoder.cpp +++ b/webrtc-jni/src/main/cpp/src/media/video/codec/linux/VaapiH264Encoder.cpp @@ -18,8 +18,6 @@ #include "api/video/encoded_image.h" #include "api/video/i420_buffer.h" -#include "common_video/h264/h264_common.h" -#include "modules/video_coding/codecs/interface/common_constants.h" #include "modules/video_coding/include/video_codec_interface.h" #include "modules/video_coding/include/video_error_codes.h" #include "rtc_base/logging.h" @@ -33,10 +31,6 @@ namespace jni { namespace { - // The QP thresholds of WebRTC's own H.264 encoder. - constexpr int kLowH264QpThreshold = 24; - constexpr int kHighH264QpThreshold = 37; - // frame_num counts to 2^8 before it wraps. constexpr uint32_t kLog2MaxFrameNum = 8; constexpr uint32_t kMaxFrameNum = 1 << kLog2MaxFrameNum; @@ -94,23 +88,6 @@ namespace jni picture.picture_id = VA_INVALID_SURFACE; picture.flags = VA_PICTURE_H264_INVALID; } - - // Whether a key frame carries both parameter sets, which the receiver - // needs to decode it. - bool HasParameterSets(const std::vector & output) - { - bool sps = false; - bool pps = false; - - for (const webrtc::H264::NaluIndex & nalu : webrtc::H264::FindNaluIndices(output)) { - const webrtc::H264::NaluType type = webrtc::H264::ParseNaluType(output[nalu.payload_start_offset]); - - sps |= type == webrtc::H264::NaluType::kSps; - pps |= type == webrtc::H264::NaluType::kPps; - } - - return sps && pps; - } } VaapiH264Encoder::VaapiH264Encoder(VaapiLibrary & library, const webrtc::SdpVideoFormat & format) : @@ -133,13 +110,8 @@ namespace jni current(0), referenceValid(false), callback(nullptr), - packetizationMode(webrtc::H264PacketizationMode::NonInterleaved) + outputProcessor(webrtc::kVideoCodecH264, format) { - auto mode = format.parameters.find("packetization-mode"); - - if (mode == format.parameters.end() || mode->second != "1") { - packetizationMode = webrtc::H264PacketizationMode::SingleNalUnit; - } } VaapiH264Encoder::~VaapiH264Encoder() @@ -183,6 +155,7 @@ namespace jni frameNum = 0; current = 0; referenceValid = false; + outputProcessor.Reset(); if (!CreateSession()) { Release(); @@ -323,9 +296,19 @@ namespace jni DestroyFrameBuffers(); - if (idr && !HasParameterSets(output)) { - // Nothing can decode the stream then; better in software. - RTC_LOG(LS_WARNING) << implementationName << " writes no parameter sets into key frames"; + webrtc::EncodedImage image; + image._encodedWidth = codecSettings.width; + image._encodedHeight = codecSettings.height; + image.SetRtpTimestamp(frame.rtp_timestamp()); + image.capture_time_ms_ = frame.render_time_ms(); + image.ntp_time_ms_ = frame.ntp_time_ms(); + image.rotation_ = frame.rotation(); + + webrtc::CodecSpecificInfo info; + + // A driver that writes no parameter sets into key frames makes a + // stream nothing can decode; better in software then. + if (!outputProcessor.Process(output, idr, image, info)) { return WEBRTC_VIDEO_CODEC_FALLBACK_SOFTWARE; } @@ -339,28 +322,6 @@ namespace jni idrPicId = (idrPicId + 1) & 0xFFFF; } - webrtc::EncodedImage image; - image.SetEncodedData(webrtc::EncodedImageBuffer::Create(output.data(), output.size())); - image._encodedWidth = codecSettings.width; - image._encodedHeight = codecSettings.height; - image.SetRtpTimestamp(frame.rtp_timestamp()); - image.capture_time_ms_ = frame.render_time_ms(); - image.ntp_time_ms_ = frame.ntp_time_ms(); - image.rotation_ = frame.rotation(); - image.set_frame_type(idr - ? webrtc::VideoFrameType::kVideoFrameKey - : webrtc::VideoFrameType::kVideoFrameDelta); - - bitstreamParser.ParseBitstream(std::span(output.data(), output.size())); - image.qp_ = bitstreamParser.GetLastSliceQp().value_or(-1); - - webrtc::CodecSpecificInfo info; - info.codecType = webrtc::kVideoCodecH264; - info.codecSpecific.H264.packetization_mode = packetizationMode; - info.codecSpecific.H264.temporal_idx = webrtc::kNoTemporalIdx; - info.codecSpecific.H264.base_layer_sync = false; - info.codecSpecific.H264.idr_frame = idr; - callback->OnEncodedImage(image, &info); return WEBRTC_VIDEO_CODEC_OK; @@ -700,7 +661,7 @@ namespace jni info.is_hardware_accelerated = true; info.supports_native_handle = false; info.supports_simulcast = false; - info.scaling_settings = ScalingSettings(kLowH264QpThreshold, kHighH264QpThreshold); + info.scaling_settings = outputProcessor.GetScalingSettings(); // NV12 needs even dimensions. info.requested_resolution_alignment = 2; diff --git a/webrtc-jni/src/main/cpp/src/media/video/codec/nvenc/NvencLibrary.cpp b/webrtc-jni/src/main/cpp/src/media/video/codec/nvenc/NvencLibrary.cpp index 792c456d..73cea94f 100644 --- a/webrtc-jni/src/main/cpp/src/media/video/codec/nvenc/NvencLibrary.cpp +++ b/webrtc-jni/src/main/cpp/src/media/video/codec/nvenc/NvencLibrary.cpp @@ -18,7 +18,9 @@ #include "rtc_base/logging.h" +#include #include +#include namespace jni { @@ -116,11 +118,74 @@ namespace jni deviceName = name; } - RTC_LOG(LS_INFO) << "NVENC available on " << deviceName; + QueryCodecs(); + + if (!h264 && !av1) { + RTC_LOG(LS_INFO) << "NVENC: " << deviceName << " encodes neither H.264 nor AV1"; + return false; + } + + RTC_LOG(LS_INFO) << "NVENC available on " << deviceName << ", H.264: " << h264 << ", AV1: " << av1; return true; } + void NvencLibrary::QueryCodecs() + { + CUcontext context = nullptr; + + if (!RetainContext(&context)) { + return; + } + + if (PushContext(context)) { + NV_ENC_OPEN_ENCODE_SESSION_EX_PARAMS params = {}; + params.version = NV_ENC_OPEN_ENCODE_SESSION_EX_PARAMS_VER; + params.deviceType = NV_ENC_DEVICE_TYPE_CUDA; + params.device = context; + params.apiVersion = NVENCAPI_VERSION; + + void * encoder = nullptr; + + if (api.nvEncOpenEncodeSessionEx(¶ms, &encoder) == NV_ENC_SUCCESS) { + uint32_t count = 0; + + if (api.nvEncGetEncodeGUIDCount(encoder, &count) == NV_ENC_SUCCESS && count > 0) { + std::vector guids(count); + uint32_t found = 0; + + if (api.nvEncGetEncodeGUIDs(encoder, guids.data(), count, &found) == NV_ENC_SUCCESS) { + for (uint32_t i = 0; i < found; i++) { + h264 |= std::memcmp(&guids[i], &NV_ENC_CODEC_H264_GUID, sizeof(GUID)) == 0; + av1 |= std::memcmp(&guids[i], &NV_ENC_CODEC_AV1_GUID, sizeof(GUID)) == 0; + } + } + } + + api.nvEncDestroyEncoder(encoder); + } + else { + // No session left to ask right now; every NVENC GPU encodes + // H.264. + h264 = true; + } + + PopContext(); + } + + ReleaseContext(); + } + + bool NvencLibrary::SupportsH264() const + { + return h264; + } + + bool NvencLibrary::SupportsAv1() const + { + return av1; + } + const NV_ENCODE_API_FUNCTION_LIST & NvencLibrary::Api() const { return api; diff --git a/webrtc-jni/src/main/cpp/src/media/video/codec/nvenc/NvencH264Encoder.cpp b/webrtc-jni/src/main/cpp/src/media/video/codec/nvenc/NvencVideoEncoder.cpp similarity index 81% rename from webrtc-jni/src/main/cpp/src/media/video/codec/nvenc/NvencH264Encoder.cpp rename to webrtc-jni/src/main/cpp/src/media/video/codec/nvenc/NvencVideoEncoder.cpp index 0ca1d30b..9d36cc30 100644 --- a/webrtc-jni/src/main/cpp/src/media/video/codec/nvenc/NvencH264Encoder.cpp +++ b/webrtc-jni/src/main/cpp/src/media/video/codec/nvenc/NvencVideoEncoder.cpp @@ -14,12 +14,12 @@ * limitations under the License. */ -#include "media/video/codec/nvenc/NvencH264Encoder.h" +#include "media/video/codec/nvenc/NvencVideoEncoder.h" #include "media/video/codec/nvenc/CudaContextScope.h" #include "api/video/encoded_image.h" #include "api/video/i420_buffer.h" -#include "modules/video_coding/codecs/interface/common_constants.h" +#include "api/video_codecs/scalability_mode.h" #include "modules/video_coding/include/video_codec_interface.h" #include "modules/video_coding/include/video_error_codes.h" #include "rtc_base/logging.h" @@ -30,16 +30,12 @@ namespace jni { - namespace - { - // The QP thresholds of WebRTC's own H.264 encoder. - constexpr int kLowH264QpThreshold = 24; - constexpr int kHighH264QpThreshold = 37; - } - - NvencH264Encoder::NvencH264Encoder(NvencLibrary & library, const webrtc::SdpVideoFormat & format) : + NvencVideoEncoder::NvencVideoEncoder(NvencLibrary & library, webrtc::VideoCodecType codec, + const webrtc::SdpVideoFormat & format) : library(library), api(library.Api()), + codec(codec), + codecGuid(codec == webrtc::kVideoCodecAV1 ? NV_ENC_CODEC_AV1_GUID : NV_ENC_CODEC_H264_GUID), implementationName("NVENC (" + library.DeviceName() + ")"), context(nullptr), encoder(nullptr), @@ -52,21 +48,16 @@ namespace jni framerate(30), frameCount(0), callback(nullptr), - packetizationMode(webrtc::H264PacketizationMode::NonInterleaved) + outputProcessor(codec, format) { - auto mode = format.parameters.find("packetization-mode"); - - if (mode == format.parameters.end() || mode->second != "1") { - packetizationMode = webrtc::H264PacketizationMode::SingleNalUnit; - } } - NvencH264Encoder::~NvencH264Encoder() + NvencVideoEncoder::~NvencVideoEncoder() { Release(); } - int32_t NvencH264Encoder::InitEncode(const webrtc::VideoCodec * settings, const Settings & encoderSettings) + int32_t NvencVideoEncoder::InitEncode(const webrtc::VideoCodec * settings, const Settings & encoderSettings) { if (settings == nullptr || settings->width == 0 || settings->height == 0) { return WEBRTC_VIDEO_CODEC_ERR_PARAMETER; @@ -74,6 +65,11 @@ namespace jni if (settings->numberOfSimulcastStreams > 1) { return WEBRTC_VIDEO_CODEC_ERR_SIMULCAST_PARAMETERS_NOT_SUPPORTED; } + // A single layer; spatial or temporal layers are left to the + // software encoder. + if (settings->GetScalabilityMode().value_or(webrtc::ScalabilityMode::kL1T1) != webrtc::ScalabilityMode::kL1T1) { + return WEBRTC_VIDEO_CODEC_ERR_PARAMETER; + } // NV12 has chroma at half the resolution in both directions. if (settings->width % 2 != 0 || settings->height % 2 != 0) { return WEBRTC_VIDEO_CODEC_ERR_PARAMETER; @@ -85,6 +81,7 @@ namespace jni bitrateBps = std::max(1u, codecSettings.startBitrate) * 1000; framerate = std::max(1u, codecSettings.maxFramerate); frameCount = 0; + outputProcessor.Reset(); if (!library.RetainContext(&context)) { RTC_LOG(LS_WARNING) << "NVENC: failed to retain the CUDA context"; @@ -103,7 +100,7 @@ namespace jni return WEBRTC_VIDEO_CODEC_OK; } - bool NvencH264Encoder::OpenSession() + bool NvencVideoEncoder::OpenSession() { CudaContextScope scope(library, context); @@ -126,7 +123,7 @@ namespace jni return true; } - bool NvencH264Encoder::Configure() + bool NvencVideoEncoder::Configure() { CudaContextScope scope(library, context); @@ -134,7 +131,7 @@ namespace jni presetConfig.version = NV_ENC_PRESET_CONFIG_VER; presetConfig.presetCfg.version = NV_ENC_CONFIG_VER; - NVENCSTATUS status = api.nvEncGetEncodePresetConfigEx(encoder, NV_ENC_CODEC_H264_GUID, + NVENCSTATUS status = api.nvEncGetEncodePresetConfigEx(encoder, codecGuid, NV_ENC_PRESET_P4_GUID, NV_ENC_TUNING_INFO_ULTRA_LOW_LATENCY, &presetConfig); if (status != NV_ENC_SUCCESS) { @@ -144,27 +141,16 @@ namespace jni config = presetConfig.presetCfg; config.version = NV_ENC_CONFIG_VER; - config.profileGUID = NV_ENC_H264_PROFILE_BASELINE_GUID; // No B-frames, and key frames only when WebRTC asks for them. config.gopLength = NVENC_INFINITE_GOPLENGTH; config.frameIntervalP = 1; - NV_ENC_CONFIG_H264 & h264 = config.encodeCodecConfig.h264Config; - h264.idrPeriod = NVENC_INFINITE_GOPLENGTH; - // A receiver joining later needs them with the key frame it starts from. - h264.repeatSPSPPS = 1; - h264.outputAUD = 0; - h264.sliceMode = 0; - h264.sliceModeData = 0; - h264.chromaFormatIDC = 1; - h264.level = NV_ENC_LEVEL_AUTOSELECT; - // Baseline has no CABAC. - h264.entropyCodingMode = NV_ENC_H264_ENTROPY_CODING_MODE_CAVLC; + ConfigureCodec(); initParams = {}; initParams.version = NV_ENC_INITIALIZE_PARAMS_VER; - initParams.encodeGUID = NV_ENC_CODEC_H264_GUID; + initParams.encodeGUID = codecGuid; initParams.presetGUID = NV_ENC_PRESET_P4_GUID; initParams.tuningInfo = NV_ENC_TUNING_INFO_ULTRA_LOW_LATENCY; initParams.encodeWidth = codecSettings.width; @@ -216,7 +202,43 @@ namespace jni return true; } - void NvencH264Encoder::ApplyRates() + void NvencVideoEncoder::ConfigureCodec() + { + if (codec == webrtc::kVideoCodecAV1) { + config.profileGUID = NV_ENC_AV1_PROFILE_MAIN_GUID; + + NV_ENC_CONFIG_AV1 & av1 = config.encodeCodecConfig.av1Config; + av1.idrPeriod = NVENC_INFINITE_GOPLENGTH; + // A receiver joining later needs it with the key frame it + // starts from. + av1.repeatSeqHdr = 1; + av1.disableSeqHdr = 0; + // OBUs with size fields, the low overhead format WebRTC sends. + av1.outputAnnexBFormat = 0; + av1.chromaFormatIDC = 1; + av1.inputPixelBitDepthMinus8 = 0; + av1.pixelBitDepthMinus8 = 0; + av1.level = NV_ENC_LEVEL_AV1_AUTOSELECT; + av1.tier = NV_ENC_TIER_AV1_0; + return; + } + + config.profileGUID = NV_ENC_H264_PROFILE_BASELINE_GUID; + + NV_ENC_CONFIG_H264 & h264 = config.encodeCodecConfig.h264Config; + h264.idrPeriod = NVENC_INFINITE_GOPLENGTH; + // A receiver joining later needs them with the key frame it starts from. + h264.repeatSPSPPS = 1; + h264.outputAUD = 0; + h264.sliceMode = 0; + h264.sliceModeData = 0; + h264.chromaFormatIDC = 1; + h264.level = NV_ENC_LEVEL_AUTOSELECT; + // Baseline has no CABAC. + h264.entropyCodingMode = NV_ENC_H264_ENTROPY_CODING_MODE_CAVLC; + } + + void NvencVideoEncoder::ApplyRates() { initParams.frameRateNum = framerate; initParams.frameRateDen = 1; @@ -231,14 +253,14 @@ namespace jni rc.vbvInitialDelay = rc.vbvBufferSize; } - int32_t NvencH264Encoder::RegisterEncodeCompleteCallback(webrtc::EncodedImageCallback * encodeCallback) + int32_t NvencVideoEncoder::RegisterEncodeCompleteCallback(webrtc::EncodedImageCallback * encodeCallback) { callback = encodeCallback; return WEBRTC_VIDEO_CODEC_OK; } - int32_t NvencH264Encoder::Release() + int32_t NvencVideoEncoder::Release() { DestroySession(); @@ -250,7 +272,7 @@ namespace jni return WEBRTC_VIDEO_CODEC_OK; } - void NvencH264Encoder::DestroySession() + void NvencVideoEncoder::DestroySession() { if (encoder == nullptr) { return; @@ -271,7 +293,7 @@ namespace jni encoder = nullptr; } - int32_t NvencH264Encoder::Encode(const webrtc::VideoFrame & frame, + int32_t NvencVideoEncoder::Encode(const webrtc::VideoFrame & frame, const std::vector * frameTypes) { if (encoder == nullptr || callback == nullptr) { @@ -328,45 +350,35 @@ namespace jni return WEBRTC_VIDEO_CODEC_FALLBACK_SOFTWARE; } - const uint8_t * data = static_cast(bitstream.bitstreamBufferPtr); - const size_t size = bitstream.bitstreamSizeInBytes; - const bool keyFrame = bitstream.pictureType == NV_ENC_PIC_TYPE_IDR + const std::span data(static_cast(bitstream.bitstreamBufferPtr), + bitstream.bitstreamSizeInBytes); + const bool keyFrame = keyFrameRequested + || bitstream.pictureType == NV_ENC_PIC_TYPE_IDR || bitstream.pictureType == NV_ENC_PIC_TYPE_I; - webrtc::scoped_refptr encoded = webrtc::EncodedImageBuffer::Create(data, size); - - api.nvEncUnlockBitstream(encoder, outputBuffer); - webrtc::EncodedImage image; - image.SetEncodedData(encoded); image._encodedWidth = codecSettings.width; image._encodedHeight = codecSettings.height; image.SetRtpTimestamp(frame.rtp_timestamp()); image.capture_time_ms_ = frame.render_time_ms(); image.ntp_time_ms_ = frame.ntp_time_ms(); image.rotation_ = frame.rotation(); - image.set_frame_type(keyFrame - ? webrtc::VideoFrameType::kVideoFrameKey - : webrtc::VideoFrameType::kVideoFrameDelta); - - // The bitstream QP, which quality scaling compares with its - // thresholds, rather than NVENC's average. - bitstreamParser.ParseBitstream(std::span(encoded->data(), encoded->size())); - image.qp_ = bitstreamParser.GetLastSliceQp().value_or(-1); webrtc::CodecSpecificInfo info; - info.codecType = webrtc::kVideoCodecH264; - info.codecSpecific.H264.packetization_mode = packetizationMode; - info.codecSpecific.H264.temporal_idx = webrtc::kNoTemporalIdx; - info.codecSpecific.H264.base_layer_sync = false; - info.codecSpecific.H264.idr_frame = keyFrame; + const bool processed = outputProcessor.Process(data, keyFrame, image, info); + + api.nvEncUnlockBitstream(encoder, outputBuffer); + + if (!processed) { + return WEBRTC_VIDEO_CODEC_FALLBACK_SOFTWARE; + } callback->OnEncodedImage(image, &info); return WEBRTC_VIDEO_CODEC_OK; } - bool NvencH264Encoder::CopyToInput(const webrtc::VideoFrame & frame, uint32_t * pitch) + bool NvencVideoEncoder::CopyToInput(const webrtc::VideoFrame & frame, uint32_t * pitch) { webrtc::scoped_refptr i420 = frame.video_frame_buffer()->ToI420(); @@ -409,7 +421,7 @@ namespace jni return true; } - void NvencH264Encoder::SetRates(const RateControlParameters & parameters) + void NvencVideoEncoder::SetRates(const RateControlParameters & parameters) { const uint32_t bitrate = parameters.bitrate.get_sum_bps(); @@ -445,14 +457,14 @@ namespace jni } } - webrtc::VideoEncoder::EncoderInfo NvencH264Encoder::GetEncoderInfo() const + webrtc::VideoEncoder::EncoderInfo NvencVideoEncoder::GetEncoderInfo() const { EncoderInfo info; info.implementation_name = implementationName; info.is_hardware_accelerated = true; info.supports_native_handle = false; info.supports_simulcast = false; - info.scaling_settings = ScalingSettings(kLowH264QpThreshold, kHighH264QpThreshold); + info.scaling_settings = outputProcessor.GetScalingSettings(); // NV12 needs even dimensions. info.requested_resolution_alignment = 2; diff --git a/webrtc-jni/src/main/cpp/src/media/video/codec/nvenc/NvencVideoEncoderFactory.cpp b/webrtc-jni/src/main/cpp/src/media/video/codec/nvenc/NvencVideoEncoderFactory.cpp index 0f7033d7..29e28184 100644 --- a/webrtc-jni/src/main/cpp/src/media/video/codec/nvenc/NvencVideoEncoderFactory.cpp +++ b/webrtc-jni/src/main/cpp/src/media/video/codec/nvenc/NvencVideoEncoderFactory.cpp @@ -15,9 +15,10 @@ */ #include "media/video/codec/nvenc/NvencVideoEncoderFactory.h" -#include "media/video/codec/nvenc/NvencH264Encoder.h" +#include "media/video/codec/nvenc/NvencVideoEncoder.h" #include "api/video_codecs/h264_profile_level_id.h" +#include "api/video_codecs/video_codec.h" #include "modules/video_coding/codecs/h264/include/h264.h" namespace jni @@ -40,15 +41,24 @@ namespace jni std::vector NvencVideoEncoderFactory::GetSupportedFormats() const { - return { - webrtc::CreateH264Format(webrtc::H264Profile::kProfileConstrainedBaseline, webrtc::H264Level::kLevel3_1, "1"), - webrtc::CreateH264Format(webrtc::H264Profile::kProfileBaseline, webrtc::H264Level::kLevel3_1, "1") - }; + std::vector formats; + + if (library.SupportsH264()) { + formats.push_back(webrtc::CreateH264Format(webrtc::H264Profile::kProfileConstrainedBaseline, + webrtc::H264Level::kLevel3_1, "1")); + formats.push_back(webrtc::CreateH264Format(webrtc::H264Profile::kProfileBaseline, + webrtc::H264Level::kLevel3_1, "1")); + } + if (library.SupportsAv1()) { + formats.push_back(webrtc::SdpVideoFormat::AV1Profile0()); + } + + return formats; } std::unique_ptr NvencVideoEncoderFactory::Create(const webrtc::Environment & env, const webrtc::SdpVideoFormat & format) { - return std::make_unique(library, format); + return std::make_unique(library, webrtc::PayloadStringToCodecType(format.name), format); } } diff --git a/webrtc-jni/src/main/cpp/src/media/video/codec/windows/MFEncoderUtils.cpp b/webrtc-jni/src/main/cpp/src/media/video/codec/windows/MFEncoderUtils.cpp index 88def1cb..999cfe56 100644 --- a/webrtc-jni/src/main/cpp/src/media/video/codec/windows/MFEncoderUtils.cpp +++ b/webrtc-jni/src/main/cpp/src/media/video/codec/windows/MFEncoderUtils.cpp @@ -23,10 +23,10 @@ using Microsoft::WRL::ComPtr; namespace jni { - HRESULT EnumerateHardwareH264Encoders(std::vector> & encoders) + HRESULT EnumerateHardwareEncoders(const GUID & format, std::vector> & encoders) { MFT_REGISTER_TYPE_INFO input = { MFMediaType_Video, MFVideoFormat_NV12 }; - MFT_REGISTER_TYPE_INFO output = { MFMediaType_Video, MFVideoFormat_H264 }; + MFT_REGISTER_TYPE_INFO output = { MFMediaType_Video, format }; IMFActivate ** activates = nullptr; UINT32 count = 0; diff --git a/webrtc-jni/src/main/cpp/src/media/video/codec/windows/MFH264Encoder.cpp b/webrtc-jni/src/main/cpp/src/media/video/codec/windows/MFVideoEncoder.cpp similarity index 80% rename from webrtc-jni/src/main/cpp/src/media/video/codec/windows/MFH264Encoder.cpp rename to webrtc-jni/src/main/cpp/src/media/video/codec/windows/MFVideoEncoder.cpp index cfdaf464..13b2d439 100644 --- a/webrtc-jni/src/main/cpp/src/media/video/codec/windows/MFH264Encoder.cpp +++ b/webrtc-jni/src/main/cpp/src/media/video/codec/windows/MFVideoEncoder.cpp @@ -14,13 +14,13 @@ * limitations under the License. */ -#include "media/video/codec/windows/MFH264Encoder.h" +#include "media/video/codec/windows/MFVideoEncoder.h" #include "media/video/codec/windows/MFEncoderUtils.h" #include "platform/windows/ComInitializer.h" #include "api/video/encoded_image.h" #include "api/video/i420_buffer.h" -#include "common_video/h264/h264_common.h" +#include "api/video_codecs/scalability_mode.h" #include "modules/video_coding/codecs/interface/common_constants.h" #include "modules/video_coding/include/video_codec_interface.h" #include "modules/video_coding/include/video_error_codes.h" @@ -50,10 +50,6 @@ namespace jni // this. constexpr UINT32 kGopSize = 3000; - // The QP thresholds of WebRTC's own H.264 encoder. - constexpr int kLowH264QpThreshold = 24; - constexpr int kHighH264QpThreshold = 37; - // Media Foundation counts time in 100 ns units. constexpr LONGLONG kUnitsPerSecond = 10000000; @@ -107,8 +103,8 @@ namespace jni } } - MFH264Encoder::MFH264Encoder(const webrtc::SdpVideoFormat & format) : - format(format), + MFVideoEncoder::MFVideoEncoder(webrtc::VideoCodecType codec, const webrtc::SdpVideoFormat & format) : + codec(codec), implementationName("MediaFoundation"), inputStreamId(0), outputStreamId(0), @@ -120,21 +116,16 @@ namespace jni inputRequests(0), lastSampleTime(-1), keyFrameRequested(false), - packetizationMode(webrtc::H264PacketizationMode::NonInterleaved) + outputProcessor(codec, format) { - auto mode = format.parameters.find("packetization-mode"); - - if (mode == format.parameters.end() || mode->second != "1") { - packetizationMode = webrtc::H264PacketizationMode::SingleNalUnit; - } } - MFH264Encoder::~MFH264Encoder() + MFVideoEncoder::~MFVideoEncoder() { Release(); } - int32_t MFH264Encoder::InitEncode(const webrtc::VideoCodec * settings, const Settings & encoderSettings) + int32_t MFVideoEncoder::InitEncode(const webrtc::VideoCodec * settings, const Settings & encoderSettings) { if (settings == nullptr || settings->width == 0 || settings->height == 0) { return WEBRTC_VIDEO_CODEC_ERR_PARAMETER; @@ -142,6 +133,11 @@ namespace jni if (settings->numberOfSimulcastStreams > 1) { return WEBRTC_VIDEO_CODEC_ERR_SIMULCAST_PARAMETERS_NOT_SUPPORTED; } + // Hardware encoders produce a single layer; spatial or temporal + // layers are left to the software encoder. + if (settings->GetScalabilityMode().value_or(webrtc::ScalabilityMode::kL1T1) != webrtc::ScalabilityMode::kL1T1) { + return WEBRTC_VIDEO_CODEC_ERR_PARAMETER; + } // NV12 has chroma at half the resolution in both directions. if (settings->width % 2 != 0 || settings->height % 2 != 0) { return WEBRTC_VIDEO_CODEC_ERR_PARAMETER; @@ -200,11 +196,12 @@ namespace jni return WEBRTC_VIDEO_CODEC_OK; } - HRESULT MFH264Encoder::CreateTransform() + HRESULT MFVideoEncoder::CreateTransform() { std::vector> encoders; - HRESULT hr = EnumerateHardwareH264Encoders(encoders); + HRESULT hr = EnumerateHardwareEncoders(codec == webrtc::kVideoCodecAV1 ? MFVideoFormat_AV1 : MFVideoFormat_H264, + encoders); if (FAILED(hr)) { return hr; @@ -262,7 +259,7 @@ namespace jni return hr; } - HRESULT MFH264Encoder::ConfigureTypes() + HRESULT MFVideoEncoder::ConfigureTypes() { const UINT32 width = codecSettings.width; const UINT32 height = codecSettings.height; @@ -276,24 +273,33 @@ namespace jni } outputType->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Video); - outputType->SetGUID(MF_MT_SUBTYPE, MFVideoFormat_H264); + outputType->SetGUID(MF_MT_SUBTYPE, codec == webrtc::kVideoCodecAV1 ? MFVideoFormat_AV1 : MFVideoFormat_H264); outputType->SetUINT32(MF_MT_AVG_BITRATE, bitrateBps); outputType->SetUINT32(MF_MT_INTERLACE_MODE, MFVideoInterlace_Progressive); MFSetAttributeSize(outputType.Get(), MF_MT_FRAME_SIZE, width, height); MFSetAttributeRatio(outputType.Get(), MF_MT_FRAME_RATE, framerate, 1); MFSetAttributeRatio(outputType.Get(), MF_MT_PIXEL_ASPECT_RATIO, 1, 1); - // Constrained Baseline is what peers expect, and is a subset of - // Baseline too. Older encoders only know Baseline, which peers - // decode as well, since WebRTC encoders use no Baseline-only tools. - outputType->SetUINT32(MF_MT_MPEG2_PROFILE, eAVEncH264VProfile_ConstrainedBase); + if (codec == webrtc::kVideoCodecAV1) { + // Main profile, 8-bit 4:2:0: AV1 profile 0, what WebRTC offers. + outputType->SetUINT32(MF_MT_VIDEO_PROFILE, eAVEncAV1VProfile_Main_420_8); - hr = transform->SetOutputType(outputStreamId, outputType.Get(), 0); - - if (FAILED(hr)) { - outputType->SetUINT32(MF_MT_MPEG2_PROFILE, eAVEncH264VProfile_Base); + hr = transform->SetOutputType(outputStreamId, outputType.Get(), 0); + } + else { + // Constrained Baseline is what peers expect, and is a subset of + // Baseline too. Older encoders only know Baseline, which peers + // decode as well, since WebRTC encoders use no Baseline-only + // tools. + outputType->SetUINT32(MF_MT_MPEG2_PROFILE, eAVEncH264VProfile_ConstrainedBase); hr = transform->SetOutputType(outputStreamId, outputType.Get(), 0); + + if (FAILED(hr)) { + outputType->SetUINT32(MF_MT_MPEG2_PROFILE, eAVEncH264VProfile_Base); + + hr = transform->SetOutputType(outputStreamId, outputType.Get(), 0); + } } if (FAILED(hr)) { return hr; @@ -317,7 +323,7 @@ namespace jni return transform->SetInputType(inputStreamId, inputType.Get(), 0); } - void MFH264Encoder::ConfigureCodec() + void MFVideoEncoder::ConfigureCodec() { if (!codecApi) { return; @@ -332,14 +338,14 @@ namespace jni SetCodecValue(codecApi.Get(), CODECAPI_AVEncCommonMeanBitRate, bitrateBps); } - int32_t MFH264Encoder::RegisterEncodeCompleteCallback(webrtc::EncodedImageCallback * encodeCallback) + int32_t MFVideoEncoder::RegisterEncodeCompleteCallback(webrtc::EncodedImageCallback * encodeCallback) { callback = encodeCallback; return WEBRTC_VIDEO_CODEC_OK; } - int32_t MFH264Encoder::Release() + int32_t MFVideoEncoder::Release() { ShutdownTransform(); @@ -358,10 +364,13 @@ namespace jni lastSampleTime = -1; keyFrameRequested = false; + // No events arrive any more. + outputProcessor.Reset(); + return WEBRTC_VIDEO_CODEC_OK; } - void MFH264Encoder::ShutdownTransform() + void MFVideoEncoder::ShutdownTransform() { // Detached first, so that no event handler touches the transform // while it shuts down, nor this encoder afterwards. @@ -384,7 +393,7 @@ namespace jni } } - int32_t MFH264Encoder::Encode(const webrtc::VideoFrame & frame, + int32_t MFVideoEncoder::Encode(const webrtc::VideoFrame & frame, const std::vector * frameTypes) { webrtc::EncodedImageCallback * encodeCallback = callback.load(); @@ -472,7 +481,7 @@ namespace jni return WEBRTC_VIDEO_CODEC_OK; } - HRESULT MFH264Encoder::CreateInputSample(const webrtc::VideoFrame & frame, IMFSample ** sample) + HRESULT MFVideoEncoder::CreateInputSample(const webrtc::VideoFrame & frame, IMFSample ** sample) { webrtc::scoped_refptr i420 = frame.video_frame_buffer()->ToI420(); @@ -544,7 +553,7 @@ namespace jni return S_OK; } - void MFH264Encoder::SetRates(const RateControlParameters & parameters) + void MFVideoEncoder::SetRates(const RateControlParameters & parameters) { const uint32_t bitrate = parameters.bitrate.get_sum_bps(); @@ -564,21 +573,21 @@ namespace jni } } - webrtc::VideoEncoder::EncoderInfo MFH264Encoder::GetEncoderInfo() const + webrtc::VideoEncoder::EncoderInfo MFVideoEncoder::GetEncoderInfo() const { EncoderInfo info; info.implementation_name = implementationName; info.is_hardware_accelerated = true; info.supports_native_handle = false; info.supports_simulcast = false; - info.scaling_settings = ScalingSettings(kLowH264QpThreshold, kHighH264QpThreshold); + info.scaling_settings = outputProcessor.GetScalingSettings(); // NV12 needs even dimensions. info.requested_resolution_alignment = 2; return info; } - void MFH264Encoder::OnTransformEvent(MediaEventType type, HRESULT status) + void MFVideoEncoder::OnTransformEvent(MediaEventType type, HRESULT status) { switch (type) { case METransformNeedInput: @@ -605,7 +614,7 @@ namespace jni } } - void MFH264Encoder::ProcessOutput() + void MFVideoEncoder::ProcessOutput() { MFT_OUTPUT_STREAM_INFO info = {}; HRESULT hr = transform->GetOutputStreamInfo(outputStreamId, &info); @@ -674,7 +683,7 @@ namespace jni DeliverOutput(sample.Get()); } - void MFH264Encoder::DeliverOutput(IMFSample * sample) + void MFVideoEncoder::DeliverOutput(IMFSample * sample) { LONGLONG sampleTime = 0; sample->GetSampleTime(&sampleTime); @@ -693,47 +702,6 @@ namespace jni std::span bitstream(data, length); - // Encoders put the parameter sets in front of the first key frame, - // not necessarily in front of every one, but a receiver that joins - // later needs them with the key frame it starts from. - std::vector parameterSetsFound; - bool hasSps = false; - bool hasPps = false; - - for (const webrtc::H264::NaluIndex & nalu : webrtc::H264::FindNaluIndices(bitstream)) { - const size_t end = nalu.payload_start_offset + nalu.payload_size; - const webrtc::H264::NaluType type = webrtc::H264::ParseNaluType(data[nalu.payload_start_offset]); - - if (type == webrtc::H264::NaluType::kSps || type == webrtc::H264::NaluType::kPps) { - hasSps |= type == webrtc::H264::NaluType::kSps; - hasPps |= type == webrtc::H264::NaluType::kPps; - - parameterSetsFound.insert(parameterSetsFound.end(), data + nalu.start_offset, data + end); - } - else if (type == webrtc::H264::NaluType::kIdr) { - keyFrame = true; - } - } - - if (hasSps && hasPps) { - parameterSets = std::move(parameterSetsFound); - } - - const bool prependParameterSets = keyFrame && !hasSps && !parameterSets.empty(); - const size_t size = length + (prependParameterSets ? parameterSets.size() : 0); - - webrtc::scoped_refptr encoded = webrtc::EncodedImageBuffer::Create(size); - uint8_t * target = encoded->data(); - - if (prependParameterSets) { - std::copy(parameterSets.begin(), parameterSets.end(), target); - target += parameterSets.size(); - } - - std::copy(data, data + length, target); - - buffer->Unlock(); - PendingFrame pending; { @@ -743,6 +711,7 @@ namespace jni if (found == pendingFrames.end()) { RTC_LOG(LS_WARNING) << "Media Foundation encoder produced a frame for no input, time " << sampleTime; + buffer->Unlock(); return; } @@ -753,26 +722,22 @@ namespace jni } webrtc::EncodedImage image; - image.SetEncodedData(encoded); image._encodedWidth = codecSettings.width; image._encodedHeight = codecSettings.height; image.SetRtpTimestamp(pending.rtpTimestamp); image.capture_time_ms_ = pending.captureTimeMs; image.ntp_time_ms_ = pending.ntpTimeMs; image.rotation_ = pending.rotation; - image.set_frame_type(keyFrame - ? webrtc::VideoFrameType::kVideoFrameKey - : webrtc::VideoFrameType::kVideoFrameDelta); - - bitstreamParser.ParseBitstream(std::span(encoded->data(), encoded->size())); - image.qp_ = bitstreamParser.GetLastSliceQp().value_or(-1); webrtc::CodecSpecificInfo info; - info.codecType = webrtc::kVideoCodecH264; - info.codecSpecific.H264.packetization_mode = packetizationMode; - info.codecSpecific.H264.temporal_idx = webrtc::kNoTemporalIdx; - info.codecSpecific.H264.base_layer_sync = false; - info.codecSpecific.H264.idr_frame = keyFrame; + const bool processed = outputProcessor.Process(bitstream, keyFrame, image, info); + + buffer->Unlock(); + + if (!processed) { + failed = true; + return; + } webrtc::EncodedImageCallback * encodeCallback = callback.load(); diff --git a/webrtc-jni/src/main/cpp/src/media/video/codec/windows/MFVideoEncoderFactory.cpp b/webrtc-jni/src/main/cpp/src/media/video/codec/windows/MFVideoEncoderFactory.cpp index 7869653d..5e127518 100644 --- a/webrtc-jni/src/main/cpp/src/media/video/codec/windows/MFVideoEncoderFactory.cpp +++ b/webrtc-jni/src/main/cpp/src/media/video/codec/windows/MFVideoEncoderFactory.cpp @@ -16,56 +16,88 @@ #include "media/video/codec/windows/MFVideoEncoderFactory.h" #include "media/video/codec/windows/MFEncoderUtils.h" -#include "media/video/codec/windows/MFH264Encoder.h" +#include "media/video/codec/windows/MFVideoEncoder.h" #include "platform/windows/ComInitializer.h" #include "platform/windows/MFInitializer.h" #include "api/video_codecs/h264_profile_level_id.h" #include "modules/video_coding/codecs/h264/include/h264.h" +#include "api/video/video_codec_type.h" +#include "api/video_codecs/video_codec.h" #include "rtc_base/logging.h" using Microsoft::WRL::ComPtr; namespace jni { + namespace + { + bool HasHardwareEncoder(const GUID & format) + { + std::vector> encoders; + + if (FAILED(EnumerateHardwareEncoders(format, encoders)) || encoders.empty()) { + return false; + } + + RTC_LOG(LS_INFO) << "Media Foundation hardware encoder: " << GetTransformName(encoders.front().Get()); + + return true; + } + } + std::unique_ptr MFVideoEncoderFactory::Create() { - std::vector> encoders; + bool h264 = false; + bool av1 = false; try { ComInitializer comInitializer; MFInitializer mfInitializer; - if (FAILED(EnumerateHardwareH264Encoders(encoders))) { - return nullptr; - } + h264 = HasHardwareEncoder(MFVideoFormat_H264); + av1 = HasHardwareEncoder(MFVideoFormat_AV1); } catch (...) { return nullptr; } - if (encoders.empty()) { - RTC_LOG(LS_INFO) << "No Media Foundation hardware H.264 encoder found"; - + if (!h264 && !av1) { + RTC_LOG(LS_INFO) << "No Media Foundation hardware encoder found"; return nullptr; } - RTC_LOG(LS_INFO) << "Media Foundation hardware H.264 encoder: " << GetTransformName(encoders.front().Get()); + return std::unique_ptr(new MFVideoEncoderFactory(h264, av1)); + } - return std::unique_ptr(new MFVideoEncoderFactory()); + MFVideoEncoderFactory::MFVideoEncoderFactory(bool h264, bool av1) : + h264(h264), + av1(av1) + { } std::vector MFVideoEncoderFactory::GetSupportedFormats() const { - return { - webrtc::CreateH264Format(webrtc::H264Profile::kProfileConstrainedBaseline, webrtc::H264Level::kLevel3_1, "1"), - webrtc::CreateH264Format(webrtc::H264Profile::kProfileBaseline, webrtc::H264Level::kLevel3_1, "1") - }; + std::vector formats; + + if (h264) { + formats.push_back(webrtc::CreateH264Format(webrtc::H264Profile::kProfileConstrainedBaseline, + webrtc::H264Level::kLevel3_1, "1")); + formats.push_back(webrtc::CreateH264Format(webrtc::H264Profile::kProfileBaseline, + webrtc::H264Level::kLevel3_1, "1")); + } + if (av1) { + formats.push_back(webrtc::SdpVideoFormat::AV1Profile0()); + } + + return formats; } std::unique_ptr MFVideoEncoderFactory::Create(const webrtc::Environment & env, const webrtc::SdpVideoFormat & format) { - return std::make_unique(format); + const webrtc::VideoCodecType codec = webrtc::PayloadStringToCodecType(format.name); + + return std::make_unique(codec, format); } } diff --git a/webrtc/src/main/java/dev/onvoid/webrtc/media/video/codec/HardwareVideoEncoderFactory.java b/webrtc/src/main/java/dev/onvoid/webrtc/media/video/codec/HardwareVideoEncoderFactory.java index af1b373c..365c6a9e 100644 --- a/webrtc/src/main/java/dev/onvoid/webrtc/media/video/codec/HardwareVideoEncoderFactory.java +++ b/webrtc/src/main/java/dev/onvoid/webrtc/media/video/codec/HardwareVideoEncoderFactory.java @@ -36,7 +36,9 @@ *

* H.264 is encoded on an NVIDIA GPU with NVENC, on Windows and Linux. On * other GPUs it is encoded with the Media Foundation encoder of the driver on - * Windows, and with its VA-API encoder on Linux. + * Windows, and with its VA-API encoder on Linux. AV1 is encoded on GPUs that + * have an AV1 encoder, with NVENC or Media Foundation, as a single layer; a + * stream that asks for SVC is encoded in software. * A hardware encoder that fails to start, for example because the GPU has no * encoder sessions left, or fails while encoding, is replaced by the next one * in line, and finally by the software encoder of the same codec, so a stream diff --git a/webrtc/src/test/java/dev/onvoid/webrtc/HardwareVideoEncoderIntegrationTest.java b/webrtc/src/test/java/dev/onvoid/webrtc/HardwareVideoEncoderIntegrationTest.java index 8777a2c4..6e80037a 100644 --- a/webrtc/src/test/java/dev/onvoid/webrtc/HardwareVideoEncoderIntegrationTest.java +++ b/webrtc/src/test/java/dev/onvoid/webrtc/HardwareVideoEncoderIntegrationTest.java @@ -41,7 +41,8 @@ *

* A machine without a hardware encoder skips the tests that need one, as CI * runners do. Set the system property {@code webrtc.test.hardwareEncoder} to - * {@code true} on a machine that has one, to make those tests fail instead. + * {@code true} on a machine that has one, to make those tests fail instead, + * and {@code webrtc.test.hardwareAv1Encoder} for a GPU that encodes AV1. */ @Execution(ExecutionMode.SAME_THREAD) class HardwareVideoEncoderIntegrationTest extends TestBase { @@ -50,6 +51,8 @@ class HardwareVideoEncoderIntegrationTest extends TestBase { private static final boolean HARDWARE_REQUIRED = Boolean.getBoolean("webrtc.test.hardwareEncoder"); + private static final boolean HARDWARE_AV1_REQUIRED = Boolean.getBoolean("webrtc.test.hardwareAv1Encoder"); + private static final String OS = System.getProperty("os.name").toLowerCase(Locale.ROOT); /** @@ -61,6 +64,8 @@ class HardwareVideoEncoderIntegrationTest extends TestBase { && "1".equals(codec.getSDPFmtp().get("packetization-mode")) && codec.getSDPFmtp().getOrDefault("profile-level-id", "").startsWith("42e0"); + private static final Predicate AV1 = codec -> "AV1".equalsIgnoreCase(codec.getName()); + @Test void hardwareKeepsCodecs() { @@ -80,18 +85,27 @@ void hardwareEncodesH264() throws Exception { .build(); try { - String implementation = encoderImplementation(hardware); + assertHardware(encoderImplementation(hardware, H264), HARDWARE_REQUIRED); + } + finally { + hardware.dispose(); + } + } - boolean hardwareUsed = implementation.startsWith("NVENC") - || implementation.startsWith("VA-API") - || implementation.contains("MediaFoundation"); + @Test + void hardwareEncodesAv1() throws Exception { + assumeTrue(OS.contains("win") || OS.contains("linux"), + "hardware encoders are implemented on Windows and Linux only"); - if (HARDWARE_REQUIRED) { - assertTrue(hardwareUsed, implementation); - } - else { - assumeTrue(hardwareUsed, "no hardware encoder: " + implementation); - } + PeerConnectionFactory hardware = PeerConnectionFactory.builder() + .setAudioDeviceModule(audioDevModule) + .setVideoEncoderFactory(new HardwareVideoEncoderFactory()) + .build(); + + try { + // Frames arrive either way: from the GPU, or from libaom where the + // GPU has no AV1 encoder. + assertHardware(encoderImplementation(hardware, AV1), HARDWARE_AV1_REQUIRED); } finally { hardware.dispose(); @@ -103,19 +117,33 @@ void defaultEncodesH264InSoftware() throws Exception { assumeFalse(OS.contains("mac"), "macOS encodes H.264 through VideoToolbox by default"); // The shared factory uses the default encoders. - String implementation = encoderImplementation(factory); + String implementation = encoderImplementation(factory, H264); assertTrue(implementation.contains("OpenH264"), implementation); } + private static void assertHardware(String implementation, boolean required) { + boolean hardwareUsed = implementation.startsWith("NVENC") + || implementation.startsWith("VA-API") + || implementation.contains("MediaFoundation"); + + if (required) { + assertTrue(hardwareUsed, implementation); + } + else { + assumeTrue(hardwareUsed, "no hardware encoder: " + implementation); + } + } + /** - * Sends H.264 through a call until frames arrive, and returns what the - * sender reports its encoder to be. + * Sends video in the preferred codec through a call until frames arrive, + * and returns what the sender reports its encoder to be. */ - private static String encoderImplementation(PeerConnectionFactory factory) throws Exception { + private static String encoderImplementation(PeerConnectionFactory factory, + Predicate codec) throws Exception { CountDownLatch received = new CountDownLatch(10); - try (TestMediaCall call = new TestMediaCall(factory, true, false, H264)) { + try (TestMediaCall call = new TestMediaCall(factory, true, false, codec)) { call.negotiate(); RTCRtpReceiver receiver = call.getReceiver("video");