feat: decode H.264 and VP9 with NVDEC on NVIDIA GPUs on Linux

[devopvoid]

Important

Not built or run on Linux, and never on an NVIDIA GPU. It was written on a Mac. The sources compile against WebRTC's headers there (syntax only, with Linux defines); the Linux CI jobs compile and link it for the first time, and it needs a run on a machine with an NVIDIA GPU: see Needs testing below.

HardwareVideoDecoderFactory now decodes on Linux where there is an NVIDIA GPU: H.264 in all its profiles and VP9 in profile 0, through the NVDEC engine. It is the decoder counterpart of the NVENC encoder (#312), and loads the same way. The Java API is unchanged, and so is what gets negotiated.

Platform	HardwareVideoDecoderFactory
Windows	H.264, AV1 and VP9 on the GPU, through Media Foundation (#315, #319)
Linux	H.264 and VP9 (profile 0) on NVIDIA GPUs, through NVDEC; VA-API (Intel, AMD) is not there yet
macOS	H.264 and VP9 through VideoToolbox (#318)

Behavior

• Loading: libcuda.so.1 and libnvcuvid.so.1 are opened with dlopen when the factory is made, so a machine without the NVIDIA driver is not affected and decodes in software as before. The factory is offered only where the first CUDA device decodes at least one of the codecs (cuvidGetDecoderCaps, 8 bit 4:2:0 into NV12).
• Negotiation: the formats are the ones the software factory offers (H.264 in all its profiles, VP9 profile 0), so negotiation does not change.
• Fallback: FallbackVideoDecoder hands the stream to the software decoder when NVDEC returns WEBRTC_VIDEO_CODEC_FALLBACK_SOFTWARE. That happens for another profile or bit depth, a size the GPU does not decode (checked against the caps of the device), a decoder that fails, and VP9 frames with spatial layers: their layers reach a decoder without the superframe index that says where each ends, and NVDEC is not known to take that, where libvpx does. A first frame that is not a key frame asks WebRTC for one.
• Diagnostics: the decoder shows in the decoderImplementation stat of inbound-rtp as NVDEC (<GPU name>).

Native side

• NvdecVideoDecoder uses NVDEC's own parser, which finds the pictures in an encoded image and calls back for the sequence (which creates the decoder), each picture to decode, and each to show. With no display delay all of that happens inside Decode, on the caller's thread. A decoded picture is NV12 in GPU memory; it is copied to system memory with cuMemcpy2D, cropped to the picture within the surface (1088 rows for 1080), and converted to I420 with libyuv, as MFVideoDecoder does.
  A frame is matched to its input by a timestamp counted per image, so the decoder does not depend on frames coming out in input order. A resolution change at a key frame makes the decoder again.
• NvdecLibrary loads the two libraries once, probes the device, and keeps the primary CUDA context, the way NvencLibrary does. NvdecContextScope makes it current for the calls.
• NvdecVideoDecoderFactory and the Linux platform hook (LinuxHardwareVideoCodecFactories.cpp), which had no decoders.
• Headers: dependencies/nvdec has dynlink_cuda.h, dynlink_cuviddec.h and dynlink_nvcuvid.h of nv-codec-headers at tag n12.0.16.1, unchanged: the version of the NVENC header, and the same files webrtc-java-media carries for FFmpeg (feat: decode video in hardware in the media player #320). They define the structures NVDEC's parser and decoder take, which would be error-prone to declare by hand. They are compile-time only. Their licenses are gathered in LICENSE and installed into the Linux platform jars under META-INF/licenses/nvdec.
• CMake: the NVDEC sources and include path are for Linux only; Windows still decodes through Media Foundation.
• Left out on purpose: AV1, until it can be tried on an RTX 30 or newer; VA-API for Intel and AMD GPUs, which needs the slice and frame headers parsed by hand.

Testing

• HardwareVideoDecoderIntegrationTest runs on Linux too: the H.264 test and the VP9 tests (hardwareDecodesVp9, hardwareFollowsResolutionChange, vp9NeedsNoKeyFrames, vp9TemporalLayers, vp9SpatialLayers) check there what they check on Windows, and the implementation name NVDEC counts as hardware. A decoder is required with -Dwebrtc.test.hardwareDecoder=true (H.264) and -Dwebrtc.test.hardwareVp9Decoder=true (VP9); without them the tests are skipped where there is no decoder.
• On an Apple M2 (macOS 14.5) the build and the decoder tests run as before (8 passed, 2 skipped). The new sources were compiled with clang++ -std=c++20 -Wall -Wextra -fsyntax-only against the project's WebRTC headers with Linux defines, with a negative control to show that the check reports errors. The only warning is the unused env parameter that the NVENC factory has too.

Not verified yet:

• Everything on Linux: the build and the link, and NVDEC on real H.264 and VP9 streams.
• The layout of the decoded surface. The chroma plane is read from pitch x aligned height and cropped on the copy, following NVIDIA's sample; it could not be checked.
• Several decoders at once on the one CUDA context. cuvidCtxLock is not used.
• ARM64 Linux with an NVIDIA GPU, which CI builds but cannot run.
• Streams that reorder frames: the frames are matched by timestamp, but nothing here sends B-frames.
• CI runners have no GPU, so the tests are skipped there.

Needs testing

On Linux with an NVIDIA GPU and its driver (libcuda.so.1 and libnvcuvid.so.1 present):

mvn -pl webrtc test -Dtest=HardwareVideoDecoderIntegrationTest -Dwebrtc.test.hardwareDecoder=true -Dwebrtc.test.hardwareVp9Decoder=true

This fails unless H.264 and VP9 are decoded in hardware. The decoderImplementation stat of a call should read NVDEC (<GPU name>). The log line NVDEC available on <GPU>, H.264: .., VP9: .. tells what the device offers. Worth looking at:

• vp9NeedsNoKeyFrames and hardwareFollowsResolutionChange: how many key frames the receiver asks for, and whether a new size makes the decoder again.
• the picture itself, for a stream whose height is not a multiple of 16 (the crop).
• a call with several video streams at once.

Docs

The video codecs guide (docs/guide/advanced/video-codecs.md) has NVDEC in the Linux row and says what it needs. The Javadoc of HardwareVideoDecoderFactory has the same.