feat: encode H.264 with NVENC on NVIDIA GPUs, on Windows and Linux

[devopvoid]

Stacked on #310 (which is stacked on #303): this PR targets its branch and should be merged after it.

Important

Not yet run on NVIDIA hardware. It was developed on a machine with an AMD GPU and no NVIDIA driver, so the NVENC path itself has only been built and syntax-checked. See Needs testing on NVIDIA below.

HardwareVideoEncoderFactory now encodes H.264 with NVENC where there is an NVIDIA GPU, on Windows and Linux. The Java API is unchanged.

Platform	HardwareVideoEncoderFactory encodes H.264 with
Windows	NVENC on NVIDIA GPUs, otherwise Media Foundation (AMD, Intel), otherwise OpenH264
Linux	NVENC on NVIDIA GPUs, otherwise OpenH264
macOS	VideoToolbox (unchanged)

Loading

• NVENC (nvEncodeAPI64.dll / libnvidia-encode.so.1) and the CUDA driver (nvcuda.dll / libcuda.so.1) are loaded at run time. Nothing is linked, so machines without an NVIDIA driver are unaffected.
• The build needs only nvEncodeAPI.h, vendored unchanged from FFmpeg's nv-codec-headers at tag n12.0.16.1 (NVENC API 12.0, MIT license). The license is installed into the platform jar under META-INF/licenses/nvenc/. dependencies/nvenc/README.md records the source and version.
• API 12.0 needs driver 522 or newer on Windows, 520 or newer on Linux. Older drivers are detected through NvEncodeAPIGetMaxSupportedVersion, and NVENC is then left out.
• The CUDA types NVENC needs are declared locally, so building doesn't need the CUDA toolkit.

Encoder

NvencH264Encoder runs on the primary CUDA context of the first device and encodes synchronously on the encoder thread:

• Preset P4 with ultra-low-latency tuning.
• Baseline profile with CAVLC.
• Constant bitrate with a one-frame VBV buffer.
• No B-frames, and an infinite GOP with IDR frames only when WebRTC requests a key frame.
• repeatSPSPPS, so SPS/PPS come with every IDR frame.
• Bitrate and frame rate changes applied with nvEncReconfigureEncoder, without a reset.
• Frames passed as NV12 into an input buffer NVENC allocates (nvEncCreateInputBuffer), the same approach FFmpeg uses for frames in system memory.

Fallback chain

The native HardwareVideoEncoderFactory now takes several hardware factories in order of preference and chains their encoders with FallbackVideoEncoder. Each encoder falls back to the next one and finally to software, both when it fails to start and when it fails mid-stream (e.g. when the GPU runs out of encoder sessions, which consumer GPUs limit). On Windows the order is NVENC, then Media Foundation.

Testing

• This machine (Windows, AMD Radeon RX 9070 XT, no NVIDIA driver):
  • NVENC reports "no NVIDIA driver" and H.264 goes through MediaFoundation (AMDh264Encoder), checked with -Dwebrtc.test.hardwareEncoder=true.
  • mvn -pl webrtc test: 197 tests pass.
  • -Pjni-check: 197 tests pass, no FATAL ERROR in native method.
• Linux: the NVENC and Linux-only sources pass g++ -std=c++20 -fsyntax-only against Linux headers in WSL. CI (clang) will be the first full Linux build.
• HardwareVideoEncoderIntegrationTest now runs on Linux too, and accepts NVENC (...) as well as MediaFoundation (...).

Needs testing on NVIDIA

On Windows, and ideally on Linux:

mvn -pl webrtc test -Dtest=HardwareVideoEncoderIntegrationTest -Dwebrtc.test.hardwareEncoder=true

This fails unless H.264 is actually encoded on the GPU. The encoderImplementation stat of a call should read NVENC (<GPU name>). Also worth checking:

• bitrate adaptation over a longer call
• the fallback when the GPU runs out of NVENC sessions, by opening more streams than a consumer card allows