feat: decode H.264 and VP9 with NVDEC on NVIDIA GPUs on Linux

docs/guide/advanced/video-codecs.md

@@ -52,14 +52,16 @@ PeerConnectionFactory factory = PeerConnectionFactory.builder()
 | Platform | Hardware decoding |
 |---|---|
 | Windows | H.264, AV1 and VP9 (profile 0), on GPUs that decode them, through the Media Foundation decoders of Windows on Direct3D 11 (DXVA) |
-| Linux | Not yet; decoding is in software |
+| Linux | H.264 and VP9 (profile 0) on NVIDIA GPUs, through NVDEC; VA-API, for Intel and AMD GPUs, is not there yet |
 | macOS | H.264 through VideoToolbox, as with `DefaultVideoDecoderFactory`, and VP9 (profile 0) through VideoToolbox's VP9 decoder, on Macs that have one |
 
+On Linux, NVDEC needs an NVIDIA driver, which provides `libcuda.so.1` and `libnvcuvid.so.1`; both are loaded when the factory is made, so a machine without the driver is not affected. Streams with spatial layers are decoded in software.
+
 AV1 on Windows needs the *AV1 Video Extension*, which Windows 11 includes, and VP9 the *VP9 Video Extensions*, a free component of the Microsoft Store that a system may not have; without it, VP9 is decoded in software. Streams with spatial layers (SVC) are decoded in software too, since their layers reach a decoder without the index that says where each ends. Decoded frames are copied from GPU memory back to system memory, where WebRTC's frames are, so hardware decoding pays off mostly at high resolutions and with many streams; at low resolutions WebRTC's software decoders are about as cheap. A hardware decoder that fails, or turns out to decode in software, is replaced by the software decoder, which starts with the next key frame.
 
 On macOS, VP9 decoding in hardware saves CPU: in a measurement on an Apple M2 it took a fraction of the processor time libvpx needs, but multi-threaded libvpx was about as fast on the clock. VideoToolbox decodes VP9 in hardware only where the Mac has a decoder for it, which Apple silicon does; a Mac without one decodes VP9 with libvpx as before. Profile 2 (10 bit), sizes below 64x64 or above 4096x4096, and streams with spatial layers (SVC) are decoded with libvpx too.
 
-Which decoder a stream uses shows in the `decoderImplementation` statistic of its `inbound-rtp` stats, e.g. `MediaFoundation (Microsoft H264 Video Decoder MFT)`, `MediaFoundation (AV1VideoExtension)` or `VideoToolbox (VP9)`.
+Which decoder a stream uses shows in the `decoderImplementation` statistic of its `inbound-rtp` stats, e.g. `MediaFoundation (Microsoft H264 Video Decoder MFT)`, `MediaFoundation (AV1VideoExtension)`, `VideoToolbox (VP9)` or `NVDEC (NVIDIA GeForce RTX 4070)`.
 
 ### Native Codecs
 

webrtc-jni/src/main/cpp/CMakeLists.txt

@@ -63,6 +63,11 @@ file(GLOB SOURCES_MEDIA_VIDEO_CODEC_OS "src/media/video/codec/${SOURCE_TARGET}/*
 if(WIN32 OR LINUX)
     file(GLOB SOURCES_MEDIA_VIDEO_CODEC_NVENC "src/media/video/codec/nvenc/*.cpp")
 endif()
+# NVDEC the same, for decoding. Only Linux has a decoder built on it so far;
+# Windows decodes through Media Foundation.
+if(LINUX)
+    file(GLOB SOURCES_MEDIA_VIDEO_CODEC_NVDEC "src/media/video/codec/nvdec/*.cpp")
+endif()
 file(GLOB SOURCES_MEDIA_VIDEO_DESKTOP "src/media/video/desktop/*.cpp")
 file(GLOB SOURCES_MEDIA_VIDEO_DESKTOP_OS "src/media/video/desktop/${SOURCE_TARGET}/*.cpp")
 file(GLOB SOURCES_MEDIA_VIDEO_OS "src/media/video/${SOURCE_TARGET}/*.cpp")
@@ -80,6 +85,7 @@ list(APPEND SOURCES
     ${SOURCES_MEDIA_VIDEO_CODEC}
     ${SOURCES_MEDIA_VIDEO_CODEC_OS}
     ${SOURCES_MEDIA_VIDEO_CODEC_NVENC}
+    ${SOURCES_MEDIA_VIDEO_CODEC_NVDEC}
     ${SOURCES_MEDIA_VIDEO_DESKTOP}
     ${SOURCES_MEDIA_VIDEO_DESKTOP_OS}
     ${SOURCES_MEDIA_VIDEO_OS}
@@ -113,6 +119,9 @@ endif()
 if(LINUX)
     # libva is loaded at run time, so building needs its headers only.
     target_include_directories(${PROJECT_NAME} PRIVATE dependencies/libva/include)
+
+    # So is NVDEC, and the CUDA driver it runs on.
+    target_include_directories(${PROJECT_NAME} PRIVATE dependencies/nvdec/include)
 endif()
 
 set_target_properties(${PROJECT_NAME} PROPERTIES
@@ -180,4 +189,9 @@ if(LINUX)
         DESTINATION "${CMAKE_INSTALL_PREFIX}/META-INF/licenses/libva"
         COMPONENT Runtime
     )
+    install(FILES
+        "${CMAKE_CURRENT_SOURCE_DIR}/dependencies/nvdec/LICENSE"
+        DESTINATION "${CMAKE_INSTALL_PREFIX}/META-INF/licenses/nvdec"
+        COMPONENT Runtime
+    )
 endif()

webrtc-jni/src/main/cpp/dependencies/nvdec/LICENSE

@@ -0,0 +1,88 @@
+The headers in include/ffnvcodec come from FFmpeg's nv-codec-headers, tag n12.0.16.1, and are
+compile-time only: the library loads the NVIDIA driver's libraries at run time. Each file carries its
+own license, which applies to that file only, and they are reproduced here as they stand at the head
+of each file.
+
+==============================================================================
+dynlink_cuda.h
+==============================================================================
+This copyright notice applies to this header file only:
+
+Copyright (c) 2016
+
+Permission is hereby granted, free of charge, to any person
+obtaining a copy of this software and associated documentation
+files (the "Software"), to deal in the Software without
+restriction, including without limitation the rights to use,
+copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the software, and to permit persons to whom the
+software is furnished to do so, subject to the following
+conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+OTHER DEALINGS IN THE SOFTWARE.
+
+==============================================================================
+dynlink_cuviddec.h
+==============================================================================
+This copyright notice applies to this header file only:
+
+Copyright (c) 2010-2022 NVIDIA Corporation
+
+Permission is hereby granted, free of charge, to any person
+obtaining a copy of this software and associated documentation
+files (the "Software"), to deal in the Software without
+restriction, including without limitation the rights to use,
+copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the software, and to permit persons to whom the
+software is furnished to do so, subject to the following
+conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+OTHER DEALINGS IN THE SOFTWARE.
+
+==============================================================================
+dynlink_nvcuvid.h
+==============================================================================
+This copyright notice applies to this header file only:
+
+Copyright (c) 2010-2022 NVIDIA Corporation
+
+Permission is hereby granted, free of charge, to any person
+obtaining a copy of this software and associated documentation
+files (the "Software"), to deal in the Software without
+restriction, including without limitation the rights to use,
+copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the software, and to permit persons to whom the
+software is furnished to do so, subject to the following
+conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+OTHER DEALINGS IN THE SOFTWARE.

webrtc-jni/src/main/cpp/dependencies/nvdec/README.md

@@ -0,0 +1,18 @@
+# NVDEC headers
+
+`include/ffnvcodec` holds three headers of FFmpeg's `nv-codec-headers` at tag `n12.0.16.1`, taken
+unchanged:
+
+https://github.com/FFmpeg/nv-codec-headers/tree/n12.0.16.1/include/ffnvcodec
+
+- `dynlink_cuda.h`: the types and function signatures of the CUDA driver API that decoding needs.
+- `dynlink_cuviddec.h` and `dynlink_nvcuvid.h`: the decoder and the bitstream parser of NVDEC
+  (`libnvcuvid`).
+
+The library loads the CUDA driver (`libcuda.so.1`) and `libnvcuvid.so.1` at run time, so the headers
+are all the build needs, and a machine without an NVIDIA driver is unaffected. Version 12.0.16.1 is
+the one the NVENC header (`../nvenc`) comes from. `webrtc-java-media` carries the same files for
+FFmpeg's NVDEC hwaccels.
+
+`LICENSE` gathers the notices at the head of the files. It is installed into the Linux platform jars
+under `META-INF/licenses/nvdec`.