feat: decode video in hardware in the media player

docs/guide/media/media-files.md

@@ -204,6 +204,29 @@ The player takes over the reader it is given. That reader must not be used or cl
 It also keeps the native side of the media sources it feeds alive until it is closed, so disposing of a source or its track while the player runs is safe.
 :::
 
+## Decoding in Hardware
+
+Decoding H.264 and VP9 takes a good part of a processor at high resolutions, and for every stream of a camera wall. A player can decode them on the media engine or GPU of the machine instead. It does so when asked to, with a flag on `MediaFileSource` or `MediaPlayer`:
+
+```java
+// A file whose video is decoded in hardware, where the platform can.
+MediaFileSource source = new MediaFileSource(path, true);
+
+// The same for a player of your own.
+MediaPlayer player = new MediaPlayer(new MediaReader(path), videoSource, null, true);
+
+// What is actually in use: false if the platform or the stream has no
+// hardware decoder for it.
+boolean hardware = source.getPlayer().isHardwareDecoding();
+```
+
+The pictures are the same ones software decoding gives; in the tests, every frame of H.264 and VP9 media is compared with its software counterpart. Hardware decoding is off unless asked for.
+
+- **Platforms:** macOS through VideoToolbox, Windows through Direct3D 11, with DXVA2 for what has no Direct3D 11 decoder, and Linux through NVDEC on NVIDIA GPUs (x86-64 and ARM64; the driver provides `libcuda.so.1` and `libnvcuvid.so.1`, which are loaded when a player asks for hardware). The flag is accepted everywhere; where the platform has no hardware decoder, or the FFmpeg build has none for it (32-bit ARM, VA-API on Intel and AMD GPUs), the video is decoded in software, and `isHardwareDecoding()` says so.
+- **Codecs:** H.264 and VP9. VP8, MPEG-4, MJPEG and H.265/HEVC are decoded in software.
+- **Fallback:** a stream the hardware does not take, such as a profile it cannot decode, is decoded in software without the player noticing more than `isHardwareDecoding()` turning `false`. If the hardware fails after it has produced a picture, decoding goes on in software from the next key frame.
+- **What it saves** is processor time, not the copy: a decoded picture is read back from the media engine into system memory, and converted to I420 as WebRTC wants it. On an Apple M2, a 1080p H.264 stream took 11.6 ms of processor time per frame in software and 1.4 ms in hardware, and a 4K stream 35.7 ms against 4.2 ms, about seven to eight times less. Per frame, software on all cores is faster on the clock, which does not matter at playback speed. These are figures for one machine and for synthetic media, not a promise.
+
 ## Playing a Live Stream
 
 A source can also be a live stream behind an `rtsp://` URL, which is what IP cameras, video recorders and most media servers offer. Everything above works the same way; only the URL differs:

webrtc-java-media/src/main/cpp/CMakeLists.txt

@@ -170,3 +170,16 @@ add_custom_command(TARGET ${PROJECT_NAME} POST_BUILD
     COMMAND ${CMAKE_COMMAND} -E copy_if_different
             "${FFMPEG_NOTICE_FILE}" "${FFMPEG_LICENSES_DIR}/NOTICE.txt"
     COMMENT "Collecting the FFmpeg license and notice for the platform jar")
+
+# The headers FFmpeg was built against for NVDEC have licenses of their own,
+# which go along as well.
+if(FFMPEG_NVDEC)
+    set(FFNVCODEC_LICENSES_DIR "${NATIVES_DIR}/META-INF/licenses/ffnvcodec")
+
+    add_custom_command(TARGET ${PROJECT_NAME} POST_BUILD
+        COMMAND ${CMAKE_COMMAND} -E make_directory "${FFNVCODEC_LICENSES_DIR}"
+        COMMAND ${CMAKE_COMMAND} -E copy_if_different
+                "${CMAKE_CURRENT_SOURCE_DIR}/dependencies/ffnvcodec/LICENSE"
+                "${FFNVCODEC_LICENSES_DIR}/LICENSE"
+        COMMENT "Collecting the license of the NVDEC headers for the platform jar")
+endif()

webrtc-java-media/src/main/cpp/dependencies/ffmpeg/CMakeLists.txt

@@ -120,6 +120,48 @@ set(FFMPEG_COMPONENTS
     --enable-protocol=tcp
 )
 
+# Decoding on the media engine or GPU, which a player does when it is asked to.
+# H.264 and VP9 only: the codecs the module plays that platform decoders have
+# too, and ones it can fall back to software for.
+if(APPLE)
+    # VideoToolbox is part of macOS, so nothing is added to what the libraries
+    # need to load.
+    list(APPEND FFMPEG_COMPONENTS
+        --enable-videotoolbox
+        --enable-hwaccel=h264_videotoolbox
+        --enable-hwaccel=vp9_videotoolbox
+    )
+elseif(WIN32)
+    # Direct3D 11 first, DXVA2 for what has no D3D11 decoder. FFmpeg loads
+    # d3d11.dll, dxgi.dll, d3d9.dll and dxva2.dll when a device is created, so
+    # a machine without a GPU decoder loads the libraries all the same.
+    list(APPEND FFMPEG_COMPONENTS
+        --enable-d3d11va
+        --enable-dxva2
+        --enable-hwaccel=h264_d3d11va2
+        --enable-hwaccel=vp9_d3d11va2
+        --enable-hwaccel=h264_dxva2
+        --enable-hwaccel=vp9_dxva2
+    )
+elseif(LINUX AND CMAKE_SYSTEM_PROCESSOR MATCHES "^(x86_64|AMD64|amd64|aarch64|arm64|ARM64)$")
+    # NVDEC, on NVIDIA GPUs. FFmpeg loads the CUDA driver and libnvcuvid with
+    # dlopen when a device is created, so the libraries load on a machine with
+    # no NVIDIA driver. The build needs the headers of nv-codec-headers, which
+    # dependencies/ffnvcodec carries, found through a pkg-config file made
+    # below. Not for 32-bit ARM, which has no NVIDIA driver to load.
+    set(FFMPEG_NVDEC TRUE)
+
+    list(APPEND FFMPEG_COMPONENTS
+        --enable-ffnvcodec
+        --enable-nvdec
+        --enable-hwaccel=h264_nvdec
+        --enable-hwaccel=vp9_nvdec
+    )
+endif()
+
+# The parent adds the license of the NVDEC headers to the platform jar.
+set(FFMPEG_NVDEC "${FFMPEG_NVDEC}" PARENT_SCOPE)
+
 # Written next to an install once it is built, naming the components it was
 # built with. Without it, an install from before a component was added would
 # be taken as good, and a file needing that component would fail to play with
@@ -485,6 +527,30 @@ fi
         set(FFMPEG_MAKE "${MAKE_EXECUTABLE}")
     endif()
 
+    if(FFMPEG_NVDEC)
+        # configure looks for the NVDEC headers through pkg-config, so that
+        # needs to be there, and to know where the vendored headers are.
+        find_program(PKG_CONFIG_EXECUTABLE NAMES pkg-config pkgconf)
+
+        if(NOT PKG_CONFIG_EXECUTABLE)
+            message(FATAL_ERROR
+                "No pkg-config, which FFmpeg's configure uses to find the NVDEC headers. "
+                "Install it, for example with: sudo apt-get install pkg-config")
+        endif()
+
+        get_filename_component(FFNVCODEC_DIR "${CMAKE_CURRENT_SOURCE_DIR}/../ffnvcodec" ABSOLUTE)
+        set(FFNVCODEC_PC_DIR "${CMAKE_CURRENT_BINARY_DIR}/ffnvcodec-pc")
+
+        configure_file("${FFNVCODEC_DIR}/ffnvcodec.pc.in" "${FFNVCODEC_PC_DIR}/ffnvcodec.pc" @ONLY)
+
+        # Only our directory: a sysroot or library path of the host would put
+        # the includedir somewhere it is not.
+        set(FFMPEG_SHELL_PREAMBLE
+"${FFMPEG_SHELL_PREAMBLE}export PKG_CONFIG_PATH=\"${FFNVCODEC_PC_DIR}:$PKG_CONFIG_PATH\"
+unset PKG_CONFIG_SYSROOT_DIR PKG_CONFIG_LIBDIR
+")
+    endif()
+
     set(FFMPEG_BUILD_SCRIPT "${CMAKE_CURRENT_BINARY_DIR}/build-ffmpeg.sh")
 
     file(WRITE "${FFMPEG_BUILD_SCRIPT}"

webrtc-java-media/src/main/cpp/dependencies/ffnvcodec/LICENSE

@@ -0,0 +1,145 @@
+The headers in include/ffnvcodec come from FFmpeg's nv-codec-headers, tag n12.0.16.1, and are
+compile-time only: FFmpeg builds against them and loads the NVIDIA driver's libraries at run time.
+Each file carries its own license, which applies to that file only, and they are reproduced here
+as they stand at the head of each file.
+
+==============================================================================
+dynlink_cuda.h
+==============================================================================
+This copyright notice applies to this header file only:
+
+Copyright (c) 2016
+
+Permission is hereby granted, free of charge, to any person
+obtaining a copy of this software and associated documentation
+files (the "Software"), to deal in the Software without
+restriction, including without limitation the rights to use,
+copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the software, and to permit persons to whom the
+software is furnished to do so, subject to the following
+conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+OTHER DEALINGS IN THE SOFTWARE.
+
+==============================================================================
+dynlink_cuviddec.h
+==============================================================================
+This copyright notice applies to this header file only:
+
+Copyright (c) 2010-2022 NVIDIA Corporation
+
+Permission is hereby granted, free of charge, to any person
+obtaining a copy of this software and associated documentation
+files (the "Software"), to deal in the Software without
+restriction, including without limitation the rights to use,
+copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the software, and to permit persons to whom the
+software is furnished to do so, subject to the following
+conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+OTHER DEALINGS IN THE SOFTWARE.
+
+==============================================================================
+dynlink_nvcuvid.h
+==============================================================================
+This copyright notice applies to this header file only:
+
+Copyright (c) 2010-2022 NVIDIA Corporation
+
+Permission is hereby granted, free of charge, to any person
+obtaining a copy of this software and associated documentation
+files (the "Software"), to deal in the Software without
+restriction, including without limitation the rights to use,
+copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the software, and to permit persons to whom the
+software is furnished to do so, subject to the following
+conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+OTHER DEALINGS IN THE SOFTWARE.
+
+==============================================================================
+dynlink_loader.h
+==============================================================================
+This copyright notice applies to this header file only:
+
+Copyright (c) 2016
+
+Permission is hereby granted, free of charge, to any person
+obtaining a copy of this software and associated documentation
+files (the "Software"), to deal in the Software without
+restriction, including without limitation the rights to use,
+copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the software, and to permit persons to whom the
+software is furnished to do so, subject to the following
+conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+OTHER DEALINGS IN THE SOFTWARE.
+
+==============================================================================
+nvEncodeAPI.h
+==============================================================================
+nvEncodeAPI.h, the NVIDIA Video Codec SDK header this library is built with
+(API version 12.0), is licensed as follows:
+
+Copyright (c) 2010-2022 NVIDIA Corporation
+
+Permission is hereby granted, free of charge, to any person
+obtaining a copy of this software and associated documentation
+files (the "Software"), to deal in the Software without
+restriction, including without limitation the rights to use,
+copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the software, and to permit persons to whom the
+software is furnished to do so, subject to the following
+conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+OTHER DEALINGS IN THE SOFTWARE.

webrtc-java-media/src/main/cpp/dependencies/ffnvcodec/README.md

@@ -0,0 +1,21 @@
+# ffnvcodec headers
+
+`include/ffnvcodec` holds the headers of FFmpeg's `nv-codec-headers` at tag `n12.0.16.1`, taken
+unchanged:
+
+https://github.com/FFmpeg/nv-codec-headers/tree/n12.0.16.1/include/ffnvcodec
+
+- `dynlink_cuda.h`, `dynlink_cuviddec.h`, `dynlink_nvcuvid.h` and `dynlink_loader.h`: what FFmpeg
+  needs to decode with NVDEC (the `h264_nvdec` and `vp9_nvdec` hwaccels). FFmpeg loads the CUDA
+  driver (`libcuda.so.1`) and `libnvcuvid.so.1` with `dlopen` when a device is created, so the
+  build needs the headers only and a machine without an NVIDIA driver loads the libraries all the
+  same.
+- `nvEncodeAPI.h`: the same file `webrtc-jni` vendors for NVENC. FFmpeg's configure checks for it
+  too, so it is part of the set.
+
+`ffnvcodec.pc.in` is the pkg-config file FFmpeg's configure looks for; CMake fills in the path and
+puts the result where configure finds it. Version 12.0.16.1 is one of those FFmpeg n8.1 accepts
+(`ffnvcodec >= 12.0.16.1 ffnvcodec < 12.1`), and the one the NVENC header comes from.
+
+`LICENSE` gathers the notices at the head of the files. It is installed into the Linux platform
+jars under `META-INF/licenses/ffnvcodec`.

webrtc-java-media/src/main/cpp/dependencies/ffnvcodec/ffnvcodec.pc.in

@@ -0,0 +1,7 @@
+prefix=@FFNVCODEC_DIR@
+includedir=${prefix}/include
+
+Name: ffnvcodec
+Description: FFmpeg version of Nvidia Codec SDK headers
+Version: 12.0.16.1
+Cflags: -I${includedir}