diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 348149c..8971761 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -108,30 +108,52 @@ jobs: with: toolchain: "1.92.0" - uses: Swatinem/rust-cache@v2 - # `rawshift-video` is parked and unpublished for v1, and the facade no - # longer has a `video` feature, so this builds the crate directly. - - run: cargo build -p rawshift-video --all-features - # Assert the video build pulls in zero image crates — this is the contract - # the image/video workspace split exists to guarantee, and it must keep - # holding while the crate is parked. + - run: cargo build -p rawshift-video --features full + # The contract the image/video workspace split exists to guarantee: + # video shares only rawshift-core and rawshift-hwdec with image, and + # pulls in none of the image codec stack. - name: Assert no image crates in the video dependency tree run: | - tree=$(cargo tree -p rawshift-video --all-features --prefix none --no-dedupe) + tree=$(cargo tree -p rawshift-video --features full --prefix none --no-dedupe) echo "$tree" if echo "$tree" | grep -qiE 'rawshift-image|zune-|libheif|libwebp|img-parts|little_exif|resvg|ravif|jxl-oxide|^image '; then echo "::error::video build pulled in image crates" exit 1 fi echo "OK: video build is free of image crates" - # Assert the parked crate stays out of the facade entirely. - - name: Assert the facade does not depend on rawshift-video + # Opacity: no backend type may appear in a public signature. The crates + # wrap every backend rather than re-exporting it, so swapping one is not + # a breaking change for callers — this keeps that true. + - name: Assert backend types stay out of the public API run: | - tree=$(cargo tree -p rawshift --all-features --prefix none --no-dedupe) - if echo "$tree" | grep -q 'rawshift-video'; then - echo "::error::rawshift facade depends on the parked rawshift-video crate" + set -o pipefail + leaks=$(grep -rn --include='*.rs' \ + -E '^\s*pub (fn|struct|enum|trait|type|const)[^;{]*\b(symphonia_[a-z_]*|mp4_atom|rawshift_hwdec)::' \ + crates/rawshift-video/src crates/rawshift-video-core/src \ + crates/rawshift-video-isobmff/src crates/rawshift-video-matroska/src \ + crates/rawshift-video-hwdec/src || true) + if [ -n "$leaks" ]; then + echo "$leaks" + echo "::error::a backend type appears in a public signature" exit 1 fi - echo "OK: facade is free of rawshift-video" + echo "OK: no backend type in a public signature" + # The facade must expose video, and must still build without it. + - name: Assert the facade wires video behind its own feature + run: | + cargo tree -p rawshift --features video --prefix none --no-dedupe \ + | grep -q 'rawshift-video' \ + || { echo "::error::the facade's video feature does not pull in rawshift-video"; exit 1; } + if cargo tree -p rawshift --no-default-features --features image --prefix none --no-dedupe \ + | grep -q 'rawshift-video'; then + echo "::error::an image-only facade build pulled in rawshift-video" + exit 1 + fi + echo "OK: video is behind its own facade feature" + # Container parsing and metadata must work with no decoder at all, which + # is what every backend-less target gets. + - name: Build the containers with no codec features + run: cargo build -p rawshift-video --no-default-features --features mp4,mkv # ── Compile boundaries (issue #34) ────────────────────────────────────────── # The `hw-*` backend pins are *verified* feature flags: rawshift-hwdec holds diff --git a/AGENTS.md b/AGENTS.md index d2368c9..909084c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -38,6 +38,43 @@ dependencies are not permitted because they prevent publishing rawshift. that only updates the version requirements and lockfile, a full test + benchmark run, and a CHANGELOG.md note for any behavioral change. +### Video is outside gamut's charter + +The policy above governs **image** work. It does not apply to video, because +gamut has declared video permanently out of scope. From gamut's README: + +> **gamut is image-first.** Even where a format's codec (AV1, AV2, VVC, HEVC) +> is fundamentally a video codec, gamut implements only the intra-frame, +> still-image subset those formats use — no inter-frame prediction, no motion +> compensation, no video sequences. […] gamut will not grow video primitives. + +`gamut-isobmff` says the same at the container level: *"Image sequences/tracks +and item protection are out of scope."* It parses the HEIF still-image item +model (`meta`/`iloc`/`iinf`/`iprp`), not the movie model (`moov`/`trak`/`stbl`), +so it cannot back MP4/MOV video however much the two formats share a box +grammar. + +Consequently, for `rawshift-video` and the `rawshift-video-*` crates: + +- Do **not** open gamut issues for video containers, video codecs, or + multi-frame primitives. They will be declined as out of charter, and + `blocked-upstream` is the wrong label for them. +- Third-party dependencies are permitted where a mature library exists — + judged by the `PRINCIPLES.md` rule ("reinvent the wheel only when + necessary"), not by the upstream-first rule. Current choices: + `symphonia-format-isomp4` and `symphonia-format-mkv` for demuxing (MPL-2.0, + matching rawshift's own license; pure Rust with no build script). +- Video **decode** stays in `rawshift-hwdec`. `docs/SUPPORT.md` fixes the + reasons: software HEVC is ruled out by the patent posture, and FFmpeg/libav + is ruled out on license and portability grounds (see "Excluded" there). +- Everything else in this file still applies to video: no git dependencies, + the fixed target list, the MSRV rule, and a CHANGELOG note for behavioral + change. + +Image work is unaffected — gamut remains the upstream home for image +primitives, color, metadata, still-image container parsing, and codecs, and the +upstream-first sequence above is still mandatory there. + ## Testing Methodology Try to unit test the bulk majority of the code but functions that take in external inputs such as image/video file(s) should use test fixtures derived from external sources (which may require human sourcing as prerequisite). Also extend example binaries in `examples/` as necessary to show that each feature actually works. diff --git a/CHANGELOG.md b/CHANGELOG.md index 8728a8b..140ae5e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,10 +14,52 @@ from the published [gamut](https://github.com/visualcommons/gamut) crates. dependency is a registry package. 0.x source compatibility is a non-goal (see `docs/V1_DESIGN.md`). +### Added + +#### Video (`rawshift-video`) + +`rawshift-video` is no longer a parked placeholder: it ships containers, +tracks, timelines, metadata, and hardware-backed frame decode, and is wired +into the `rawshift` facade behind a new `video` feature. Six crates make it up +— `rawshift-video-core` (the shared vocabulary), `-isobmff`, `-matroska`, +`-symphonia` (an internal bridge), `-hwdec`, and the `rawshift-video` facade. + +- **Containers** — MP4, M4V and QuickTime on `mp4-atom` with a rawshift-owned + sample index; Matroska and WebM on `symphonia-format-mkv`. Both read tracks, + timelines, rotation, colour signalling and metadata. AVI and MXF are named + in the API and return a matchable `UnsupportedContainer`. +- **Decode** — HEVC random access points through `rawshift-hwdec`, which + covers keyframe extraction, thumbnails, scrubbing, and All-Intra camera + modes end to end. H.264's seam is final and its backend path is a follow-up. +- **Everything but decode works with no hardware decoder**, on every target + in `docs/SUPPORT.md` including `wasm32` and `musl`. A missing backend + surfaces as `VideoError::HwDecoderUnavailable`, and + `hw_decode_available()` reports capability up front. + +No software H.264 or HEVC decoder ships, and none will: `docs/SUPPORT.md` +records why. FFmpeg/libav is excluded as a dependency in any form, on license +and portability grounds. + +Video is outside gamut's charter — gamut states it "will not grow video +primitives" — so `AGENTS.md` now carves video out of the Upstream-First +Policy: video containers and codecs take third-party dependencies judged by +the `PRINCIPLES.md` maturity rule instead. + ### Changed All entries below are **breaking**, grouped by area. +#### Video and hardware decode + +- `HwCodec` gains `H264`. The enum is documented as deliberately exhaustive, + so this is that decision taken explicitly; matches on it need a new arm. + H.264 is reachable only through the new sequence seam — + `decoder(HwCodec::H264)` returns `None` and `available_codecs()` never lists + it — because no rawshift still format uses H.264. +- `MetadataNamespace` is now `#[non_exhaustive]` and gains `Quicktime` and + `Matroska`. Matches on it need a `_` arm. Both changes land together while + the workspace is pre-1.0, after which new namespaces are additive forever. + #### Package boundaries - Image formats now live in 17 independently publishable diff --git a/Cargo.lock b/Cargo.lock index f33e310..23c432b 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -365,6 +365,27 @@ version = "0.3.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "be1e0bca6c3637f992fc1cc7cbc52a78c1ef6db076dbf1059c4323d6a2048376" +[[package]] +name = "derive_more" +version = "2.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d751e9e49156b02b44f9c1815bcb94b984cdcc4396ecc32521c739452808b134" +dependencies = [ + "derive_more-impl", +] + +[[package]] +name = "derive_more-impl" +version = "2.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "799a97264921d8623a957f6c3b9011f3b5492f557bbb7a5a19b7fa6d06ba8dcb" +dependencies = [ + "proc-macro2", + "quote", + "rustc_version", + "syn", +] + [[package]] name = "digest" version = "0.10.7" @@ -539,7 +560,7 @@ version = "2.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b17c79c8672b675d538dceffda1dd5d00ba279cbcca4f75c7b4fa847741d5d31" dependencies = [ - "thiserror", + "thiserror 2.0.18", ] [[package]] @@ -575,7 +596,7 @@ checksum = "c9ce8735a1fc71d77b5361f4f126676abab8811cdd3797a6bacdb920cd20f85d" dependencies = [ "gamut-core", "gamut-ifd", - "thiserror", + "thiserror 2.0.18", ] [[package]] @@ -598,7 +619,7 @@ checksum = "09916f0aea3f9dbe1c5ac67e0e29956b9af43c8c86a0fbe6b47c597b63fd8cee" dependencies = [ "gamut-core", "md-5", - "thiserror", + "thiserror 2.0.18", ] [[package]] @@ -618,7 +639,7 @@ checksum = "99b7bb8412adf69f48657d207ff5358750ebe4efe81c2dd7b32091034f354d6b" dependencies = [ "gamut-core", "gamut-xmp", - "thiserror", + "thiserror 2.0.18", ] [[package]] @@ -684,7 +705,7 @@ dependencies = [ "gamut-icc", "gamut-iptc", "gamut-xmp", - "thiserror", + "thiserror 2.0.18", ] [[package]] @@ -730,7 +751,7 @@ checksum = "fd39a4ebd0d14800751e3ae64a0c86a07e00b77f3ac564570e0e8987d41945bb" dependencies = [ "gamut-core", "quick-xml", - "thiserror", + "thiserror 2.0.18", ] [[package]] @@ -862,7 +883,7 @@ dependencies = [ "jxl_transforms", "num-derive", "num-traits", - "thiserror", + "thiserror 2.0.18", ] [[package]] @@ -1001,6 +1022,19 @@ dependencies = [ "windows-sys", ] +[[package]] +name = "mp4-atom" +version = "0.15.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f3d7433df407a636db7047600422af07145adf6ee25e0622745b0131516dbc3c" +dependencies = [ + "derive_more", + "num", + "pastey", + "thiserror 1.0.69", + "tracing", +] + [[package]] name = "nu-ansi-term" version = "0.50.3" @@ -1010,6 +1044,39 @@ dependencies = [ "windows-sys", ] +[[package]] +name = "num" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "35bd024e8b2ff75562e5f34e7f4905839deb4b22955ef5e73d2fea1b9813cb23" +dependencies = [ + "num-bigint", + "num-complex", + "num-integer", + "num-iter", + "num-rational", + "num-traits", +] + +[[package]] +name = "num-bigint" +version = "0.4.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c89e69e7e0f03bea5ef08013795c25018e101932225a656383bd384495ecc367" +dependencies = [ + "num-integer", + "num-traits", +] + +[[package]] +name = "num-complex" +version = "0.4.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "73f88a1307638156682bada9d7604135552957b7818057dcef22705b4d509495" +dependencies = [ + "num-traits", +] + [[package]] name = "num-derive" version = "0.4.2" @@ -1021,6 +1088,36 @@ dependencies = [ "syn", ] +[[package]] +name = "num-integer" +version = "0.1.47" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7ce2d95d4b3734dc35aa2f45e1aa22cd416814592a4f9d9205e11affd5b8e10b" +dependencies = [ + "num-traits", +] + +[[package]] +name = "num-iter" +version = "0.1.46" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c92800bd69a1eac91786bcfe9da64a897eb72911b8dc3095decbd07429e8048b" +dependencies = [ + "num-integer", + "num-traits", +] + +[[package]] +name = "num-rational" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f83d14da390562dca69fc84082e73e548e1ad308d24accdedd2720017cb37824" +dependencies = [ + "num-bigint", + "num-integer", + "num-traits", +] + [[package]] name = "num-traits" version = "0.2.19" @@ -1071,6 +1168,12 @@ dependencies = [ "windows-link", ] +[[package]] +name = "pastey" +version = "0.2.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2ee67f1008b1ba2321834326597b8e186293b049a023cdef258527550b9935b4" + [[package]] name = "pico-args" version = "0.5.0" @@ -1184,6 +1287,7 @@ name = "rawshift" version = "0.1.1" dependencies = [ "rawshift-image", + "rawshift-video", ] [[package]] @@ -1202,7 +1306,7 @@ version = "0.1.1" dependencies = [ "gamut-color", "libloading", - "thiserror", + "thiserror 2.0.18", ] [[package]] @@ -1255,7 +1359,7 @@ dependencies = [ "resvg", "serde", "serde_json", - "thiserror", + "thiserror 2.0.18", "tiff", "tokio", "tracing", @@ -1297,7 +1401,7 @@ dependencies = [ "rawshift-core", "rawshift-hwdec", "serde", - "thiserror", + "thiserror 2.0.18", ] [[package]] @@ -1487,6 +1591,62 @@ dependencies = [ [[package]] name = "rawshift-video" version = "0.1.1" +dependencies = [ + "rawshift-core", + "rawshift-video-core", + "rawshift-video-hwdec", + "rawshift-video-isobmff", + "rawshift-video-matroska", +] + +[[package]] +name = "rawshift-video-core" +version = "0.1.1" +dependencies = [ + "gamut-color", + "rawshift-core", + "serde", + "thiserror 2.0.18", +] + +[[package]] +name = "rawshift-video-hwdec" +version = "0.1.1" +dependencies = [ + "rawshift-core", + "rawshift-hwdec", + "rawshift-video-core", +] + +[[package]] +name = "rawshift-video-isobmff" +version = "0.1.1" +dependencies = [ + "mp4-atom", + "rawshift-core", + "rawshift-video-core", + "tracing", +] + +[[package]] +name = "rawshift-video-matroska" +version = "0.1.1" +dependencies = [ + "rawshift-core", + "rawshift-video-core", + "rawshift-video-symphonia", + "symphonia-core", + "symphonia-format-mkv", +] + +[[package]] +name = "rawshift-video-symphonia" +version = "0.1.1" +dependencies = [ + "rawshift-core", + "rawshift-video-core", + "symphonia-core", +] [[package]] name = "rayon" @@ -1540,6 +1700,12 @@ dependencies = [ "regex-syntax", ] +[[package]] +name = "regex-lite" +version = "0.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cab834c73d247e67f4fae452806d17d3c7501756d98c8808d7c9c7aa7d18f973" + [[package]] name = "regex-syntax" version = "0.8.10" @@ -1578,6 +1744,15 @@ version = "0.20.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6c20b6793b5c2fa6553b250154b78d6d0db37e72700ae35fad9387a46f487c97" +[[package]] +name = "rustc_version" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92" +dependencies = [ + "semver", +] + [[package]] name = "rustversion" version = "1.0.22" @@ -1617,6 +1792,12 @@ version = "1.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" +[[package]] +name = "semver" +version = "1.0.28" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd" + [[package]] name = "serde" version = "1.0.228" @@ -1756,6 +1937,56 @@ dependencies = [ "siphasher", ] +[[package]] +name = "symphonia-common" +version = "0.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2acc3fcc18ec9b8cdd48614e259c4cf0d27b71d41e5d9b120b42c5adab12d7c4" +dependencies = [ + "log", + "symphonia-core", + "symphonia-metadata", +] + +[[package]] +name = "symphonia-core" +version = "0.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "01c412864d599d4750d0c3d684d7e093ec05e5309681ef5252cc1096a437f6e0" +dependencies = [ + "bitflags 2.11.0", + "bytemuck", + "lazy_static", + "log", + "num-complex", + "smallvec", +] + +[[package]] +name = "symphonia-format-mkv" +version = "0.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d015c5c0558864665894b3f4cbd95e10abb01b9c868e751c72670f326a56360e" +dependencies = [ + "lazy_static", + "log", + "symphonia-common", + "symphonia-core", +] + +[[package]] +name = "symphonia-metadata" +version = "0.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "83713a97705d77bdef7cdbc0768fd6e5a54e4cd7e48d60a806ae85639e2c87c6" +dependencies = [ + "lazy_static", + "log", + "regex-lite", + "smallvec", + "symphonia-core", +] + [[package]] name = "syn" version = "2.0.117" @@ -1767,13 +1998,33 @@ dependencies = [ "unicode-ident", ] +[[package]] +name = "thiserror" +version = "1.0.69" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6aaf5339b578ea85b50e080feb250a3e8ae8cfcdff9a461c9ec2904bc923f52" +dependencies = [ + "thiserror-impl 1.0.69", +] + [[package]] name = "thiserror" version = "2.0.18" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4288b5bcbc7920c07a1149a35cf9590a2aa808e0bc1eafaade0b80947865fbc4" dependencies = [ - "thiserror-impl", + "thiserror-impl 2.0.18", +] + +[[package]] +name = "thiserror-impl" +version = "1.0.69" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4fee6c4efc90059e10f81e6d42c60a18f76588c3d74cb83a0b242a2b6c7504c1" +dependencies = [ + "proc-macro2", + "quote", + "syn", ] [[package]] diff --git a/Cargo.toml b/Cargo.toml index dd400d3..8e70e41 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -27,6 +27,11 @@ members = [ "crates/rawshift-image-tiff", "crates/rawshift-image-webp", "crates/rawshift-video", + "crates/rawshift-video-core", + "crates/rawshift-video-isobmff", + "crates/rawshift-video-matroska", + "crates/rawshift-video-symphonia", + "crates/rawshift-video-hwdec", ] [workspace.package] @@ -61,6 +66,25 @@ gamut-heic = "0.2.2" # backend-less HEIC enumeration/decode-pipeline tests. gamut-isobmff = "2.0.1" +# symphonia — demuxing for rawshift's video crates. MPL-2.0 (rawshift's own +# license), pure Rust with no build script, MSRV 1.85, so it builds on every +# target in docs/SUPPORT.md including wasm32, musl, iOS and Android. gamut's +# charter excludes video, so this is a direct dependency rather than an +# upstream ask — see "Video is outside gamut's charter" in AGENTS.md. +# default-features off keeps symphonia-core's optional rustfft (audio DSP) out +# of the tree; rawshift never decodes audio. +symphonia-core = { version = "0.6.1", default-features = false } +symphonia-format-mkv = "0.6.1" + +# mp4-atom — the ISOBMFF box layer under rawshift-video-isobmff. MIT, pure +# Rust, no build script. It replaced symphonia-format-isomp4, which is +# audio-first and caps the hvcC record at 1 KB — real HEVC records are 2-3 KB, +# so every HEVC MP4 and MOV failed to open. mp4-atom additionally surfaces the +# tkhd matrix, the colr box and ProRes-shaped sample entries, all of which the +# audio-first reader hides. rawshift owns the sample-index and demux semantics +# on top; mp4-atom deliberately does no interpretation. +mp4-atom = "0.15.0" + # Shared infrastructure dependencies (used across multiple workspace crates). thiserror = "2.0" tracing = "0.1" @@ -92,7 +116,12 @@ rawshift-image-ppm = { path = "crates/rawshift-image-ppm", version = "0.1.1", de rawshift-image-svg = { path = "crates/rawshift-image-svg", version = "0.1.1", default-features = false } rawshift-image-tiff = { path = "crates/rawshift-image-tiff", version = "0.1.1", default-features = false } rawshift-image-webp = { path = "crates/rawshift-image-webp", version = "0.1.1", default-features = false } -rawshift-video = { path = "crates/rawshift-video", version = "0.1.1" } +rawshift-video = { path = "crates/rawshift-video", version = "0.1.1", default-features = false } +rawshift-video-core = { path = "crates/rawshift-video-core", version = "0.1.1" } +rawshift-video-isobmff = { path = "crates/rawshift-video-isobmff", version = "0.1.1" } +rawshift-video-matroska = { path = "crates/rawshift-video-matroska", version = "0.1.1" } +rawshift-video-symphonia = { path = "crates/rawshift-video-symphonia", version = "0.1.1" } +rawshift-video-hwdec = { path = "crates/rawshift-video-hwdec", version = "0.1.1", default-features = false } # Profiles must live at the workspace root — Cargo ignores profiles declared in # member manifests. diff --git a/PRINCIPLES.md b/PRINCIPLES.md index 0a55769..02769ff 100644 --- a/PRINCIPLES.md +++ b/PRINCIPLES.md @@ -2,7 +2,7 @@ - Stateless: The library should assume nothing about the state to support portability and parallelization. - Separation of IO and CPU: Writing good IO-heavy and CPU-heavy code can be tough in different ways so we separate it where possible to simplify benching. -- Reinvent the wheel only when necessary: We should aim to use existing mature libraries for functionality; but often there are libraries that are either: lacking low-level features, non-performant for specific use cases, or insufficiently mature. For image primitives, codecs, containers, and metadata, the mature library is [gamut](https://github.com/visualcommons/gamut) — improve it upstream rather than reimplementing here (see the Upstream-First Policy in AGENTS.md). Accepted exceptions: GIF (`gif`), SVG (`resvg`), PPM (`zune-ppm`). +- Reinvent the wheel only when necessary: We should aim to use existing mature libraries for functionality; but often there are libraries that are either: lacking low-level features, non-performant for specific use cases, or insufficiently mature. For image primitives, codecs, containers, and metadata, the mature library is [gamut](https://github.com/visualcommons/gamut) — improve it upstream rather than reimplementing here (see the Upstream-First Policy in AGENTS.md). Accepted exceptions: GIF (`gif`), SVG (`resvg`), PPM (`zune-ppm`). For **video** containers and codecs there is no upstream to improve — gamut's charter excludes video (see "Video is outside gamut's charter" in AGENTS.md) — so this principle applies directly: `symphonia` backs demuxing, and platform hardware decoders back frame decode. ## Safety Boundaries @@ -13,6 +13,10 @@ - `crates/rawshift-image/src/processing`: Unsafe Rust is acceptable as long as it is constrained to hot paths. - `crates/rawshift-image/src/transforms`: Unsafe Rust is acceptable as long as it is constrained to hot paths. - `crates/rawshift-hwdec`: Unsafe platform FFI is permitted — `#![deny(unsafe_op_in_unsafe_fn)]`, every public item is a safe wrapper, every unsafe block documents its invariants. + - `crates/rawshift-hwdec/src/codec`: Safe Rust is strictly required. Bitstream parsing, picture-order derivation, and reference-picture management are pure functions over bytes; they carry no platform types and must stay testable on every target, so they live outside the backend `cfg` and outside the FFI. +- `crates/rawshift-video-core`: `#![forbid(unsafe_code)]`. +- `crates/rawshift-video-*`: Safe Rust is strictly required. +- `crates/rawshift-video`: Safe Rust is strictly required. All platform FFI for video decode lives in `rawshift-hwdec`, never here. - `**/**`: TBD ## Testing Strategy diff --git a/crates/rawshift-core/src/lib.rs b/crates/rawshift-core/src/lib.rs index 2595ef6..d55e673 100644 --- a/crates/rawshift-core/src/lib.rs +++ b/crates/rawshift-core/src/lib.rs @@ -25,8 +25,8 @@ //! now keeps the boundary from drifting into "everything shared-ish lives in //! core". The division is: //! -//! **Genuinely video-shared** — media-agnostic, and video will consume these -//! as-is: +//! **Genuinely video-shared** — media-agnostic, and consumed as-is by +//! `rawshift-video-core` and the `rawshift-video-*` crates: //! //! - Geometry — [`image::Dimensions`], [`image::Point`], [`image::Rect`]. //! - Codec descriptors — [`codec::CodecId`], [`codec::CodecInfo`], diff --git a/crates/rawshift-core/src/metadata.rs b/crates/rawshift-core/src/metadata.rs index e048093..ac22154 100644 --- a/crates/rawshift-core/src/metadata.rs +++ b/crates/rawshift-core/src/metadata.rs @@ -250,8 +250,13 @@ pub enum MetadataValue { } /// Namespace identifying the origin of a generic metadata tag. +/// +/// Marked `#[non_exhaustive]`: new namespaces arrive whenever a new container +/// or format grows container-level facts worth surfacing, and that must not be +/// a breaking change. Match with a `_` arm. #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] #[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))] +#[non_exhaustive] pub enum MetadataNamespace { /// Standard TIFF/EXIF IFD tags. Exif, @@ -267,6 +272,12 @@ pub enum MetadataNamespace { Heic, /// AVIF container-level facts. Avif, + /// QuickTime / ISOBMFF movie-level facts — `moov`-tree values and the + /// `com.apple.quicktime.*` `udta`/`mdta` keys iOS writes. + Quicktime, + /// Matroska / WebM segment-level facts — `Tags`, `Info`, and `TrackEntry` + /// elements. + Matroska, /// Vendor/format-specific, identified by the accompanying tag string. Other, } diff --git a/crates/rawshift-hwdec/src/lib.rs b/crates/rawshift-hwdec/src/lib.rs index 5d77541..52bd807 100644 --- a/crates/rawshift-hwdec/src/lib.rs +++ b/crates/rawshift-hwdec/src/lib.rs @@ -89,26 +89,43 @@ pub use gamut_color::{ChromaSubsampling, ColorRange}; // ── Codec / backend identity ──────────────────────────────────────────────── -/// A codec this crate can decode still frames of. +/// A codec this crate can decode. /// -/// The set is fixed at v1 (see `docs/SUPPORT.md`): HEVC for HEIC and AV1 for -/// AVIF. It is deliberately exhaustive — a new codec is a deliberate, -/// breaking decision, not an additive one. +/// The set is fixed by `docs/SUPPORT.md` and is deliberately exhaustive — a +/// new codec is a deliberate, breaking decision, not an additive one. +/// +/// # Seams +/// +/// Not every codec is decodable on both seams, and the set differs by design +/// rather than by omission: +/// +/// - [`decoder`] (still frames) answers for [`Hevc`](Self::Hevc) (HEIC) and +/// [`Av1`](Self::Av1) (AVIF). +/// - [`video_decoder`] (coded sequences) answers for [`Hevc`](Self::Hevc) and +/// [`H264`](Self::H264). +/// +/// [`H264`](Self::H264) exists only on the sequence seam: no rawshift still +/// format uses H.264, and offering an untested still path for it would +/// misreport what this crate can do. [`Av1`](Self::Av1) is still-only for now; +/// AV1 video sequences are a follow-up over the same backend. #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] pub enum HwCodec { - /// HEVC / H.265 intra (HEIC still images). + /// HEVC / H.265 — intra stills (HEIC) and coded video sequences. Hevc, /// AV1 intra (AVIF still images). Av1, + /// H.264 / AVC coded video sequences. + H264, } impl HwCodec { - /// The conventional display name of the codec (`"HEVC"` / `"AV1"`). + /// The conventional display name of the codec. #[must_use] pub fn name(self) -> &'static str { match self { HwCodec::Hevc => "HEVC", HwCodec::Av1 => "AV1", + HwCodec::H264 => "H.264", } } } @@ -470,6 +487,188 @@ pub enum HwDecodeError { }, } +// ── Video sequence seam ───────────────────────────────────────────────────── + +/// The codec configuration record of a video-sequence decode session. +/// +/// **The variant is the codec**, so a mismatched codec/config pair cannot be +/// constructed. Separate from [`CodecConfig`] deliberately: adding an `Avcc` +/// variant there would make "H.264 still image" representable, which is not a +/// rawshift use case and would ship an untested path. +/// +/// The bytes are the record **body** as the container stores it, with no box +/// header — an `AVCDecoderConfigurationRecord` (ISO/IEC 14496-15 §5.3.3.1) or +/// an `HEVCDecoderConfigurationRecord` (§8.3.3.1). Both carry the parameter +/// sets and the NAL length-prefix width every [`VideoPacket::data`] of the +/// track uses, so a session cannot be opened without one. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum VideoConfig<'a> { + /// An `avcC` record body; the codec is [`HwCodec::H264`]. + Avcc(&'a [u8]), + /// An `hvcC` record body; the codec is [`HwCodec::Hevc`]. + Hvcc(&'a [u8]), +} + +impl<'a> VideoConfig<'a> { + /// The codec this configuration selects. + #[must_use] + pub fn codec(&self) -> HwCodec { + match self { + VideoConfig::Avcc(_) => HwCodec::H264, + VideoConfig::Hvcc(_) => HwCodec::Hevc, + } + } + + /// The configuration record body. + #[must_use] + pub fn bytes(&self) -> &'a [u8] { + match self { + VideoConfig::Avcc(b) | VideoConfig::Hvcc(b) => b, + } + } +} + +/// One access unit: the coded picture for a single presentation time. +/// +/// `data` is the sample payload exactly as the container stores it — a +/// **length-prefixed** NAL stream, never Annex B — with the prefix width taken +/// from the session's [`VideoConfig`]. Exactly one coded picture per packet, +/// as ISO/IEC 14496-15 requires of a sample. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct VideoPacket<'a> { + /// The access unit's length-prefixed NAL units. + pub data: &'a [u8], + /// Container presentation timestamp, carried through to the frame this + /// picture becomes and never used for ordering. + pub pts: Option, + /// Container decode timestamp. Diagnostics only: output order comes from + /// the bitstream's picture order count, which is authoritative where a + /// container's timestamps are absent or wrong. + pub dts: Option, + /// Whether the container marks this as a sync sample. Advisory; the + /// bitstream decides. + pub is_sync: bool, +} + +/// A decoded picture in output (display) order, with the timestamp of the +/// packet it was coded from. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct VideoFrame { + frame: DecodedFrame, + pts: Option, + poc: i32, +} + +impl VideoFrame { + /// Wrap a decoded picture with its timing. + #[must_use] + pub fn new(frame: DecodedFrame, pts: Option, poc: i32) -> Self { + Self { frame, pts, poc } + } + + /// The decoded picture. + #[must_use] + pub fn frame(&self) -> &DecodedFrame { + &self.frame + } + + /// Take ownership of the decoded picture. + #[must_use] + pub fn into_frame(self) -> DecodedFrame { + self.frame + } + + /// The `pts` of the packet this picture was coded from. + #[must_use] + pub fn pts(&self) -> Option { + self.pts + } + + /// The picture order count derived from the bitstream. + /// + /// Strictly increasing across the frames emitted between two random access + /// points, which makes it the reliable ordering key even for a container + /// whose timestamps are missing or wrong. + #[must_use] + pub fn poc(&self) -> i32 { + self.poc + } +} + +/// A hardware decoder for one coded video sequence, from [`video_decoder`]. +/// +/// **Send-packet / receive-frame**, because coded order and output order +/// differ whenever a stream reorders (B pictures, hierarchical GOPs): one +/// [`send_packet`](Self::send_packet) may make zero, one, or several frames +/// available. Drive it as: +/// +/// ```text +/// for packet in access_units { +/// decoder.send_packet(&packet)?; +/// while let Some(frame) = decoder.receive_frame()? { sink(frame); } +/// } +/// decoder.flush()?; +/// while let Some(frame) = decoder.receive_frame()? { sink(frame); } +/// ``` +/// +/// **This decoder owns the picture buffer and emits frames in output order**, +/// because VAAPI is a slice-level API: the caller must hand it fully-formed +/// picture parameters including the reference lists, so the buffer and the +/// ordering cannot live above it. Callers must not reorder the output; +/// [`VideoFrame::poc`] is already monotonic within a coded video sequence. +pub trait HwVideoDecoder: Send { + /// The codec this session decodes. + fn codec(&self) -> HwCodec; + + /// Submit one access unit, making zero or more frames available to + /// [`receive_frame`](Self::receive_frame). + /// + /// After [`reset`](Self::reset), packets before the next random access + /// point reference pictures that no longer exist; they are dropped and + /// `Ok(())` returned, so seeking into the middle of a group of pictures is + /// not an error. + /// + /// # Errors + /// + /// [`HwDecodeError::Decode`] when the access unit is malformed or uses a + /// coding tool outside the backend's scope, and + /// [`HwDecodeError::Unavailable`] when in-band parameter sets change the + /// stream to something the driver cannot decode. + fn send_packet(&mut self, packet: &VideoPacket<'_>) -> Result<(), HwDecodeError>; + + /// Take the next frame in output order, or `None` when none is ready. + /// + /// `None` is the normal state while a reordering stream fills its buffer: + /// it means "not yet", not "end of stream". Only [`flush`](Self::flush) + /// establishes the latter. + /// + /// # Errors + /// + /// [`HwDecodeError::Decode`] when reading a decoded surface back fails, + /// or [`HwDecodeError::InvalidFrame`] when the readback is inconsistent. + fn receive_frame(&mut self) -> Result, HwDecodeError>; + + /// Signal end of stream: every picture still held for output becomes + /// available to [`receive_frame`](Self::receive_frame). + /// + /// Without this the last frames of every reordering stream are never + /// emitted. The session stays usable — a later + /// [`send_packet`](Self::send_packet) starts a new coded video sequence + /// and must begin at a random access point. + /// + /// # Errors + /// + /// As [`send_packet`](Self::send_packet). + fn flush(&mut self) -> Result<(), HwDecodeError>; + + /// Discard all decoder state **without** emitting pending frames — the + /// seek path. + /// + /// Reference pictures and queued output are dropped; the parameter sets + /// from the session's [`VideoConfig`] are retained. + fn reset(&mut self); +} + // ── Backend discovery ─────────────────────────────────────────────────────── /// Returns a decoder for `codec`, or `None` when no compiled-in backend can @@ -494,6 +693,52 @@ pub fn decoder(codec: HwCodec) -> Option> { } } +/// Opens a video-sequence decoder for `config`. +/// +/// The configuration record is parsed and validated up front — parameter sets, +/// profile, bit depth, chroma format — and the driver is probed for the +/// matching profile, so an unsupported stream fails here rather than on the +/// first packet. +/// +/// Unlike [`decoder`] this returns a `Result`: the caller supplied a real +/// configuration record, so there is a specific, reportable reason for +/// failure, and a consumer needs it for its own error surface. +/// +/// # Errors +/// +/// [`HwDecodeError::Unavailable`] when no backend is compiled in or the driver +/// cannot decode the stream's profile, and [`HwDecodeError::Decode`] when the +/// configuration record is malformed or out of scope. +pub fn video_decoder(config: VideoConfig<'_>) -> Result, HwDecodeError> { + #[cfg(hwdec_backend = "vaapi")] + { + vaapi::video_decoder(config) + } + #[cfg(not(hwdec_backend = "vaapi"))] + { + Err(HwDecodeError::Unavailable { + codec: config.codec(), + reason: "no hardware decode backend is compiled in for this target".to_string(), + }) + } +} + +/// The codecs [`video_decoder`] can open a session for on this machine. +/// +/// Empty when no backend is compiled in, or when the runtime probe finds no +/// usable driver — never a link or startup failure. +#[must_use] +pub fn available_video_codecs() -> &'static [HwCodec] { + #[cfg(hwdec_backend = "vaapi")] + { + vaapi::available_video_codecs() + } + #[cfg(not(hwdec_backend = "vaapi"))] + { + &[] + } +} + /// The platform backend compiled into this build and usable at runtime, or /// `None`. /// diff --git a/crates/rawshift-hwdec/src/vaapi/mod.rs b/crates/rawshift-hwdec/src/vaapi/mod.rs index faf0aed..89418b6 100644 --- a/crates/rawshift-hwdec/src/vaapi/mod.rs +++ b/crates/rawshift-hwdec/src/vaapi/mod.rs @@ -40,6 +40,7 @@ mod av1; mod bits; mod hevc; mod sys; +mod video; use std::ffi::c_int; use std::fs::File; @@ -59,6 +60,10 @@ pub fn decoder(codec: HwCodec) -> Option> { let supported = match codec { HwCodec::Hevc => caps.hevc_main || caps.hevc_main10, HwCodec::Av1 => caps.av1_profile0, + // H.264 exists only on the sequence seam: no rawshift still format + // uses it, so offering a still decoder would advertise an untested + // path. See `HwCodec`'s "Seams" documentation. + HwCodec::H264 => false, }; if !supported { return None; @@ -74,7 +79,8 @@ pub fn decoder(codec: HwCodec) -> Option> { /// Backend hook for [`crate::backend`]. pub fn backend() -> Option { let caps = probe(); - (caps.hevc_main || caps.hevc_main10 || caps.av1_profile0).then_some(HwBackend::Vaapi) + (caps.hevc_main || caps.hevc_main10 || caps.av1_profile0 || caps.h264_any()) + .then_some(HwBackend::Vaapi) } /// Backend hook for [`crate::available_codecs`]. @@ -92,6 +98,32 @@ pub fn available_codecs() -> &'static [HwCodec] { } } +/// Backend hook for [`crate::video_decoder`]. +pub fn video_decoder( + config: crate::VideoConfig<'_>, +) -> Result, HwDecodeError> { + video::open(config) +} + +/// Backend hook for [`crate::available_video_codecs`]. +/// +/// Reports only what [`video_decoder`] can actually open. The driver on this +/// machine may well decode H.264 — the probe tracks that in +/// [`Caps::h264_any`] — but this crate has no H.264 picture-parameter path +/// yet, so advertising it would promise a session that fails to open. +/// Discovery that lies is worse than discovery that is narrow. +pub fn available_video_codecs() -> &'static [HwCodec] { + const NONE: &[HwCodec] = &[]; + const HEVC_ONLY: &[HwCodec] = &[HwCodec::Hevc]; + + let caps = probe(); + if caps.hevc_main || caps.hevc_main10 { + HEVC_ONLY + } else { + NONE + } +} + // ── Runtime probe ─────────────────────────────────────────────────────────── /// What the driver actually decodes, per `vaQueryConfigProfiles` + @@ -101,6 +133,16 @@ struct Caps { hevc_main: bool, hevc_main10: bool, av1_profile0: bool, + h264_constrained_baseline: bool, + h264_main: bool, + h264_high: bool, +} + +impl Caps { + /// Whether the driver decodes H.264 at any profile rawshift accepts. + fn h264_any(&self) -> bool { + self.h264_constrained_baseline || self.h264_main || self.h264_high + } } /// Probe once per process: dlopen, open a render node, init a display, @@ -137,7 +179,12 @@ fn probe_uncached() -> Option { for &profile in &profiles { let interesting = matches!( profile, - sys::VA_PROFILE_HEVC_MAIN | sys::VA_PROFILE_HEVC_MAIN10 | sys::VA_PROFILE_AV1_PROFILE0 + sys::VA_PROFILE_HEVC_MAIN + | sys::VA_PROFILE_HEVC_MAIN10 + | sys::VA_PROFILE_AV1_PROFILE0 + | sys::VA_PROFILE_H264_CONSTRAINED_BASELINE + | sys::VA_PROFILE_H264_MAIN + | sys::VA_PROFILE_H264_HIGH ); if !interesting { continue; @@ -149,6 +196,9 @@ fn probe_uncached() -> Option { sys::VA_PROFILE_HEVC_MAIN => caps.hevc_main = true, sys::VA_PROFILE_HEVC_MAIN10 => caps.hevc_main10 = true, sys::VA_PROFILE_AV1_PROFILE0 => caps.av1_profile0 = true, + sys::VA_PROFILE_H264_CONSTRAINED_BASELINE => caps.h264_constrained_baseline = true, + sys::VA_PROFILE_H264_MAIN => caps.h264_main = true, + sys::VA_PROFILE_H264_HIGH => caps.h264_high = true, _ => {} } } diff --git a/crates/rawshift-hwdec/src/vaapi/sys.rs b/crates/rawshift-hwdec/src/vaapi/sys.rs index 97bb3db..ed08b74 100644 --- a/crates/rawshift-hwdec/src/vaapi/sys.rs +++ b/crates/rawshift-hwdec/src/vaapi/sys.rs @@ -42,6 +42,12 @@ pub const VA_INVALID_ID: u32 = 0xffff_ffff; pub const VA_INVALID_SURFACE: VASurfaceID = VA_INVALID_ID; // VAProfile values (va.h). +/// `VAProfileH264ConstrainedBaseline` (va.h). +pub const VA_PROFILE_H264_CONSTRAINED_BASELINE: VAProfile = 13; +/// `VAProfileH264Main` (va.h). +pub const VA_PROFILE_H264_MAIN: VAProfile = 6; +/// `VAProfileH264High` (va.h). +pub const VA_PROFILE_H264_HIGH: VAProfile = 7; pub const VA_PROFILE_HEVC_MAIN: VAProfile = 17; pub const VA_PROFILE_HEVC_MAIN10: VAProfile = 18; pub const VA_PROFILE_AV1_PROFILE0: VAProfile = 32; diff --git a/crates/rawshift-hwdec/src/vaapi/video.rs b/crates/rawshift-hwdec/src/vaapi/video.rs new file mode 100644 index 0000000..a68886c --- /dev/null +++ b/crates/rawshift-hwdec/src/vaapi/video.rs @@ -0,0 +1,211 @@ +//! VAAPI decoding of coded video sequences. +//! +//! Where [`super::still`]-shaped decoding handles one independent picture, +//! this owns a *session*: parameter sets that persist across access units, a +//! picture buffer, and the output-ordering that reordered streams require. +//! +//! # Current scope +//! +//! **Random access points only.** Every IRAP (HEVC) and IDR (H.264) access +//! unit decodes; an access unit that references other pictures is reported as +//! [`HwDecodeError::Decode`] rather than decoded incorrectly. +//! +//! That is a real capability, not a stub: it covers keyframe extraction, +//! poster frames, timeline thumbnails and scrubbing, and it decodes All-Intra +//! camera modes (Sony XAVC S-I and XAVC HS All-I) completely, since every +//! access unit in those is a random access point. +//! +//! Inter-frame decoding is a purely internal change behind this unchanged +//! API: it adds a decoded picture buffer, picture order counts, reference +//! marking and reference lists, and removes the rejection below. No part of +//! the public seam moves, so it needs no second breaking release. + +use crate::{ + CodecConfig, DecodedFrame, HwCodec, HwDecodeError, HwVideoDecoder, StillDecodeRequest, + VideoConfig, VideoFrame, VideoPacket, +}; + +use super::hevc; + +/// Open a sequence decoder for `config`. +pub fn open(config: VideoConfig<'_>) -> Result, HwDecodeError> { + let codec = config.codec(); + let caps = super::probe(); + + let supported = match codec { + HwCodec::Hevc => caps.hevc_main || caps.hevc_main10, + // The driver may well decode H.264, but this crate has no H.264 + // picture-parameter path yet. Refusing here keeps `open` consistent + // with `available_video_codecs`, which does not advertise it. + HwCodec::H264 | HwCodec::Av1 => false, + }; + if !supported { + let reason = match codec { + HwCodec::H264 if caps.h264_any() => { + "the driver decodes H.264, but this build has no H.264 sequence path yet" + } + HwCodec::Av1 => "AV1 video sequences are not implemented; AV1 stills use `decoder()`", + _ => "the VAAPI driver exposes no decode entry point for this codec", + }; + return Err(HwDecodeError::Unavailable { + codec, + reason: reason.to_string(), + }); + } + + // Validate the configuration record now, so a stream that cannot work + // fails at open rather than on the first packet. + let record = config.bytes().to_vec(); + let nal_length_size = match config { + VideoConfig::Hvcc(bytes) => { + hevc::parse_hvcc(bytes) + .map_err(|e| HwDecodeError::Decode { + codec, + message: format!("hvcC: {e}"), + })? + .nal_length_size + } + VideoConfig::Avcc(bytes) => { + avcc_nal_length_size(bytes).ok_or_else(|| HwDecodeError::Decode { + codec, + message: "avcC: truncated or illegal length-prefix width".to_string(), + })? + } + }; + + let still = + super::decoder(codec_for_still(codec)).ok_or_else(|| HwDecodeError::Unavailable { + codec, + reason: "the VAAPI driver exposes no decode session for this codec".to_string(), + })?; + + Ok(Box::new(VaapiVideoDecoder { + codec, + record, + nal_length_size, + still, + ready: Vec::new(), + next_poc: 0, + })) +} + +/// The still-seam codec used to decode one random access point. +/// +/// Only HEVC has a still path today; H.264 sessions are rejected at +/// [`open`] before reaching here. +fn codec_for_still(codec: HwCodec) -> HwCodec { + codec +} + +/// Read `lengthSizeMinusOne` from an `avcC` record (ISO/IEC 14496-15 +/// §5.3.3.1, byte 4, low two bits). +/// +/// `None` for a truncated record or the illegal 3-byte width. +fn avcc_nal_length_size(record: &[u8]) -> Option { + let field = record.get(4)?; + match (field & 0b11) + 1 { + n @ (1 | 2 | 4) => Some(n as usize), + _ => None, + } +} + +struct VaapiVideoDecoder { + codec: HwCodec, + /// The configuration record, kept because every decode submits it and it + /// must survive [`HwVideoDecoder::reset`]. + record: Vec, + nal_length_size: usize, + still: Box, + /// Frames decoded and awaiting collection, in output order. + ready: Vec, + /// Picture order count assigned to the next frame. + /// + /// With random access points only there is no reordering, so output order + /// is submission order and a monotonic counter is the correct picture + /// order. Real picture order counts arrive with inter-frame support. + next_poc: i32, +} + +impl VaapiVideoDecoder { + /// Whether `access_unit` is a random access point this build can decode. + fn is_random_access_point(&self, access_unit: &[u8]) -> bool { + let Ok(nals) = hevc::split_length_prefixed(access_unit, self.nal_length_size) else { + return false; + }; + nals.iter().any(|nal| match self.codec { + // HEVC: types 16..=23 are the IRAP range (BLA, IDR, CRA). + HwCodec::Hevc => hevc::nal_type(nal).is_ok_and(hevc::is_irap), + // H.264: type 5 is an IDR slice. + HwCodec::H264 => nal.first().is_some_and(|h| h & 0x1f == 5), + HwCodec::Av1 => false, + }) + } +} + +impl HwVideoDecoder for VaapiVideoDecoder { + fn codec(&self) -> HwCodec { + self.codec + } + + fn send_packet(&mut self, packet: &VideoPacket<'_>) -> Result<(), HwDecodeError> { + if !self.is_random_access_point(packet.data) { + return Err(HwDecodeError::Decode { + codec: self.codec, + message: "this build decodes random access points only; inter-frame decoding \ + needs the decoded picture buffer that is not implemented yet" + .to_string(), + }); + } + + let frame = self.decode_random_access_point(packet.data)?; + let poc = self.next_poc; + self.next_poc = self.next_poc.saturating_add(1); + self.ready.push(VideoFrame::new(frame, packet.pts, poc)); + Ok(()) + } + + fn receive_frame(&mut self) -> Result, HwDecodeError> { + if self.ready.is_empty() { + return Ok(None); + } + Ok(Some(self.ready.remove(0))) + } + + fn flush(&mut self) -> Result<(), HwDecodeError> { + // Nothing is held back: with no reordering every decoded picture is + // already in `ready`. Inter-frame support drains the picture buffer + // here instead. + Ok(()) + } + + fn reset(&mut self) { + self.ready.clear(); + self.next_poc = 0; + } +} + +impl VaapiVideoDecoder { + fn decode_random_access_point(&mut self, data: &[u8]) -> Result { + let config = match self.codec { + HwCodec::Hevc => CodecConfig::Hvcc(&self.record), + codec => { + return Err(HwDecodeError::Decode { + codec, + message: "no still-decode path exists for this codec".to_string(), + }); + } + }; + + let request = StillDecodeRequest { + config, + payload: data, + // Advisory only; the still path takes the real geometry from the + // stream's own parameter sets. + width: 0, + height: 0, + bit_depth: 0, + chroma: gamut_color::ChromaSubsampling::Cs420, + }; + self.still.decode_still(&request) + } +} diff --git a/crates/rawshift-hwdec/tests/common/mod.rs b/crates/rawshift-hwdec/tests/common/mod.rs new file mode 100644 index 0000000..3e0ef75 --- /dev/null +++ b/crates/rawshift-hwdec/tests/common/mod.rs @@ -0,0 +1,116 @@ +//! Helpers shared by the VAAPI device tests. +//! +//! These live in one place because they encode fiddly, easy-to-get-wrong +//! details — the `hvcC` record layout in particular, where putting +//! `numOfArrays` one byte off yields a record that parses but carries no +//! parameter sets. Two copies of that drift, and the second copy is where the +//! bug hides. + +#![allow(dead_code)] + +use rawshift_hwdec::DecodedFrame; +use std::process::Command; + +/// Whether any DRM render node exists (cheap pre-check for skip messages). +pub fn has_render_node() -> bool { + (128..136).any(|minor| std::path::Path::new(&format!("/dev/dri/renderD{minor}")).exists()) +} + +/// Run ffmpeg, returning false (→ skip) when it is missing or fails. +pub fn ffmpeg(args: &[&str]) -> bool { + match Command::new("ffmpeg") + .args(["-y", "-hide_banner", "-loglevel", "error"]) + .args(args) + .status() + { + Ok(status) => status.success(), + Err(_) => false, + } +} + +/// Per-plane sample variance must be non-zero for a real test pattern — +/// catches "decode succeeded but wrote a blank surface". +pub fn luma_variance(frame: &DecodedFrame) -> f64 { + let plane = &frame.planes()[0]; + let bps = frame.format().bytes_per_sample(); + let width = frame.width() as usize; + let mut samples: Vec = Vec::new(); + for row in 0..frame.height() as usize { + let start = row * plane.stride; + for px in 0..width { + let value = if bps == 1 { + f64::from(plane.data[start + px]) + } else { + f64::from(u16::from_le_bytes([ + plane.data[start + px * 2], + plane.data[start + px * 2 + 1], + ])) + }; + samples.push(value); + } + } + let mean = samples.iter().sum::() / samples.len() as f64; + samples.iter().map(|s| (s - mean).powi(2)).sum::() / samples.len() as f64 +} + +/// Split an Annex-B stream into NAL units (3- or 4-byte start codes). +pub fn split_annex_b(data: &[u8]) -> Vec> { + let mut starts = Vec::new(); + let mut i = 0; + while i + 3 <= data.len() { + if data[i] == 0 && data[i + 1] == 0 && data[i + 2] == 1 { + starts.push(i); + i += 3; + } else { + i += 1; + } + } + let mut nals = Vec::new(); + for (n, &start) in starts.iter().enumerate() { + let end = starts.get(n + 1).copied().unwrap_or(data.len()); + let mut nal = &data[start + 3..end]; + // Trim the trailing zero(s) that belong to the next 4-byte start + // code / trailing_zero_8bits. + while let Some((&0, rest)) = nal.split_last() { + nal = rest; + } + if !nal.is_empty() { + nals.push(nal.to_vec()); + } + } + nals +} + +/// Build an hvcC record (4-byte length prefixes) carrying `param_sets` and +/// a length-prefixed payload from `slices`. +pub fn build_hvcc_and_payload(nals: &[Vec]) -> (Vec, Vec) { + let mut hvcc = vec![0u8; 23]; + hvcc[0] = 1; // configurationVersion + hvcc[21] = 0x03; // lengthSizeMinusOne = 3 + let mut arrays: Vec<(u8, Vec<&[u8]>)> = Vec::new(); + let mut payload = Vec::new(); + for nal in nals { + let nal_type = (nal[0] >> 1) & 0x3f; + match nal_type { + 32..=34 => match arrays.iter_mut().find(|(t, _)| *t == nal_type) { + Some((_, list)) => list.push(nal), + None => arrays.push((nal_type, vec![nal])), + }, + _ if nal_type < 32 => { + payload.extend_from_slice(&(nal.len() as u32).to_be_bytes()); + payload.extend_from_slice(nal); + } + _ => {} // SEI etc. + } + } + hvcc[22] = arrays.len() as u8; + for (nal_type, list) in &arrays { + hvcc.push(0x80 | nal_type); + hvcc.extend_from_slice(&(list.len() as u16).to_be_bytes()); + for nal in list { + hvcc.extend_from_slice(&(nal.len() as u16).to_be_bytes()); + hvcc.extend_from_slice(nal); + } + } + (hvcc, payload) +} diff --git a/crates/rawshift-hwdec/tests/vaapi_device.rs b/crates/rawshift-hwdec/tests/vaapi_device.rs index 10f80f8..21b4195 100644 --- a/crates/rawshift-hwdec/tests/vaapi_device.rs +++ b/crates/rawshift-hwdec/tests/vaapi_device.rs @@ -14,19 +14,17 @@ #![cfg(hwdec_backend = "vaapi")] +mod common; + +use common::{build_hvcc_and_payload, ffmpeg, has_render_node, luma_variance, split_annex_b}; + use rawshift_hwdec::{ ChromaSubsampling, CodecConfig, DecodedFrame, HwBackend, HwCodec, PixelFormat, StillDecodeRequest, available_codecs, backend, decoder, }; -use std::process::Command; // ── gating helpers ────────────────────────────────────────────────────────── -/// Whether any DRM render node exists (cheap pre-check for skip messages). -fn has_render_node() -> bool { - (128..136).any(|minor| std::path::Path::new(&format!("/dev/dri/renderD{minor}")).exists()) -} - macro_rules! device_or_skip { ($codec:expr) => { match decoder($codec) { @@ -44,107 +42,8 @@ macro_rules! device_or_skip { }; } -/// Run ffmpeg, returning false (→ skip) when it is missing or fails. -fn ffmpeg(args: &[&str]) -> bool { - match Command::new("ffmpeg") - .args(["-y", "-hide_banner", "-loglevel", "error"]) - .args(args) - .status() - { - Ok(status) => status.success(), - Err(_) => false, - } -} - -/// Per-plane sample variance must be non-zero for a real test pattern — -/// catches "decode succeeded but wrote a blank surface". -fn luma_variance(frame: &DecodedFrame) -> f64 { - let plane = &frame.planes()[0]; - let bps = frame.format().bytes_per_sample(); - let width = frame.width() as usize; - let mut samples: Vec = Vec::new(); - for row in 0..frame.height() as usize { - let start = row * plane.stride; - for px in 0..width { - let value = if bps == 1 { - f64::from(plane.data[start + px]) - } else { - f64::from(u16::from_le_bytes([ - plane.data[start + px * 2], - plane.data[start + px * 2 + 1], - ])) - }; - samples.push(value); - } - } - let mean = samples.iter().sum::() / samples.len() as f64; - samples.iter().map(|s| (s - mean).powi(2)).sum::() / samples.len() as f64 -} - // ── Annex-B → hvcC + length-prefixed payload (test-side container glue) ───── -/// Split an Annex-B stream into NAL units (3- or 4-byte start codes). -fn split_annex_b(data: &[u8]) -> Vec> { - let mut starts = Vec::new(); - let mut i = 0; - while i + 3 <= data.len() { - if data[i] == 0 && data[i + 1] == 0 && data[i + 2] == 1 { - starts.push(i); - i += 3; - } else { - i += 1; - } - } - let mut nals = Vec::new(); - for (n, &start) in starts.iter().enumerate() { - let end = starts.get(n + 1).copied().unwrap_or(data.len()); - let mut nal = &data[start + 3..end]; - // Trim the trailing zero(s) that belong to the next 4-byte start - // code / trailing_zero_8bits. - while let Some((&0, rest)) = nal.split_last() { - nal = rest; - } - if !nal.is_empty() { - nals.push(nal.to_vec()); - } - } - nals -} - -/// Build an hvcC record (4-byte length prefixes) carrying `param_sets` and -/// a length-prefixed payload from `slices`. -fn build_hvcc_and_payload(nals: &[Vec]) -> (Vec, Vec) { - let mut hvcc = vec![0u8; 23]; - hvcc[0] = 1; // configurationVersion - hvcc[21] = 0x03; // lengthSizeMinusOne = 3 - let mut arrays: Vec<(u8, Vec<&[u8]>)> = Vec::new(); - let mut payload = Vec::new(); - for nal in nals { - let nal_type = (nal[0] >> 1) & 0x3f; - match nal_type { - 32..=34 => match arrays.iter_mut().find(|(t, _)| *t == nal_type) { - Some((_, list)) => list.push(nal), - None => arrays.push((nal_type, vec![nal])), - }, - _ if nal_type < 32 => { - payload.extend_from_slice(&(nal.len() as u32).to_be_bytes()); - payload.extend_from_slice(nal); - } - _ => {} // SEI etc. - } - } - hvcc[22] = arrays.len() as u8; - for (nal_type, list) in &arrays { - hvcc.push(0x80 | nal_type); - hvcc.extend_from_slice(&(list.len() as u16).to_be_bytes()); - for nal in list { - hvcc.extend_from_slice(&(nal.len() as u16).to_be_bytes()); - hvcc.extend_from_slice(nal); - } - } - (hvcc, payload) -} - /// Generate one intra HEVC frame with ffmpeg/libx265 and decode it through /// the hardware; returns `None` when the encoder is unavailable (→ skip). fn decode_generated_hevc(pix_fmt: &str, tag: &str) -> Option { diff --git a/crates/rawshift-hwdec/tests/vaapi_video_device.rs b/crates/rawshift-hwdec/tests/vaapi_video_device.rs new file mode 100644 index 0000000..8762050 --- /dev/null +++ b/crates/rawshift-hwdec/tests/vaapi_video_device.rs @@ -0,0 +1,255 @@ +//! Device-gated VAAPI sequence-decode tests. +//! +//! Skips gracefully (eprintln + return) when the machine has no render node, +//! libva cannot be dlopen'd, the driver lacks the codec, or ffmpeg is +//! unavailable — CI without a GPU stays green. On a machine with a working +//! driver these exercise the real path: an ffmpeg-generated bitstream through +//! `video_decoder()` to `VideoFrame` pixels. + +#![cfg(hwdec_backend = "vaapi")] + +mod common; + +use common::{build_hvcc_and_payload, ffmpeg, has_render_node, luma_variance, split_annex_b}; +use rawshift_hwdec::{HwCodec, VideoConfig, VideoPacket, available_video_codecs, video_decoder}; + +// ── gating helpers ────────────────────────────────────────────────────────── + +/// Open a sequence decoder, or skip when the driver cannot provide one. +macro_rules! decoder_or_skip { + ($config:expr) => { + match video_decoder($config) { + Ok(decoder) => decoder, + Err(e) => { + eprintln!( + "Skipping VAAPI sequence test: {e} (render node present: {})", + has_render_node() + ); + return; + } + } + }; +} + +/// A one-frame all-intra HEVC bitstream, or `None` to skip. +fn hevc_keyframe() -> Option> { + let path = std::env::temp_dir().join("rawshift-hwdec-seq-intra.h265"); + if !path.exists() { + let ok = ffmpeg(&[ + "-f", + "lavfi", + "-i", + "testsrc2=size=128x96:duration=1:rate=1", + "-c:v", + "libx265", + "-x265-params", + "log-level=error:keyint=1", + "-pix_fmt", + "yuv420p", + "-f", + "hevc", + path.to_str()?, + ]); + if !ok { + eprintln!("Skipping: ffmpeg could not produce an HEVC fixture (libx265 missing?)"); + return None; + } + } + std::fs::read(&path).ok() +} + +// ── tests ─────────────────────────────────────────────────────────────────── + +#[test] +fn a_keyframe_decodes_to_real_pixels() { + let Some(stream) = hevc_keyframe() else { + return; + }; + let nals = split_annex_b(&stream); + let (hvcc, payload) = build_hvcc_and_payload(&nals); + + let mut decoder = decoder_or_skip!(VideoConfig::Hvcc(&hvcc)); + assert_eq!(decoder.codec(), HwCodec::Hevc); + + decoder + .send_packet(&VideoPacket { + data: &payload, + pts: Some(4_242), + dts: Some(4_242), + is_sync: true, + }) + .expect("a random access point must decode"); + + let frame = decoder + .receive_frame() + .expect("receive succeeds") + .expect("a frame must be ready after a keyframe"); + + assert_eq!(frame.pts(), Some(4_242), "the timestamp must be carried"); + assert_eq!(frame.frame().width(), 128); + assert_eq!(frame.frame().height(), 96); + assert!( + luma_variance(frame.frame()) > 1.0, + "decoded a blank surface, not the test pattern" + ); +} + +#[test] +fn frames_arrive_only_after_a_packet_and_flush_leaves_nothing_behind() { + let Some(stream) = hevc_keyframe() else { + return; + }; + let nals = split_annex_b(&stream); + let (hvcc, payload) = build_hvcc_and_payload(&nals); + + let mut decoder = decoder_or_skip!(VideoConfig::Hvcc(&hvcc)); + + // Nothing has been submitted: "not ready" is None, not an error. + assert!(decoder.receive_frame().expect("receive succeeds").is_none()); + + decoder + .send_packet(&VideoPacket { + data: &payload, + pts: Some(0), + dts: Some(0), + is_sync: true, + }) + .expect("decodes"); + assert!(decoder.receive_frame().expect("receive").is_some()); + + // Drained: flush must not invent a frame, and the decoder stays usable. + decoder.flush().expect("flush succeeds"); + assert!(decoder.receive_frame().expect("receive").is_none()); +} + +#[test] +fn reset_discards_pending_output_rather_than_emitting_it() { + let Some(stream) = hevc_keyframe() else { + return; + }; + let nals = split_annex_b(&stream); + let (hvcc, payload) = build_hvcc_and_payload(&nals); + + let mut decoder = decoder_or_skip!(VideoConfig::Hvcc(&hvcc)); + decoder + .send_packet(&VideoPacket { + data: &payload, + pts: Some(0), + dts: Some(0), + is_sync: true, + }) + .expect("decodes"); + + // A seek must drop what was decoded, not play it out. + decoder.reset(); + assert!( + decoder.receive_frame().expect("receive").is_none(), + "reset must discard pending output" + ); + + // ... and the session must still work afterwards. + decoder + .send_packet(&VideoPacket { + data: &payload, + pts: Some(99), + dts: Some(99), + is_sync: true, + }) + .expect("decodes after reset"); + let frame = decoder.receive_frame().expect("receive").expect("a frame"); + assert_eq!(frame.pts(), Some(99)); +} + +#[test] +fn an_inter_access_unit_is_refused_rather_than_decoded_wrongly() { + // Until the picture buffer lands, a packet that references other pictures + // must produce a clear error, never a plausible-looking wrong picture. + let Some(stream) = hevc_keyframe() else { + return; + }; + let nals = split_annex_b(&stream); + let (hvcc, _) = build_hvcc_and_payload(&nals); + + let mut decoder = decoder_or_skip!(VideoConfig::Hvcc(&hvcc)); + + // NAL type 1 (TRAIL_R) is a non-IRAP coded slice. + let mut inter = Vec::new(); + let nal = [0x02u8, 0x01, 0xd0, 0x09]; + inter.extend_from_slice(&(nal.len() as u32).to_be_bytes()); + inter.extend_from_slice(&nal); + + let err = decoder + .send_packet(&VideoPacket { + data: &inter, + pts: Some(0), + dts: Some(0), + is_sync: false, + }) + .expect_err("an inter picture must be refused"); + assert!( + err.to_string().contains("random access"), + "the error must say why: {err}" + ); +} + +#[test] +fn a_malformed_configuration_record_fails_at_open() { + // Opening validates the record, so a stream that cannot work fails here + // rather than on the first packet. + // `Box` is not Debug, so expect_err does not apply. + match video_decoder(VideoConfig::Hvcc(&[0x01, 0x02])) { + Ok(_) => panic!("a truncated hvcC must not open"), + Err(e) => assert!(!e.to_string().is_empty()), + } + + // lengthSizeMinusOne = 2 means an illegal 3-byte prefix. + let mut bad = vec![0u8; 23]; + bad[0] = 1; + bad[21] = 0x02; + assert!(video_decoder(VideoConfig::Hvcc(&bad)).is_err()); +} + +#[test] +fn discovery_agrees_with_what_can_actually_be_opened() { + // Every advertised codec must genuinely open a session. Advertising one + // the driver supports but this crate cannot drive would make callers + // choose a path that then fails. + let Some(stream) = hevc_keyframe() else { + return; + }; + let nals = split_annex_b(&stream); + let (hvcc, _) = build_hvcc_and_payload(&nals); + + for &codec in available_video_codecs() { + let config = match codec { + HwCodec::Hevc => VideoConfig::Hvcc(&hvcc), + other => panic!("{other} is advertised but this test cannot build a config for it"), + }; + assert!( + video_decoder(config).is_ok(), + "{codec} is advertised yet fails to open" + ); + } +} + +#[test] +fn an_unimplemented_codec_is_refused_with_a_reason_not_advertised() { + // H.264 must be absent from discovery, and asking anyway must explain + // itself rather than failing opaquely. + assert!(!available_video_codecs().contains(&HwCodec::H264)); + + let mut avcc = vec![0u8; 8]; + avcc[0] = 1; + avcc[4] = 0xff; // lengthSizeMinusOne = 3 + match video_decoder(VideoConfig::Avcc(&avcc)) { + Ok(_) => panic!("H.264 has no sequence path yet and must not open"), + Err(e) => assert!( + e.to_string().contains("H.264 sequence path") + || e.to_string().contains("no decode entry point"), + "the error must say why: {e}" + ), + } + + // And H.264 must never appear on the still seam either. + assert!(!rawshift_hwdec::available_codecs().contains(&HwCodec::H264)); +} diff --git a/crates/rawshift-video-core/Cargo.toml b/crates/rawshift-video-core/Cargo.toml new file mode 100644 index 0000000..e1dc8b0 --- /dev/null +++ b/crates/rawshift-video-core/Cargo.toml @@ -0,0 +1,23 @@ +[package] +name = "rawshift-video-core" +version.workspace = true +edition.workspace = true +rust-version.workspace = true +license.workspace = true +repository.workspace = true +homepage.workspace = true +description = "Shared vocabulary for rawshift's video crates: errors, identity, tracks, packets, frames, and the demux/decode contracts" +documentation = "https://docs.rs/rawshift-video-core" +keywords = ["video", "demux", "container", "codec"] +categories = ["multimedia::video"] +readme = "README.md" + +[dependencies] +rawshift-core = { workspace = true } +gamut-color = { workspace = true } +thiserror = { workspace = true } +serde = { workspace = true, optional = true } + +[features] +default = [] +serde = ["dep:serde", "rawshift-core/serde"] diff --git a/crates/rawshift-video-core/README.md b/crates/rawshift-video-core/README.md new file mode 100644 index 0000000..9c9e649 --- /dev/null +++ b/crates/rawshift-video-core/README.md @@ -0,0 +1,21 @@ +# rawshift-video-core + +Shared vocabulary for [rawshift](https://github.com/visualcommons/rawshift)'s +video crates — the video counterpart to `rawshift-image-core`. + +This crate holds the types every `rawshift-video-*` crate agrees on and no +implementation of its own: errors, format and codec identity, the track / +packet / frame model, container-level metadata, and the `Demuxer` / +`VideoDecoder` / `VideoEncoder` contracts. + +It exists because the video crates cannot reuse `rawshift-image-core`: the +workspace enforces (in CI) that no `rawshift-image*` crate appears in +`rawshift-video`'s dependency tree, so the two media share only +`rawshift-core`. + +`#![forbid(unsafe_code)]`. No I/O, no platform code, no container or codec +dependencies. + +## License + +Licensed under [MPL-2.0](../../LICENSE). diff --git a/crates/rawshift-video-core/src/decode.rs b/crates/rawshift-video-core/src/decode.rs new file mode 100644 index 0000000..0e26760 --- /dev/null +++ b/crates/rawshift-video-core/src/decode.rs @@ -0,0 +1,492 @@ +//! The decoding contracts and the backend registry. + +use rawshift_core::CodecId; + +use crate::error::{VideoError, VideoResult}; +use crate::frame::VideoFrame; +use crate::id::VideoCodecId; +use crate::packet::Packet; +use crate::track::VideoTrack; + +/// A decoder for one video track. +/// +/// **Send-packet / receive-frame**, not `decode(packet) -> frame`. Coded order +/// and display order differ whenever a stream reorders (B pictures, +/// hierarchical GOPs), so one packet may make zero, one, or several frames +/// available. The split is also the shape every hardware decode API natively +/// has. +/// +/// Drive it as: +/// +/// ```text +/// while let Some(packet) = demuxer.next_packet()? { +/// decoder.send_packet(&packet)?; +/// while let Some(frame) = decoder.receive_frame()? { sink(frame); } +/// } +/// decoder.flush()?; +/// while let Some(frame) = decoder.receive_frame()? { sink(frame); } +/// ``` +/// +/// Frames arrive in **display order**; a caller must not reorder them. +pub trait VideoDecoder: Send { + /// The codec this decoder decodes. + fn codec(&self) -> VideoCodecId; + + /// Submit one access unit, making zero or more frames available to + /// [`receive_frame`](Self::receive_frame). + /// + /// After [`reset`](Self::reset), packets before the next random access + /// point reference pictures that are gone; they are dropped and `Ok(())` + /// is returned, so seeking into the middle of a GOP is not an error. + /// + /// # Errors + /// + /// [`VideoError::Decode`] when the access unit is malformed or the backend + /// fails. + fn send_packet(&mut self, packet: &Packet) -> VideoResult<()>; + + /// Take the next frame in display order, or `None` when none is ready. + /// + /// `None` is the normal state while a reordering stream fills its buffer; + /// it means "not yet", not "end of stream". Only + /// [`flush`](Self::flush) establishes the latter. + /// + /// # Errors + /// + /// [`VideoError::Decode`] when reading the decoded picture back fails. + fn receive_frame(&mut self) -> VideoResult>; + + /// Signal end of stream: every buffered picture becomes available to + /// [`receive_frame`](Self::receive_frame). + /// + /// Without this the last few frames of every reordered stream are never + /// emitted. The decoder stays usable afterwards. + /// + /// # Errors + /// + /// As [`send_packet`](Self::send_packet). + fn flush(&mut self) -> VideoResult<()>; + + /// Discard all state **without** emitting buffered frames — the seek path. + fn reset(&mut self); +} + +/// An encoder for one video track. +/// +/// The seam only: rawshift ships no video encoder yet. It is defined now so +/// that adding one is additive, and so the shape is settled while the decode +/// side is fresh. +pub trait VideoEncoder: Send { + /// The codec this encoder produces. + fn codec(&self) -> VideoCodecId; + + /// Submit one raw frame for encoding. + /// + /// # Errors + /// + /// Backend-specific encode failures. + fn send_frame(&mut self, frame: &VideoFrame) -> VideoResult<()>; + + /// Take the next coded access unit, or `None` when none is ready. + /// + /// # Errors + /// + /// Backend-specific encode failures. + fn receive_packet(&mut self) -> VideoResult>; + + /// Signal end of stream and drain the encoder. + /// + /// # Errors + /// + /// Backend-specific encode failures. + fn flush(&mut self) -> VideoResult<()>; +} + +/// Opens decoders for tracks it supports. +/// +/// One factory per backend per codec. Registered in a [`DecoderRegistry`], +/// which picks between them. +pub trait VideoDecoderFactory: Send + Sync { + /// Stable `"{format}/{impl}"` identifier, e.g. `"h264/vaapi"`. + fn id(&self) -> CodecId; + + /// The codec this factory decodes. + fn codec(&self) -> VideoCodecId; + + /// Whether this backend can decode this specific track **right now**. + /// + /// Answers about the runtime too, not just the codec: a VAAPI factory + /// returns `false` with no libva, no device, or no driver support for the + /// track's profile. This is the **only** signal that lets the registry try + /// the next backend — a factory that accepts a track and then fails in + /// [`open`](Self::open) is reported to the caller rather than retried. + fn supports(&self, track: &VideoTrack) -> bool; + + /// Open a decoder for `track`. + /// + /// # Errors + /// + /// Any failure to start a session. Reaching here after `supports` returned + /// `true` is terminal for the whole open, by the contract above. + fn open(&self, track: &VideoTrack) -> VideoResult>; +} + +/// Backends to try, in order. +/// +/// **Push order is priority order**, and `supports() == false` is the only +/// thing that falls through to the next entry. A backend that accepts a track +/// and then fails produces an error rather than a silent retry, so a genuine +/// backend fault is never disguised as "unsupported". +/// +/// This mirrors the fallback contract `gamut-codec-abi` documents for image +/// codestream backends, adopted deliberately so both media behave the same. +#[derive(Default)] +pub struct DecoderRegistry { + factories: Vec>, +} + +impl DecoderRegistry { + /// An empty registry. + #[must_use] + pub fn new() -> Self { + Self::default() + } + + /// Append a factory, at lowest priority. + pub fn push(&mut self, factory: Box) { + self.factories.push(factory); + } + + /// Append a factory and return the registry, for chaining at startup. + #[must_use] + pub fn with(mut self, factory: Box) -> Self { + self.push(factory); + self + } + + /// Every registered factory, in priority order. + #[must_use] + pub fn factories(&self) -> &[Box] { + &self.factories + } + + /// Whether any registered backend can decode `track` right now. + #[must_use] + pub fn supports(&self, track: &VideoTrack) -> bool { + self.factories.iter().any(|f| f.supports(track)) + } + + /// Open a decoder for `track` using the first backend that supports it. + /// + /// # Errors + /// + /// [`VideoError::UnsupportedCodec`] when no factory for the codec is + /// registered at all, and + /// [`VideoError::HwDecoderUnavailable`] when factories for the codec exist + /// but none supports this track on this machine — the distinction between + /// "not built" and "no device", which callers act on differently. + pub fn open(&self, track: &VideoTrack) -> VideoResult> { + let mut saw_codec = false; + for factory in &self.factories { + if factory.codec() != track.codec { + continue; + } + saw_codec = true; + if factory.supports(track) { + return factory.open(track); + } + } + + if saw_codec { + Err(VideoError::hw_unavailable( + track.codec, + "a decoder is compiled in, but no backend on this machine supports this track", + )) + } else { + Err(VideoError::UnsupportedCodec { codec: track.codec }) + } + } +} + +impl std::fmt::Debug for DecoderRegistry { + /// Lists the registered ids: the factories themselves are trait objects + /// with nothing else printable, and the order is the interesting part. + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("DecoderRegistry") + .field( + "factories", + &self.factories.iter().map(|f| f.id().id).collect::>(), + ) + .finish() + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::track::{CicpColor, CodecConfig, Rotation}; + use gamut_color::ChromaSubsampling; + use rawshift_core::metadata::URational; + use rawshift_core::{BitDepth, Dimensions}; + use std::sync::atomic::{AtomicUsize, Ordering}; + + fn track(codec: VideoCodecId) -> VideoTrack { + VideoTrack { + id: crate::id::TrackId::new(1), + codec, + dimensions: Dimensions { + width: 1920, + height: 1080, + }, + color: CicpColor::UNSPECIFIED, + bit_depth: BitDepth::Eight, + chroma: ChromaSubsampling::Cs420, + frame_rate: None, + time_base: URational::new(1, 1000), + duration: None, + frame_count: None, + rotation: Rotation::None, + codec_config: CodecConfig::default(), + } + } + + struct StubDecoder(VideoCodecId); + impl VideoDecoder for StubDecoder { + fn codec(&self) -> VideoCodecId { + self.0 + } + fn send_packet(&mut self, _p: &Packet) -> VideoResult<()> { + Ok(()) + } + fn receive_frame(&mut self) -> VideoResult> { + Ok(None) + } + fn flush(&mut self) -> VideoResult<()> { + Ok(()) + } + fn reset(&mut self) {} + } + + struct StubFactory { + id: &'static str, + codec: VideoCodecId, + supports: bool, + opens: &'static AtomicUsize, + open_fails: bool, + } + + impl VideoDecoderFactory for StubFactory { + fn id(&self) -> CodecId { + CodecId::new(self.id) + } + fn codec(&self) -> VideoCodecId { + self.codec + } + fn supports(&self, _t: &VideoTrack) -> bool { + self.supports + } + fn open(&self, _t: &VideoTrack) -> VideoResult> { + self.opens.fetch_add(1, Ordering::SeqCst); + if self.open_fails { + Err(VideoError::decode(self.codec, "backend fault")) + } else { + Ok(Box::new(StubDecoder(self.codec))) + } + } + } + + /// `Box` is not `Debug`, so `expect_err` does not apply. + fn expect_err(result: VideoResult>, context: &str) -> VideoError { + match result { + Ok(_) => panic!("expected an error: {context}"), + Err(e) => e, + } + } + + fn expect_ok(result: VideoResult>, context: &str) { + if let Err(e) = result { + panic!("expected success ({context}), got {e}"); + } + } + + static FIRST: AtomicUsize = AtomicUsize::new(0); + static SECOND: AtomicUsize = AtomicUsize::new(0); + + fn factory( + id: &'static str, + codec: VideoCodecId, + supports: bool, + opens: &'static AtomicUsize, + open_fails: bool, + ) -> Box { + Box::new(StubFactory { + id, + codec, + supports, + opens, + open_fails, + }) + } + + #[test] + fn push_order_is_priority_order() { + FIRST.store(0, Ordering::SeqCst); + SECOND.store(0, Ordering::SeqCst); + let reg = DecoderRegistry::new() + .with(factory( + "h264/first", + VideoCodecId::H264, + true, + &FIRST, + false, + )) + .with(factory( + "h264/second", + VideoCodecId::H264, + true, + &SECOND, + false, + )); + + expect_ok(reg.open(&track(VideoCodecId::H264)), "a supporting backend"); + assert_eq!(FIRST.load(Ordering::SeqCst), 1, "first should have won"); + assert_eq!( + SECOND.load(Ordering::SeqCst), + 0, + "second should not be tried" + ); + } + + #[test] + fn an_unsupporting_backend_falls_through_to_the_next() { + FIRST.store(0, Ordering::SeqCst); + SECOND.store(0, Ordering::SeqCst); + let reg = DecoderRegistry::new() + .with(factory( + "h264/first", + VideoCodecId::H264, + false, + &FIRST, + false, + )) + .with(factory( + "h264/second", + VideoCodecId::H264, + true, + &SECOND, + false, + )); + + expect_ok(reg.open(&track(VideoCodecId::H264)), "a supporting backend"); + assert_eq!( + FIRST.load(Ordering::SeqCst), + 0, + "unsupporting must not open" + ); + assert_eq!(SECOND.load(Ordering::SeqCst), 1); + } + + #[test] + fn a_backend_that_accepts_then_fails_is_not_retried_elsewhere() { + // The core of the fallback contract: only supports()==false falls + // through, so a genuine backend fault surfaces instead of being + // disguised as "unsupported" by a later backend succeeding. + FIRST.store(0, Ordering::SeqCst); + SECOND.store(0, Ordering::SeqCst); + let reg = DecoderRegistry::new() + .with(factory( + "h264/first", + VideoCodecId::H264, + true, + &FIRST, + true, + )) + .with(factory( + "h264/second", + VideoCodecId::H264, + true, + &SECOND, + false, + )); + + let err = expect_err( + reg.open(&track(VideoCodecId::H264)), + "the fault must propagate", + ); + assert!(err.to_string().contains("backend fault"), "{err}"); + assert_eq!(SECOND.load(Ordering::SeqCst), 0, "must not fall through"); + } + + #[test] + fn no_factory_for_the_codec_is_unsupported_not_unavailable() { + let reg = DecoderRegistry::new().with(factory( + "h264/vaapi", + VideoCodecId::H264, + true, + &FIRST, + false, + )); + let err = expect_err(reg.open(&track(VideoCodecId::ProRes)), "no ProRes factory"); + assert!( + matches!(err, VideoError::UnsupportedCodec { .. }), + "expected UnsupportedCodec, got {err}" + ); + } + + #[test] + fn a_compiled_codec_with_no_usable_backend_is_unavailable_not_unsupported() { + // The distinction a caller acts on: "rebuild with the feature" versus + // "this machine has no GPU". + let reg = DecoderRegistry::new().with(factory( + "hevc/vaapi", + VideoCodecId::Hevc, + false, + &FIRST, + false, + )); + let err = expect_err( + reg.open(&track(VideoCodecId::Hevc)), + "no backend supports it", + ); + assert!( + matches!(err, VideoError::HwDecoderUnavailable { .. }), + "expected HwDecoderUnavailable, got {err}" + ); + assert!(err.is_capability_gap()); + } + + #[test] + fn an_empty_registry_reports_unsupported() { + let reg = DecoderRegistry::new(); + assert!(!reg.supports(&track(VideoCodecId::H264))); + assert!(matches!( + reg.open(&track(VideoCodecId::H264)), + Err(VideoError::UnsupportedCodec { .. }) + )); + } + + #[test] + fn debug_lists_ids_in_priority_order() { + let reg = DecoderRegistry::new() + .with(factory( + "hevc/vaapi", + VideoCodecId::Hevc, + true, + &FIRST, + false, + )) + .with(factory( + "h264/vaapi", + VideoCodecId::H264, + true, + &SECOND, + false, + )); + let printed = format!("{reg:?}"); + assert!(printed.contains("hevc/vaapi"), "{printed}"); + assert!( + printed.find("hevc/vaapi") < printed.find("h264/vaapi"), + "order must be visible: {printed}" + ); + } +} diff --git a/crates/rawshift-video-core/src/demux.rs b/crates/rawshift-video-core/src/demux.rs new file mode 100644 index 0000000..a1dcbc8 --- /dev/null +++ b/crates/rawshift-video-core/src/demux.rs @@ -0,0 +1,154 @@ +//! The demuxing contract. + +use crate::error::VideoResult; +use crate::id::{ContainerId, TrackId}; +use crate::metadata::VideoMetadata; +use crate::packet::{Packet, SeekMode, SeekTo}; +use crate::track::Track; + +/// Cheap signature detection for one container. +/// +/// Deliberately a static method on a marker type rather than a method on a +/// demuxer: detection must be answerable from a handful of leading bytes, +/// before anything is opened or allocated. +pub trait ContainerSniffer { + /// The container this sniffer recognises. + const CONTAINER: ContainerId; + + /// Whether `data` looks like this container. + /// + /// `data` may be short; return `false` rather than panicking. A `true` + /// here is a claim about the signature only, not that the file is valid. + fn matches(data: &[u8]) -> bool; +} + +/// A container reader: tracks, metadata, and a packet stream. +/// +/// Object-safe, and held behind `Box` by the aggregator, so the +/// set of compiled-in containers is a runtime property rather than a type +/// parameter that would leak into every caller's signature. +pub trait Demuxer: Send { + /// Which container this is reading. + fn container(&self) -> ContainerId; + + /// Every track in the file, in the container's own order. + /// + /// Available immediately after opening, on every target, with no decoder: + /// track enumeration never needs a codec. + fn tracks(&self) -> &[Track]; + + /// The file's container-level metadata. + fn metadata(&self) -> &VideoMetadata; + + /// The next packet from any track, in stored order, or `None` at end of + /// file. + /// + /// Packets from every track are interleaved as the container stores them. + /// Filter on [`Packet::track`] to follow one track; a caller that reads + /// only one track must still drain the others or they accumulate. + /// + /// # Errors + /// + /// Propagates I/O failures and container errors. An error is not + /// necessarily terminal, but a caller that cannot re-synchronise should + /// stop. + fn next_packet(&mut self) -> VideoResult>; + + /// Seek `track` to `to`, returning the timestamp actually landed on. + /// + /// With [`SeekMode::Precise`] the landing point is at or before the + /// request and always a random access point, so decoding can resume; + /// reaching an exact frame means decoding forward from there and + /// discarding. Any decoder consuming this demuxer must be + /// [`reset`](crate::decode::VideoDecoder::reset) afterwards. + /// + /// # Errors + /// + /// [`VideoError::NoSuchTrack`](crate::error::VideoError::NoSuchTrack) for + /// an unknown track, plus I/O and container errors. Seeking a + /// non-seekable source fails rather than silently reading forward. + fn seek(&mut self, track: TrackId, to: SeekTo, mode: SeekMode) -> VideoResult; + + /// The track with this identifier, if the file has one. + /// + /// Provided: a linear scan over [`tracks`](Self::tracks). Track counts are + /// single digits in practice, so no index is worth maintaining. + fn track(&self, id: TrackId) -> Option<&Track> { + self.tracks().iter().find(|t| t.id() == id) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::error::VideoError; + use crate::id::TrackKind; + use crate::track::OtherTrack; + use rawshift_core::metadata::URational; + + struct StubDemuxer { + tracks: Vec, + metadata: VideoMetadata, + } + + fn other_track(id: u32) -> Track { + Track::Other(OtherTrack { + id: TrackId::new(id), + kind: TrackKind::Data, + time_base: URational::new(1, 1000), + duration: None, + language: None, + }) + } + + impl Demuxer for StubDemuxer { + fn container(&self) -> ContainerId { + ContainerId::Mp4 + } + fn tracks(&self) -> &[Track] { + &self.tracks + } + fn metadata(&self) -> &VideoMetadata { + &self.metadata + } + fn next_packet(&mut self) -> VideoResult> { + Ok(None) + } + fn seek(&mut self, track: TrackId, _to: SeekTo, _mode: SeekMode) -> VideoResult { + Err(VideoError::NoSuchTrack { id: track }) + } + } + + #[test] + fn the_provided_track_lookup_matches_on_container_id_not_index() { + // Container-assigned ids are 1-based and need not be contiguous, so a + // lookup must never be an index into the slice. + let d = StubDemuxer { + tracks: vec![other_track(3), other_track(9)], + metadata: VideoMetadata::default(), + }; + + assert_eq!( + d.track(TrackId::new(3)).map(Track::id), + Some(TrackId::new(3)) + ); + assert_eq!( + d.track(TrackId::new(9)).map(Track::id), + Some(TrackId::new(9)) + ); + // Id 1 exists as neither a track id nor a valid index shortcut. + assert!(d.track(TrackId::new(1)).is_none()); + assert!(d.track(TrackId::new(0)).is_none()); + } + + #[test] + fn a_demuxer_is_object_safe() { + // The aggregator stores Box; this fails to compile if a + // signature above ever stops being object-safe. + let d: Box = Box::new(StubDemuxer { + tracks: Vec::new(), + metadata: VideoMetadata::default(), + }); + assert_eq!(d.container(), ContainerId::Mp4); + } +} diff --git a/crates/rawshift-video-core/src/error.rs b/crates/rawshift-video-core/src/error.rs new file mode 100644 index 0000000..1b0bed9 --- /dev/null +++ b/crates/rawshift-video-core/src/error.rs @@ -0,0 +1,218 @@ +//! The error type every rawshift video crate returns. +//! +//! Mirrors `rawshift-image-core`'s `RawError` in shape and in one rule: a +//! backend's own error type never crosses a rawshift boundary. Demuxer and +//! decoder backends are stringified into [`VideoError::Container`] or +//! [`VideoError::Decode`] at the leaf crate that owns them, so swapping a +//! backend is not a breaking change for callers. + +use std::io; + +use thiserror::Error; + +use crate::id::{ContainerId, TrackId, VideoCodecId}; + +/// Anything that can go wrong reading or decoding a video file. +/// +/// `#[non_exhaustive]`: new containers and codecs bring new failure modes, and +/// that must not be a breaking change. Match with a `_` arm. +#[derive(Debug, Error)] +#[non_exhaustive] +pub enum VideoError { + /// The underlying reader failed. + #[error("I/O error: {0}")] + Io(#[from] io::Error), + + /// The container is malformed, truncated, or uses a structure the demuxer + /// does not implement. + /// + /// `container` names the demuxer that rejected the file so the message can + /// be attributed; `message` is the backend's own text, flattened to a + /// string precisely so the backend's error type stays private. + #[error("{container} container error: {message}")] + Container { + /// The demuxer that produced this error. + container: &'static str, + /// Backend-supplied detail. + message: String, + }, + + /// The bytes are not a container rawshift can open. + /// + /// `detected` is `Some` when rawshift recognised the format but has no + /// demuxer compiled in for it (AVI and MXF are named but unimplemented; + /// MP4 and Matroska are feature-gated), and `None` when nothing matched. + #[error("unsupported container: {}", match .detected { + Some(c) => c.name(), + None => "unrecognised", + })] + UnsupportedContainer { + /// The container, when it was recognised. + detected: Option, + }, + + /// The track's codec has no decoder in this build. + /// + /// Distinct from [`HwDecoderUnavailable`](Self::HwDecoderUnavailable): + /// this one means no decoder for the codec was compiled in at all, not + /// that a compiled decoder found no backend at runtime. + #[error("unsupported codec: {codec}")] + UnsupportedCodec { + /// The codec the track carries. + codec: VideoCodecId, + }, + + /// A decoder was found, and decoding failed. + #[error("{codec} decode failed: {message}")] + Decode { + /// The codec being decoded. + codec: VideoCodecId, + /// Backend-supplied detail. + message: String, + }, + + /// A decoder for this codec exists in the build, but no hardware backend + /// could be found at runtime. + /// + /// This is the expected outcome on every target with no hardware decode + /// API — `wasm32`, `musl`, Windows — and on Linux hosts with no libva or + /// no capable driver. It is deliberately matchable so a caller can fall + /// back or degrade rather than treat it as corruption. Container parsing, + /// track enumeration, and metadata all still work: see `docs/SUPPORT.md`. + #[error("no hardware decoder available for {codec}: {reason}")] + HwDecoderUnavailable { + /// The codec no backend was found for. + codec: VideoCodecId, + /// Why — no backend compiled in, no device, or no driver support. + reason: String, + }, + + /// The file has no track with the requested identifier. + #[error("no such track: {id}")] + NoSuchTrack { + /// The identifier that did not match a track. + id: TrackId, + }, + + /// Container-level metadata was present but could not be interpreted. + /// + /// Metadata failures are never fatal to opening a file — a file with + /// unreadable metadata still demuxes — so this surfaces only from the + /// explicit metadata entry points. + #[error("metadata error: {message}")] + Metadata { + /// What could not be interpreted. + message: String, + }, +} + +impl VideoError { + /// Build a [`VideoError::Container`] from a backend error. + /// + /// The one place demuxer backends are flattened, so the rule "no backend + /// type crosses a public boundary" has a single obvious implementation. + pub fn container(container: &'static str, message: impl std::fmt::Display) -> Self { + Self::Container { + container, + message: message.to_string(), + } + } + + /// Build a [`VideoError::Decode`] from a backend error. + pub fn decode(codec: VideoCodecId, message: impl std::fmt::Display) -> Self { + Self::Decode { + codec, + message: message.to_string(), + } + } + + /// Build a [`VideoError::HwDecoderUnavailable`]. + pub fn hw_unavailable(codec: VideoCodecId, reason: impl std::fmt::Display) -> Self { + Self::HwDecoderUnavailable { + codec, + reason: reason.to_string(), + } + } + + /// Whether this error means "this build or machine cannot decode it", + /// rather than "this file is broken". + /// + /// True for [`UnsupportedContainer`](Self::UnsupportedContainer), + /// [`UnsupportedCodec`](Self::UnsupportedCodec) and + /// [`HwDecoderUnavailable`](Self::HwDecoderUnavailable). Lets a caller + /// distinguish a capability gap from corruption without matching every + /// variant of a `#[non_exhaustive]` enum. + #[must_use] + pub const fn is_capability_gap(&self) -> bool { + matches!( + self, + Self::UnsupportedContainer { .. } + | Self::UnsupportedCodec { .. } + | Self::HwDecoderUnavailable { .. } + ) + } +} + +/// `Result` alias used throughout rawshift's video crates. +pub type VideoResult = Result; + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn capability_gaps_are_distinguished_from_corruption() { + assert!( + VideoError::UnsupportedCodec { + codec: VideoCodecId::ProRes + } + .is_capability_gap() + ); + assert!(VideoError::hw_unavailable(VideoCodecId::Hevc, "no libva").is_capability_gap()); + assert!( + VideoError::UnsupportedContainer { + detected: Some(ContainerId::Mxf) + } + .is_capability_gap() + ); + + assert!(!VideoError::container("MP4", "truncated moov").is_capability_gap()); + assert!(!VideoError::decode(VideoCodecId::H264, "bad slice").is_capability_gap()); + assert!(!VideoError::Io(io::Error::from(io::ErrorKind::UnexpectedEof)).is_capability_gap()); + } + + #[test] + fn unsupported_container_reads_well_both_ways() { + let named = VideoError::UnsupportedContainer { + detected: Some(ContainerId::Avi), + }; + assert_eq!(named.to_string(), "unsupported container: AVI"); + + let unknown = VideoError::UnsupportedContainer { detected: None }; + assert_eq!(unknown.to_string(), "unsupported container: unrecognised"); + } + + #[test] + fn backend_detail_is_preserved_in_the_message() { + let e = VideoError::container("Matroska", "unexpected EBML id 0x1f43b675"); + assert!(e.to_string().contains("Matroska")); + assert!(e.to_string().contains("0x1f43b675")); + } + + #[test] + fn hw_unavailable_names_the_codec_and_reason() { + let e = VideoError::hw_unavailable(VideoCodecId::H264, "no VAAPI device"); + let s = e.to_string(); + assert!(s.contains("H.264"), "{s}"); + assert!(s.contains("no VAAPI device"), "{s}"); + } + + #[test] + fn io_errors_convert_with_the_question_mark_operator() { + fn inner() -> VideoResult<()> { + Err(io::Error::from(io::ErrorKind::PermissionDenied))?; + unreachable!() + } + assert!(matches!(inner(), Err(VideoError::Io(_)))); + } +} diff --git a/crates/rawshift-video-core/src/frame.rs b/crates/rawshift-video-core/src/frame.rs new file mode 100644 index 0000000..b0110b2 --- /dev/null +++ b/crates/rawshift-video-core/src/frame.rs @@ -0,0 +1,576 @@ +//! Decoded frames. +//! +//! A [`VideoFrame`] owns its pixel data. Hardware decoders hand back planes +//! that live in a driver-owned surface, so the decoder copies them out before +//! the surface is recycled; a frame therefore never pins a decoder resource +//! and can outlive the decoder that produced it. + +use gamut_color::{ColorRange, MatrixCoefficients}; +use rawshift_core::Dimensions; + +use crate::error::{VideoError, VideoResult}; +use crate::id::VideoCodecId; + +/// One plane of a decoded picture. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Plane { + /// Sample data, `stride` bytes per row. + pub data: Vec, + /// Bytes per row, which is at least the row's width in bytes and may + /// exceed it: hardware decoders align rows. + pub stride: usize, +} + +/// The sample layout of a decoded picture. +/// +/// These are the four 4:2:0 layouts hardware decoders produce. rawshift's own +/// type rather than the backend's, so swapping backends is not a breaking +/// change. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +#[non_exhaustive] +pub enum FramePixelFormat { + /// 8-bit, Y plane then interleaved CbCr. Two planes. + Nv12, + /// 10-bit in 16-bit words, Y plane then interleaved CbCr, value in the + /// **most** significant bits. Two planes. + P010, + /// 8-bit planar Y, Cb, Cr. Three planes. + I420, + /// 10-bit planar Y, Cb, Cr in 16-bit words, value in the **least** + /// significant bits. Three planes. + I010, +} + +impl FramePixelFormat { + /// How many planes this layout has. + #[must_use] + pub const fn plane_count(self) -> usize { + match self { + Self::Nv12 | Self::P010 => 2, + Self::I420 | Self::I010 => 3, + } + } + + /// Bytes per sample: 1 for the 8-bit layouts, 2 for the 10-bit ones. + #[must_use] + pub const fn bytes_per_sample(self) -> usize { + match self { + Self::Nv12 | Self::I420 => 1, + Self::P010 | Self::I010 => 2, + } + } + + /// Whether chroma is stored interleaved (`NV12`/`P010`) rather than as two + /// separate planes. + #[must_use] + pub const fn is_semi_planar(self) -> bool { + matches!(self, Self::Nv12 | Self::P010) + } + + /// How far to shift a stored 16-bit word right to recover the sample. + /// + /// `P010` parks the 10-bit value in the top of the word; `I010` stores it + /// at the bottom. Zero for the 8-bit layouts. + #[must_use] + pub const fn sample_shift(self) -> u32 { + match self { + Self::P010 => 6, + Self::Nv12 | Self::I420 | Self::I010 => 0, + } + } +} + +/// Everything about a decoded picture except its pixels. +/// +/// Exists so [`VideoFrame::new`] takes one descriptor rather than a row of +/// same-typed positional arguments, where transposing two is silent. +#[derive(Debug, Clone, Copy, PartialEq)] +#[non_exhaustive] +pub struct FrameDesc { + /// The sample layout. + pub format: FramePixelFormat, + /// Picture dimensions. + pub dimensions: Dimensions, + /// Bits per sample. + pub bit_depth: u8, + /// Whether samples occupy the full code range or the studio subset. + pub range: ColorRange, + /// The matrix mapping YCbCr to RGB. Must already be resolved — never + /// [`MatrixCoefficients::Unspecified`]. + pub matrix: MatrixCoefficients, + /// Presentation timestamp of the packet this picture was coded from. + pub pts: Option, + /// Picture order count derived from the bitstream. + pub poc: i32, +} + +impl FrameDesc { + /// A descriptor for a picture, with no timestamp and a zero picture order + /// count. + #[must_use] + pub const fn new( + format: FramePixelFormat, + dimensions: Dimensions, + bit_depth: u8, + range: ColorRange, + matrix: MatrixCoefficients, + ) -> Self { + Self { + format, + dimensions, + bit_depth, + range, + matrix, + pts: None, + poc: 0, + } + } + + /// Set the presentation timestamp and picture order count. + #[must_use] + pub const fn with_timing(mut self, pts: Option, poc: i32) -> Self { + self.pts = pts; + self.poc = poc; + self + } +} + +/// A decoded picture, in output (display) order. +/// +/// Construct with [`VideoFrame::new`], which validates that the planes match +/// the format and dimensions — so every accessor below can be infallible. +#[derive(Debug, Clone, PartialEq)] +pub struct VideoFrame { + format: FramePixelFormat, + dimensions: Dimensions, + bit_depth: u8, + range: ColorRange, + matrix: MatrixCoefficients, + planes: Vec, + pts: Option, + poc: i32, +} + +impl VideoFrame { + /// Assemble a frame, validating it against its stated format. + /// + /// # Errors + /// + /// [`VideoError::Decode`] when the plane count does not match the format, + /// or a plane's stride or length cannot hold the stated dimensions. This + /// catches a backend that returned a surface inconsistent with what it + /// claimed, which is otherwise an out-of-bounds read waiting to happen in + /// every consumer. + pub fn new(codec: VideoCodecId, desc: FrameDesc, planes: Vec) -> VideoResult { + let FrameDesc { + format, + dimensions, + bit_depth, + range, + matrix, + pts, + poc, + } = desc; + + if planes.len() != format.plane_count() { + return Err(VideoError::decode( + codec, + format!( + "{format:?} needs {} planes, got {}", + format.plane_count(), + planes.len() + ), + )); + } + + let bps = format.bytes_per_sample(); + let width = dimensions.width as usize; + let height = dimensions.height as usize; + // 4:2:0: chroma is half resolution on both axes, rounded up so odd + // dimensions keep their final half-populated row and column. + let chroma_width = width.div_ceil(2); + let chroma_height = height.div_ceil(2); + + for (index, plane) in planes.iter().enumerate() { + let (min_row_bytes, rows) = match (index, format.is_semi_planar()) { + (0, _) => (width * bps, height), + // Interleaved CbCr: one row holds both components. + (_, true) => (chroma_width * 2 * bps, chroma_height), + (_, false) => (chroma_width * bps, chroma_height), + }; + + if plane.stride < min_row_bytes { + return Err(VideoError::decode( + codec, + format!( + "plane {index} stride {} is below the {min_row_bytes} bytes a row needs", + plane.stride + ), + )); + } + // The last row need only be complete up to min_row_bytes; drivers + // do not always pad it out to a full stride. + let required = plane.stride * rows.saturating_sub(1) + min_row_bytes; + if plane.data.len() < required { + return Err(VideoError::decode( + codec, + format!( + "plane {index} holds {} bytes, needs {required} for {rows} rows", + plane.data.len() + ), + )); + } + } + + Ok(Self { + format, + dimensions, + bit_depth, + range, + matrix, + planes, + pts, + poc, + }) + } + + /// Picture width in pixels. + #[must_use] + pub const fn width(&self) -> u32 { + self.dimensions.width + } + + /// Picture height in pixels. + #[must_use] + pub const fn height(&self) -> u32 { + self.dimensions.height + } + + /// Picture dimensions. + #[must_use] + pub const fn dimensions(&self) -> Dimensions { + self.dimensions + } + + /// The sample layout. + #[must_use] + pub const fn format(&self) -> FramePixelFormat { + self.format + } + + /// Bits per sample — 8 or 10 for the layouts rawshift decodes. + #[must_use] + pub const fn bit_depth(&self) -> u8 { + self.bit_depth + } + + /// Whether samples occupy the full code range or the studio subset. + #[must_use] + pub const fn range(&self) -> ColorRange { + self.range + } + + /// The matrix that maps this frame's YCbCr samples to RGB. + /// + /// Already resolved: never `Unspecified`. + #[must_use] + pub const fn matrix(&self) -> MatrixCoefficients { + self.matrix + } + + /// The planes, in the order the format defines. + #[must_use] + pub fn planes(&self) -> &[Plane] { + &self.planes + } + + /// The presentation timestamp of the packet this picture was coded from, + /// in the track's `time_base` units. + #[must_use] + pub const fn pts(&self) -> Option { + self.pts + } + + /// The picture order count derived from the bitstream. + /// + /// Strictly increasing across the frames emitted between two random access + /// points, which makes it the reliable ordering key even where a container + /// carries absent or broken timestamps. + #[must_use] + pub const fn poc(&self) -> i32 { + self.poc + } + + /// Read one luma sample, or `None` if the coordinates are outside the + /// picture. + /// + /// Returned on a 0–65535 scale regardless of the source bit depth, so + /// callers need not branch on depth. + #[must_use] + pub fn luma(&self, x: u32, y: u32) -> Option { + if x >= self.dimensions.width || y >= self.dimensions.height { + return None; + } + let plane = self.planes.first()?; + let (x, y) = (x as usize, y as usize); + let sample = match self.format.bytes_per_sample() { + 1 => u16::from(*plane.data.get(y * plane.stride + x)?), + _ => { + let at = y * plane.stride + x * 2; + let lo = *plane.data.get(at)?; + let hi = *plane.data.get(at + 1)?; + u16::from_le_bytes([lo, hi]) >> self.format.sample_shift() + } + }; + Some(scale_to_u16(sample, self.bit_depth)) + } +} + +/// Scale a sample of `bit_depth` bits up to the full `u16` range. +/// +/// Replicates the sample's own high bits into the vacated low bits rather than +/// shifting in zeros. A plain `<< 8` would map 8-bit 255 to 65280, so every +/// white in the picture would come out slightly grey and the scaling would not +/// be idempotent across depths. +fn scale_to_u16(sample: u16, bit_depth: u8) -> u16 { + let depth = u32::from(bit_depth); + if depth == 0 || depth >= 16 { + return sample; + } + // Mask defensively: a backend that reports 10-bit while leaving junk in + // the top 6 bits would otherwise scale that junk into the output. + let value = u32::from(sample) & ((1u32 << depth) - 1); + + // Left-align, then tile the value downwards to fill the remaining bits. + let mut out = value << (32 - depth); + let mut filled = depth; + while filled < 32 { + out |= (value << (32 - depth)) >> filled; + filled += depth; + } + (out >> 16) as u16 +} + +#[cfg(test)] +mod tests { + use super::*; + + fn dims(w: u32, h: u32) -> Dimensions { + Dimensions { + width: w, + height: h, + } + } + + fn nv12(width: usize, height: usize) -> Vec { + vec![ + Plane { + data: vec![0u8; width * height], + stride: width, + }, + Plane { + data: vec![0u8; width * height.div_ceil(2)], + stride: width, + }, + ] + } + + fn frame_from( + planes: Vec, + format: FramePixelFormat, + w: u32, + h: u32, + ) -> VideoResult { + let desc = FrameDesc::new( + format, + dims(w, h), + 8, + ColorRange::Limited, + MatrixCoefficients::Bt709, + ) + .with_timing(Some(0), 0); + VideoFrame::new(VideoCodecId::Hevc, desc, planes) + } + + #[test] + fn plane_counts_match_the_layouts() { + assert_eq!(FramePixelFormat::Nv12.plane_count(), 2); + assert_eq!(FramePixelFormat::P010.plane_count(), 2); + assert_eq!(FramePixelFormat::I420.plane_count(), 3); + assert_eq!(FramePixelFormat::I010.plane_count(), 3); + } + + #[test] + fn only_p010_parks_its_value_in_the_high_bits() { + assert_eq!(FramePixelFormat::P010.sample_shift(), 6); + assert_eq!(FramePixelFormat::I010.sample_shift(), 0); + assert_eq!(FramePixelFormat::Nv12.sample_shift(), 0); + } + + #[test] + fn a_well_formed_frame_is_accepted() { + let f = frame_from(nv12(64, 48), FramePixelFormat::Nv12, 64, 48) + .expect("well-formed NV12 frame"); + assert_eq!(f.width(), 64); + assert_eq!(f.height(), 48); + assert_eq!(f.planes().len(), 2); + } + + #[test] + fn a_wrong_plane_count_is_rejected() { + // NV12 needs two planes; hand it three. + let mut planes = nv12(64, 48); + planes.push(Plane { + data: vec![0; 16], + stride: 4, + }); + let err = frame_from(planes, FramePixelFormat::Nv12, 64, 48) + .expect_err("three planes is not NV12"); + assert!(err.to_string().contains("needs 2 planes"), "{err}"); + } + + #[test] + fn a_stride_too_small_for_the_width_is_rejected() { + let planes = vec![ + Plane { + data: vec![0; 64 * 48], + stride: 32, // half the 64 bytes a row needs + }, + Plane { + data: vec![0; 64 * 24], + stride: 64, + }, + ]; + let err = + frame_from(planes, FramePixelFormat::Nv12, 64, 48).expect_err("understated stride"); + assert!(err.to_string().contains("stride"), "{err}"); + } + + #[test] + fn a_truncated_plane_is_rejected() { + let planes = vec![ + Plane { + data: vec![0; 64 * 10], // 10 rows where 48 are claimed + stride: 64, + }, + Plane { + data: vec![0; 64 * 24], + stride: 64, + }, + ]; + let err = + frame_from(planes, FramePixelFormat::Nv12, 64, 48).expect_err("truncated luma plane"); + assert!(err.to_string().contains("needs"), "{err}"); + } + + #[test] + fn an_aligned_stride_larger_than_the_width_is_fine() { + // Drivers align rows; a 64-wide picture in a 128-byte stride is normal. + let planes = vec![ + Plane { + data: vec![0; 128 * 48], + stride: 128, + }, + Plane { + data: vec![0; 128 * 24], + stride: 128, + }, + ]; + assert!(frame_from(planes, FramePixelFormat::Nv12, 64, 48).is_ok()); + } + + #[test] + fn odd_dimensions_round_chroma_up() { + // A 65x49 picture has 33x25 chroma, not 32x24 — rounding down would + // drop the last column and row. + let planes = vec![ + Plane { + data: vec![0; 65 * 49], + stride: 65, + }, + Plane { + data: vec![0; 66 * 25], + stride: 66, + }, + ]; + assert!(frame_from(planes, FramePixelFormat::Nv12, 65, 49).is_ok()); + } + + #[test] + fn the_final_row_need_not_be_padded_to_a_full_stride() { + // A driver may hand back exactly enough bytes to hold the last row's + // pixels, with no trailing alignment padding. + let stride = 128; + let planes = vec![ + Plane { + data: vec![0; stride * 47 + 64], + stride, + }, + Plane { + data: vec![0; stride * 23 + 64], + stride, + }, + ]; + assert!(frame_from(planes, FramePixelFormat::Nv12, 64, 48).is_ok()); + } + + #[test] + fn luma_reads_8_bit_samples_scaled_to_full_range() { + let mut planes = nv12(4, 2); + planes[0].data[0] = 255; + planes[0].data[1] = 0; + let f = frame_from(planes, FramePixelFormat::Nv12, 4, 2).expect("frame"); + // 8-bit 255 must reach u16::MAX, not 65280. + assert_eq!(f.luma(0, 0), Some(u16::MAX)); + assert_eq!(f.luma(1, 0), Some(0)); + } + + #[test] + fn luma_is_none_outside_the_picture() { + let f = frame_from(nv12(4, 2), FramePixelFormat::Nv12, 4, 2).expect("frame"); + assert_eq!(f.luma(4, 0), None); + assert_eq!(f.luma(0, 2), None); + } + + #[test] + fn scaling_maps_extremes_to_extremes_at_every_depth() { + for depth in [1u8, 4, 8, 10, 12, 15, 16] { + let max = ((1u32 << depth) - 1) as u16; + assert_eq!(scale_to_u16(0, depth), 0, "depth {depth} floor"); + assert_eq!(scale_to_u16(max, depth), u16::MAX, "depth {depth} ceiling"); + } + } + + #[test] + fn scaling_replicates_bits_rather_than_shifting_in_zeros() { + // The textbook cases: 8-bit replicates the byte, 10-bit borrows its + // own top 6 bits, 12-bit its own top 4. + assert_eq!(scale_to_u16(0xAB, 8), 0xABAB); + assert_eq!(scale_to_u16(0b11_1111_0000, 10), 0b1111_1100_0011_1111); + assert_eq!(scale_to_u16(0x800, 12), 0x8008); + } + + #[test] + fn scaling_is_monotonic() { + // Ordering must survive scaling, or gradients would band. + let mut previous = 0; + for value in 0..=1023u16 { + let scaled = scale_to_u16(value, 10); + assert!(scaled >= previous, "value {value} went backwards"); + previous = scaled; + } + } + + #[test] + fn scaling_ignores_junk_above_the_stated_depth() { + // A backend claiming 10 bits but leaving the top 6 set must not have + // that junk scaled into the result. + assert_eq!(scale_to_u16(0x03FF, 10), scale_to_u16(0xFFFF, 10)); + } + + #[test] + fn full_and_zero_depth_pass_through_untouched() { + assert_eq!(scale_to_u16(12345, 16), 12345); + assert_eq!(scale_to_u16(12345, 0), 12345); + } +} diff --git a/crates/rawshift-video-core/src/id.rs b/crates/rawshift-video-core/src/id.rs new file mode 100644 index 0000000..b2fc586 --- /dev/null +++ b/crates/rawshift-video-core/src/id.rs @@ -0,0 +1,257 @@ +//! Format and codec identity. +//! +//! Every enum here is `#[non_exhaustive]`. rawshift names containers and +//! codecs it does not yet decode, so that "we know what this is and cannot +//! handle it" is expressible — see [`crate::error::VideoError`] — and so that +//! growing support later is additive rather than breaking. + +use std::fmt; + +/// A container (file) format. +/// +/// A variant existing says only that rawshift can *name* the container. Ask +/// the aggregator whether a demuxer for it was compiled in; opening an +/// unsupported one fails with +/// [`VideoError::UnsupportedContainer`](crate::error::VideoError::UnsupportedContainer). +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +#[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))] +#[non_exhaustive] +pub enum ContainerId { + /// ISO Base Media File Format as MP4 / M4V (`isom`, `mp41`, `mp42`, …). + Mp4, + /// ISOBMFF in its QuickTime flavour (`qt `) — the `.mov` Apple writes. + Mov, + /// Matroska. + Matroska, + /// WebM, the Matroska subset restricted to royalty-free codecs. + WebM, + /// Audio Video Interleave (RIFF). Named, not implemented. + Avi, + /// Material Exchange Format, SMPTE 377M. Named, not implemented. + Mxf, +} + +impl ContainerId { + /// A short, stable, human-readable name (`"MP4"`, `"QuickTime"`, …). + /// + /// Stable across releases: it appears in error messages and in + /// [`CodecId`](rawshift_core::CodecId) strings. + #[must_use] + pub const fn name(self) -> &'static str { + match self { + Self::Mp4 => "MP4", + Self::Mov => "QuickTime", + Self::Matroska => "Matroska", + Self::WebM => "WebM", + Self::Avi => "AVI", + Self::Mxf => "MXF", + } + } + + /// Whether this container is an ISOBMFF flavour, and so shares a box + /// grammar and a demuxer with the others that are. + #[must_use] + pub const fn is_isobmff(self) -> bool { + matches!(self, Self::Mp4 | Self::Mov) + } + + /// Whether this container is a Matroska flavour. + #[must_use] + pub const fn is_matroska(self) -> bool { + matches!(self, Self::Matroska | Self::WebM) + } +} + +impl fmt::Display for ContainerId { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str(self.name()) + } +} + +/// A video codec. +/// +/// As with [`ContainerId`], a variant existing does not imply a decoder was +/// compiled in or that a backend can be found at runtime. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +#[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))] +#[non_exhaustive] +pub enum VideoCodecId { + /// H.264 / MPEG-4 AVC. + H264, + /// H.265 / HEVC. + Hevc, + /// AOMedia Video 1. + Av1, + /// VP9. + Vp9, + /// Apple ProRes. Named, not implemented — see the crate README. + ProRes, + /// A codec rawshift recognises the track as carrying but does not name. + Other, +} + +impl VideoCodecId { + /// A short, stable, human-readable name. + #[must_use] + pub const fn name(self) -> &'static str { + match self { + Self::H264 => "H.264", + Self::Hevc => "HEVC", + Self::Av1 => "AV1", + Self::Vp9 => "VP9", + Self::ProRes => "ProRes", + Self::Other => "unknown", + } + } +} + +impl fmt::Display for VideoCodecId { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str(self.name()) + } +} + +/// An audio codec. +/// +/// rawshift **enumerates** audio tracks and hands back their packets; it never +/// decodes audio. This enum exists so a caller can tell what an audio track +/// carries before routing those packets somewhere that does. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +#[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))] +#[non_exhaustive] +pub enum AudioCodecId { + /// Advanced Audio Coding. + Aac, + /// Linear PCM. + Pcm, + /// Opus. + Opus, + /// Vorbis. + Vorbis, + /// FLAC. + Flac, + /// Dolby AC-3 or E-AC-3. + Ac3, + /// A codec rawshift does not name. + Other, +} + +impl AudioCodecId { + /// A short, stable, human-readable name. + #[must_use] + pub const fn name(self) -> &'static str { + match self { + Self::Aac => "AAC", + Self::Pcm => "PCM", + Self::Opus => "Opus", + Self::Vorbis => "Vorbis", + Self::Flac => "FLAC", + Self::Ac3 => "AC-3", + Self::Other => "unknown", + } + } +} + +impl fmt::Display for AudioCodecId { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str(self.name()) + } +} + +/// What kind of media a track carries. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +#[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))] +#[non_exhaustive] +pub enum TrackKind { + /// Coded pictures. + Video, + /// Coded audio. + Audio, + /// Subtitles or captions. + Subtitle, + /// SMPTE timecode. + Timecode, + /// Anything else the container carries as a track. + Data, +} + +/// A track's identifier within one file. +/// +/// Opaque and container-assigned: for ISOBMFF this is the `tkhd` track ID, +/// which is 1-based and need not be contiguous, so never treat it as an index +/// into [`tracks`](crate::demux::Demuxer::tracks). +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)] +#[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))] +pub struct TrackId(u32); + +impl TrackId { + /// Wrap a container-assigned track identifier. + #[must_use] + pub const fn new(id: u32) -> Self { + Self(id) + } + + /// The underlying container-assigned value. + #[must_use] + pub const fn get(self) -> u32 { + self.0 + } +} + +impl fmt::Display for TrackId { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "{}", self.0) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn isobmff_flavours_are_grouped() { + assert!(ContainerId::Mp4.is_isobmff()); + assert!(ContainerId::Mov.is_isobmff()); + assert!(!ContainerId::Matroska.is_isobmff()); + assert!(!ContainerId::Avi.is_isobmff()); + } + + #[test] + fn matroska_flavours_are_grouped() { + assert!(ContainerId::Matroska.is_matroska()); + assert!(ContainerId::WebM.is_matroska()); + assert!(!ContainerId::Mp4.is_matroska()); + } + + #[test] + fn a_container_is_never_both_families() { + for c in [ + ContainerId::Mp4, + ContainerId::Mov, + ContainerId::Matroska, + ContainerId::WebM, + ContainerId::Avi, + ContainerId::Mxf, + ] { + assert!( + !(c.is_isobmff() && c.is_matroska()), + "{c} claims two container families" + ); + } + } + + #[test] + fn track_id_round_trips_and_displays() { + let id = TrackId::new(7); + assert_eq!(id.get(), 7); + assert_eq!(id.to_string(), "7"); + } + + #[test] + fn names_are_stable_and_non_empty() { + assert_eq!(ContainerId::Mov.name(), "QuickTime"); + assert_eq!(VideoCodecId::Hevc.name(), "HEVC"); + assert_eq!(AudioCodecId::Aac.name(), "AAC"); + assert!(!VideoCodecId::Other.name().is_empty()); + } +} diff --git a/crates/rawshift-video-core/src/lib.rs b/crates/rawshift-video-core/src/lib.rs new file mode 100644 index 0000000..e9c7806 --- /dev/null +++ b/crates/rawshift-video-core/src/lib.rs @@ -0,0 +1,66 @@ +//! Shared vocabulary for rawshift's video crates. +//! +//! This is the video counterpart to `rawshift-image-core`: the types every +//! `rawshift-video-*` crate agrees on, and no implementation of its own. No +//! I/O, no platform code, no container or codec dependencies. +//! +//! # Why this crate exists separately +//! +//! The workspace enforces in CI that no `rawshift-image*` crate appears in +//! `rawshift-video`'s dependency tree, so the video crates cannot reuse +//! `rawshift-image-core`'s error type or format traits. Image and video share +//! only [`rawshift_core`] — the media-agnostic geometry, codec descriptors, +//! metadata model, and colour vocabulary — and this crate adds the video +//! half on top of it. +//! +//! # The model +//! +//! - **Identity** — [`ContainerId`], [`VideoCodecId`], [`AudioCodecId`], +//! [`TrackKind`], [`TrackId`]. All `#[non_exhaustive]`, and all naming more +//! than rawshift can decode, so "recognised but unsupported" is expressible. +//! - **Errors** — [`VideoError`], which keeps every backend's own error type +//! private, and distinguishes a capability gap from a broken file via +//! [`VideoError::is_capability_gap`]. +//! - **Tracks** — [`Track`], [`VideoTrack`], [`AudioTrack`], plus +//! [`CicpColor`] and [`Rotation`]. All header-derived, so a full track list +//! is available on targets with no decoder at all. +//! - **Packets and frames** — [`Packet`] in, [`VideoFrame`] out. +//! - **Contracts** — [`Demuxer`], [`VideoDecoder`], [`VideoEncoder`], and the +//! [`DecoderRegistry`] that picks between decoder backends. +//! +//! # Decoding is a two-step conversation +//! +//! [`VideoDecoder`] splits submit from retrieve, because coded order and +//! display order differ whenever a stream reorders. See its documentation for +//! the loop, including the [`flush`](VideoDecoder::flush) without which the +//! last frames of every reordered stream are silently dropped. + +#![forbid(unsafe_code)] +#![warn(missing_docs)] + +pub mod decode; +pub mod demux; +pub mod error; +pub mod frame; +pub mod id; +pub mod metadata; +pub mod nal; +pub mod packet; +pub mod track; + +pub use decode::{DecoderRegistry, VideoDecoder, VideoDecoderFactory, VideoEncoder}; +pub use demux::{ContainerSniffer, Demuxer}; +pub use error::{VideoError, VideoResult}; +pub use frame::{FrameDesc, FramePixelFormat, Plane, VideoFrame}; +pub use id::{AudioCodecId, ContainerId, TrackId, TrackKind, VideoCodecId}; +pub use metadata::{ContainerMetadata, VideoMetadata}; +pub use nal::{NalLengthSize, is_random_access_point, split_length_prefixed}; +pub use packet::{Packet, SeekMode, SeekTo}; +pub use track::{AudioTrack, CicpColor, CodecConfig, OtherTrack, Rotation, Track, VideoTrack}; + +// The colour vocabulary is gamut's, re-exported so callers need not depend on +// gamut directly — the same arrangement `rawshift-core` already uses for the +// pixel and CICP types. +pub use gamut_color::{ + ChromaSubsampling, ColorRange, ColourPrimaries, MatrixCoefficients, TransferCharacteristics, +}; diff --git a/crates/rawshift-video-core/src/metadata.rs b/crates/rawshift-video-core/src/metadata.rs new file mode 100644 index 0000000..05b8bb7 --- /dev/null +++ b/crates/rawshift-video-core/src/metadata.rs @@ -0,0 +1,133 @@ +//! Container-level metadata. +//! +//! The EXIF/XMP/GPS core is [`rawshift_core::ImageMetadata`], reused as-is: +//! `rawshift-core` documents that model as "genuinely video-shared … EXIF/XMP +//! semantics are the same for a video file as for a still", and this is its +//! first video consumer. What video adds on top is the *movie*-level facts a +//! still has no analogue for, which is what [`ContainerMetadata`] holds. + +use std::time::Duration; + +use rawshift_core::{ImageMetadata, MetadataEntry, MetadataKey, MetadataNamespace, MetadataValue}; + +use crate::id::ContainerId; + +/// Everything rawshift read from a video file's headers. +#[derive(Debug, Clone, Default, PartialEq)] +#[non_exhaustive] +pub struct VideoMetadata { + /// The media-agnostic core: EXIF, XMP, GPS, camera identification, and the + /// generic `extra` table. Container-level facts are also projected into + /// `common.extra` under the [`MetadataNamespace::Quicktime`] and + /// [`MetadataNamespace::Matroska`] namespaces, so a caller that only knows + /// `ImageMetadata` still sees them. + pub common: ImageMetadata, + /// Movie-level facts with no still-image analogue. + pub container: ContainerMetadata, +} + +/// Movie-level facts: the parts of a container that describe the file as a +/// whole rather than any one track. +#[derive(Debug, Clone, Default, PartialEq)] +#[non_exhaustive] +pub struct ContainerMetadata { + /// Which container this came from. + pub container: Option, + /// Total presentation duration, when the container states one. + pub duration: Option, + /// The movie timescale, in ticks per second. + /// + /// The movie header's own unit, distinct from any track's `time_base`. + pub timescale: Option, + /// Creation time, as the container states it. + pub creation_time: Option, + /// Modification time, as the container states it. + pub modification_time: Option, + /// ISOBMFF `ftyp` major brand and compatible brands (`"qt "`, `"mp42"`, + /// `"isom"`, …). Empty for non-ISOBMFF containers. + pub brands: Vec, + /// The writing application or muxer, when stated. + pub writing_app: Option, +} + +impl VideoMetadata { + /// Record a container-level fact into the shared `extra` table. + /// + /// The single place container facts enter [`ImageMetadata`], so every + /// demuxer namespaces them the same way and a caller can find them all + /// without knowing which container produced them. + pub fn push_container_entry( + &mut self, + namespace: MetadataNamespace, + tag: impl Into, + value: MetadataValue, + ) { + self.common.extra.push(MetadataEntry { + key: MetadataKey::new(namespace, tag), + value, + }); + } + + /// Look up a container-level fact previously recorded. + #[must_use] + pub fn container_entry( + &self, + namespace: MetadataNamespace, + tag: &str, + ) -> Option<&MetadataValue> { + self.common + .extra + .iter() + .find(|e| e.key.namespace == namespace && e.key.tag == tag) + .map(|e| &e.value) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn container_entries_round_trip_through_the_shared_table() { + let mut md = VideoMetadata::default(); + md.push_container_entry( + MetadataNamespace::Quicktime, + "com.apple.quicktime.model", + MetadataValue::Text("iPhone 17 Pro Max".into()), + ); + + assert_eq!( + md.container_entry(MetadataNamespace::Quicktime, "com.apple.quicktime.model"), + Some(&MetadataValue::Text("iPhone 17 Pro Max".into())) + ); + // It really landed in the shared table, not a side channel. + assert_eq!(md.common.extra.len(), 1); + } + + #[test] + fn lookup_is_namespaced_not_just_tag_matched() { + let mut md = VideoMetadata::default(); + md.push_container_entry( + MetadataNamespace::Matroska, + "title", + MetadataValue::Text("mkv".into()), + ); + // Same tag, different namespace: must not match. + assert_eq!( + md.container_entry(MetadataNamespace::Quicktime, "title"), + None + ); + assert!( + md.container_entry(MetadataNamespace::Matroska, "title") + .is_some() + ); + } + + #[test] + fn a_default_is_empty_rather_than_absent() { + let md = VideoMetadata::default(); + assert!(md.common.extra.is_empty()); + assert_eq!(md.container.container, None); + assert!(md.container.brands.is_empty()); + } +} diff --git a/crates/rawshift-video-core/src/nal.rs b/crates/rawshift-video-core/src/nal.rs new file mode 100644 index 0000000..1775f96 --- /dev/null +++ b/crates/rawshift-video-core/src/nal.rs @@ -0,0 +1,350 @@ +//! NAL-unit framing for the H.264 and HEVC families. +//! +//! MP4 and Matroska store H.264/HEVC access units as **length-prefixed** NAL +//! units — never Annex B start codes — with the prefix width declared once, in +//! the track's `avcC`/`hvcC` configuration record. Both the demuxers and the +//! decoders need to walk that framing, so it lives here rather than in either. +//! +//! This module reads *framing and NAL headers only*. It never parses a slice +//! header, a parameter set, or anything else that would make it a decoder. + +use crate::id::VideoCodecId; +use crate::track::CodecConfig; + +/// The width, in bytes, of the length prefix before each NAL unit. +/// +/// Legal values are 1, 2 and 4. Three-byte prefixes are representable in the +/// configuration record's two-bit field but are not legal, and are rejected. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +pub struct NalLengthSize(u8); + +impl NalLengthSize { + /// Wrap a prefix width, rejecting anything but 1, 2, or 4. + #[must_use] + pub const fn new(bytes: u8) -> Option { + match bytes { + 1 | 2 | 4 => Some(Self(bytes)), + _ => None, + } + } + + /// The width in bytes. + #[must_use] + pub const fn get(self) -> usize { + self.0 as usize + } + + /// Read the prefix width from an `avcC` or `hvcC` record body. + /// + /// Both records store `lengthSizeMinusOne` in the low two bits of a byte: + /// offset 4 for `avcC` (ISO/IEC 14496-15 §5.3.3.1), offset 21 for `hvcC` + /// (§8.3.3.1). + /// + /// Returns `None` for a record that is too short, states an illegal + /// 3-byte prefix, or belongs to a codec with no such record. + #[must_use] + pub fn from_config(codec: VideoCodecId, config: &CodecConfig) -> Option { + let bytes = config.as_bytes(); + let offset = match codec { + VideoCodecId::H264 => 4, + VideoCodecId::Hevc => 21, + _ => return None, + }; + let field = bytes.get(offset)?; + Self::new((field & 0b11) + 1) + } +} + +/// Walk the length-prefixed NAL units of one access unit. +/// +/// Yields each NAL's payload, prefix excluded. Stops at the first prefix that +/// overruns the buffer rather than panicking or yielding a truncated unit: a +/// container is untrusted input, and a truncated tail is the common shape of +/// a damaged file. +pub fn split_length_prefixed(data: &[u8], length_size: NalLengthSize) -> NalUnits<'_> { + NalUnits { + data, + offset: 0, + length_size: length_size.get(), + } +} + +/// Iterator over the NAL units of one access unit. See +/// [`split_length_prefixed`]. +#[derive(Debug, Clone)] +pub struct NalUnits<'a> { + data: &'a [u8], + offset: usize, + length_size: usize, +} + +impl<'a> Iterator for NalUnits<'a> { + type Item = &'a [u8]; + + fn next(&mut self) -> Option<&'a [u8]> { + let header_end = self.offset.checked_add(self.length_size)?; + if header_end > self.data.len() { + return None; + } + + let mut length = 0usize; + for &byte in &self.data[self.offset..header_end] { + // At most 4 bytes, so this cannot overflow a usize on any target + // rawshift supports (all are 32-bit or wider). + length = (length << 8) | byte as usize; + } + + let payload_end = header_end.checked_add(length)?; + if payload_end > self.data.len() { + // Truncated: stop cleanly rather than yielding a short NAL. + return None; + } + + self.offset = payload_end; + Some(&self.data[header_end..payload_end]) + } +} + +/// Whether an access unit is a random access point — a picture a decoder can +/// start on with no earlier data. +/// +/// Read from the bitstream rather than from the container's sync-sample table, +/// because the bitstream is authoritative and container indices are a +/// well-known source of wrong keyframe flags. The chosen demux backend also +/// exposes no sync-sample flag at all, so this is the only source available. +/// +/// Returns `false` for codecs whose NAL framing this module does not model, +/// and for a malformed or empty access unit. +#[must_use] +pub fn is_random_access_point( + codec: VideoCodecId, + config: &CodecConfig, + access_unit: &[u8], +) -> bool { + let Some(length_size) = NalLengthSize::from_config(codec, config) else { + return false; + }; + + split_length_prefixed(access_unit, length_size).any(|nal| match codec { + // H.264 (ITU-T H.264 Table 7-1): the type is the low 5 bits of the + // one-byte header, and type 5 is an IDR slice. + VideoCodecId::H264 => nal.first().is_some_and(|h| h & 0x1f == 5), + // HEVC (ITU-T H.265 Table 7-1): the type is bits 1..6 of the + // two-byte header. Types 16..=23 are the IRAP range — BLA, IDR and + // CRA — every one of which is a random access point. + VideoCodecId::Hevc => nal + .first() + .is_some_and(|h| matches!((h >> 1) & 0x3f, 16..=23)), + _ => false, + }) +} + +#[cfg(test)] +mod tests { + use super::*; + + /// Frame `nals` with 4-byte length prefixes, as MP4 stores them. + fn framed(nals: &[&[u8]]) -> Vec { + let mut out = Vec::new(); + for nal in nals { + out.extend_from_slice(&(nal.len() as u32).to_be_bytes()); + out.extend_from_slice(nal); + } + out + } + + fn four() -> NalLengthSize { + NalLengthSize::new(4).expect("4 is legal") + } + + #[test] + fn only_legal_prefix_widths_are_accepted() { + assert!(NalLengthSize::new(1).is_some()); + assert!(NalLengthSize::new(2).is_some()); + assert!(NalLengthSize::new(4).is_some()); + // 3 is representable in the two-bit field but is not legal. + assert!(NalLengthSize::new(3).is_none()); + assert!(NalLengthSize::new(0).is_none()); + assert!(NalLengthSize::new(8).is_none()); + } + + #[test] + fn avcc_length_size_comes_from_byte_four() { + // lengthSizeMinusOne = 3 -> 4-byte prefixes. 0xff sets the reserved + // high bits too, as real records do. + let cfg = CodecConfig::new(vec![0x01, 0x64, 0x00, 0x28, 0xff]); + assert_eq!( + NalLengthSize::from_config(VideoCodecId::H264, &cfg).map(NalLengthSize::get), + Some(4) + ); + + // lengthSizeMinusOne = 1 -> 2-byte prefixes. + let cfg = CodecConfig::new(vec![0x01, 0x64, 0x00, 0x28, 0xfd]); + assert_eq!( + NalLengthSize::from_config(VideoCodecId::H264, &cfg).map(NalLengthSize::get), + Some(2) + ); + } + + #[test] + fn hvcc_length_size_comes_from_byte_twenty_one() { + let mut bytes = vec![0u8; 22]; + bytes[21] = 0xff; // lengthSizeMinusOne = 3 + let cfg = CodecConfig::new(bytes); + assert_eq!( + NalLengthSize::from_config(VideoCodecId::Hevc, &cfg).map(NalLengthSize::get), + Some(4) + ); + } + + #[test] + fn an_illegal_three_byte_prefix_is_rejected_not_rounded() { + // lengthSizeMinusOne = 2 means a 3-byte prefix, which is illegal. + let cfg = CodecConfig::new(vec![0x01, 0x64, 0x00, 0x28, 0xfe]); + assert_eq!(NalLengthSize::from_config(VideoCodecId::H264, &cfg), None); + } + + #[test] + fn a_truncated_config_record_yields_no_length_size() { + assert_eq!( + NalLengthSize::from_config(VideoCodecId::H264, &CodecConfig::new(vec![0x01, 0x64])), + None + ); + assert_eq!( + NalLengthSize::from_config(VideoCodecId::Hevc, &CodecConfig::new(vec![0u8; 10])), + None + ); + assert_eq!( + NalLengthSize::from_config(VideoCodecId::H264, &CodecConfig::default()), + None + ); + } + + #[test] + fn codecs_without_nal_framing_have_no_length_size() { + let cfg = CodecConfig::new(vec![0xff; 32]); + assert_eq!(NalLengthSize::from_config(VideoCodecId::Av1, &cfg), None); + assert_eq!(NalLengthSize::from_config(VideoCodecId::ProRes, &cfg), None); + } + + #[test] + fn splitting_yields_each_nal_without_its_prefix() { + let au = framed(&[&[0x65, 0xaa], &[0x41, 0xbb, 0xcc]]); + let nals: Vec<_> = split_length_prefixed(&au, four()).collect(); + assert_eq!(nals, vec![&[0x65, 0xaa][..], &[0x41, 0xbb, 0xcc][..]]); + } + + #[test] + fn splitting_handles_every_legal_prefix_width() { + for width in [1u8, 2, 4] { + let size = NalLengthSize::new(width).expect("legal"); + let mut au = Vec::new(); + let payload = [0x65, 0x01, 0x02]; + au.extend_from_slice(&(payload.len() as u32).to_be_bytes()[4 - size.get()..]); + au.extend_from_slice(&payload); + + let nals: Vec<_> = split_length_prefixed(&au, size).collect(); + assert_eq!(nals, vec![&payload[..]], "width {width}"); + } + } + + #[test] + fn a_truncated_trailing_nal_is_dropped_not_yielded_short() { + // A well-formed NAL, then a prefix claiming 100 bytes with 2 present. + let mut au = framed(&[&[0x65, 0xaa]]); + au.extend_from_slice(&100u32.to_be_bytes()); + au.extend_from_slice(&[0xde, 0xad]); + + let nals: Vec<_> = split_length_prefixed(&au, four()).collect(); + assert_eq!(nals, vec![&[0x65, 0xaa][..]], "the good NAL must survive"); + } + + #[test] + fn a_truncated_prefix_terminates_cleanly() { + let au = vec![0x00, 0x00]; // two bytes of a four-byte prefix + assert_eq!(split_length_prefixed(&au, four()).count(), 0); + } + + #[test] + fn empty_input_yields_nothing() { + assert_eq!(split_length_prefixed(&[], four()).count(), 0); + } + + #[test] + fn a_zero_length_nal_does_not_stall_the_iterator() { + // A zero-length NAL is degenerate but must not loop forever. + let au = framed(&[&[], &[0x65]]); + let nals: Vec<_> = split_length_prefixed(&au, four()).collect(); + assert_eq!(nals.len(), 2); + assert!(nals[0].is_empty()); + } + + fn avcc() -> CodecConfig { + CodecConfig::new(vec![0x01, 0x64, 0x00, 0x28, 0xff]) + } + + fn hvcc() -> CodecConfig { + let mut bytes = vec![0u8; 22]; + bytes[21] = 0xff; + CodecConfig::new(bytes) + } + + #[test] + fn h264_idr_slices_are_random_access_points() { + // NAL type 5 = IDR. + let idr = framed(&[&[0x65, 0x88]]); + assert!(is_random_access_point(VideoCodecId::H264, &avcc(), &idr)); + + // NAL type 1 = a non-IDR slice. + let inter = framed(&[&[0x41, 0x9a]]); + assert!(!is_random_access_point(VideoCodecId::H264, &avcc(), &inter)); + } + + #[test] + fn h264_parameter_sets_before_the_idr_do_not_hide_it() { + // Real keyframe access units lead with SPS (7) and PPS (8). + let au = framed(&[&[0x67, 0x64], &[0x68, 0xee], &[0x65, 0x88]]); + assert!(is_random_access_point(VideoCodecId::H264, &avcc(), &au)); + } + + #[test] + fn hevc_irap_types_are_random_access_points() { + // Types 16..=23 are the IRAP range; check the edges and the middle. + for nal_type in [16u8, 19, 20, 21, 23] { + let au = framed(&[&[nal_type << 1, 0x01]]); + assert!( + is_random_access_point(VideoCodecId::Hevc, &hvcc(), &au), + "type {nal_type} should be IRAP" + ); + } + } + + #[test] + fn hevc_trailing_pictures_are_not_random_access_points() { + // Types 0..=15 are the non-IRAP leading/trailing range; 24+ is + // reserved or non-VCL. + for nal_type in [0u8, 1, 15, 24, 32] { + let au = framed(&[&[nal_type << 1, 0x01]]); + assert!( + !is_random_access_point(VideoCodecId::Hevc, &hvcc(), &au), + "type {nal_type} should not be IRAP" + ); + } + } + + #[test] + fn a_codec_without_framing_is_never_a_random_access_point() { + let au = framed(&[&[0x65, 0x88]]); + assert!(!is_random_access_point(VideoCodecId::ProRes, &avcc(), &au)); + } + + #[test] + fn a_malformed_access_unit_is_not_a_random_access_point() { + assert!(!is_random_access_point(VideoCodecId::H264, &avcc(), &[])); + assert!(!is_random_access_point( + VideoCodecId::H264, + &avcc(), + &[0x00, 0x00] + )); + } +} diff --git a/crates/rawshift-video-core/src/packet.rs b/crates/rawshift-video-core/src/packet.rs new file mode 100644 index 0000000..1100c9d --- /dev/null +++ b/crates/rawshift-video-core/src/packet.rs @@ -0,0 +1,173 @@ +//! Demuxed packets and the seek vocabulary. + +use std::time::Duration; + +use rawshift_core::metadata::URational; + +use crate::id::TrackId; + +/// One demuxed access unit — the coded data for a single presentation time on +/// one track. +/// +/// For H.264 and HEVC in MP4 or Matroska this is a **length-prefixed** NAL +/// stream, never Annex B; the prefix width comes from the track's +/// [`CodecConfig`](crate::track::CodecConfig). +#[derive(Debug, Clone, PartialEq, Eq)] +#[non_exhaustive] +pub struct Packet { + /// The track this packet belongs to. + pub track: TrackId, + /// Presentation timestamp, in the track's + /// [`time_base`](crate::track::VideoTrack::time_base) units. + pub pts: Option, + /// Decode timestamp, in the same units. + /// + /// Differs from `pts` exactly when the stream reorders. rawshift never + /// orders output by `dts` — the bitstream's picture order is + /// authoritative — so this is carried for diagnostics and for callers + /// remuxing rather than decoding. + pub dts: Option, + /// How long this packet occupies, in the same units, when stated. + pub duration: Option, + /// Whether the container marks this as a sync sample (a random access + /// point). + /// + /// Advisory. Containers get this wrong, and the bitstream is + /// authoritative; a decoder uses it to find a starting point, not to + /// decide how to decode. + pub is_keyframe: bool, + /// The coded bytes. + pub data: Vec, +} + +impl Packet { + /// A packet carrying `data` for `track`, with no timing and not marked as + /// a random access point. + /// + /// [`Packet`] is `#[non_exhaustive]`, so demuxers in other crates build + /// one through this and then set the fields the container supplied. + #[must_use] + pub const fn new(track: TrackId, data: Vec) -> Self { + Self { + track, + pts: None, + dts: None, + duration: None, + is_keyframe: false, + data, + } + } + + /// Convert a timestamp in `time_base` units to a [`Duration`]. + /// + /// Returns `None` for an absent or negative timestamp: a `Duration` cannot + /// be negative, and negative presentation times are real (an edit list can + /// place a track's first samples before the presentation origin). + #[must_use] + pub fn timestamp_to_duration(timestamp: i64, time_base: URational) -> Option { + if timestamp < 0 || time_base.denominator == 0 { + return None; + } + let seconds = + timestamp as f64 * f64::from(time_base.numerator) / f64::from(time_base.denominator); + Duration::try_from_secs_f64(seconds).ok() + } + + /// This packet's presentation time, when it has one. + #[must_use] + pub fn presentation_time(&self, time_base: URational) -> Option { + self.pts + .and_then(|pts| Self::timestamp_to_duration(pts, time_base)) + } +} + +/// Where to seek to. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[non_exhaustive] +pub enum SeekTo { + /// A timestamp in the target track's `time_base` units. + Timestamp(i64), + /// A position from the start of the track. + Time(Duration), +} + +/// How precisely to seek. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] +#[non_exhaustive] +pub enum SeekMode { + /// Land at or before the request, on a random access point. + /// + /// The default, and the only mode that lets decoding resume: a decoder + /// cannot start mid-GOP. Reaching the exact frame means decoding forward + /// and discarding, which is the caller's choice to make. + #[default] + Precise, + /// Land near the request, wherever the container's index is coarsest. + /// + /// May land *after* the requested position. For scrubbing, where latency + /// matters more than landing before a specific frame. + Coarse, +} + +#[cfg(test)] +mod tests { + use super::*; + + const MS: URational = URational { + numerator: 1, + denominator: 1000, + }; + + #[test] + fn timestamps_convert_against_the_time_base() { + // 1500 ticks of 1/1000 s is 1.5 s. + assert_eq!( + Packet::timestamp_to_duration(1500, MS), + Some(Duration::from_millis(1500)) + ); + assert_eq!(Packet::timestamp_to_duration(0, MS), Some(Duration::ZERO)); + } + + #[test] + fn negative_timestamps_have_no_duration() { + // Edit lists legitimately place samples before the presentation + // origin; a Duration cannot express that, so it is None rather than a + // panic or a wrapped huge value. + assert_eq!(Packet::timestamp_to_duration(-1, MS), None); + } + + #[test] + fn a_zero_denominator_time_base_does_not_divide_by_zero() { + let bad = URational { + numerator: 1, + denominator: 0, + }; + assert_eq!(Packet::timestamp_to_duration(1000, bad), None); + } + + #[test] + fn presentation_time_uses_pts_and_tolerates_its_absence() { + let mut p = Packet { + track: TrackId::new(1), + pts: Some(90_000), + dts: None, + duration: None, + is_keyframe: true, + data: Vec::new(), + }; + // A 90 kHz time base: 90000 ticks is one second. + let hz90 = URational { + numerator: 1, + denominator: 90_000, + }; + assert_eq!(p.presentation_time(hz90), Some(Duration::from_secs(1))); + + p.pts = None; + assert_eq!(p.presentation_time(hz90), None); + } + + #[test] + fn precise_is_the_default_seek_mode() { + assert_eq!(SeekMode::default(), SeekMode::Precise); + } +} diff --git a/crates/rawshift-video-core/src/track.rs b/crates/rawshift-video-core/src/track.rs new file mode 100644 index 0000000..0979f81 --- /dev/null +++ b/crates/rawshift-video-core/src/track.rs @@ -0,0 +1,642 @@ +//! The track model: what a file contains, before any of it is decoded. +//! +//! Every field here comes from the container's headers, so a full track list +//! is available on any target with no decoder at all — see the "Behavior +//! without a hardware decoder" section of `docs/SUPPORT.md`. + +use std::fmt; +use std::time::Duration; + +use gamut_color::{ + ChromaSubsampling, ColorRange, ColourPrimaries, MatrixCoefficients, TransferCharacteristics, +}; +use rawshift_core::metadata::URational; +use rawshift_core::{BitDepth, ColorDescription, Dimensions}; + +use crate::id::{AudioCodecId, TrackId, TrackKind, VideoCodecId}; + +/// The codec configuration record a track carries out of band. +/// +/// For ISOBMFF this is the `avcC` or `hvcC` box *body* from the sample entry; +/// for Matroska it is `CodecPrivate`, which holds the same records. It carries +/// the parameter sets (SPS/PPS, and VPS for HEVC) and the NAL length-prefix +/// width every packet of the track uses, so a decoder cannot be opened without +/// it. +/// +/// Deliberately opaque: the bytes are a codec-defined structure, and rawshift +/// exposes them rather than a parsed form so that the parsed form stays an +/// implementation detail of whichever decoder consumes it. +#[derive(Clone, PartialEq, Eq, Hash, Default)] +#[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))] +pub struct CodecConfig(Vec); + +impl CodecConfig { + /// Wrap a configuration record body. + #[must_use] + pub const fn new(bytes: Vec) -> Self { + Self(bytes) + } + + /// The record body. + #[must_use] + pub fn as_bytes(&self) -> &[u8] { + &self.0 + } + + /// Whether the track carried no configuration record. + /// + /// Legal in some containers and for some codecs, but H.264 and HEVC + /// tracks without one cannot be decoded: there are no parameter sets and + /// no length-prefix width. + #[must_use] + pub fn is_empty(&self) -> bool { + self.0.is_empty() + } +} + +impl fmt::Debug for CodecConfig { + /// Prints the length rather than the bytes — a configuration record is + /// tens to hundreds of opaque bytes and dumping it buries every other + /// field of a `{:?}`-printed track. + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "CodecConfig({} bytes)", self.0.len()) + } +} + +/// Display rotation the container asks for, in degrees clockwise. +/// +/// Phones record in the sensor's native orientation and store the correction +/// in the container rather than rotating pixels, so honouring this is not +/// optional: ignoring it shows most phone video sideways. Decoded frames are +/// **not** rotated by rawshift — this is the instruction, applied by whoever +/// presents the frames. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Default)] +#[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))] +pub enum Rotation { + /// No rotation. + #[default] + None, + /// 90° clockwise. + Clockwise90, + /// 180°. + Clockwise180, + /// 270° clockwise (90° counter-clockwise). + Clockwise270, +} + +impl Rotation { + /// Degrees clockwise: 0, 90, 180, or 270. + #[must_use] + pub const fn degrees(self) -> u16 { + match self { + Self::None => 0, + Self::Clockwise90 => 90, + Self::Clockwise180 => 180, + Self::Clockwise270 => 270, + } + } + + /// Whether applying this rotation swaps width and height. + #[must_use] + pub const fn swaps_axes(self) -> bool { + matches!(self, Self::Clockwise90 | Self::Clockwise270) + } + + /// Nearest quarter turn to `degrees` clockwise, normalising any multiple + /// of 360 and any negative value. + /// + /// Containers store rotation as a transform matrix rather than an angle, + /// so a demuxer recovers an angle that may be slightly off or expressed + /// as a negative; snapping to the nearest quarter turn is the standard + /// reading and keeps a 89.98°-from-matrix value from becoming `None`. + #[must_use] + pub fn from_degrees(degrees: f64) -> Self { + if !degrees.is_finite() { + return Self::None; + } + // rem_euclid keeps the result non-negative, so -90 becomes 270. + let normalised = degrees.rem_euclid(360.0); + match ((normalised / 90.0).round() as i64) % 4 { + 1 => Self::Clockwise90, + 2 => Self::Clockwise180, + 3 => Self::Clockwise270, + _ => Self::None, + } + } + + /// Recover the rotation from an ISOBMFF `tkhd` display matrix. + /// + /// The matrix is stored as nine fixed-point values `[a, b, u, c, d, v, x, + /// y, w]`; only `a`, `b`, `c`, `d` carry rotation. They are 16.16 fixed + /// point, so callers pass them already converted to `f64`. + #[must_use] + pub fn from_display_matrix(a: f64, b: f64, c: f64, d: f64) -> Self { + // atan2(b, a) is the angle the matrix rotates by, counter-clockwise in + // the mathematical convention; the container's convention is clockwise. + // `c` and `d` are accepted so a caller passes the whole rotation + // sub-matrix, and are used only to reject a degenerate matrix. + if a == 0.0 && b == 0.0 && c == 0.0 && d == 0.0 { + return Self::None; + } + Self::from_degrees(b.atan2(a).to_degrees()) + } +} + +impl fmt::Display for Rotation { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "{}°", self.degrees()) + } +} + +/// A video track's header-derived description. +/// # Serialization +/// +/// Not `Serialize`/`Deserialize` even under the `serde` feature. The track +/// types embed upstream geometry and colour vocabulary (`Dimensions`, +/// `BitDepth`, `ChromaSubsampling`) that gamut does not implement serde for, +/// and skipping those fields would serialize a track that silently lost its +/// resolution and pixel format — worse than not serializing at all. The leaf +/// types here ([`CicpColor`], [`Rotation`], [`CodecConfig`], and everything in +/// [`crate::id`]) do support serde, so a caller can build and serialize its +/// own projection. +#[derive(Debug, Clone, PartialEq)] +#[non_exhaustive] +pub struct VideoTrack { + /// Container-assigned identifier. + pub id: TrackId, + /// The codec the samples are coded with. + pub codec: VideoCodecId, + /// Coded picture size, before [`rotation`](Self::rotation) is applied. + pub dimensions: Dimensions, + /// The track's colour signalling. + pub color: CicpColor, + /// Bits per sample. + pub bit_depth: BitDepth, + /// Chroma subsampling. + pub chroma: ChromaSubsampling, + /// Average frame rate, when the container states or implies one. + /// + /// Absent for variable-frame-rate recordings that state no average. Not a + /// substitute for per-packet timestamps, which remain authoritative. + pub frame_rate: Option, + /// The unit of this track's timestamps, in seconds. + /// + /// Every `pts`/`dts` on a [`Packet`](crate::packet::Packet) of this track + /// is a count of these. + pub time_base: URational, + /// Track duration, when the container states one. + pub duration: Option, + /// Number of coded pictures, when the container states one. + pub frame_count: Option, + /// Display rotation the container asks for. + pub rotation: Rotation, + /// The out-of-band codec configuration record. + pub codec_config: CodecConfig, +} + +impl VideoTrack { + /// A video track with `codec`, `dimensions` and `time_base` set and every + /// other field at its neutral default. + /// + /// These types are `#[non_exhaustive]`, so demuxers in other crates build + /// one through this and then assign the fields their container supplied. + /// A new field therefore arrives with a default rather than breaking every + /// demuxer. + #[must_use] + pub fn new( + id: TrackId, + codec: VideoCodecId, + dimensions: Dimensions, + time_base: URational, + ) -> Self { + Self { + id, + codec, + dimensions, + color: CicpColor::UNSPECIFIED, + bit_depth: BitDepth::Eight, + chroma: ChromaSubsampling::Cs420, + frame_rate: None, + time_base, + duration: None, + frame_count: None, + rotation: Rotation::None, + codec_config: CodecConfig::default(), + } + } +} + +/// A track's colour signalling, as the container states it. +/// +/// Stored as raw CICP (ITU-T H.273) code points rather than as typed enums, +/// for two reasons: it is exactly what the container carries, and a code point +/// no library names yet survives round-tripping instead of collapsing to +/// "unspecified". The accessors below type them where a type exists. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Default)] +#[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))] +pub struct CicpColor { + /// CICP colour primaries code point. 2 means unspecified. + pub primaries: u16, + /// CICP transfer characteristics code point. 2 means unspecified. + pub transfer: u16, + /// CICP matrix coefficients code point. 2 means unspecified. + /// + /// Its own axis, separate from the primaries: this is what maps YCbCr to + /// RGB, and getting it wrong tints the entire picture. + pub matrix: u16, + /// Whether samples occupy the full code range or the studio subset. + pub full_range: bool, +} + +impl CicpColor { + /// The code points meaning "unspecified" on every axis, studio range — + /// what an untagged camera file effectively asserts. + pub const UNSPECIFIED: Self = Self { + primaries: 2, + transfer: 2, + matrix: 2, + full_range: false, + }; + + /// The typed colour primaries, when the code point is one gamut names. + #[must_use] + pub fn primaries(self) -> Option { + ColourPrimaries::from_code_point(self.primaries) + } + + /// The typed transfer characteristics, when the code point is one gamut + /// names. + #[must_use] + pub fn transfer(self) -> Option { + TransferCharacteristics::from_code_point(self.transfer) + } + + /// The typed matrix coefficients, when the code point is one gamut names. + #[must_use] + pub fn matrix(self) -> Option { + MatrixCoefficients::from_code_point(self.matrix) + } + + /// Whether samples occupy the full code range or the studio subset. + #[must_use] + pub fn range(self) -> ColorRange { + if self.full_range { + ColorRange::Full + } else { + ColorRange::Limited + } + } + + /// The primaries-and-transfer pair as rawshift's [`ColorDescription`], + /// falling back to its `UNSPECIFIED` value for code points gamut does not + /// name. + #[must_use] + pub fn description(self) -> ColorDescription { + match (self.primaries(), self.transfer()) { + (Some(primaries), Some(transfer)) => ColorDescription { + primaries, + transfer, + }, + _ => ColorDescription::UNSPECIFIED, + } + } + + /// The matrix to actually decode with, resolving "unspecified" against the + /// coded picture height. + /// + /// Containers frequently omit the matrix, and decoders in the field all + /// fall back the same way: standard definition means BT.601, anything + /// larger means BT.709. Defaulting to BT.2020 is never right — files that + /// use it say so, because HDR is unusable if mistagged. + #[must_use] + pub fn resolved_matrix(self, coded_height: u32) -> MatrixCoefficients { + match self.matrix() { + Some(MatrixCoefficients::Unspecified) | None => { + if coded_height <= 576 { + MatrixCoefficients::Bt601 + } else { + MatrixCoefficients::Bt709 + } + } + Some(explicit) => explicit, + } + } +} + +/// An audio track's header-derived description. +/// +/// rawshift enumerates audio and yields its packets; it never decodes audio. +/// # Serialization +/// +/// Not `Serialize`/`Deserialize` even under the `serde` feature. The track +/// types embed upstream geometry and colour vocabulary (`Dimensions`, +/// `BitDepth`, `ChromaSubsampling`) that gamut does not implement serde for, +/// and skipping those fields would serialize a track that silently lost its +/// resolution and pixel format — worse than not serializing at all. The leaf +/// types here ([`CicpColor`], [`Rotation`], [`CodecConfig`], and everything in +/// [`crate::id`]) do support serde, so a caller can build and serialize its +/// own projection. +#[derive(Debug, Clone, PartialEq)] +#[non_exhaustive] +pub struct AudioTrack { + /// Container-assigned identifier. + pub id: TrackId, + /// The codec the samples are coded with. + pub codec: AudioCodecId, + /// Sampling rate in Hz, when stated. + pub sample_rate: Option, + /// Channel count, when stated. + pub channels: Option, + /// The unit of this track's timestamps, in seconds. + pub time_base: URational, + /// Track duration, when the container states one. + pub duration: Option, + /// The out-of-band codec configuration record, when the track has one. + pub codec_config: CodecConfig, + /// BCP 47 / ISO 639 language tag, when the container states one. + pub language: Option, +} + +impl AudioTrack { + /// An audio track with `codec` and `time_base` set and every other field + /// at its neutral default. See [`VideoTrack::new`]. + #[must_use] + pub fn new(id: TrackId, codec: AudioCodecId, time_base: URational) -> Self { + Self { + id, + codec, + sample_rate: None, + channels: None, + time_base, + duration: None, + codec_config: CodecConfig::default(), + language: None, + } + } +} + +/// A track rawshift enumerates but models no further. +/// # Serialization +/// +/// Not `Serialize`/`Deserialize` even under the `serde` feature. The track +/// types embed upstream geometry and colour vocabulary (`Dimensions`, +/// `BitDepth`, `ChromaSubsampling`) that gamut does not implement serde for, +/// and skipping those fields would serialize a track that silently lost its +/// resolution and pixel format — worse than not serializing at all. The leaf +/// types here ([`CicpColor`], [`Rotation`], [`CodecConfig`], and everything in +/// [`crate::id`]) do support serde, so a caller can build and serialize its +/// own projection. +#[derive(Debug, Clone, PartialEq)] +#[non_exhaustive] +pub struct OtherTrack { + /// Container-assigned identifier. + pub id: TrackId, + /// What the track carries. + pub kind: TrackKind, + /// The unit of this track's timestamps, in seconds. + pub time_base: URational, + /// Track duration, when the container states one. + pub duration: Option, + /// BCP 47 / ISO 639 language tag, when the container states one. + pub language: Option, +} + +impl OtherTrack { + /// A track of `kind` with `time_base` set and every other field at its + /// neutral default. See [`VideoTrack::new`]. + #[must_use] + pub fn new(id: TrackId, kind: TrackKind, time_base: URational) -> Self { + Self { + id, + kind, + time_base, + duration: None, + language: None, + } + } +} + +/// One track of any kind. +/// # Serialization +/// +/// Not `Serialize`/`Deserialize` even under the `serde` feature. The track +/// types embed upstream geometry and colour vocabulary (`Dimensions`, +/// `BitDepth`, `ChromaSubsampling`) that gamut does not implement serde for, +/// and skipping those fields would serialize a track that silently lost its +/// resolution and pixel format — worse than not serializing at all. The leaf +/// types here ([`CicpColor`], [`Rotation`], [`CodecConfig`], and everything in +/// [`crate::id`]) do support serde, so a caller can build and serialize its +/// own projection. +#[derive(Debug, Clone, PartialEq)] +#[non_exhaustive] +pub enum Track { + /// A video track. + Video(VideoTrack), + /// An audio track. + Audio(AudioTrack), + /// A subtitle, timecode, or data track. + Other(OtherTrack), +} + +impl Track { + /// The track's container-assigned identifier. + #[must_use] + pub fn id(&self) -> TrackId { + match self { + Self::Video(t) => t.id, + Self::Audio(t) => t.id, + Self::Other(t) => t.id, + } + } + + /// What the track carries. + #[must_use] + pub fn kind(&self) -> TrackKind { + match self { + Self::Video(_) => TrackKind::Video, + Self::Audio(_) => TrackKind::Audio, + Self::Other(t) => t.kind, + } + } + + /// The video track, if this is one. + #[must_use] + pub fn as_video(&self) -> Option<&VideoTrack> { + match self { + Self::Video(t) => Some(t), + _ => None, + } + } + + /// The audio track, if this is one. + #[must_use] + pub fn as_audio(&self) -> Option<&AudioTrack> { + match self { + Self::Audio(t) => Some(t), + _ => None, + } + } + + /// The unit of this track's timestamps, in seconds. + #[must_use] + pub fn time_base(&self) -> URational { + match self { + Self::Video(t) => t.time_base, + Self::Audio(t) => t.time_base, + Self::Other(t) => t.time_base, + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn rotation_degrees_round_trip() { + for r in [ + Rotation::None, + Rotation::Clockwise90, + Rotation::Clockwise180, + Rotation::Clockwise270, + ] { + assert_eq!(Rotation::from_degrees(f64::from(r.degrees())), r); + } + } + + #[test] + fn rotation_normalises_negative_and_wrapped_angles() { + assert_eq!(Rotation::from_degrees(-90.0), Rotation::Clockwise270); + assert_eq!(Rotation::from_degrees(450.0), Rotation::Clockwise90); + assert_eq!(Rotation::from_degrees(720.0), Rotation::None); + assert_eq!(Rotation::from_degrees(-360.0), Rotation::None); + } + + #[test] + fn rotation_snaps_near_misses_from_fixed_point_matrices() { + // A 16.16 matrix cannot represent 90° exactly; the recovered angle is + // a hair off and must not collapse to None. + assert_eq!(Rotation::from_degrees(89.98), Rotation::Clockwise90); + assert_eq!(Rotation::from_degrees(270.03), Rotation::Clockwise270); + } + + #[test] + fn rotation_rejects_non_finite_angles() { + assert_eq!(Rotation::from_degrees(f64::NAN), Rotation::None); + assert_eq!(Rotation::from_degrees(f64::INFINITY), Rotation::None); + } + + #[test] + fn display_matrix_recovers_the_quarter_turns_phones_write() { + // Identity: no rotation. + assert_eq!( + Rotation::from_display_matrix(1.0, 0.0, 0.0, 1.0), + Rotation::None + ); + // The matrix iOS writes for a portrait capture: 90° clockwise. + assert_eq!( + Rotation::from_display_matrix(0.0, 1.0, -1.0, 0.0), + Rotation::Clockwise90 + ); + // Upside-down landscape. + assert_eq!( + Rotation::from_display_matrix(-1.0, 0.0, 0.0, -1.0), + Rotation::Clockwise180 + ); + // 270°. + assert_eq!( + Rotation::from_display_matrix(0.0, -1.0, 1.0, 0.0), + Rotation::Clockwise270 + ); + } + + #[test] + fn degenerate_display_matrix_is_not_a_rotation() { + assert_eq!( + Rotation::from_display_matrix(0.0, 0.0, 0.0, 0.0), + Rotation::None + ); + } + + #[test] + fn only_quarter_turns_swap_axes() { + assert!(Rotation::Clockwise90.swaps_axes()); + assert!(Rotation::Clockwise270.swaps_axes()); + assert!(!Rotation::None.swaps_axes()); + assert!(!Rotation::Clockwise180.swaps_axes()); + } + + #[test] + fn unspecified_matrix_resolves_by_height() { + let c = CicpColor::UNSPECIFIED; + // SD heights take BT.601 ... + assert_eq!(c.resolved_matrix(480), MatrixCoefficients::Bt601); + assert_eq!(c.resolved_matrix(576), MatrixCoefficients::Bt601); + // ... anything larger takes BT.709. + assert_eq!(c.resolved_matrix(720), MatrixCoefficients::Bt709); + assert_eq!(c.resolved_matrix(2160), MatrixCoefficients::Bt709); + } + + #[test] + fn an_unnamed_matrix_code_point_still_resolves_rather_than_panicking() { + // Code point 7 (SMPTE 240M) is real but not modelled by gamut. It must + // fall back like "unspecified" rather than being lost or fatal. + let exotic = CicpColor { + matrix: 7, + ..CicpColor::UNSPECIFIED + }; + assert_eq!(exotic.matrix(), None); + assert_eq!(exotic.resolved_matrix(1080), MatrixCoefficients::Bt709); + // ... and the raw code point survives for a caller that does know it. + assert_eq!(exotic.matrix, 7); + } + + #[test] + fn resolving_never_overrides_an_explicit_matrix() { + // A 4K BT.601 file stays BT.601, and an SD BT.2020 file stays BT.2020. + let bt601 = CicpColor { + matrix: MatrixCoefficients::Bt601.code_point(), + ..CicpColor::UNSPECIFIED + }; + assert_eq!(bt601.resolved_matrix(2160), MatrixCoefficients::Bt601); + + let bt2020 = CicpColor { + matrix: MatrixCoefficients::Bt2020Ncl.code_point(), + ..CicpColor::UNSPECIFIED + }; + assert_eq!(bt2020.resolved_matrix(480), MatrixCoefficients::Bt2020Ncl); + } + + #[test] + fn range_defaults_to_studio_which_is_what_cameras_write() { + assert_eq!(CicpColor::UNSPECIFIED.range(), ColorRange::Limited); + assert_eq!(CicpColor::default().range(), ColorRange::Limited); + let full = CicpColor { + full_range: true, + ..CicpColor::UNSPECIFIED + }; + assert_eq!(full.range(), ColorRange::Full); + } + + #[test] + fn typed_accessors_round_trip_named_code_points() { + let rec709 = CicpColor { + primaries: ColourPrimaries::Bt709.code_point(), + transfer: TransferCharacteristics::Bt709.code_point(), + matrix: MatrixCoefficients::Bt709.code_point(), + full_range: false, + }; + assert_eq!(rec709.primaries(), Some(ColourPrimaries::Bt709)); + assert_eq!(rec709.matrix(), Some(MatrixCoefficients::Bt709)); + assert_eq!(rec709.description().primaries, ColourPrimaries::Bt709); + } + + #[test] + fn codec_config_debug_reports_length_not_bytes() { + let cfg = CodecConfig::new(vec![0xde, 0xad, 0xbe, 0xef]); + assert_eq!(format!("{cfg:?}"), "CodecConfig(4 bytes)"); + assert_eq!(cfg.as_bytes(), &[0xde, 0xad, 0xbe, 0xef]); + assert!(!cfg.is_empty()); + assert!(CodecConfig::default().is_empty()); + } +} diff --git a/crates/rawshift-video-hwdec/Cargo.toml b/crates/rawshift-video-hwdec/Cargo.toml new file mode 100644 index 0000000..812aac8 --- /dev/null +++ b/crates/rawshift-video-hwdec/Cargo.toml @@ -0,0 +1,32 @@ +[package] +name = "rawshift-video-hwdec" +version.workspace = true +edition.workspace = true +rust-version.workspace = true +license.workspace = true +repository.workspace = true +homepage.workspace = true +description = "Hardware-backed H.264 and HEVC video decoders for rawshift" +documentation = "https://docs.rs/rawshift-video-hwdec" +keywords = ["video", "h264", "hevc", "decode", "hardware"] +categories = ["multimedia::video"] +readme = "README.md" + +[dependencies] +rawshift-core = { workspace = true } +rawshift-video-core = { workspace = true } +rawshift-hwdec = { workspace = true } + +[features] +default = [] +# One per codec, so a build takes only the decoders it needs. +h264 = [] +hevc = [] + +# Hardware backend selection, forwarded to rawshift-hwdec. These are verified +# flags: `hw` is portable and picks the native backend for the target, while +# the pinned ones fail to compile off-target on purpose. +hw = ["rawshift-hwdec/hw"] +hw-vaapi = ["rawshift-hwdec/vaapi"] +hw-videotoolbox = ["rawshift-hwdec/videotoolbox"] +hw-mediacodec = ["rawshift-hwdec/mediacodec"] diff --git a/crates/rawshift-video-hwdec/README.md b/crates/rawshift-video-hwdec/README.md new file mode 100644 index 0000000..330fa73 --- /dev/null +++ b/crates/rawshift-video-hwdec/README.md @@ -0,0 +1,27 @@ +# rawshift-video-hwdec + +H.264 and HEVC video decoders for +[rawshift](https://github.com/visualcommons/rawshift), backed by the platform +hardware decoders in `rawshift-hwdec`. + +## Why one crate for two codecs + +The image side has a leaf crate per format because each wraps a different +codec library. Here both codecs go through the same seam and the same backend, +and the two implementations differ only in which configuration record they +carry — two crates would be near-identical copies. The `h264` and `hevc` +features give a build the same narrowing a split would, without the +duplication. + +## Software decode + +There is none, deliberately. `docs/SUPPORT.md` rules out shipping software +H.264 or HEVC: both are patent-encumbered independently of any +implementation's own license. Hardware decoders carry the device OEM's +licence, so that is the only path rawshift offers. Where no backend exists the +decoders report `VideoError::HwDecoderUnavailable`, and container parsing, +track enumeration and metadata keep working regardless. + +## License + +Licensed under [MPL-2.0](../../LICENSE). diff --git a/crates/rawshift-video-hwdec/src/lib.rs b/crates/rawshift-video-hwdec/src/lib.rs new file mode 100644 index 0000000..0bb887d --- /dev/null +++ b/crates/rawshift-video-hwdec/src/lib.rs @@ -0,0 +1,298 @@ +//! H.264 and HEVC video decoders backed by platform hardware. +//! +//! Binds `rawshift-video-core`'s [`VideoDecoder`] contract to +//! `rawshift-hwdec`'s sequence seam, translating between rawshift's video +//! vocabulary and the hardware layer's. No `rawshift_hwdec` type appears in +//! this crate's public API. +//! +//! # No software fallback, deliberately +//! +//! `docs/SUPPORT.md` rules out shipping software H.264 or HEVC: both are +//! patent-encumbered independently of any implementation's own license, and +//! the OpenH264 royalty grant covers Cisco's prebuilt binary rather than a +//! from-source or clean-slate build. Hardware decoders carry the device OEM's +//! licence, so that is the only path offered. +//! +//! Where no backend exists — `wasm32`, `musl`, Windows, or a Linux host with +//! no libva — [`VideoError::HwDecoderUnavailable`] is reported, and container +//! parsing, track enumeration and metadata keep working regardless. + +#![forbid(unsafe_code)] +#![warn(missing_docs)] + +use rawshift_core::CodecId; +use rawshift_hwdec as hw; +use rawshift_video_core::{ + ColorRange, FrameDesc, FramePixelFormat, MatrixCoefficients, Packet, Plane, VideoCodecId, + VideoDecoder, VideoDecoderFactory, VideoError, VideoFrame, VideoResult, VideoTrack, +}; + +/// Register every hardware-backed decoder compiled into this build. +/// +/// Push order is priority order; within a codec there is only one backend, so +/// the order between codecs is irrelevant. +pub fn register(registry: &mut rawshift_video_core::DecoderRegistry) { + #[cfg(feature = "hevc")] + registry.push(Box::new(HwDecoderFactory::new(VideoCodecId::Hevc))); + #[cfg(feature = "h264")] + registry.push(Box::new(HwDecoderFactory::new(VideoCodecId::H264))); + let _ = registry; +} + +/// Opens hardware-backed decoders for one codec. +#[derive(Debug, Clone, Copy)] +pub struct HwDecoderFactory { + codec: VideoCodecId, +} + +impl HwDecoderFactory { + /// A factory for `codec`. + #[must_use] + pub const fn new(codec: VideoCodecId) -> Self { + Self { codec } + } + + /// The hardware layer's name for this codec, or `None` if it has no + /// sequence path. + fn hw_codec(self) -> Option { + match self.codec { + VideoCodecId::H264 => Some(hw::HwCodec::H264), + VideoCodecId::Hevc => Some(hw::HwCodec::Hevc), + _ => None, + } + } +} + +/// Build the hardware configuration for a track. +/// +/// The variant is the codec on both sides, so this cannot pair a record with +/// the wrong codec. +fn hw_config<'a>(codec: VideoCodecId, record: &'a [u8]) -> Option> { + match codec { + VideoCodecId::H264 => Some(hw::VideoConfig::Avcc(record)), + VideoCodecId::Hevc => Some(hw::VideoConfig::Hvcc(record)), + _ => None, + } +} + +impl VideoDecoderFactory for HwDecoderFactory { + fn id(&self) -> CodecId { + match self.codec { + VideoCodecId::H264 => CodecId::new("h264/hwdec"), + VideoCodecId::Hevc => CodecId::new("hevc/hwdec"), + _ => CodecId::new("unknown/hwdec"), + } + } + + fn codec(&self) -> VideoCodecId { + self.codec + } + + fn supports(&self, track: &VideoTrack) -> bool { + if track.codec != self.codec { + return false; + } + // A track with no configuration record has no parameter sets and no + // NAL length-prefix width, so no session could be opened for it. + if track.codec_config.is_empty() { + return false; + } + // Answer about this machine, not just the codec: the registry treats + // `false` as "try the next backend", so claiming support without a + // usable driver would strand the track here. + self.hw_codec() + .is_some_and(|codec| hw::available_video_codecs().contains(&codec)) + } + + fn open(&self, track: &VideoTrack) -> VideoResult> { + let record = track.codec_config.as_bytes(); + let config = hw_config(self.codec, record) + .ok_or(VideoError::UnsupportedCodec { codec: self.codec })?; + + let inner = hw::video_decoder(config).map_err(|e| match e { + hw::HwDecodeError::Unavailable { reason, .. } => { + VideoError::hw_unavailable(self.codec, reason) + } + other => VideoError::decode(self.codec, other), + })?; + + Ok(Box::new(HwVideoDecoder { + codec: self.codec, + inner, + // Resolved once: the matrix a frame is decoded with cannot depend + // on which frame it is. + matrix: track.color.resolved_matrix(track.dimensions.height), + range: track.color.range(), + })) + } +} + +/// A decoder driving one hardware sequence session. +struct HwVideoDecoder { + codec: VideoCodecId, + inner: Box, + matrix: MatrixCoefficients, + range: ColorRange, +} + +impl VideoDecoder for HwVideoDecoder { + fn codec(&self) -> VideoCodecId { + self.codec + } + + fn send_packet(&mut self, packet: &Packet) -> VideoResult<()> { + self.inner + .send_packet(&hw::VideoPacket { + data: &packet.data, + pts: packet.pts, + dts: packet.dts, + is_sync: packet.is_keyframe, + }) + .map_err(|e| VideoError::decode(self.codec, e)) + } + + fn receive_frame(&mut self) -> VideoResult> { + let Some(frame) = self + .inner + .receive_frame() + .map_err(|e| VideoError::decode(self.codec, e))? + else { + return Ok(None); + }; + self.lift(frame).map(Some) + } + + fn flush(&mut self) -> VideoResult<()> { + self.inner + .flush() + .map_err(|e| VideoError::decode(self.codec, e)) + } + + fn reset(&mut self) { + self.inner.reset(); + } +} + +impl HwVideoDecoder { + /// Translate a hardware frame into rawshift's own, so no backend type + /// crosses the boundary. + fn lift(&self, frame: hw::VideoFrame) -> VideoResult { + let (pts, poc) = (frame.pts(), frame.poc()); + let decoded = frame.into_frame(); + + let format = match decoded.format() { + hw::PixelFormat::Nv12 => FramePixelFormat::Nv12, + hw::PixelFormat::P010 => FramePixelFormat::P010, + hw::PixelFormat::I420 => FramePixelFormat::I420, + hw::PixelFormat::I010 => FramePixelFormat::I010, + }; + + let dimensions = rawshift_core::Dimensions { + width: decoded.width(), + height: decoded.height(), + }; + // The decoded frame's own range wins over the container's: the + // bitstream's VUI is what the samples were actually coded against, + // and a container that disagrees is mistagged. + let range = match decoded.range() { + hw::ColorRange::Full => ColorRange::Full, + _ => self.range, + }; + + let planes = decoded + .planes() + .iter() + .map(|p| Plane { + data: p.data.clone(), + stride: p.stride, + }) + .collect(); + + let desc = FrameDesc::new(format, dimensions, decoded.bit_depth(), range, self.matrix) + .with_timing(pts, poc); + + VideoFrame::new(self.codec, desc, planes) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use rawshift_video_core::{CicpColor, CodecConfig, DecoderRegistry, TrackId}; + + fn track(codec: VideoCodecId, config: Vec) -> VideoTrack { + let mut t = VideoTrack::new( + TrackId::new(1), + codec, + rawshift_core::Dimensions { + width: 1920, + height: 1080, + }, + rawshift_core::metadata::URational::new(1, 1000), + ); + t.codec_config = CodecConfig::new(config); + t.color = CicpColor::UNSPECIFIED; + t + } + + #[test] + fn a_factory_only_claims_its_own_codec() { + let hevc = HwDecoderFactory::new(VideoCodecId::Hevc); + assert!(!hevc.supports(&track(VideoCodecId::H264, vec![0u8; 32]))); + } + + #[test] + fn a_track_without_a_configuration_record_is_never_claimed() { + // No parameter sets and no NAL length-prefix width means no session + // could be opened, so claiming it would strand the track. + let hevc = HwDecoderFactory::new(VideoCodecId::Hevc); + assert!(!hevc.supports(&track(VideoCodecId::Hevc, Vec::new()))); + } + + #[test] + fn support_tracks_what_the_machine_can_actually_do() { + // supports() must answer about this machine, since the registry reads + // `false` as "try the next backend". + let hevc = HwDecoderFactory::new(VideoCodecId::Hevc); + let claimed = hevc.supports(&track(VideoCodecId::Hevc, vec![0u8; 32])); + let available = hw::available_video_codecs().contains(&hw::HwCodec::Hevc); + assert_eq!(claimed, available); + } + + #[test] + fn codec_ids_are_stable_and_namespaced() { + assert_eq!( + HwDecoderFactory::new(VideoCodecId::H264).id().id, + "h264/hwdec" + ); + assert_eq!( + HwDecoderFactory::new(VideoCodecId::Hevc).id().id, + "hevc/hwdec" + ); + } + + #[test] + fn registration_matches_the_compiled_features() { + let mut registry = DecoderRegistry::new(); + register(&mut registry); + + let expected = usize::from(cfg!(feature = "hevc")) + usize::from(cfg!(feature = "h264")); + assert_eq!(registry.factories().len(), expected); + } + + #[test] + fn opening_without_a_backend_reports_unavailable_not_unsupported() { + // On a machine or build with no hardware decoder this must be a + // capability gap a caller can act on, never a claim the codec is + // unknown. + let hevc = HwDecoderFactory::new(VideoCodecId::Hevc); + if hw::available_video_codecs().contains(&hw::HwCodec::Hevc) { + eprintln!("skipping: this machine has an HEVC decoder"); + return; + } + match hevc.open(&track(VideoCodecId::Hevc, vec![0u8; 32])) { + Ok(_) => panic!("no backend, yet a decoder opened"), + Err(e) => assert!(e.is_capability_gap(), "{e}"), + } + } +} diff --git a/crates/rawshift-video-isobmff/Cargo.toml b/crates/rawshift-video-isobmff/Cargo.toml new file mode 100644 index 0000000..dcbfef2 --- /dev/null +++ b/crates/rawshift-video-isobmff/Cargo.toml @@ -0,0 +1,22 @@ +[package] +name = "rawshift-video-isobmff" +version.workspace = true +edition.workspace = true +rust-version.workspace = true +license.workspace = true +repository.workspace = true +homepage.workspace = true +description = "MP4, M4V and QuickTime (MOV) demuxing and metadata for rawshift" +documentation = "https://docs.rs/rawshift-video-isobmff" +keywords = ["video", "mp4", "mov", "quicktime", "isobmff"] +categories = ["multimedia::video"] +readme = "README.md" + +[dependencies] +rawshift-core = { workspace = true } +rawshift-video-core = { workspace = true } +mp4-atom = { workspace = true } +tracing = { workspace = true } + +[features] +default = [] diff --git a/crates/rawshift-video-isobmff/README.md b/crates/rawshift-video-isobmff/README.md new file mode 100644 index 0000000..511d663 --- /dev/null +++ b/crates/rawshift-video-isobmff/README.md @@ -0,0 +1,44 @@ +# rawshift-video-isobmff + +MP4, M4V and QuickTime (MOV) support for +[rawshift](https://github.com/visualcommons/rawshift). + +The box layer is [`mp4-atom`](https://crates.io/crates/mp4-atom) (MIT, pure +Rust, no build script). It performs no interpretation, so this crate owns the +parts that give a container meaning: the sample index reconstructed from the +`stbl` tables, the track model, and the demux and seek semantics. + +## Why not an off-the-shelf demuxer + +This crate first used `symphonia-format-isomp4`, which is audio-first and caps +the `hvcC` configuration record at 1 KB. Real HEVC records are 2–3 KB, so +**every HEVC MP4 and MOV failed to open** — that is XAVC HS and default iPhone +video, the two highest-priority formats on rawshift's roadmap. It also hides +the `tkhd` matrix (and so rotation), the `colr` box (and so all colour +signalling), the sync-sample table, and ProRes-shaped sample entries. + +Owning the sample index on a box-level library costs a few hundred lines and +removes all five limitations at once. Matroska keeps its symphonia backend, +which has none of these problems. + +## What it reads + +`ftyp` brands, the `mvhd` timescale and times, per-track `tkhd` rotation, +`mdhd` timescale and language, the `stsd` sample entry (including `colr` +colour signalling and the `avcC`/`hvcC` record a decoder needs), the full +`stbl` sample tables, and the `elst` presentation shift — without which every +timestamp is offset by the encoder's reorder delay and disagrees with every +other tool. + +Samples are read on demand, so opening a file costs only its headers however +large the media is. + +## Not covered + +Fragmented MP4 (`mvex`/`moof`) is rejected with a clear error rather than +presented as a track with no samples. Camera-origin files — rawshift's subject +— are not fragmented. + +## License + +Licensed under [MPL-2.0](../../LICENSE). diff --git a/crates/rawshift-video-isobmff/examples/probe.rs b/crates/rawshift-video-isobmff/examples/probe.rs new file mode 100644 index 0000000..7145ad1 --- /dev/null +++ b/crates/rawshift-video-isobmff/examples/probe.rs @@ -0,0 +1,54 @@ +//! Print what the ISOBMFF demuxer sees in a file. +//! +//! `cargo run -p rawshift-video-isobmff --example probe -- FILE` + +use std::fs::File; + +use rawshift_video_core::{Demuxer, Track}; +use rawshift_video_isobmff::IsoBmffDemuxer; + +fn main() -> Result<(), Box> { + let path = std::env::args().nth(1).ok_or("usage: probe FILE")?; + let mut demuxer = IsoBmffDemuxer::open(File::open(&path)?)?; + + println!("container: {}", demuxer.container()); + let md = demuxer.metadata(); + println!("brands: {:?}", md.container.brands); + println!("timescale: {:?}", md.container.timescale); + println!("duration: {:?}", md.container.duration); + println!("created: {:?}", md.container.creation_time); + + for track in demuxer.tracks() { + match track { + Track::Video(v) => println!( + "video #{} {} {}x{} rot={} tb={}/{} frames={:?} dur={:?} cfg={}B", + v.id, + v.codec, + v.dimensions.width, + v.dimensions.height, + v.rotation, + v.time_base.numerator, + v.time_base.denominator, + v.frame_count, + v.duration, + v.codec_config.as_bytes().len(), + ), + Track::Audio(a) => println!("audio #{} {}", a.id, a.codec), + Track::Other(o) => println!("other #{} {:?}", o.id, o.kind), + other => println!("other #{}", other.id()), + } + } + + let (mut packets, mut keyframes, mut first_pts) = (0usize, 0usize, None); + while let Some(p) = demuxer.next_packet()? { + if first_pts.is_none() { + first_pts = p.pts; + } + packets += 1; + if p.is_keyframe { + keyframes += 1; + } + } + println!("packets: {packets} ({keyframes} random access points), first pts {first_pts:?}"); + Ok(()) +} diff --git a/crates/rawshift-video-isobmff/src/demux.rs b/crates/rawshift-video-isobmff/src/demux.rs new file mode 100644 index 0000000..6ff3c86 --- /dev/null +++ b/crates/rawshift-video-isobmff/src/demux.rs @@ -0,0 +1,172 @@ +//! The MP4 / QuickTime demuxer. + +use std::io::{Read, Seek, SeekFrom}; + +use rawshift_video_core::{ + ContainerId, Demuxer, Packet, SeekMode, SeekTo, Track, TrackId, VideoError, VideoMetadata, + VideoResult, nal, +}; + +use crate::index::SampleIndex; +use crate::movie::{CONTAINER_NAME, Movie}; + +/// A demuxer for MP4, M4V and QuickTime files. +/// +/// Samples are read on demand from the source rather than held in memory, so +/// opening a file costs only its headers however large the media is. +pub struct IsoBmffDemuxer { + source: R, + tracks: Vec, + indices: Vec, + /// Read cursor into each track's sample index, parallel to `tracks`. + cursors: Vec, + container: ContainerId, + metadata: VideoMetadata, +} + +impl IsoBmffDemuxer { + /// Open an ISOBMFF file. + /// + /// # Errors + /// + /// [`VideoError::Container`] for a file with no `moov`, a malformed atom, + /// or a fragmented file, and [`VideoError::Io`] for a reader failure. + pub fn open(mut source: R) -> VideoResult { + let movie = Movie::read(&mut source)?; + let cursors = vec![0; movie.tracks.len()]; + + Ok(Self { + source, + tracks: movie.tracks, + indices: movie.indices, + cursors, + container: movie.container, + metadata: movie.metadata, + }) + } + + /// The position of a track in the parallel `tracks`/`indices`/`cursors` + /// vectors. + /// + /// Container-assigned track ids are 1-based and need not be contiguous, so + /// this is a search rather than an index. + fn slot_of(&self, id: TrackId) -> Option { + self.tracks.iter().position(|t| t.id() == id) + } + + /// The track whose next unread sample decodes earliest. + /// + /// Packets are delivered in the file's own interleaving so that a caller + /// reading every track streams the file in one forward pass. Ordering by + /// decode timestamp reproduces that interleaving from the per-track + /// indices. + fn next_slot(&self) -> Option { + self.cursors + .iter() + .enumerate() + .filter_map(|(slot, &cursor)| { + self.indices[slot] + .samples() + .get(cursor) + .map(|sample| (slot, sample.dts, sample.offset)) + }) + // Ties break on file offset, which keeps the order stable and + // matches how the samples are actually laid out. + .min_by_key(|&(_, dts, offset)| (dts, offset)) + .map(|(slot, _, _)| slot) + } + + /// Read one sample's bytes. + fn read_sample(&mut self, offset: u64, size: u32) -> VideoResult> { + self.source.seek(SeekFrom::Start(offset))?; + let mut data = vec![0u8; size as usize]; + self.source.read_exact(&mut data)?; + Ok(data) + } +} + +impl Demuxer for IsoBmffDemuxer { + fn container(&self) -> ContainerId { + self.container + } + + fn tracks(&self) -> &[Track] { + &self.tracks + } + + fn metadata(&self) -> &VideoMetadata { + &self.metadata + } + + fn next_packet(&mut self) -> VideoResult> { + let Some(slot) = self.next_slot() else { + return Ok(None); + }; + let sample = self.indices[slot].samples()[self.cursors[slot]]; + self.cursors[slot] += 1; + + let data = self.read_sample(sample.offset, sample.size)?; + let track = self.tracks[slot].id(); + + // The container's sync table is advisory; for the codecs rawshift + // decodes, the bitstream is authoritative and disagrees often enough + // to matter. Fall back to the table for anything else. + let is_keyframe = match self.tracks[slot].as_video() { + Some(video) if !video.codec_config.is_empty() => { + nal::is_random_access_point(video.codec, &video.codec_config, &data) + } + _ => sample.is_sync, + }; + + let mut packet = Packet::new(track, data); + packet.pts = Some(sample.pts); + packet.dts = Some(sample.dts); + packet.duration = Some(u64::from(sample.duration)); + packet.is_keyframe = is_keyframe; + Ok(Some(packet)) + } + + fn seek(&mut self, track: TrackId, to: SeekTo, mode: SeekMode) -> VideoResult { + let Some(slot) = self.slot_of(track) else { + return Err(VideoError::NoSuchTrack { id: track }); + }; + + let time_base = self.tracks[slot].time_base(); + let timestamp = match to { + SeekTo::Timestamp(ts) => ts, + SeekTo::Time(time) => { + if time_base.numerator == 0 { + return Err(VideoError::container( + CONTAINER_NAME, + "track has a degenerate time base; seek by timestamp instead", + )); + } + // ticks = seconds * (denominator / numerator) + (time.as_secs_f64() * f64::from(time_base.denominator) + / f64::from(time_base.numerator)) as i64 + } + other => { + return Err(VideoError::container( + CONTAINER_NAME, + format!("unsupported seek target: {other:?}"), + )); + } + }; + + // Both seek modes land on a random access point: a decoder cannot + // resume anywhere else. Coarse differs only in that the caller has said + // it will not decode forward to reach the exact frame, which changes + // nothing about where the demuxer can put the cursor. + let _ = mode; + let index = &self.indices[slot]; + let Some(landing) = index.sync_sample_at_or_before(timestamp) else { + return Err(VideoError::container( + CONTAINER_NAME, + "track has no samples to seek within", + )); + }; + + self.cursors[slot] = landing; + Ok(index.samples()[landing].pts) + } +} diff --git a/crates/rawshift-video-isobmff/src/index.rs b/crates/rawshift-video-isobmff/src/index.rs new file mode 100644 index 0000000..3a3a871 --- /dev/null +++ b/crates/rawshift-video-isobmff/src/index.rs @@ -0,0 +1,899 @@ +//! The sample index: turning a track's sample tables into a flat, ordered list +//! of samples with byte ranges and timestamps. +//! +//! ISOBMFF does not store a sample table. It stores five or six *compressed* +//! tables — sizes, chunk offsets, samples-per-chunk runs, decode-time runs, +//! composition offsets, and sync sample numbers — that have to be walked +//! together to recover where each sample lives and when it is shown. That +//! reconstruction is what this module does, and it is the part `mp4-atom` +//! deliberately leaves to its caller. +//! +//! Every table is attacker-controlled, so the walk is total: an inconsistent +//! table truncates the index rather than panicking or producing a byte range +//! that points outside the file. + +use mp4_atom::{Edts, Stbl, StszSamples}; + +/// One sample: where it is, when it decodes, and when it is shown. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct Sample { + /// Byte offset from the start of the file. + pub offset: u64, + /// Size in bytes. + pub size: u32, + /// Decode timestamp, in the track's timescale. + pub dts: i64, + /// Presentation timestamp, in the track's timescale. + /// + /// Equal to `dts` unless a `ctts` composition offset applies, which is how + /// reordered streams say a picture is decoded before it is shown. + pub pts: i64, + /// How long the sample occupies, in the track's timescale. + pub duration: u32, + /// Whether the container marks this as a sync sample. + /// + /// Advisory only: rawshift derives random access from the bitstream, which + /// is authoritative. Kept because it is what seeking indexes against. + pub is_sync: bool, +} + +/// A track's samples, in decode order. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct SampleIndex { + samples: Vec, +} + +impl SampleIndex { + /// Reconstruct the index from a track's sample tables, applying `edts`. + /// + /// `file_len` bounds every table against reality. The sample tables are + /// attacker-controlled, and a sample occupies at least one byte, so no + /// honest table can claim more samples than the file has bytes, and no + /// sample can extend past the end. Without that bound an `stsz` claiming + /// `u32::MAX` samples allocates tens of gigabytes before a single byte of + /// media is read. + /// + /// Never fails: a malformed or inconsistent table stops the walk and + /// yields the samples recovered so far, because a partly-readable track is + /// more useful than none and the caller cannot act on a parse error here + /// anyway. + #[must_use] + pub fn build(stbl: &Stbl, edts: Option<&Edts>, file_len: u64) -> Self { + let shift = presentation_shift(edts); + let sizes = sample_sizes(&stbl.stsz.samples, file_len); + let sample_count = sizes.len(); + if sample_count == 0 { + return Self::default(); + } + + let chunk_offsets: Vec = match (&stbl.stco, &stbl.co64) { + // co64 wins when both are present: a file large enough to need + // 64-bit offsets has a stco that cannot address it. + (_, Some(co64)) => co64.entries.clone(), + (Some(stco), None) => stco.entries.iter().map(|&o| u64::from(o)).collect(), + (None, None) => return Self::default(), + }; + + let offsets = sample_offsets(&stbl.stsc.entries, &chunk_offsets, &sizes); + let times = sample_times(&stbl.stts.entries, sample_count); + let composition = composition_offsets(stbl.ctts.as_ref(), sample_count); + + // stss lists the sync samples by 1-based number. Its *absence* means + // every sample is a sync sample — the convention for all-intra tracks — + // which is the opposite of treating an absent table as "none". + let sync: Option> = stbl + .stss + .as_ref() + .map(|stss| stss.entries.iter().copied().collect()); + + let usable = offsets.len().min(times.len()).min(sample_count); + let samples = (0..usable) + // A sample that runs past the end of the file is not readable, so + // indexing it would only produce a failing read later. + .filter(|&i| { + offsets[i] + .checked_add(u64::from(sizes[i])) + .is_some_and(|end| end <= file_len) + }) + .map(|i| { + let (dts, duration) = times[i]; + let composition_shift = composition.get(i).copied().unwrap_or(0); + Sample { + offset: offsets[i], + size: sizes[i], + dts: dts.saturating_sub(shift), + pts: dts.saturating_add(composition_shift).saturating_sub(shift), + duration, + is_sync: sync + .as_ref() + .is_none_or(|set| set.contains(&(i as u32 + 1))), + } + }) + .collect(); + + Self { samples } + } + + /// The samples, in decode order. + #[must_use] + pub fn samples(&self) -> &[Sample] { + &self.samples + } + + /// How many samples the track has. + #[must_use] + pub fn len(&self) -> usize { + self.samples.len() + } + + /// Whether the track has no samples. + #[must_use] + pub fn is_empty(&self) -> bool { + self.samples.is_empty() + } + + /// The index of the last sync sample at or before `timestamp` in + /// presentation order, or the first sample when none precedes it. + /// + /// This is the seek landing point: decoding can only resume at a random + /// access point, so a request between two of them lands on the earlier + /// one and the caller decodes forward. + #[must_use] + pub fn sync_sample_at_or_before(&self, timestamp: i64) -> Option { + if self.samples.is_empty() { + return None; + } + let landing = self + .samples + .iter() + .enumerate() + .rfind(|(_, s)| s.is_sync && s.pts <= timestamp) + .map(|(i, _)| i); + + // Before the first sync sample there is nothing to resume from, so the + // start of the track is the only honest answer. + Some(landing.unwrap_or(0)) + } +} + +/// How far to shift the timeline so presentation starts at zero. +/// +/// An `elst` maps presentation time onto media time. Encoders that reorder +/// write a single edit whose `media_time` equals the composition delay, which +/// says "presentation begins this far into the media" — the first frame is +/// shown at time zero even though its composition timestamp is not. Ignoring +/// it leaves every timestamp offset by the reorder delay, disagreeing with +/// every other tool that reads the file. +/// +/// Only the single-entry case is honoured, which is what encoders write. A +/// multi-entry edit list is a genuine cut list that would have to trim and +/// reorder samples; that is out of scope, and ignoring it yields untrimmed +/// media rather than wrong media. +fn presentation_shift(edts: Option<&Edts>) -> i64 { + let Some(elst) = edts.and_then(|e| e.elst.as_ref()) else { + return 0; + }; + let [entry] = elst.entries.as_slice() else { + return 0; + }; + // An empty edit (media_time == -1, spelled None here) is a dwell with no + // media and shifts nothing. + let Some(media_time) = entry.media_time else { + return 0; + }; + i64::try_from(media_time).unwrap_or(0) +} + +/// Expand `stsz` into a per-sample size list, bounded by the file length. +/// +/// The count is clamped because it is attacker-controlled and drives an +/// allocation: a sample occupies at least one byte, so a file cannot hold more +/// samples than it has bytes. +fn sample_sizes(samples: &StszSamples, file_len: u64) -> Vec { + let ceiling = usize::try_from(file_len).unwrap_or(usize::MAX); + match samples { + StszSamples::Identical { count, size } => { + vec![*size; (*count as usize).min(ceiling)] + } + StszSamples::Different { sizes } => { + let mut sizes = sizes.clone(); + sizes.truncate(ceiling); + sizes + } + } +} + +/// Walk `stsc` and the chunk offsets to place every sample in the file. +/// +/// `stsc` stores runs: an entry `(first_chunk, samples_per_chunk, ..)` applies +/// from `first_chunk` until the next entry's `first_chunk`. Within a chunk, +/// samples are laid out back to back from the chunk's offset. +fn sample_offsets(stsc: &[mp4_atom::StscEntry], chunk_offsets: &[u64], sizes: &[u32]) -> Vec { + let mut offsets = Vec::with_capacity(sizes.len()); + let mut sample = 0usize; + + for (index, entry) in stsc.iter().enumerate() { + // Chunk numbers are 1-based; a zero would underflow the range below. + if entry.first_chunk == 0 { + break; + } + let first = entry.first_chunk as usize - 1; + let last = stsc + .get(index + 1) + .map_or(chunk_offsets.len(), |next| { + (next.first_chunk as usize).saturating_sub(1) + }) + .min(chunk_offsets.len()); + + for chunk in first..last { + let Some(&base) = chunk_offsets.get(chunk) else { + return offsets; + }; + let mut cursor = base; + for _ in 0..entry.samples_per_chunk { + let Some(&size) = sizes.get(sample) else { + return offsets; + }; + offsets.push(cursor); + // A crafted table can push the cursor past the file; the + // demuxer bounds-checks the read, so saturating here is enough + // to keep the walk total. + cursor = cursor.saturating_add(u64::from(size)); + sample += 1; + } + } + } + + offsets +} + +/// Expand the `stts` runs into per-sample `(dts, duration)`. +fn sample_times(stts: &[mp4_atom::SttsEntry], sample_count: usize) -> Vec<(i64, u32)> { + let mut times = Vec::with_capacity(sample_count); + let mut dts: i64 = 0; + + for entry in stts { + for _ in 0..entry.sample_count { + if times.len() >= sample_count { + return times; + } + times.push((dts, entry.sample_delta)); + dts = dts.saturating_add(i64::from(entry.sample_delta)); + } + } + + // A track whose stts covers fewer samples than stsz claims is malformed. + // Extend with zero-duration entries at the last timestamp rather than + // dropping the samples: their bytes are still readable. + while times.len() < sample_count { + times.push((dts, 0)); + } + times +} + +/// Expand the `ctts` runs into per-sample composition offsets. +fn composition_offsets(ctts: Option<&mp4_atom::Ctts>, sample_count: usize) -> Vec { + let Some(ctts) = ctts else { + return vec![0; sample_count]; + }; + + let mut offsets = Vec::with_capacity(sample_count); + for entry in &ctts.entries { + for _ in 0..entry.sample_count { + if offsets.len() >= sample_count { + return offsets; + } + offsets.push(entry.sample_offset); + } + } + offsets.resize(sample_count, 0); + offsets +} + +#[cfg(test)] +mod tests { + use super::*; + /// Generous bound for tests that are not about the file-length clamp. + const TEST_FILE_LEN: u64 = 1 << 20; + + use mp4_atom::{ + Co64, Ctts, CttsEntry, Elst, ElstEntry, Stco, Stsc, StscEntry, Stsd, Stss, Stsz, Stts, + SttsEntry, + }; + + /// A minimal `stbl` from the parts a test cares about. + fn stbl( + sizes: StszSamples, + stsc: Vec, + chunks: Vec, + stts: Vec, + ctts: Option>, + stss: Option>, + ) -> Stbl { + Stbl { + stsd: Stsd { codecs: vec![] }, + stts: Stts { entries: stts }, + ctts: ctts.map(|entries| Ctts { entries }), + stss: stss.map(|entries| Stss { entries }), + stsc: Stsc { entries: stsc }, + stsz: Stsz { samples: sizes }, + stco: Some(Stco { entries: chunks }), + co64: None, + sbgp: vec![], + sgpd: vec![], + subs: vec![], + saiz: vec![], + saio: vec![], + cslg: None, + } + } + + fn run(count: u32, delta: u32) -> SttsEntry { + SttsEntry { + sample_count: count, + sample_delta: delta, + } + } + + #[test] + fn samples_are_laid_out_back_to_back_within_a_chunk() { + // Two chunks of two samples each, at offsets 100 and 500. + let index = SampleIndex::build( + &stbl( + StszSamples::Different { + sizes: vec![10, 20, 30, 40], + }, + vec![StscEntry { + first_chunk: 1, + samples_per_chunk: 2, + sample_description_index: 1, + }], + vec![100, 500], + vec![run(4, 512)], + None, + None, + ), + None, + TEST_FILE_LEN, + ); + + let offsets: Vec<_> = index.samples().iter().map(|s| s.offset).collect(); + assert_eq!(offsets, vec![100, 110, 500, 530]); + let sizes: Vec<_> = index.samples().iter().map(|s| s.size).collect(); + assert_eq!(sizes, vec![10, 20, 30, 40]); + } + + #[test] + fn a_second_stsc_run_changes_the_samples_per_chunk() { + // Chunk 1 holds one sample; chunks 2 and 3 hold two each. + let index = SampleIndex::build( + &stbl( + StszSamples::Identical { count: 5, size: 10 }, + vec![ + StscEntry { + first_chunk: 1, + samples_per_chunk: 1, + sample_description_index: 1, + }, + StscEntry { + first_chunk: 2, + samples_per_chunk: 2, + sample_description_index: 1, + }, + ], + vec![1000, 2000, 3000], + vec![run(5, 100)], + None, + None, + ), + None, + TEST_FILE_LEN, + ); + + let offsets: Vec<_> = index.samples().iter().map(|s| s.offset).collect(); + assert_eq!(offsets, vec![1000, 2000, 2010, 3000, 3010]); + } + + #[test] + fn decode_timestamps_accumulate_across_stts_runs() { + let index = SampleIndex::build( + &stbl( + StszSamples::Identical { count: 5, size: 1 }, + vec![StscEntry { + first_chunk: 1, + samples_per_chunk: 5, + sample_description_index: 1, + }], + vec![0], + // Three samples of 100, then two of 50. + vec![run(3, 100), run(2, 50)], + None, + None, + ), + None, + TEST_FILE_LEN, + ); + + let dts: Vec<_> = index.samples().iter().map(|s| s.dts).collect(); + assert_eq!(dts, vec![0, 100, 200, 300, 350]); + let durations: Vec<_> = index.samples().iter().map(|s| s.duration).collect(); + assert_eq!(durations, vec![100, 100, 100, 50, 50]); + } + + #[test] + fn composition_offsets_separate_presentation_from_decode_order() { + // The classic reordered pattern: an I frame, a P frame shown last, and + // two B frames shown in between. + let index = SampleIndex::build( + &stbl( + StszSamples::Identical { count: 4, size: 1 }, + vec![StscEntry { + first_chunk: 1, + samples_per_chunk: 4, + sample_description_index: 1, + }], + vec![0], + vec![run(4, 100)], + Some(vec![ + CttsEntry { + sample_count: 1, + sample_offset: 0, + }, + CttsEntry { + sample_count: 1, + sample_offset: 300, + }, + CttsEntry { + sample_count: 2, + sample_offset: 0, + }, + ]), + None, + ), + None, + TEST_FILE_LEN, + ); + + let dts: Vec<_> = index.samples().iter().map(|s| s.dts).collect(); + let pts: Vec<_> = index.samples().iter().map(|s| s.pts).collect(); + assert_eq!(dts, vec![0, 100, 200, 300]); + // The second sample decodes at 100 but is shown at 400, after the two + // B frames at 200 and 300. + assert_eq!(pts, vec![0, 400, 200, 300]); + } + + #[test] + fn a_negative_composition_offset_is_honoured() { + // Version 1 ctts allows negative offsets, which move presentation + // earlier than decode. + let index = SampleIndex::build( + &stbl( + StszSamples::Identical { count: 2, size: 1 }, + vec![StscEntry { + first_chunk: 1, + samples_per_chunk: 2, + sample_description_index: 1, + }], + vec![0], + vec![run(2, 100)], + Some(vec![CttsEntry { + sample_count: 2, + sample_offset: -100, + }]), + None, + ), + None, + TEST_FILE_LEN, + ); + + let pts: Vec<_> = index.samples().iter().map(|s| s.pts).collect(); + assert_eq!(pts, vec![-100, 0]); + } + + #[test] + fn an_absent_stss_means_every_sample_is_a_sync_sample() { + // The all-intra convention. Reading an absent table as "no sync + // samples" would make such a track unseekable. + let index = SampleIndex::build( + &stbl( + StszSamples::Identical { count: 3, size: 1 }, + vec![StscEntry { + first_chunk: 1, + samples_per_chunk: 3, + sample_description_index: 1, + }], + vec![0], + vec![run(3, 100)], + None, + None, + ), + None, + TEST_FILE_LEN, + ); + + assert!(index.samples().iter().all(|s| s.is_sync)); + } + + #[test] + fn stss_marks_sync_samples_by_one_based_number() { + let index = SampleIndex::build( + &stbl( + StszSamples::Identical { count: 5, size: 1 }, + vec![StscEntry { + first_chunk: 1, + samples_per_chunk: 5, + sample_description_index: 1, + }], + vec![0], + vec![run(5, 100)], + None, + // Samples 1 and 4, one-based: indices 0 and 3. + Some(vec![1, 4]), + ), + None, + TEST_FILE_LEN, + ); + + let sync: Vec<_> = index.samples().iter().map(|s| s.is_sync).collect(); + assert_eq!(sync, vec![true, false, false, true, false]); + } + + #[test] + fn co64_wins_over_stco_when_both_are_present() { + // A file large enough to need 64-bit offsets has an stco that cannot + // address it, so preferring stco would place samples wrongly. + let mut tables = stbl( + StszSamples::Identical { count: 1, size: 4 }, + vec![StscEntry { + first_chunk: 1, + samples_per_chunk: 1, + sample_description_index: 1, + }], + vec![42], + vec![run(1, 100)], + None, + None, + ); + tables.co64 = Some(Co64 { + entries: vec![0x1_0000_0000], + }); + + // Addressing past 4 GB is the whole point of co64, so this needs a + // file length that can hold it. + let index = SampleIndex::build(&tables, None, 0x1_0000_0100); + assert_eq!(index.samples()[0].offset, 0x1_0000_0000); + } + + #[test] + fn seeking_lands_on_the_sync_sample_at_or_before_the_request() { + let index = SampleIndex::build( + &stbl( + StszSamples::Identical { count: 6, size: 1 }, + vec![StscEntry { + first_chunk: 1, + samples_per_chunk: 6, + sample_description_index: 1, + }], + vec![0], + vec![run(6, 100)], + None, + Some(vec![1, 4]), // sync at pts 0 and 300 + ), + None, + TEST_FILE_LEN, + ); + + // Exactly on a sync sample. + assert_eq!(index.sync_sample_at_or_before(300), Some(3)); + // Between two: must land on the earlier one, never the later. + assert_eq!(index.sync_sample_at_or_before(250), Some(0)); + assert_eq!(index.sync_sample_at_or_before(450), Some(3)); + // Past the end: the last sync sample. + assert_eq!(index.sync_sample_at_or_before(99_999), Some(3)); + // Before the first: the start of the track is the only resumable point. + assert_eq!(index.sync_sample_at_or_before(-100), Some(0)); + } + + fn edts(media_time: Option) -> Edts { + Edts { + elst: Some(Elst { + entries: vec![ElstEntry { + segment_duration: 1_000, + media_time, + media_rate: 1.into(), + }], + }), + } + } + + fn reordered_tables() -> Stbl { + // Two frames with a composition delay of 100, as a reordering encoder + // writes: composition timestamps start at 100, not 0. + stbl( + StszSamples::Identical { count: 2, size: 1 }, + vec![StscEntry { + first_chunk: 1, + samples_per_chunk: 2, + sample_description_index: 1, + }], + vec![0], + vec![run(2, 100)], + Some(vec![CttsEntry { + sample_count: 2, + sample_offset: 100, + }]), + None, + ) + } + + #[test] + fn an_edit_list_shifts_presentation_to_start_at_zero() { + // Without the edit list every timestamp is offset by the reorder + // delay, and rawshift would disagree with every other tool reading the + // same file. + let unshifted = SampleIndex::build(&reordered_tables(), None, TEST_FILE_LEN); + assert_eq!(unshifted.samples()[0].pts, 100); + + let shifted = + SampleIndex::build(&reordered_tables(), Some(&edts(Some(100))), TEST_FILE_LEN); + assert_eq!(shifted.samples()[0].pts, 0, "presentation must start at 0"); + // Decode time goes negative, which is correct: the frame is decoded + // before the presentation timeline begins. + assert_eq!(shifted.samples()[0].dts, -100); + } + + #[test] + fn an_empty_edit_shifts_nothing() { + // media_time == -1 is a dwell with no media, not an offset. + let index = SampleIndex::build(&reordered_tables(), Some(&edts(None)), TEST_FILE_LEN); + assert_eq!(index.samples()[0].pts, 100); + } + + #[test] + fn a_multi_entry_edit_list_is_ignored_rather_than_half_applied() { + // A real cut list would have to trim and reorder samples. Ignoring it + // yields untrimmed media, which is recoverable; applying only its + // first entry would yield wrong media, which is not. + let cut_list = Edts { + elst: Some(Elst { + entries: vec![ + ElstEntry { + segment_duration: 500, + media_time: Some(100), + media_rate: 1.into(), + }, + ElstEntry { + segment_duration: 500, + media_time: Some(900), + media_rate: 1.into(), + }, + ], + }), + }; + let index = SampleIndex::build(&reordered_tables(), Some(&cut_list), TEST_FILE_LEN); + assert_eq!(index.samples()[0].pts, 100, "left untrimmed, not shifted"); + } + + #[test] + fn seeking_an_empty_index_yields_nothing() { + assert_eq!(SampleIndex::default().sync_sample_at_or_before(0), None); + assert!(SampleIndex::default().is_empty()); + } + + #[test] + fn a_chunk_offset_table_shorter_than_stsc_claims_truncates_cleanly() { + // stsc describes three chunks; stco lists one. The walk must stop at + // what it can place rather than indexing past the table. + let index = SampleIndex::build( + &stbl( + StszSamples::Identical { count: 6, size: 10 }, + vec![StscEntry { + first_chunk: 1, + samples_per_chunk: 2, + sample_description_index: 1, + }], + vec![100], + vec![run(6, 100)], + None, + None, + ), + None, + TEST_FILE_LEN, + ); + + assert_eq!(index.len(), 2, "only the samples in the known chunk"); + } + + #[test] + fn a_sample_count_beyond_the_file_length_is_clamped() { + // An stsz claiming u32::MAX samples would otherwise allocate tens of + // gigabytes before a byte of media is read. A sample occupies at least + // one byte, so the file length is the honest ceiling. + let tables = stbl( + StszSamples::Identical { + count: u32::MAX, + size: 1, + }, + vec![StscEntry { + first_chunk: 1, + samples_per_chunk: 4, + sample_description_index: 1, + }], + vec![0], + vec![run(4, 100)], + None, + None, + ); + + // A 64-byte file cannot hold four billion samples. + let index = SampleIndex::build(&tables, None, 64); + assert!( + index.len() <= 64, + "clamped to the file length, got {}", + index.len() + ); + } + + #[test] + fn an_explicit_size_table_longer_than_the_file_is_truncated() { + let tables = stbl( + StszSamples::Different { + sizes: vec![1; 10_000], + }, + vec![StscEntry { + first_chunk: 1, + samples_per_chunk: 10_000, + sample_description_index: 1, + }], + vec![0], + vec![run(10_000, 1)], + None, + None, + ); + assert!(SampleIndex::build(&tables, None, 32).len() <= 32); + } + + #[test] + fn a_sample_running_past_the_end_of_the_file_is_dropped() { + // Indexing it would only produce a failing read later, and a size of + // u32::MAX would allocate four gigabytes to do it. + let tables = stbl( + StszSamples::Different { + sizes: vec![10, u32::MAX], + }, + vec![StscEntry { + first_chunk: 1, + samples_per_chunk: 2, + sample_description_index: 1, + }], + vec![0], + vec![run(2, 100)], + None, + None, + ); + + let index = SampleIndex::build(&tables, None, 100); + assert_eq!(index.len(), 1, "only the sample that fits"); + assert_eq!(index.samples()[0].size, 10); + } + + #[test] + fn an_offset_plus_size_that_overflows_is_dropped_rather_than_wrapping() { + // Only a 64-bit co64 offset can get near the top of the range; two + // 32-bit values cannot overflow a u64 between them. + let mut tables = stbl( + StszSamples::Different { + sizes: vec![u32::MAX], + }, + vec![StscEntry { + first_chunk: 1, + samples_per_chunk: 1, + sample_description_index: 1, + }], + vec![0], + vec![run(1, 100)], + None, + None, + ); + tables.stco = None; + tables.co64 = Some(Co64 { + entries: vec![u64::MAX - 1], + }); + + // Wrapping would place the sample at a small offset and read the + // wrong bytes; it must be dropped instead. + assert!(SampleIndex::build(&tables, None, u64::MAX).is_empty()); + } + + #[test] + fn a_zero_first_chunk_does_not_underflow() { + // first_chunk is 1-based; a zero is malformed and must not wrap. + let index = SampleIndex::build( + &stbl( + StszSamples::Identical { count: 2, size: 10 }, + vec![StscEntry { + first_chunk: 0, + samples_per_chunk: 2, + sample_description_index: 1, + }], + vec![100], + vec![run(2, 100)], + None, + None, + ), + None, + TEST_FILE_LEN, + ); + assert!(index.is_empty()); + } + + #[test] + fn a_track_with_no_chunk_offsets_yields_no_samples() { + let mut tables = stbl( + StszSamples::Identical { count: 4, size: 10 }, + vec![StscEntry { + first_chunk: 1, + samples_per_chunk: 2, + sample_description_index: 1, + }], + vec![], + vec![run(4, 100)], + None, + None, + ); + tables.stco = None; + assert!(SampleIndex::build(&tables, None, TEST_FILE_LEN).is_empty()); + } + + #[test] + fn an_stts_shorter_than_stsz_still_yields_every_sample() { + // Malformed, but the sample bytes are readable, so they are kept with + // the last known timestamp rather than dropped. + let index = SampleIndex::build( + &stbl( + StszSamples::Identical { count: 4, size: 10 }, + vec![StscEntry { + first_chunk: 1, + samples_per_chunk: 4, + sample_description_index: 1, + }], + vec![0], + vec![run(2, 100)], + None, + None, + ), + None, + TEST_FILE_LEN, + ); + + assert_eq!(index.len(), 4); + assert_eq!(index.samples()[3].dts, 200); + assert_eq!(index.samples()[3].duration, 0); + } + + #[test] + fn an_identical_size_table_expands_to_every_sample() { + let index = SampleIndex::build( + &stbl( + StszSamples::Identical { + count: 3, + size: 1234, + }, + vec![StscEntry { + first_chunk: 1, + samples_per_chunk: 3, + sample_description_index: 1, + }], + vec![0], + vec![run(3, 100)], + None, + None, + ), + None, + TEST_FILE_LEN, + ); + + assert_eq!(index.len(), 3); + assert!(index.samples().iter().all(|s| s.size == 1234)); + } +} diff --git a/crates/rawshift-video-isobmff/src/lib.rs b/crates/rawshift-video-isobmff/src/lib.rs new file mode 100644 index 0000000..ad12118 --- /dev/null +++ b/crates/rawshift-video-isobmff/src/lib.rs @@ -0,0 +1,40 @@ +//! MP4, M4V and QuickTime (MOV) support for rawshift. +//! +//! The box layer is [`mp4_atom`] — MIT, pure Rust, no build script, so it +//! compiles on every target in `docs/SUPPORT.md`. It performs no +//! interpretation, so rawshift owns the parts that give a container meaning: +//! the [sample index](index) reconstructed from the `stbl` tables, the track +//! model, and the demux and seek semantics. +//! +//! gamut's charter excludes video, so this is a direct dependency rather than +//! an upstream ask; see "Video is outside gamut's charter" in `AGENTS.md`. +//! +//! # Why not an off-the-shelf demuxer +//! +//! This crate first used `symphonia-format-isomp4`, which is audio-first and +//! caps the `hvcC` configuration record at 1 KB. Real HEVC records are 2–3 KB, +//! so **every HEVC MP4 and MOV failed to open** — that is XAVC HS and default +//! iPhone video, the two highest-priority formats on rawshift's roadmap. It +//! also hides the `tkhd` matrix (and so rotation), the `colr` box (and so all +//! colour signalling), the sync-sample table, and ProRes-shaped sample +//! entries. Owning the sample index on a box-level library costs a few hundred +//! lines and removes all five limitations at once. +//! +//! # Not covered +//! +//! Fragmented MP4 (`mvex`/`moof`) is rejected with a clear error rather than +//! presented as a track with no samples. Camera-origin files — rawshift's +//! subject — are not fragmented; streaming-shaped input is a follow-up. + +#![forbid(unsafe_code)] +#![warn(missing_docs)] + +pub mod index; + +mod demux; +mod movie; +mod sniff; + +pub use demux::IsoBmffDemuxer; +pub use index::{Sample, SampleIndex}; +pub use sniff::{IsoBmff, detect}; diff --git a/crates/rawshift-video-isobmff/src/movie.rs b/crates/rawshift-video-isobmff/src/movie.rs new file mode 100644 index 0000000..2822f67 --- /dev/null +++ b/crates/rawshift-video-isobmff/src/movie.rs @@ -0,0 +1,504 @@ +//! Reading the movie header, and mapping its tracks onto rawshift's model. + +use std::io::{Read, Seek, SeekFrom}; +use std::time::Duration; + +use mp4_atom::{Atom, Codec, Colr, Encode, Ftyp, Header, Moov, ReadAtom, ReadFrom, Trak}; + +use rawshift_core::metadata::URational; +use rawshift_core::{Dimensions, MetadataNamespace, MetadataValue}; +use rawshift_video_core::{ + AudioCodecId, AudioTrack, CicpColor, CodecConfig, ContainerId, OtherTrack, Rotation, Track, + TrackId, TrackKind, VideoCodecId, VideoError, VideoMetadata, VideoResult, VideoTrack, +}; + +use crate::index::SampleIndex; + +/// The name that appears in [`VideoError::Container`] messages. +pub const CONTAINER_NAME: &str = "MP4"; + +/// The movie header, and everything derived from it. +pub struct Movie { + /// One entry per track, in the file's own order. + pub tracks: Vec, + /// Sample indices, parallel to `tracks`. + pub indices: Vec, + /// The container flavour. + pub container: ContainerId, + /// File-level metadata. + pub metadata: VideoMetadata, +} + +impl Movie { + /// Read `ftyp` and `moov`, and build the tracks and their sample indices. + /// + /// Walks the top-level atoms by header so that `mdat` — which is the whole + /// media payload and routinely gigabytes — is skipped rather than read + /// into memory. + /// + /// # Errors + /// + /// [`VideoError::Container`] when no `moov` is present or an atom is + /// malformed, and [`VideoError::Io`] for a reader failure. + pub fn read(reader: &mut R) -> VideoResult { + let end = reader.seek(SeekFrom::End(0))?; + reader.seek(SeekFrom::Start(0))?; + + let mut ftyp: Option = None; + let mut moov: Option = None; + + while reader.stream_position()? < end { + let header = match Header::read_from(reader) { + Ok(header) => header, + // A trailing partial atom is where damaged files end; stop + // rather than failing a file whose moov we may already have. + Err(_) => break, + }; + + match header.kind { + Ftyp::KIND => { + ftyp = Ftyp::read_atom(&header, reader).ok(); + } + Moov::KIND => { + moov = Some( + Moov::read_atom(&header, reader) + .map_err(|e| VideoError::container(CONTAINER_NAME, e))?, + ); + } + _ => { + // Skip the body. An atom with no stated size runs to the + // end of the file, so there is nothing after it. + match header.size { + Some(size) => { + reader.seek(SeekFrom::Current(size as i64))?; + } + None => break, + } + } + } + + if ftyp.is_some() && moov.is_some() { + break; + } + } + + let Some(moov) = moov else { + return Err(VideoError::container( + CONTAINER_NAME, + "no moov atom: not a playable ISOBMFF file", + )); + }; + + // Fragmented files carry their sample tables in moof boxes rather than + // in stbl, so the index built here would be empty. Say so plainly + // instead of presenting a track with no samples. + if moov.mvex.is_some() { + return Err(VideoError::container( + CONTAINER_NAME, + "fragmented MP4 (mvex present) is not supported yet", + )); + } + + let container = container_of(ftyp.as_ref()); + let mut tracks = Vec::with_capacity(moov.trak.len()); + let mut indices = Vec::with_capacity(moov.trak.len()); + + for trak in &moov.trak { + tracks.push(map_track(trak)); + indices.push(SampleIndex::build( + &trak.mdia.minf.stbl, + trak.edts.as_ref(), + end, + )); + } + + let metadata = build_metadata(&moov, ftyp.as_ref(), container); + + Ok(Self { + tracks, + indices, + container, + metadata, + }) + } +} + +/// The container flavour implied by the `ftyp` brands. +/// +/// The QuickTime brand `qt ` means [`ContainerId::Mov`]; anything else with a +/// `ftyp` is MP4. A file with no `ftyp` is old-style QuickTime. +fn container_of(ftyp: Option<&Ftyp>) -> ContainerId { + let Some(ftyp) = ftyp else { + return ContainerId::Mov; + }; + let quicktime = ftyp.major_brand.as_ref() == b"qt " + || ftyp.compatible_brands.iter().any(|b| b.as_ref() == b"qt "); + if quicktime { + ContainerId::Mov + } else { + ContainerId::Mp4 + } +} + +fn map_track(trak: &Trak) -> Track { + let id = TrackId::new(trak.tkhd.track_id); + let mdhd = &trak.mdia.mdhd; + // The media timescale is the unit of this track's timestamps. + let time_base = URational::new(1, mdhd.timescale.max(1)); + let duration = ticks_to_duration(mdhd.duration, time_base); + let language = (!mdhd.language.is_empty()).then(|| mdhd.language.clone()); + + let stbl = &trak.mdia.minf.stbl; + let Some(codec) = stbl.stsd.codecs.first() else { + let mut other = OtherTrack::new(id, TrackKind::Data, time_base); + other.duration = duration; + other.language = language; + return Track::Other(other); + }; + + match codec { + Codec::Avc1(_) | Codec::Hvc1(_) | Codec::Hev1(_) | Codec::Av01(_) | Codec::Vp09(_) => { + let mut video = map_video_track(id, codec, time_base); + video.duration = duration; + video.frame_count = Some(stbl_sample_count(stbl)); + video.rotation = rotation_of(trak); + Track::Video(video) + } + Codec::Mp4a(_) | Codec::Opus(_) | Codec::Flac(_) | Codec::Ac3(_) | Codec::Eac3(_) => { + let mut audio = AudioTrack::new(id, audio_codec_of(codec), time_base); + audio.duration = duration; + audio.language = language; + Track::Audio(audio) + } + _ => { + let mut other = OtherTrack::new(id, TrackKind::Data, time_base); + other.duration = duration; + other.language = language; + Track::Other(other) + } + } +} + +fn map_video_track(id: TrackId, codec: &Codec, time_base: URational) -> VideoTrack { + let (video_codec, visual, colr, config) = match codec { + Codec::Avc1(a) => ( + VideoCodecId::H264, + &a.visual, + a.colr.as_ref(), + encode_body(&a.avcc), + ), + Codec::Hvc1(h) => ( + VideoCodecId::Hevc, + &h.visual, + h.colr.as_ref(), + encode_body(&h.hvcc), + ), + Codec::Hev1(h) => ( + VideoCodecId::Hevc, + &h.visual, + h.colr.as_ref(), + encode_body(&h.hvcc), + ), + Codec::Av01(a) => ( + VideoCodecId::Av1, + &a.visual, + a.colr.as_ref(), + encode_body(&a.av1c), + ), + Codec::Vp09(v) => (VideoCodecId::Vp9, &v.visual, None, Vec::new()), + // map_track only routes video sample entries here. + _ => unreachable!("map_video_track called with a non-video sample entry"), + }; + + let mut track = VideoTrack::new( + id, + video_codec, + Dimensions { + width: u32::from(visual.width), + height: u32::from(visual.height), + }, + time_base, + ); + track.color = colour_of(colr); + track.codec_config = CodecConfig::new(config); + track +} + +/// Re-encode a configuration atom back to its record body. +/// +/// The box layer parses `avcC`/`hvcC` into fields, but a decoder wants the +/// record exactly as the file stored it — that is what a hardware decoder's +/// configuration input is defined in terms of. Encoding the body back is the +/// faithful round trip, and it keeps rawshift from having to re-serialise the +/// record itself. +fn encode_body(atom: &A) -> Vec { + let mut buf = Vec::new(); + // Encoding into a Vec cannot fail for these small, fully-owned atoms. + if atom.encode(&mut buf).is_err() { + return Vec::new(); + } + // `encode` writes the full box; the record body is what follows the + // 8-byte size and type header. + buf.split_off(8.min(buf.len())) +} + +/// Translate a `colr` box into rawshift's colour signalling. +/// +/// `nclx` carries the full CICP triple plus the range flag; the older `nclc` +/// has no range flag, and studio range is the correct reading for it. An +/// absent or ICC-profile-only `colr` leaves everything unspecified, which +/// resolves by picture height at decode time. +fn colour_of(colr: Option<&Colr>) -> CicpColor { + match colr { + Some(Colr::Nclx { + colour_primaries, + transfer_characteristics, + matrix_coefficients, + full_range_flag, + }) => CicpColor { + primaries: *colour_primaries, + transfer: *transfer_characteristics, + matrix: *matrix_coefficients, + full_range: *full_range_flag, + }, + Some(Colr::Nclc { + colour_primaries, + transfer_characteristics, + matrix_coefficients, + }) => CicpColor { + primaries: *colour_primaries, + transfer: *transfer_characteristics, + matrix: *matrix_coefficients, + full_range: false, + }, + _ => CicpColor::UNSPECIFIED, + } +} + +/// Recover display rotation from the `tkhd` matrix. +/// +/// The matrix values are 16.16 fixed point; only `a`, `b`, `c`, `d` carry +/// rotation. +fn rotation_of(trak: &Trak) -> Rotation { + let m = &trak.tkhd.matrix; + let fixed = |v: i32| f64::from(v) / 65536.0; + Rotation::from_display_matrix(fixed(m.a), fixed(m.b), fixed(m.c), fixed(m.d)) +} + +fn audio_codec_of(codec: &Codec) -> AudioCodecId { + match codec { + Codec::Mp4a(_) => AudioCodecId::Aac, + Codec::Opus(_) => AudioCodecId::Opus, + Codec::Flac(_) => AudioCodecId::Flac, + Codec::Ac3(_) | Codec::Eac3(_) => AudioCodecId::Ac3, + _ => AudioCodecId::Other, + } +} + +fn stbl_sample_count(stbl: &mp4_atom::Stbl) -> u64 { + match &stbl.stsz.samples { + mp4_atom::StszSamples::Identical { count, .. } => u64::from(*count), + mp4_atom::StszSamples::Different { sizes } => sizes.len() as u64, + } +} + +/// Convert a tick count in `time_base` units to a [`Duration`]. +pub fn ticks_to_duration(ticks: u64, time_base: URational) -> Option { + if time_base.denominator == 0 { + return None; + } + let seconds = ticks as f64 * f64::from(time_base.numerator) / f64::from(time_base.denominator); + Duration::try_from_secs_f64(seconds).ok() +} + +fn build_metadata(moov: &Moov, ftyp: Option<&Ftyp>, container: ContainerId) -> VideoMetadata { + let mut md = VideoMetadata::default(); + let mvhd = &moov.mvhd; + + md.container.container = Some(container); + md.container.timescale = Some(mvhd.timescale); + md.container.duration = + ticks_to_duration(mvhd.duration, URational::new(1, mvhd.timescale.max(1))); + md.container.creation_time = Some(format_iso_time(mvhd.creation_time)); + md.container.modification_time = Some(format_iso_time(mvhd.modification_time)); + + if let Some(ftyp) = ftyp { + md.container.brands = std::iter::once(&ftyp.major_brand) + .chain(ftyp.compatible_brands.iter()) + .map(|b| brand_string(b.as_ref())) + .collect(); + } + + for brand in md.container.brands.clone() { + md.push_container_entry( + MetadataNamespace::Quicktime, + "ftyp.brand", + MetadataValue::Text(brand), + ); + } + md.push_container_entry( + MetadataNamespace::Quicktime, + "mvhd.timescale", + MetadataValue::U64(u64::from(mvhd.timescale)), + ); + + md +} + +/// Render a four-character brand, replacing non-printable bytes. +/// +/// A crafted brand must not smuggle control characters into a log line or an +/// error message. +fn brand_string(bytes: &[u8]) -> String { + bytes + .iter() + .map(|&b| { + if b.is_ascii_graphic() || b == b' ' { + b as char + } else { + '?' + } + }) + .collect() +} + +/// Render an ISOBMFF timestamp as an ISO 8601 date-time in UTC. +/// +/// ISOBMFF counts seconds from 1904-01-01, not the Unix epoch. Formatted here +/// rather than by pulling in a date library for one field. +pub fn format_iso_time(seconds_since_1904: u64) -> String { + /// Seconds between 1904-01-01 and 1970-01-01. + const EPOCH_OFFSET: u64 = 2_082_844_800; + + if seconds_since_1904 < EPOCH_OFFSET { + // Before the Unix epoch: report the raw value rather than a wrong date. + return format!("{seconds_since_1904} (seconds since 1904-01-01)"); + } + let unix = seconds_since_1904 - EPOCH_OFFSET; + + let (days, time) = (unix / 86_400, unix % 86_400); + let (hour, minute, second) = (time / 3_600, (time % 3_600) / 60, time % 60); + let (year, month, day) = civil_from_days(days as i64); + + format!("{year:04}-{month:02}-{day:02}T{hour:02}:{minute:02}:{second:02}Z") +} + +/// Convert days since 1970-01-01 to a proleptic Gregorian date. +/// +/// Howard Hinnant's `civil_from_days`: exact over the whole representable +/// range and table-free. +fn civil_from_days(days: i64) -> (i64, u32, u32) { + let z = days + 719_468; + let era = if z >= 0 { z } else { z - 146_096 } / 146_097; + let doe = (z - era * 146_097) as u64; + let yoe = (doe - doe / 1_460 + doe / 36_524 - doe / 146_096) / 365; + let y = yoe as i64 + era * 400; + let doy = doe - (365 * yoe + yoe / 4 - yoe / 100); + let mp = (5 * doy + 2) / 153; + let d = (doy - (153 * mp + 2) / 5 + 1) as u32; + let m = if mp < 10 { mp + 3 } else { mp - 9 } as u32; + (if m <= 2 { y + 1 } else { y }, m, d) +} + +#[cfg(test)] +mod tests { + use super::*; + use mp4_atom::FourCC; + + #[test] + fn the_isobmff_epoch_is_offset_from_unix_not_equal_to_it() { + assert_eq!(format_iso_time(2_082_844_800), "1970-01-01T00:00:00Z"); + assert_eq!( + format_iso_time(2_082_844_800 + 1_705_321_845), + "2024-01-15T12:30:45Z" + ); + } + + #[test] + fn a_timestamp_before_the_unix_epoch_reports_raw_rather_than_a_wrong_date() { + let s = format_iso_time(0); + assert!(s.contains("1904"), "{s}"); + assert!(!s.starts_with("1970"), "{s}"); + } + + #[test] + fn civil_from_days_handles_leap_years_and_epoch_edges() { + assert_eq!(civil_from_days(0), (1970, 1, 1)); + assert_eq!(civil_from_days(59), (1970, 3, 1)); + assert_eq!(civil_from_days(11_016), (2000, 2, 29)); + assert_eq!(civil_from_days(19_782), (2024, 2, 29)); + assert_eq!(civil_from_days(19_783), (2024, 3, 1)); + } + + #[test] + fn the_quicktime_brand_selects_mov_from_either_position() { + let major = Ftyp { + major_brand: FourCC::new(b"qt "), + minor_version: 0, + compatible_brands: vec![], + }; + assert_eq!(container_of(Some(&major)), ContainerId::Mov); + + let compatible = Ftyp { + major_brand: FourCC::new(b"isom"), + minor_version: 0, + compatible_brands: vec![FourCC::new(b"qt ")], + }; + assert_eq!(container_of(Some(&compatible)), ContainerId::Mov); + } + + #[test] + fn other_brands_are_mp4_and_a_missing_ftyp_is_legacy_quicktime() { + let mp4 = Ftyp { + major_brand: FourCC::new(b"isom"), + minor_version: 512, + compatible_brands: vec![FourCC::new(b"mp41")], + }; + assert_eq!(container_of(Some(&mp4)), ContainerId::Mp4); + assert_eq!(container_of(None), ContainerId::Mov); + } + + #[test] + fn nclx_colour_carries_the_full_cicp_triple_and_range() { + let colr = Colr::Nclx { + colour_primaries: 9, + transfer_characteristics: 16, + matrix_coefficients: 9, + full_range_flag: true, + }; + let c = colour_of(Some(&colr)); + assert_eq!((c.primaries, c.transfer, c.matrix), (9, 16, 9)); + assert!(c.full_range, "the range flag must survive"); + } + + #[test] + fn nclc_colour_has_no_range_flag_and_reads_as_studio() { + let colr = Colr::Nclc { + colour_primaries: 1, + transfer_characteristics: 1, + matrix_coefficients: 1, + }; + let c = colour_of(Some(&colr)); + assert_eq!(c.primaries, 1); + assert!( + !c.full_range, + "nclc has no range flag; studio is the reading" + ); + } + + #[test] + fn an_absent_or_icc_only_colr_leaves_colour_unspecified() { + assert_eq!(colour_of(None), CicpColor::UNSPECIFIED); + let icc = Colr::Prof { + profile: vec![0u8; 16], + }; + assert_eq!(colour_of(Some(&icc)), CicpColor::UNSPECIFIED); + } + + #[test] + fn brands_are_sanitised_of_control_bytes() { + assert_eq!(brand_string(b"isom"), "isom"); + assert_eq!(brand_string(b"qt "), "qt "); + assert_eq!(brand_string(&[0x00, 0x1b, b'[', b'A']), "??[A"); + } +} diff --git a/crates/rawshift-video-isobmff/src/sniff.rs b/crates/rawshift-video-isobmff/src/sniff.rs new file mode 100644 index 0000000..00c9a81 --- /dev/null +++ b/crates/rawshift-video-isobmff/src/sniff.rs @@ -0,0 +1,112 @@ +//! Signature detection for the ISOBMFF family. + +use rawshift_video_core::{ContainerId, ContainerSniffer}; + +/// Marker for MP4 / M4V / QuickTime detection and demuxing. +/// +/// One type for the whole family: MP4 and QuickTime share a box grammar and a +/// demuxer, and differ only in the `ftyp` brand, so splitting them into two +/// sniffers would duplicate every byte of the logic. +pub struct IsoBmff; + +impl ContainerSniffer for IsoBmff { + // The family's representative. Use [`detect`] for the MP4/MOV + // distinction, which needs the brand rather than just the signature. + const CONTAINER: ContainerId = ContainerId::Mp4; + + fn matches(data: &[u8]) -> bool { + detect(data).is_some() + } +} + +/// Which ISOBMFF flavour `data` looks like, if any. +/// +/// Recognises a leading `ftyp` box, and the legacy QuickTime layout that +/// leads with `moov`, `mdat`, `wide`, `free`, or `skip` instead — old `.mov` +/// files have no `ftyp` at all. +#[must_use] +pub fn detect(data: &[u8]) -> Option { + let kind = data.get(4..8)?; + + match kind { + b"ftyp" => { + // The major brand follows the box header. QuickTime's is `qt `; + // every other brand in practice means MP4. + let brand = data.get(8..12); + if brand == Some(b"qt ") { + Some(ContainerId::Mov) + } else { + Some(ContainerId::Mp4) + } + } + // No ftyp: legacy QuickTime. + b"moov" | b"mdat" | b"wide" | b"skip" | b"free" => Some(ContainerId::Mov), + _ => None, + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn with_header(kind: &[u8; 4], rest: &[u8]) -> Vec { + let mut out = ((rest.len() + 8) as u32).to_be_bytes().to_vec(); + out.extend_from_slice(kind); + out.extend_from_slice(rest); + out + } + + #[test] + fn an_mp4_brand_is_mp4() { + for brand in [b"isom", b"mp42", b"avc1", b"iso5", b"M4V "] { + let data = with_header(b"ftyp", brand); + assert_eq!(detect(&data), Some(ContainerId::Mp4), "{brand:?}"); + } + } + + #[test] + fn the_quicktime_brand_is_mov() { + let data = with_header(b"ftyp", b"qt "); + assert_eq!(detect(&data), Some(ContainerId::Mov)); + } + + #[test] + fn legacy_quicktime_without_an_ftyp_is_still_recognised() { + for kind in [b"moov", b"mdat", b"wide", b"free", b"skip"] { + let data = with_header(kind, &[0u8; 8]); + assert_eq!(detect(&data), Some(ContainerId::Mov), "{kind:?}"); + } + } + + #[test] + fn unrelated_formats_are_not_claimed() { + // PNG, JPEG, Matroska, RIFF, and plain text must all be rejected. + assert_eq!(detect(b"\x89PNG\r\n\x1a\n\x00\x00\x00\x00"), None); + assert_eq!(detect(&[0xff, 0xd8, 0xff, 0xe0, 0, 0, 0, 0]), None); + assert_eq!(detect(&[0x1a, 0x45, 0xdf, 0xa3, 0, 0, 0, 0]), None); + assert_eq!(detect(b"RIFF____AVI LIST"), None); + assert_eq!(detect(b"this is not a video"), None); + } + + #[test] + fn short_input_is_rejected_rather_than_panicking() { + for len in 0..8 { + assert_eq!(detect(&vec![0u8; len]), None, "len {len}"); + } + } + + #[test] + fn an_ftyp_truncated_before_its_brand_still_reads_as_mp4() { + // The signature is present even though the brand is not; treating it + // as MP4 lets the demuxer produce the real error. + let data = with_header(b"ftyp", &[]); + assert_eq!(detect(&data), Some(ContainerId::Mp4)); + } + + #[test] + fn the_sniffer_trait_agrees_with_detect() { + let mp4 = with_header(b"ftyp", b"isom"); + assert!(IsoBmff::matches(&mp4)); + assert!(!IsoBmff::matches(b"\x89PNG\r\n\x1a\n\x00\x00\x00\x00")); + } +} diff --git a/crates/rawshift-video-isobmff/tests/common/mod.rs b/crates/rawshift-video-isobmff/tests/common/mod.rs new file mode 100644 index 0000000..fa429f0 --- /dev/null +++ b/crates/rawshift-video-isobmff/tests/common/mod.rs @@ -0,0 +1,152 @@ +//! Shared fixture generation for the container integration tests. +//! +//! Fixtures are generated with `ffmpeg` at test time rather than committed, +//! following `TEST_FIXTURES.md`: generated inputs stay out of the repository, +//! and **tests skip gracefully when the tooling is absent**, so `cargo test` +//! passes on a machine with no ffmpeg. + +use std::path::{Path, PathBuf}; +use std::process::Command; + +/// Whether ffmpeg is available to generate fixtures. +pub fn ffmpeg_available() -> bool { + Command::new("ffmpeg") + .arg("-version") + .stdout(std::process::Stdio::null()) + .stderr(std::process::Stdio::null()) + .status() + .is_ok_and(|s| s.success()) +} + +/// Where generated fixtures live: a temp directory, never the repository. +fn fixture_dir() -> PathBuf { + let dir = std::env::temp_dir().join("rawshift-video-fixtures"); + let _ = std::fs::create_dir_all(&dir); + dir +} + +/// Generate a test clip, returning its path. +/// +/// Cached across runs: regenerating clips for every test would dominate the +/// suite's runtime. Returns `None` when ffmpeg is missing or lacks the +/// encoder, which callers treat as a skip — the tests assert rawshift's +/// behaviour, not the machine's codec inventory. +pub fn clip(name: &str, args: &[&str]) -> Option { + if !ffmpeg_available() { + return None; + } + let path = fixture_dir().join(name); + if path.exists() { + return Some(path); + } + + // Generate to a unique name and rename into place. Tests run in parallel + // and share this cache; writing directly to `path` would make it exist — + // and so look ready — while ffmpeg is still filling it, and the other + // tests would open a half-written file. + let staging = path.with_extension(format!( + "tmp.{}.{}", + std::process::id(), + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map_or(0, |d| d.as_nanos()) + )); + + let ok = Command::new("ffmpeg") + .args(["-y", "-loglevel", "error"]) + .args(args) + .arg(&staging) + .status() + .is_ok_and(|s| s.success()); + + if !ok || !staging.exists() { + let _ = std::fs::remove_file(&staging); + return None; + } + // Rename is atomic within a filesystem, so `path` goes from absent to + // complete with nothing observable in between. A concurrent generator + // renaming its own copy over ours is harmless: both are complete. + if std::fs::rename(&staging, &path).is_err() { + let _ = std::fs::remove_file(&staging); + return None; + } + Some(path) +} + +/// A short H.264 clip with B-frames and a 12-frame GOP. +pub fn h264_mp4() -> Option { + clip( + "h264.mp4", + &[ + "-f", + "lavfi", + "-i", + "testsrc2=size=128x96:duration=2:rate=25", + "-c:v", + "libx264", + "-bf", + "3", + "-g", + "12", + "-profile:v", + "high", + "-pix_fmt", + "yuv420p", + ], + ) +} + +/// The same content as HEVC — the case that motivated owning the sample index. +pub fn hevc_mp4() -> Option { + clip( + "hevc.mp4", + &[ + "-f", + "lavfi", + "-i", + "testsrc2=size=128x96:duration=2:rate=25", + "-c:v", + "libx265", + "-x265-params", + "log-level=error", + "-g", + "12", + "-pix_fmt", + "yuv420p", + ], + ) +} + +/// H.264 in a QuickTime container. +pub fn h264_mov() -> Option { + clip( + "h264.mov", + &[ + "-f", + "lavfi", + "-i", + "testsrc2=size=128x96:duration=1:rate=25", + "-c:v", + "libx264", + "-g", + "12", + "-pix_fmt", + "yuv420p", + ], + ) +} + +/// Ask ffprobe for values, as ground truth to compare rawshift against. +pub fn ffprobe(path: &Path, entries: &str, extra: &[&str]) -> Option { + let out = Command::new("ffprobe") + .args(["-v", "error", "-select_streams", "v:0", "-show_entries"]) + .arg(entries) + .args(extra) + .args(["-of", "default=noprint_wrappers=1:nokey=1"]) + .arg(path) + .output() + .ok()?; + out.status + .success() + .then(|| String::from_utf8_lossy(&out.stdout).trim().to_string()) +} diff --git a/crates/rawshift-video-isobmff/tests/demux_fixtures.rs b/crates/rawshift-video-isobmff/tests/demux_fixtures.rs new file mode 100644 index 0000000..83fe1e3 --- /dev/null +++ b/crates/rawshift-video-isobmff/tests/demux_fixtures.rs @@ -0,0 +1,235 @@ +//! End-to-end demuxing of real files, checked against ffprobe. +//! +//! These assert the behaviours that unit tests over synthetic tables cannot: +//! that rawshift agrees with the rest of the world about a file ffmpeg wrote. +//! Every test skips cleanly when ffmpeg is unavailable. + +mod common; + +use std::fs::File; + +use rawshift_video_core::{ContainerId, Demuxer, Track, VideoCodecId}; +use rawshift_video_isobmff::IsoBmffDemuxer; + +/// Skip the test when the fixture could not be generated. +macro_rules! fixture { + ($maker:expr, $what:literal) => { + match $maker { + Some(path) => path, + None => { + eprintln!("skipping: cannot generate the {} fixture (ffmpeg missing or without the encoder)", $what); + return; + } + } + }; +} + +fn open(path: &std::path::Path) -> IsoBmffDemuxer { + IsoBmffDemuxer::open(File::open(path).expect("fixture is readable")).expect("file opens") +} + +fn only_video_track(demuxer: &IsoBmffDemuxer) -> &rawshift_video_core::VideoTrack { + demuxer + .tracks() + .iter() + .find_map(Track::as_video) + .expect("the fixture has a video track") +} + +#[test] +fn h264_in_mp4_reads_its_track_and_every_sample() { + let path = fixture!(common::h264_mp4(), "H.264 MP4"); + let mut demuxer = open(&path); + + assert_eq!(demuxer.container(), ContainerId::Mp4); + + let video = only_video_track(&demuxer); + assert_eq!(video.codec, VideoCodecId::H264); + assert_eq!(video.dimensions.width, 128); + assert_eq!(video.dimensions.height, 96); + // An avcC must be present, or no decoder could be opened for the track. + assert!( + !video.codec_config.is_empty(), + "the track must carry its avcC" + ); + + // 2 seconds at 25 fps. + let mut packets = 0; + while demuxer.next_packet().expect("packets read").is_some() { + packets += 1; + } + assert_eq!(packets, 50, "every sample must be delivered exactly once"); +} + +#[test] +fn hevc_in_mp4_opens_despite_a_multi_kilobyte_hvcc() { + // The regression this crate's rewrite exists for: the previous backend + // capped hvcC at 1 KB, and every real HEVC file failed to open. + let path = fixture!(common::hevc_mp4(), "HEVC MP4"); + let mut demuxer = open(&path); + + let video = only_video_track(&demuxer); + assert_eq!(video.codec, VideoCodecId::Hevc); + assert!( + video.codec_config.as_bytes().len() > 1024, + "this fixture's hvcC should exceed the 1 KB that used to be fatal, got {} bytes", + video.codec_config.as_bytes().len() + ); + + let mut packets = 0; + while demuxer.next_packet().expect("packets read").is_some() { + packets += 1; + } + assert_eq!(packets, 50); +} + +#[test] +fn quicktime_is_distinguished_from_mp4_by_its_brand() { + let path = fixture!(common::h264_mov(), "H.264 MOV"); + let demuxer = open(&path); + assert_eq!(demuxer.container(), ContainerId::Mov); +} + +#[test] +fn presentation_timestamps_agree_with_ffprobe() { + // The edit list must be applied: without it every timestamp carries the + // encoder's reorder delay and rawshift disagrees with every other tool. + let path = fixture!(common::h264_mp4(), "H.264 MP4"); + let mut demuxer = open(&path); + + let first = demuxer + .next_packet() + .expect("first packet") + .expect("the file is not empty"); + + let Some(expected) = common::ffprobe(&path, "packet=pts", &["-read_intervals", "%+#1"]) else { + eprintln!("skipping the ffprobe comparison: ffprobe is unavailable"); + return; + }; + let expected: i64 = expected.trim().parse().expect("ffprobe prints an integer"); + assert_eq!( + first.pts, + Some(expected), + "rawshift's first PTS must match ffprobe's" + ); +} + +#[test] +fn random_access_points_match_the_encoders_gop_length() { + // Derived from the bitstream, not from the container's sync table. A + // 50-frame clip with -g 12 has keyframes at 0, 12, 24, 36 and 48. + let path = fixture!(common::h264_mp4(), "H.264 MP4"); + let mut demuxer = open(&path); + + let mut keyframes = 0; + while let Some(packet) = demuxer.next_packet().expect("packets read") { + if packet.is_keyframe { + keyframes += 1; + } + } + assert_eq!(keyframes, 5, "one per 12-frame GOP across 50 frames"); +} + +#[test] +fn packets_are_delivered_in_decode_order() { + // Decode order is what a decoder must be fed; presentation order is + // recovered afterwards from the timestamps. + let path = fixture!(common::h264_mp4(), "H.264 MP4"); + let mut demuxer = open(&path); + + let mut previous = i64::MIN; + while let Some(packet) = demuxer.next_packet().expect("packets read") { + let dts = packet.dts.expect("MP4 samples always carry a decode time"); + assert!(dts >= previous, "decode timestamps went backwards: {dts}"); + previous = dts; + } +} + +#[test] +fn a_reordered_stream_has_presentation_diverge_from_decode() { + // With -bf 3 the encoder reorders, so at least one packet must be shown + // later than it is decoded. If this ever stops holding, the ctts handling + // has silently become a no-op. + let path = fixture!(common::h264_mp4(), "H.264 MP4"); + let mut demuxer = open(&path); + + let mut reordered = false; + while let Some(packet) = demuxer.next_packet().expect("packets read") { + if packet.pts != packet.dts { + reordered = true; + } + } + assert!( + reordered, + "a B-frame stream must have composition offsets applied" + ); +} + +#[test] +fn seeking_lands_on_a_random_access_point_at_or_before_the_request() { + use rawshift_video_core::{SeekMode, SeekTo}; + + let path = fixture!(common::h264_mp4(), "H.264 MP4"); + let mut demuxer = open(&path); + let track = only_video_track(&demuxer).id; + + // Ask for a point mid-GOP; the demuxer must land at or before it. + let requested = 20_000; + let landed = demuxer + .seek(track, SeekTo::Timestamp(requested), SeekMode::Precise) + .expect("seek succeeds"); + assert!( + landed <= requested, + "landed at {landed}, after the requested {requested}" + ); + + // And the next packet must be decodable from cold, i.e. a keyframe. + let packet = demuxer + .next_packet() + .expect("packet after seek") + .expect("not at end of file"); + assert!( + packet.is_keyframe, + "a seek must land somewhere decoding can resume" + ); +} + +#[test] +fn seeking_an_unknown_track_is_reported_rather_than_ignored() { + use rawshift_video_core::{SeekMode, SeekTo, TrackId, VideoError}; + + let path = fixture!(common::h264_mp4(), "H.264 MP4"); + let mut demuxer = open(&path); + + let err = demuxer + .seek(TrackId::new(999), SeekTo::Timestamp(0), SeekMode::Precise) + .expect_err("track 999 does not exist"); + assert!(matches!(err, VideoError::NoSuchTrack { .. }), "{err}"); +} + +#[test] +fn a_non_isobmff_file_is_rejected_rather_than_half_read() { + use std::io::Cursor; + + let not_a_movie = Cursor::new(b"\x89PNG\r\n\x1a\n and then some bytes".to_vec()); + assert!( + IsoBmffDemuxer::open(not_a_movie).is_err(), + "a PNG must not open as a movie" + ); +} + +#[test] +fn a_truncated_file_fails_to_open_rather_than_panicking() { + use std::io::Cursor; + + let path = fixture!(common::h264_mp4(), "H.264 MP4"); + let whole = std::fs::read(&path).expect("fixture is readable"); + + // Truncating mid-header is the shape of a partial download. + for fraction in [1, 2, 8] { + let cut = whole.len() / fraction / 2; + let truncated = Cursor::new(whole[..cut].to_vec()); + // Either it opens with what it found or it errors; it must not panic. + let _ = IsoBmffDemuxer::open(truncated); + } +} diff --git a/crates/rawshift-video-matroska/Cargo.toml b/crates/rawshift-video-matroska/Cargo.toml new file mode 100644 index 0000000..c9c710c --- /dev/null +++ b/crates/rawshift-video-matroska/Cargo.toml @@ -0,0 +1,23 @@ +[package] +name = "rawshift-video-matroska" +version.workspace = true +edition.workspace = true +rust-version.workspace = true +license.workspace = true +repository.workspace = true +homepage.workspace = true +description = "Matroska and WebM demuxing and metadata for rawshift" +documentation = "https://docs.rs/rawshift-video-matroska" +keywords = ["video", "matroska", "mkv", "webm", "demux"] +categories = ["multimedia::video"] +readme = "README.md" + +[dependencies] +rawshift-core = { workspace = true } +rawshift-video-core = { workspace = true } +rawshift-video-symphonia = { workspace = true } +symphonia-core = { workspace = true } +symphonia-format-mkv = { workspace = true } + +[features] +default = [] diff --git a/crates/rawshift-video-matroska/README.md b/crates/rawshift-video-matroska/README.md new file mode 100644 index 0000000..60942e8 --- /dev/null +++ b/crates/rawshift-video-matroska/README.md @@ -0,0 +1,17 @@ +# rawshift-video-matroska + +Matroska and WebM support for +[rawshift](https://github.com/visualcommons/rawshift). + +Backed by [`symphonia-format-mkv`](https://crates.io/crates/symphonia-format-mkv) +(MPL-2.0, pure Rust, no build script), which maps `V_MPEG4/ISO/AVC` and +`V_MPEGH/ISO/HEVC` tracks and surfaces their `CodecPrivate` — the same `avcC` +and `hvcC` records MP4 carries — as codec extra data. + +As with the ISOBMFF crate, random access points are read from the bitstream +rather than from a container flag, and no symphonia type appears in the public +API. + +## License + +Licensed under [MPL-2.0](../../LICENSE). diff --git a/crates/rawshift-video-matroska/examples/probe.rs b/crates/rawshift-video-matroska/examples/probe.rs new file mode 100644 index 0000000..c4a9ab5 --- /dev/null +++ b/crates/rawshift-video-matroska/examples/probe.rs @@ -0,0 +1,60 @@ +//! Print what the ISOBMFF demuxer sees in a file. +//! +//! `cargo run -p rawshift-video-isobmff --example probe -- FILE` + +use std::fs::File; + +use rawshift_video_core::{Demuxer, Track}; +use rawshift_video_matroska::MatroskaDemuxer; + +fn main() -> Result<(), Box> { + let path = std::env::args().nth(1).ok_or("usage: probe FILE")?; + let mut demuxer = MatroskaDemuxer::open( + File::open(&path)?, + rawshift_video_matroska::detect( + &std::fs::read(&path)?[..64.min(std::fs::metadata(&path)?.len() as usize)], + ) + .ok_or("not matroska")?, + )?; + + println!("container: {}", demuxer.container()); + let md = demuxer.metadata(); + println!("brands: {:?}", md.container.brands); + println!("timescale: {:?}", md.container.timescale); + println!("duration: {:?}", md.container.duration); + println!("created: {:?}", md.container.creation_time); + + for track in demuxer.tracks() { + match track { + Track::Video(v) => println!( + "video #{} {} {}x{} rot={} tb={}/{} frames={:?} dur={:?} cfg={}B", + v.id, + v.codec, + v.dimensions.width, + v.dimensions.height, + v.rotation, + v.time_base.numerator, + v.time_base.denominator, + v.frame_count, + v.duration, + v.codec_config.as_bytes().len(), + ), + Track::Audio(a) => println!("audio #{} {}", a.id, a.codec), + Track::Other(o) => println!("other #{} {:?}", o.id, o.kind), + other => println!("other #{}", other.id()), + } + } + + let (mut packets, mut keyframes, mut first_pts) = (0usize, 0usize, None); + while let Some(p) = demuxer.next_packet()? { + if first_pts.is_none() { + first_pts = p.pts; + } + packets += 1; + if p.is_keyframe { + keyframes += 1; + } + } + println!("packets: {packets} ({keyframes} random access points), first pts {first_pts:?}"); + Ok(()) +} diff --git a/crates/rawshift-video-matroska/src/demux.rs b/crates/rawshift-video-matroska/src/demux.rs new file mode 100644 index 0000000..2692e5a --- /dev/null +++ b/crates/rawshift-video-matroska/src/demux.rs @@ -0,0 +1,130 @@ +//! The Matroska / WebM demuxer. + +use std::io::{Read, Seek}; + +use symphonia_core::formats::{FormatOptions, FormatReader}; +use symphonia_core::io::{MediaSourceStream, MediaSourceStreamOptions}; +use symphonia_format_mkv::MkvReader; + +use rawshift_core::MetadataNamespace; +use rawshift_core::MetadataValue; +use rawshift_video_core::{ + CodecConfig, ContainerId, Demuxer, Packet, Rotation, SeekMode, SeekTo, Track, TrackId, + VideoCodecId, VideoError, VideoMetadata, VideoResult, +}; +use rawshift_video_symphonia as bridge; + +/// The name that appears in [`VideoError::Container`] messages. +const CONTAINER_NAME: &str = "Matroska"; + +/// A demuxer for Matroska and WebM files. +pub struct MatroskaDemuxer { + reader: MkvReader<'static>, + container: ContainerId, + tracks: Vec, + metadata: VideoMetadata, +} + +impl MatroskaDemuxer { + /// Open a Matroska or WebM file. + /// + /// `container` distinguishes the two flavours; use [`crate::detect`] on a + /// prefix of the file to obtain it. + /// + /// # Errors + /// + /// [`VideoError::Container`] for a malformed or unsupported file, and + /// [`VideoError::Io`] for a reader failure. + pub fn open(source: R, container: ContainerId) -> VideoResult + where + R: Read + Seek + Send + Sync + 'static, + { + let stream = MediaSourceStream::new( + Box::new(bridge::SeekableSource::new(source)), + MediaSourceStreamOptions::default(), + ); + let reader = MkvReader::try_new(stream, FormatOptions::default()) + .map_err(|e| VideoError::container(CONTAINER_NAME, e))?; + + // Matroska states rotation in Video/Projection, which the backend does + // not surface; see the crate docs for why that costs nothing here. + let tracks = reader + .tracks() + .iter() + .map(|t| bridge::map_track(t, Rotation::None)) + .collect(); + + let mut metadata = VideoMetadata::default(); + metadata.container.container = Some(container); + let info = reader.media_info(); + if let (Some(tb), Some(dur)) = (info.time_base, info.duration) { + let time_base = bridge::map_time_base(Some(tb)); + metadata.container.duration = bridge::ticks_to_duration(dur.get(), time_base); + metadata.container.timescale = Some(time_base.denominator); + } + metadata.push_container_entry( + MetadataNamespace::Matroska, + "DocType", + MetadataValue::Text( + if container == ContainerId::WebM { + "webm" + } else { + "matroska" + } + .to_string(), + ), + ); + + Ok(Self { + reader, + container, + tracks, + metadata, + }) + } + + fn video_codec_of(&self, track: TrackId) -> Option<(VideoCodecId, &CodecConfig)> { + self.tracks + .iter() + .find(|t| t.id() == track) + .and_then(Track::as_video) + .map(|v| (v.codec, &v.codec_config)) + } +} + +impl Demuxer for MatroskaDemuxer { + fn container(&self) -> ContainerId { + self.container + } + + fn tracks(&self) -> &[Track] { + &self.tracks + } + + fn metadata(&self) -> &VideoMetadata { + &self.metadata + } + + fn next_packet(&mut self) -> VideoResult> { + // Peek the track before consuming, so the keyframe flag can be derived + // from the right track's codec configuration. + let result = self.reader.next_packet(); + let video = match &result { + Ok(Some(p)) => self.video_codec_of(TrackId::new(p.track_id)), + _ => None, + }; + bridge::map_packet_result(CONTAINER_NAME, result, video) + } + + fn seek(&mut self, track: TrackId, to: SeekTo, mode: SeekMode) -> VideoResult { + if self.track(track).is_none() { + return Err(VideoError::NoSuchTrack { id: track }); + } + let (sym_mode, target) = bridge::map_seek(CONTAINER_NAME, track, to, mode)?; + + self.reader + .seek(sym_mode, target) + .map(|landed| landed.actual_ts.get()) + .map_err(|e| VideoError::container(CONTAINER_NAME, e)) + } +} diff --git a/crates/rawshift-video-matroska/src/lib.rs b/crates/rawshift-video-matroska/src/lib.rs new file mode 100644 index 0000000..f36db52 --- /dev/null +++ b/crates/rawshift-video-matroska/src/lib.rs @@ -0,0 +1,28 @@ +//! Matroska and WebM support for rawshift. +//! +//! Backed by `symphonia-format-mkv` — MPL-2.0, pure Rust, no build script — +//! which maps `V_MPEG4/ISO/AVC` and `V_MPEGH/ISO/HEVC` tracks and surfaces +//! their `CodecPrivate` as codec extra data. That `CodecPrivate` holds the +//! same `avcC`/`hvcC` records MP4 carries, so decoders take the same input +//! from either container. +//! +//! As in `rawshift-video-isobmff`, random access points are read from the +//! bitstream rather than from a container flag, and no symphonia type appears +//! in the public API. +//! +//! # Not covered +//! +//! Matroska expresses display rotation through `Video/Projection`, which the +//! backend does not surface; tracks therefore report +//! [`Rotation::None`](rawshift_video_core::Rotation::None). This costs nothing +//! in practice — rotation is a phone-capture concern, and phones write MP4 or +//! QuickTime, not Matroska. + +#![forbid(unsafe_code)] +#![warn(missing_docs)] + +mod demux; +mod sniff; + +pub use demux::MatroskaDemuxer; +pub use sniff::{Matroska, detect}; diff --git a/crates/rawshift-video-matroska/src/sniff.rs b/crates/rawshift-video-matroska/src/sniff.rs new file mode 100644 index 0000000..915611f --- /dev/null +++ b/crates/rawshift-video-matroska/src/sniff.rs @@ -0,0 +1,104 @@ +//! Signature detection for Matroska and WebM. + +use rawshift_video_core::{ContainerId, ContainerSniffer}; + +/// The EBML magic every Matroska and WebM file starts with. +const EBML_MAGIC: [u8; 4] = [0x1a, 0x45, 0xdf, 0xa3]; + +/// Marker for Matroska / WebM detection and demuxing. +pub struct Matroska; + +impl ContainerSniffer for Matroska { + const CONTAINER: ContainerId = ContainerId::Matroska; + + fn matches(data: &[u8]) -> bool { + detect(data).is_some() + } +} + +/// Which Matroska flavour `data` looks like, if any. +/// +/// WebM is Matroska restricted to royalty-free codecs and declares itself in +/// the `DocType` string of the EBML header. That header is small and sits at +/// the very start, so scanning a short prefix for the string is enough and +/// avoids pulling in an EBML parser just to name the flavour. +#[must_use] +pub fn detect(data: &[u8]) -> Option { + if data.len() < 4 || data[..4] != EBML_MAGIC { + return None; + } + + // The DocType lives within the first few dozen bytes; bound the scan so a + // crafted file cannot make this walk the whole buffer. + let horizon = &data[..data.len().min(64)]; + if horizon.windows(4).any(|w| w == b"webm") { + Some(ContainerId::WebM) + } else { + Some(ContainerId::Matroska) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn ebml(doctype: &[u8]) -> Vec { + let mut out = EBML_MAGIC.to_vec(); + out.extend_from_slice(&[0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x1f]); + out.extend_from_slice(&[0x42, 0x82]); // DocType element id + out.push(doctype.len() as u8); + out.extend_from_slice(doctype); + out + } + + #[test] + fn a_matroska_doctype_is_matroska() { + assert_eq!(detect(&ebml(b"matroska")), Some(ContainerId::Matroska)); + } + + #[test] + fn a_webm_doctype_is_webm() { + assert_eq!(detect(&ebml(b"webm")), Some(ContainerId::WebM)); + } + + #[test] + fn the_ebml_magic_is_required() { + // Right length, wrong magic. + let mut wrong = ebml(b"webm"); + wrong[0] = 0x00; + assert_eq!(detect(&wrong), None); + } + + #[test] + fn unrelated_formats_are_not_claimed() { + assert_eq!(detect(b"\x89PNG\r\n\x1a\n"), None); + assert_eq!(detect(b"\x00\x00\x00\x18ftypisom"), None); + assert_eq!(detect(b"RIFF____AVI "), None); + } + + #[test] + fn short_input_is_rejected_rather_than_panicking() { + for len in 0..4 { + assert_eq!(detect(&vec![0x1au8; len]), None, "len {len}"); + } + // Exactly the magic and nothing else: recognised, flavour unknown, so + // the more general one. + assert_eq!(detect(&EBML_MAGIC), Some(ContainerId::Matroska)); + } + + #[test] + fn the_doctype_scan_is_bounded_and_does_not_match_late_content() { + // "webm" appearing far into the file is track content, not a DocType, + // and must not reclassify the container. + let mut data = ebml(b"matroska"); + data.extend(vec![0u8; 4096]); + data.extend_from_slice(b"webm"); + assert_eq!(detect(&data), Some(ContainerId::Matroska)); + } + + #[test] + fn the_sniffer_trait_agrees_with_detect() { + assert!(Matroska::matches(&ebml(b"webm"))); + assert!(!Matroska::matches(b"\x89PNG\r\n\x1a\n")); + } +} diff --git a/crates/rawshift-video-symphonia/Cargo.toml b/crates/rawshift-video-symphonia/Cargo.toml new file mode 100644 index 0000000..f52eda7 --- /dev/null +++ b/crates/rawshift-video-symphonia/Cargo.toml @@ -0,0 +1,21 @@ +[package] +name = "rawshift-video-symphonia" +version.workspace = true +edition.workspace = true +rust-version.workspace = true +license.workspace = true +repository.workspace = true +homepage.workspace = true +description = "Internal symphonia bridge shared by rawshift's container crates" +documentation = "https://docs.rs/rawshift-video-symphonia" +keywords = ["video", "demux", "symphonia", "internal"] +categories = ["multimedia::video"] +readme = "README.md" + +[dependencies] +rawshift-core = { workspace = true } +rawshift-video-core = { workspace = true } +symphonia-core = { workspace = true } + +[features] +default = [] diff --git a/crates/rawshift-video-symphonia/README.md b/crates/rawshift-video-symphonia/README.md new file mode 100644 index 0000000..c0b3bdc --- /dev/null +++ b/crates/rawshift-video-symphonia/README.md @@ -0,0 +1,16 @@ +# rawshift-video-symphonia + +Internal bridge between [symphonia](https://github.com/pdeljanov/Symphonia) and +[rawshift](https://github.com/visualcommons/rawshift)'s video model. + +Not a container crate and not a public API. It exists so +`rawshift-video-isobmff` and `rawshift-video-matroska` share one translation of +symphonia's tracks, packets, timestamps and seek vocabulary instead of two +copies that drift. + +The translation is one-directional and total: no symphonia type leaves this +crate, which is what lets the container crates keep that guarantee too. + +## License + +Licensed under [MPL-2.0](../../LICENSE). diff --git a/crates/rawshift-video-symphonia/src/lib.rs b/crates/rawshift-video-symphonia/src/lib.rs new file mode 100644 index 0000000..e0c8459 --- /dev/null +++ b/crates/rawshift-video-symphonia/src/lib.rs @@ -0,0 +1,424 @@ +//! Internal bridge between symphonia and rawshift's video model. +//! +//! Not a container crate and not a public API: it exists so +//! `rawshift-video-isobmff` and `rawshift-video-matroska` share one translation +//! instead of two copies that drift apart. +//! +//! The translation is one-directional and total — **no symphonia type leaves +//! this crate**. That is what lets the container crates make the same promise, +//! and it is the property to preserve when editing here: every function takes +//! symphonia types and returns rawshift ones. + +#![forbid(unsafe_code)] +#![warn(missing_docs)] + +use std::io::{Read, Seek, SeekFrom}; +use std::time::Duration; + +use symphonia_core::codecs::CodecParameters; +use symphonia_core::codecs::video::well_known as vwk; +use symphonia_core::codecs::video::well_known::extra_data as vxd; +use symphonia_core::codecs::video::{VideoCodecParameters, VideoExtraData}; +use symphonia_core::errors::Error as SymError; +use symphonia_core::formats::{SeekMode as SymSeekMode, SeekTo as SymSeekTo, Track as SymTrack}; +use symphonia_core::io::MediaSource; +use symphonia_core::packet::Packet as SymPacket; +use symphonia_core::units::{Time, TimeBase, Timestamp}; + +use rawshift_core::Dimensions; +use rawshift_core::metadata::URational; +use rawshift_video_core::{ + AudioCodecId, AudioTrack, CodecConfig, OtherTrack, Packet, SeekMode, SeekTo, Track, TrackId, + TrackKind, VideoCodecId, VideoError, VideoResult, VideoTrack, nal, +}; + +/// The time base rawshift assumes when a track states none: milliseconds. +const DEFAULT_TIME_BASE: URational = URational { + numerator: 1, + denominator: 1000, +}; + +/// Adapts any `Read + Seek` source to symphonia's `MediaSource`. +/// +/// rawshift's public API requires `Seek`, so `is_seekable` is unconditionally +/// true. `byte_len` returns `None` rather than seeking to the end on every +/// call: symphonia treats an unknown length as "find out by reading", which is +/// correct, and probing it here would be a hidden syscall on a hot path. +pub struct SeekableSource(R); + +impl SeekableSource { + /// Wrap a source. + pub const fn new(source: R) -> Self { + Self(source) + } +} + +impl Read for SeekableSource { + fn read(&mut self, buf: &mut [u8]) -> std::io::Result { + self.0.read(buf) + } +} + +impl Seek for SeekableSource { + fn seek(&mut self, pos: SeekFrom) -> std::io::Result { + self.0.seek(pos) + } +} + +impl MediaSource for SeekableSource { + fn is_seekable(&self) -> bool { + true + } + + fn byte_len(&self) -> Option { + None + } +} + +/// Translate a symphonia track, using `rotation` for video tracks. +/// +/// Rotation is a parameter because no container surfaces it through +/// symphonia; each container crate recovers it its own way, or passes the +/// default. +#[must_use] +pub fn map_track(track: &SymTrack, rotation: rawshift_video_core::Rotation) -> Track { + let id = TrackId::new(track.id); + let time_base = map_time_base(track.time_base); + let duration = track + .duration + .and_then(|d| ticks_to_duration(d.get(), time_base)); + + match &track.codec_params { + Some(CodecParameters::Video(v)) => { + let mut out = map_video_track(id, v, time_base); + out.duration = duration; + out.frame_count = track.num_frames; + out.rotation = rotation; + Track::Video(out) + } + Some(CodecParameters::Audio(a)) => { + let mut out = AudioTrack::new(id, AudioCodecId::Other, time_base); + out.sample_rate = a.sample_rate; + out.channels = a.channels.as_ref().map(|c| c.count() as u16); + out.duration = duration; + out.language = track.language.clone(); + Track::Audio(out) + } + _ => { + let mut out = OtherTrack::new(id, TrackKind::Data, time_base); + out.duration = duration; + out.language = track.language.clone(); + Track::Other(out) + } + } +} + +fn map_video_track(id: TrackId, params: &VideoCodecParameters, time_base: URational) -> VideoTrack { + let codec = map_video_codec(params.codec); + let dimensions = Dimensions { + width: u32::from(params.width.unwrap_or(0)), + height: u32::from(params.height.unwrap_or(0)), + }; + + let mut track = VideoTrack::new(id, codec, dimensions, time_base); + track.codec_config = CodecConfig::new(config_record(codec, ¶ms.extra_data)); + track +} + +/// Pick the decoder configuration record matching `codec`. +/// +/// Selected by id rather than by position: a track may carry several extra +/// data blobs — a Dolby Vision configuration accompanies an HEVC base layer, +/// for instance — and taking the first would hand a decoder the wrong record. +#[must_use] +pub fn config_record(codec: VideoCodecId, extra: &[VideoExtraData]) -> Vec { + let wanted = match codec { + VideoCodecId::H264 => vxd::VIDEO_EXTRA_DATA_ID_AVC_DECODER_CONFIG, + VideoCodecId::Hevc => vxd::VIDEO_EXTRA_DATA_ID_HEVC_DECODER_CONFIG, + VideoCodecId::Av1 => vxd::VIDEO_EXTRA_DATA_ID_AV1_DECODER_CONFIG, + VideoCodecId::Vp9 => vxd::VIDEO_EXTRA_DATA_ID_VP9_DECODER_CONFIG, + _ => return Vec::new(), + }; + extra + .iter() + .find(|e| e.id == wanted) + .map(|e| e.data.to_vec()) + .unwrap_or_default() +} + +/// Translate a symphonia video codec id. +/// +/// A codec rawshift does not model becomes [`VideoCodecId::Other`] rather than +/// being dropped, so the track still appears in the track list. +#[must_use] +pub fn map_video_codec(codec: symphonia_core::codecs::video::VideoCodecId) -> VideoCodecId { + match codec { + vwk::CODEC_ID_H264 => VideoCodecId::H264, + vwk::CODEC_ID_HEVC => VideoCodecId::Hevc, + vwk::CODEC_ID_AV1 => VideoCodecId::Av1, + vwk::CODEC_ID_VP9 => VideoCodecId::Vp9, + _ => VideoCodecId::Other, + } +} + +/// Translate a symphonia time base, falling back to milliseconds. +#[must_use] +pub fn map_time_base(time_base: Option) -> URational { + time_base.map_or(DEFAULT_TIME_BASE, |tb| { + URational::new(tb.numer.get(), tb.denom.get()) + }) +} + +/// Convert a tick count in `time_base` units to a [`Duration`]. +/// +/// `None` for a zero denominator or a span no `Duration` can hold. +#[must_use] +pub fn ticks_to_duration(ticks: u64, time_base: URational) -> Option { + if time_base.denominator == 0 { + return None; + } + let seconds = ticks as f64 * f64::from(time_base.numerator) / f64::from(time_base.denominator); + Duration::try_from_secs_f64(seconds).ok() +} + +/// Translate a symphonia packet, deriving the keyframe flag from the +/// bitstream. +/// +/// `codec` and `config` describe the packet's track; pass `None` for a +/// non-video track, whose keyframe flag is left false. symphonia exposes no +/// sync-sample flag, and the bitstream is the authoritative source anyway. +#[must_use] +pub fn map_packet(packet: SymPacket, video: Option<(VideoCodecId, &CodecConfig)>) -> Packet { + let data = packet.data.into_vec(); + let is_keyframe = + video.is_some_and(|(codec, config)| nal::is_random_access_point(codec, config, &data)); + + let mut out = Packet::new(TrackId::new(packet.track_id), data); + out.pts = Some(packet.pts.get()); + out.dts = Some(packet.dts.get()); + out.duration = Some(packet.dur.get()); + out.is_keyframe = is_keyframe; + out +} + +/// Translate a `next_packet` result into rawshift's end-of-stream convention. +/// +/// The two demuxers disagree about how a stream ends: the ISOBMFF reader +/// returns `Ok(None)`, while the Matroska reader surfaces an +/// `UnexpectedEof` I/O error. Both mean the same thing, and a caller looping +/// until `Ok(None)` must not see one of them as a failure. +/// +/// # Errors +/// +/// Any genuine container or I/O failure, as [`VideoError`]. +pub fn map_packet_result( + container: &'static str, + result: Result, SymError>, + video: Option<(VideoCodecId, &CodecConfig)>, +) -> VideoResult> { + match result { + Ok(Some(packet)) => Ok(Some(map_packet(packet, video))), + Ok(None) => Ok(None), + // End of stream, spelled as an error by some readers. + Err(SymError::IoError(e)) if e.kind() == std::io::ErrorKind::UnexpectedEof => Ok(None), + Err(e) => Err(VideoError::container(container, e)), + } +} + +/// Translate a seek request. +/// +/// # Errors +/// +/// [`VideoError::Container`] for a `#[non_exhaustive]` target this bridge does +/// not know how to express, which is reported rather than silently +/// approximated to a different position. +pub fn map_seek( + container: &'static str, + track: TrackId, + to: SeekTo, + mode: SeekMode, +) -> VideoResult<(SymSeekMode, SymSeekTo)> { + let target = match to { + SeekTo::Timestamp(ts) => SymSeekTo::Timestamp { + ts: Timestamp::new(ts), + track_id: track.get(), + }, + SeekTo::Time(time) => SymSeekTo::Time { + time: Time::try_from_nanos_u128(time.as_nanos()).unwrap_or(Time::ZERO), + track_id: Some(track.get()), + }, + other => { + return Err(VideoError::container( + container, + format!("unsupported seek target: {other:?}"), + )); + } + }; + + let sym_mode = match mode { + // symphonia's "Accurate" means "at or before the request", which is + // what rawshift calls Precise. + SeekMode::Precise => SymSeekMode::Accurate, + SeekMode::Coarse => SymSeekMode::Coarse, + // SeekMode is #[non_exhaustive]; an unknown mode takes the + // conservative option, which is always safe to resume decoding from. + _ => SymSeekMode::Accurate, + }; + + Ok((sym_mode, target)) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn a_missing_time_base_falls_back_to_milliseconds() { + assert_eq!(map_time_base(None), URational::new(1, 1000)); + } + + #[test] + fn durations_convert_against_the_time_base() { + // 90000 ticks of a 90 kHz base is one second. + assert_eq!( + ticks_to_duration(90_000, URational::new(1, 90_000)), + Some(Duration::from_secs(1)) + ); + assert_eq!( + ticks_to_duration(1_500, URational::new(1, 1_000)), + Some(Duration::from_millis(1_500)) + ); + } + + #[test] + fn a_zero_denominator_does_not_divide_by_zero() { + assert_eq!(ticks_to_duration(1_000, URational::new(1, 0)), None); + } + + #[test] + fn the_config_record_is_selected_by_codec_not_position() { + // An HEVC track carrying a Dolby Vision blob first must still yield + // the hvcC, not whatever sits at index zero. + let extra = vec![ + VideoExtraData { + id: vxd::VIDEO_EXTRA_DATA_ID_DOLBY_VISION_CONFIG, + data: vec![0xDD; 4].into_boxed_slice(), + }, + VideoExtraData { + id: vxd::VIDEO_EXTRA_DATA_ID_HEVC_DECODER_CONFIG, + data: vec![0xAA; 4].into_boxed_slice(), + }, + ]; + assert_eq!(config_record(VideoCodecId::Hevc, &extra), vec![0xAA; 4]); + } + + #[test] + fn a_track_with_no_matching_record_yields_an_empty_one() { + let extra = vec![VideoExtraData { + id: vxd::VIDEO_EXTRA_DATA_ID_AVC_DECODER_CONFIG, + data: vec![0x01; 4].into_boxed_slice(), + }]; + // Asking for HEVC when only an avcC is present must not return it. + assert!(config_record(VideoCodecId::Hevc, &extra).is_empty()); + assert!(config_record(VideoCodecId::ProRes, &extra).is_empty()); + } + + #[test] + fn known_codecs_map_and_unknown_ones_degrade_to_other() { + assert_eq!(map_video_codec(vwk::CODEC_ID_H264), VideoCodecId::H264); + assert_eq!(map_video_codec(vwk::CODEC_ID_HEVC), VideoCodecId::Hevc); + assert_eq!(map_video_codec(vwk::CODEC_ID_AV1), VideoCodecId::Av1); + assert_eq!(map_video_codec(vwk::CODEC_ID_VP9), VideoCodecId::Vp9); + assert_eq!(map_video_codec(vwk::CODEC_ID_MJPEG), VideoCodecId::Other); + } + + #[test] + fn end_of_stream_is_ok_none_however_the_reader_spells_it() { + // The ISOBMFF reader says Ok(None); the Matroska reader raises + // UnexpectedEof. A caller looping until Ok(None) must not see the + // second as a failure. + let explicit: Result, SymError> = Ok(None); + assert!(matches!(map_packet_result("MP4", explicit, None), Ok(None))); + + let as_error: Result, SymError> = Err(SymError::IoError( + std::io::Error::from(std::io::ErrorKind::UnexpectedEof), + )); + assert!(matches!( + map_packet_result("Matroska", as_error, None), + Ok(None) + )); + } + + #[test] + fn a_genuine_io_failure_is_still_an_error() { + // Only UnexpectedEof means end of stream; a permission error must not + // be silently swallowed into a clean end. + let denied: Result, SymError> = Err(SymError::IoError( + std::io::Error::from(std::io::ErrorKind::PermissionDenied), + )); + assert!(map_packet_result("MP4", denied, None).is_err()); + + let malformed: Result, SymError> = Err(SymError::DecodeError("bad atom")); + assert!(map_packet_result("MP4", malformed, None).is_err()); + } + + #[test] + fn seek_modes_translate_with_precise_meaning_at_or_before() { + let (mode, _) = map_seek( + "MP4", + TrackId::new(1), + SeekTo::Timestamp(0), + SeekMode::Precise, + ) + .expect("precise"); + assert_eq!(mode, SymSeekMode::Accurate); + + let (mode, _) = map_seek( + "MP4", + TrackId::new(1), + SeekTo::Timestamp(0), + SeekMode::Coarse, + ) + .expect("coarse"); + assert_eq!(mode, SymSeekMode::Coarse); + } + + #[test] + fn a_time_seek_carries_sub_second_precision() { + let (_, target) = map_seek( + "MP4", + TrackId::new(3), + SeekTo::Time(Duration::from_millis(1_500)), + SeekMode::Precise, + ) + .expect("time seek"); + + match target { + SymSeekTo::Time { time, track_id } => { + assert_eq!(track_id, Some(3)); + // 1.5 s must survive as 1.5 s, not truncate to 1 s. + assert!((time.as_secs_f64() - 1.5).abs() < 1e-9, "{time:?}"); + } + _ => panic!("expected a time seek"), + } + } + + #[test] + fn a_timestamp_seek_is_relative_to_the_named_track() { + let (_, target) = map_seek( + "MKV", + TrackId::new(7), + SeekTo::Timestamp(4_242), + SeekMode::Precise, + ) + .expect("timestamp seek"); + + match target { + SymSeekTo::Timestamp { ts, track_id } => { + assert_eq!(ts.get(), 4_242); + assert_eq!(track_id, 7); + } + _ => panic!("expected a timestamp seek"), + } + } +} diff --git a/crates/rawshift-video/Cargo.toml b/crates/rawshift-video/Cargo.toml index 26f5319..1ad0d05 100644 --- a/crates/rawshift-video/Cargo.toml +++ b/crates/rawshift-video/Cargo.toml @@ -6,49 +6,55 @@ rust-version.workspace = true license.workspace = true repository.workspace = true homepage.workspace = true -description = "Video format support for rawshift (not yet implemented)" +description = "Video decoding and metadata for rawshift: containers, tracks, timelines, and hardware-backed frame decode" documentation = "https://docs.rs/rawshift-video" -keywords = ["video", "raw", "metadata"] +keywords = ["video", "mp4", "matroska", "metadata", "decode"] categories = ["multimedia::video"] readme = "README.md" -# Parked for v1: rawshift v1 ships image only. This crate is an unpublished -# placeholder that holds the workspace slot and roadmap for post-v1 video work. -# It is excluded from the publish set and from the `rawshift` facade until it -# has an implementation to publish. See README.md and release-plz.toml. -publish = false [dependencies] +rawshift-core = { workspace = true } +rawshift-video-core = { workspace = true } +rawshift-video-isobmff = { workspace = true, optional = true } +rawshift-video-matroska = { workspace = true, optional = true } +rawshift-video-hwdec = { workspace = true, optional = true } -# Feature flags mirror the tier structure of `rawshift-image`, so the two -# libraries present a consistent surface. Video is not yet implemented: there -# are no tier-4 implementation features and no dependencies — the tiers below -# exist so the feature tree, public API, and `rawshift` facade can be laid out -# ahead of the decoder work. See the README "Video" section for the roadmap. -# -# tier 1 bundle video, full -# tier 2 format xavc-hs, xavc-s, hevc, h264, prores -# tier 3 direction xavc-hs-decode, … (decode-only for now) [features] default = [] -# ── Tier 1: bundle features ─────────────────────────────────────────────────── -full = ["video"] -video = ["xavc-hs", "xavc-s", "hevc", "h264", "prores"] - -# ── Tier 2: format features ─────────────────────────────────────────────────── -# One per roadmap format. Each is the codec+container pairing produced by a -# supported device (see the device support table in the README). -xavc-hs = ["xavc-hs-decode"] # Sony XAVC HS — H.265/HEVC in MP4 -xavc-s = ["xavc-s-decode"] # Sony XAVC S — H.264/AVC in MP4 -hevc = ["hevc-decode"] # HEVC (H.265) in QuickTime — default iPhone video -h264 = ["h264-decode"] # H.264 (AVC) in QuickTime — legacy/compatibility -prores = ["prores-decode"] # Apple ProRes in QuickTime - -# ── Tier 3: direction features ──────────────────────────────────────────────── -# Decode-only for now. Encode directions and tier-4 implementation features -# will be added when concrete decoder backends land. -xavc-hs-decode = [] -xavc-s-decode = [] -hevc-decode = [] -h264-decode = [] -prores-decode = [] +# ── Tier 1: bundles ────────────────────────────────────────────────────────── +full = ["video", "hw"] +# Every container and codec rawshift implements. +video = ["mp4", "mkv", "h264", "hevc"] + +# ── Tier 2: containers ─────────────────────────────────────────────────────── +# MP4 / M4V / QuickTime. One feature: they share a box grammar and a demuxer. +mp4 = ["dep:rawshift-video-isobmff"] +# Matroska / WebM. +mkv = ["dep:rawshift-video-matroska"] + +# ── Tier 2: codecs ─────────────────────────────────────────────────────────── +h264 = ["h264-decode"] +hevc = ["hevc-decode"] + +# ── Tier 3: directions ─────────────────────────────────────────────────────── +# Decode only. Encode is a finished trait seam with no implementations, so +# there is nothing for an `-encode` feature to gate yet. +h264-decode = ["dep:rawshift-video-hwdec", "rawshift-video-hwdec/h264"] +hevc-decode = ["dep:rawshift-video-hwdec", "rawshift-video-hwdec/hevc"] + +# ── Tier 4: hardware backends ──────────────────────────────────────────────── +# Verified flags, forwarded to rawshift-hwdec: `hw` is portable and selects the +# native backend for the target, the pinned ones fail to compile off-target. +hw = ["rawshift-video-hwdec?/hw"] +hw-vaapi = ["rawshift-video-hwdec?/hw-vaapi"] +hw-videotoolbox = ["rawshift-video-hwdec?/hw-videotoolbox"] +hw-mediacodec = ["rawshift-video-hwdec?/hw-mediacodec"] + +# ── Device-oriented aliases ────────────────────────────────────────────────── +# The roadmap's names, kept so the documented surface survives now that +# container and codec are independent axes. +xavc-hs = ["mp4", "hevc-decode"] # Sony XAVC HS — HEVC in MP4 +xavc-s = ["mp4", "h264-decode"] # Sony XAVC S — H.264 in MP4 + +serde = ["rawshift-video-core/serde"] diff --git a/crates/rawshift-video/README.md b/crates/rawshift-video/README.md index 23f7b96..d931334 100644 --- a/crates/rawshift-video/README.md +++ b/crates/rawshift-video/README.md @@ -1,44 +1,78 @@ # rawshift-video -Video format support for [rawshift](https://github.com/visualcommons/rawshift). - -> **Status: parked for v1, unpublished.** rawshift v1 ships image only. No -> video code ships today, this crate is marked `publish = false`, and it is -> **not** a dependency of the [`rawshift`](https://crates.io/crates/rawshift) -> facade — there is no `video` feature to enable. The crate remains in the -> workspace to hold the roadmap below and the workspace slot for post-v1 work. -> -> It is re-added to the publish set and to the facade when it has an -> implementation to publish. Until then the feature flags below gate no code and -> should be treated as a design sketch, not a supported surface. - -## Roadmap - -The formats below are prioritised by the cameras in rawshift's supported device -list: - -| Format / Codec | Container | Status | Notes | -| -------------------- | --------------- | ------- | ---------------------------------------------- | -| XAVC HS (H.265/HEVC) | MP4 | Planned | Sony mirrorless video. | -| XAVC S (H.264/AVC) | MP4 | Planned | Sony mirrorless video. | -| Apple ProRes | QuickTime (MOV) | Planned | iPhone Pro and professional editing workflows. | -| HEVC (H.265) | QuickTime (MOV) | Planned | Default iPhone video. | -| H.264 (AVC) | QuickTime (MOV) | Planned | Legacy and compatibility video. | - -Initial work will focus on container parsing and metadata extraction, reusing -the in-repo ISOBMFF parser already used for Canon CR3 (both MP4 and QuickTime -are ISOBMFF-based). Codec-level decoding is a later milestone. +Video support for [rawshift](https://github.com/visualcommons/rawshift): +containers, tracks, timelines, metadata, and hardware-backed frame decode. + +The video counterpart to `rawshift-image` — a feature facade over the +container and decoder crates, plus the entry points that tie them together. + +```rust,no_run +use rawshift_video::prelude::*; + +let mut video = VideoFile::open(std::fs::File::open("clip.mp4")?)?; +println!("{} {:?}", video.container(), video.metadata().container.duration); + +for track in video.video_tracks() { + println!("{} {}x{} rotation {}", track.codec, track.dimensions.width, + track.dimensions.height, track.rotation); +} + +if let Some(frame) = video.thumbnail()? { + println!("{}x{}", frame.width(), frame.height()); +} +# Ok::<(), Box>(()) +``` + +## What works without a hardware decoder + +Container parsing, track enumeration, timelines, packet access and metadata +work on **every** target in `docs/SUPPORT.md`, including `wasm32` and `musl` +where no hardware decode API can exist. Only frame decode needs a backend; +without one it reports `VideoError::HwDecoderUnavailable`, and +`hw_decode_available()` lets a caller check before committing. + +## Support + +| Container | Status | Backing | +| --- | --- | --- | +| MP4 / M4V | ✅ | `mp4-atom` boxes, rawshift sample index | +| QuickTime (MOV) | ✅ | the same | +| Matroska / WebM | ✅ | `symphonia-format-mkv` | +| AVI | named, not implemented | no viable Rust crate | +| MXF | named, not implemented | the only `mxf` crate dates from 2017 | + +| Codec | Decode | Notes | +| --- | --- | --- | +| HEVC / H.265 | ✅ random access points | via `rawshift-hwdec`; see below | +| H.264 / AVC | seam ready, no backend path yet | the API is final; the VAAPI picture-parameter path is a follow-up | +| ProRes | named, not implemented | | +| AV1, VP9 | named, not implemented | AV1 stills decode through `rawshift-image` | + +**Decode scope.** Every random access point decodes, which covers keyframe +extraction, poster frames, thumbnails and scrubbing, and decodes All-Intra +camera modes (Sony XAVC S-I, XAVC HS All-I) completely. Inter-frame decoding +is a purely internal change behind the same public API — it adds a decoded +picture buffer and reference handling — so it needs no second breaking +release. + +**No software decode**, deliberately. `docs/SUPPORT.md` rules out shipping +software H.264 or HEVC: both are patent-encumbered independently of any +implementation's own license. Hardware decoders carry the device OEM's +licence. ## Feature Flags -Video features mirror the `rawshift-image` tier structure but currently gate no -code or dependencies. They are a design sketch for post-v1 work and are not -reachable through the `rawshift` facade — see the status note above. +- **Bundles** — `video` (every container and codec), `full` (adds `hw`). +- **Containers** — `mp4`, `mkv`. +- **Codecs** — `h264`, `hevc`. +- **Directions** — `h264-decode`, `hevc-decode`. Encode is a finished trait + seam with no implementations, so no `-encode` flag gates anything yet. +- **Backends** — `hw` (portable), `hw-vaapi` / `hw-videotoolbox` / + `hw-mediacodec` (pinned; these fail to compile off-target by design). +- **Device aliases** — `xavc-hs` (HEVC in MP4), `xavc-s` (H.264 in MP4). -- **Bundles** — `video` (all formats), `full`. -- **Formats** — `xavc-hs`, `xavc-s`, `hevc`, `h264`, `prores`. -- **Directions** — `xavc-hs-decode`, `xavc-s-decode`, `hevc-decode`, - `h264-decode`, `prores-decode` (decode-only for now). +A container feature without a codec feature is a valid metadata-and-timeline +build — the same shape as `heic`/`avif` without `hw` on the image side. ## License diff --git a/crates/rawshift-video/examples/video_inspect.rs b/crates/rawshift-video/examples/video_inspect.rs new file mode 100644 index 0000000..853ccfb --- /dev/null +++ b/crates/rawshift-video/examples/video_inspect.rs @@ -0,0 +1,83 @@ +//! Inspect a video file and decode its first frame, through the public API. +//! +//! ```text +//! cargo run -p rawshift-video --features full --example video_inspect -- FILE +//! ``` +//! +//! Everything except the final decode works on every supported target with no +//! hardware decoder at all; the decode step reports why when none is +//! available rather than failing the run. + +use std::fs::File; + +use rawshift_video::prelude::*; + +fn main() -> Result<(), Box> { + let path = std::env::args().nth(1).ok_or("usage: video_inspect FILE")?; + + let mut video = VideoFile::open(File::open(&path)?)?; + + println!("container : {}", video.container()); + let md = video.metadata(); + if !md.container.brands.is_empty() { + println!("brands : {}", md.container.brands.join(", ")); + } + println!("duration : {:?}", md.container.duration); + if let Some(created) = &md.container.creation_time { + println!("created : {created}"); + } + + println!("\ntracks"); + for track in video.tracks() { + match track { + Track::Video(v) => println!( + " #{} video {} {}x{} rotation {} {:?} frames, config {} bytes", + v.id, + v.codec, + v.dimensions.width, + v.dimensions.height, + v.rotation, + v.frame_count, + v.codec_config.as_bytes().len(), + ), + Track::Audio(a) => println!( + " #{} audio {} {:?} Hz, {:?} channels", + a.id, a.codec, a.sample_rate, a.channels + ), + other => println!(" #{} {:?}", other.id(), other.kind()), + } + } + + println!("\ndecoders compiled in"); + for codec in available_decoders() { + println!(" {} ({})", codec.id, codec.version); + } + + let Some(track) = video.tracks().iter().find_map(Track::as_video) else { + println!("\nno video track to decode"); + return Ok(()); + }; + println!( + "\n{} decode available here: {}", + track.codec, + hw_decode_available(track.codec) + ); + + match video.thumbnail() { + Ok(Some(frame)) => println!( + "first frame: {}x{} {:?} {}-bit, pts {:?}", + frame.width(), + frame.height(), + frame.format(), + frame.bit_depth(), + frame.pts(), + ), + Ok(None) => println!("first frame: none decodable"), + // A missing backend is the expected outcome on most targets, and is + // reported rather than treated as a failure of the file. + Err(e) if e.is_capability_gap() => println!("first frame: unavailable — {e}"), + Err(e) => return Err(e.into()), + } + + Ok(()) +} diff --git a/crates/rawshift-video/src/file.rs b/crates/rawshift-video/src/file.rs new file mode 100644 index 0000000..977d6dc --- /dev/null +++ b/crates/rawshift-video/src/file.rs @@ -0,0 +1,371 @@ +//! [`VideoFile`]: opening a file and working with what is in it. + +use std::io::{Read, Seek}; + +use rawshift_video_core::{ + ContainerId, Demuxer, Packet, SeekMode, SeekTo, Track, TrackId, VideoDecoder, VideoError, + VideoFrame, VideoMetadata, VideoResult, VideoTrack, +}; + +/// A source [`VideoFile`] can read. +/// +/// Blanket-implemented for every `Read + Seek + Send + Sync + 'static`, which +/// covers `File`, `&[u8]` and `Cursor>`. +/// +/// It exists as a **sealed** trait rather than as a bare set of bounds because +/// the bounds are not rawshift's own: the Matroska backend needs `Sync` and +/// `'static`, while the ISOBMFF one does not. Naming the requirement once +/// keeps that backend detail out of rawshift's public contract, so relaxing it +/// later is not a breaking change. +pub trait VideoSource: sealed::Sealed + Read + Seek + Send + Sync + 'static {} + +impl VideoSource for T {} + +mod sealed { + /// Prevents outside implementations, so the supertrait bounds above stay + /// an implementation detail rather than a promise. + pub trait Sealed {} + impl Sealed for T {} +} + +/// How many leading bytes container detection needs. +/// +/// Enough for an ISOBMFF `ftyp` header and brand, and for the EBML header a +/// Matroska `DocType` sits in. +const SNIFF_BYTES: usize = 64; + +/// Identify the container from a prefix of a file. +/// +/// Returns `None` when nothing matches, and may name a container this build +/// cannot open — [`ContainerId`] is a vocabulary of what rawshift can +/// *recognise*, which is deliberately wider than what it can demux. +#[must_use] +pub fn detect_container(data: &[u8]) -> Option { + #[cfg(feature = "mp4")] + if let Some(container) = rawshift_video_isobmff::detect(data) { + return Some(container); + } + #[cfg(feature = "mkv")] + if let Some(container) = rawshift_video_matroska::detect(data) { + return Some(container); + } + let _ = data; + None +} + +/// What [`VideoFile::probe`] reports without committing to a full open. +#[derive(Debug, Clone, PartialEq)] +#[non_exhaustive] +pub struct VideoProbe { + /// The container. + pub container: ContainerId, + /// Every track in the file. + pub tracks: Vec, + /// File-level metadata. + pub metadata: VideoMetadata, +} + +/// An open video file. +/// +/// Not generic over the reader: the containers box their sources internally, +/// so carrying `R` in the type would constrain callers without buying +/// anything. Sources need `Read + Seek + Send`, which `File`, `&[u8]` and +/// `Cursor>` all satisfy. +pub struct VideoFile { + demuxer: Box, +} + +impl std::fmt::Debug for VideoFile { + /// Summarises the file rather than dumping it: the demuxer is a trait + /// object with nothing printable, and the track list can be long. + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("VideoFile") + .field("container", &self.container()) + .field("tracks", &self.tracks().len()) + .field( + "video_tracks", + &self.video_tracks().map(|t| t.codec).collect::>(), + ) + .finish() + } +} + +impl VideoFile { + /// Open a video file. + /// + /// The container is detected from the file's leading bytes, so the caller + /// does not name it. + /// + /// # Errors + /// + /// [`VideoError::UnsupportedContainer`] when the bytes are not a container + /// this build can open — with `detected` naming it when rawshift + /// recognised the format but has no demuxer compiled in, which is the + /// difference between "rebuild with the feature" and "not a video file". + /// [`VideoError::Container`] for a malformed file, and + /// [`VideoError::Io`] for a reader failure. + pub fn open(mut source: R) -> VideoResult { + let mut header = [0u8; SNIFF_BYTES]; + let read = read_prefix(&mut source, &mut header)?; + let detected = detect_container(&header[..read]); + + match detected { + #[cfg(feature = "mp4")] + Some(container) if container.is_isobmff() => { + let demuxer = rawshift_video_isobmff::IsoBmffDemuxer::open(source)?; + Ok(Self { + demuxer: Box::new(demuxer), + }) + } + #[cfg(feature = "mkv")] + Some(container) if container.is_matroska() => { + let demuxer = rawshift_video_matroska::MatroskaDemuxer::open(source, container)?; + Ok(Self { + demuxer: Box::new(demuxer), + }) + } + // Recognised, but this build has no demuxer for it — or rawshift + // names it and implements nothing, as with AVI and MXF. + other => Err(VideoError::UnsupportedContainer { detected: other }), + } + } + + /// Open a file just far enough to report its tracks and metadata. + /// + /// # Errors + /// + /// As [`open`](Self::open). + pub fn probe(source: R) -> VideoResult { + let file = Self::open(source)?; + Ok(VideoProbe { + container: file.container(), + tracks: file.tracks().to_vec(), + metadata: file.metadata().clone(), + }) + } + + /// The container this file is in. + #[must_use] + pub fn container(&self) -> ContainerId { + self.demuxer.container() + } + + /// Every track, in the container's own order. + #[must_use] + pub fn tracks(&self) -> &[Track] { + self.demuxer.tracks() + } + + /// The video tracks. + pub fn video_tracks(&self) -> impl Iterator { + self.tracks().iter().filter_map(Track::as_video) + } + + /// The first video track, which is what "the video" means for the + /// single-stream files cameras produce. + #[must_use] + pub fn primary_video_track(&self) -> Option<&VideoTrack> { + self.video_tracks().next() + } + + /// File-level metadata. + #[must_use] + pub fn metadata(&self) -> &VideoMetadata { + self.demuxer.metadata() + } + + /// The next packet from any track, in stored order. + /// + /// # Errors + /// + /// Container and I/O failures. + pub fn next_packet(&mut self) -> VideoResult> { + self.demuxer.next_packet() + } + + /// Seek `track` to `to`, returning the timestamp landed on. + /// + /// # Errors + /// + /// [`VideoError::NoSuchTrack`] for an unknown track, plus container and + /// I/O failures. + pub fn seek(&mut self, track: TrackId, to: SeekTo, mode: SeekMode) -> VideoResult { + self.demuxer.seek(track, to, mode) + } + + /// Open a decoder for `track`. + /// + /// # Errors + /// + /// [`VideoError::NoSuchTrack`] when the track does not exist or is not + /// video, [`VideoError::UnsupportedCodec`] when no decoder for its codec + /// was compiled in, and [`VideoError::HwDecoderUnavailable`] when one was + /// but no backend on this machine can drive it. + pub fn decoder(&self, track: TrackId) -> VideoResult> { + let video = self + .tracks() + .iter() + .find(|t| t.id() == track) + .and_then(Track::as_video) + .ok_or(VideoError::NoSuchTrack { id: track })?; + crate::registry().open(video) + } + + /// Decode the file's first frame. + /// + /// The video analogue of a still's embedded thumbnail: the cheapest + /// representative image, decoded from the first random access point rather + /// than read from a preview the container may not carry. + /// + /// `Ok(None)` when the file has no video track or no decodable frame. + /// + /// # Errors + /// + /// As [`decoder`](Self::decoder), plus decode failures. + pub fn thumbnail(&mut self) -> VideoResult> { + let Some(track) = self.primary_video_track().map(|t| t.id) else { + return Ok(None); + }; + let mut decoder = self.decoder(track)?; + + while let Some(packet) = self.demuxer.next_packet()? { + if packet.track != track || !packet.is_keyframe { + continue; + } + decoder.send_packet(&packet)?; + if let Some(frame) = decoder.receive_frame()? { + return Ok(Some(frame)); + } + // A decoder that buffered the first picture has nothing more + // coming without an end-of-stream signal. + decoder.flush()?; + if let Some(frame) = decoder.receive_frame()? { + return Ok(Some(frame)); + } + } + Ok(None) + } +} + +/// Read up to `buf.len()` bytes, then rewind. +/// +/// `read` may return fewer bytes than asked for without being at end of file, +/// so this loops; a single call would misdetect a container on a reader that +/// returns short reads. +fn read_prefix(source: &mut R, buf: &mut [u8]) -> VideoResult { + use std::io::SeekFrom; + + let mut filled = 0; + while filled < buf.len() { + match source.read(&mut buf[filled..])? { + 0 => break, + n => filled += n, + } + } + source.seek(SeekFrom::Start(0))?; + Ok(filled) +} + +#[cfg(test)] +mod tests { + use super::*; + use rawshift_video_core::VideoCodecId; + use std::io::Cursor; + + #[test] + fn unrelated_bytes_are_not_a_container() { + assert_eq!(detect_container(b"\x89PNG\r\n\x1a\n\x00\x00\x00\x00"), None); + assert_eq!(detect_container(b"not a video at all"), None); + assert_eq!(detect_container(&[]), None); + } + + #[test] + fn opening_a_non_container_reports_unsupported_with_nothing_detected() { + let err = VideoFile::open(Cursor::new(b"\x89PNG\r\n\x1a\n----".to_vec())) + .expect_err("a PNG is not a video"); + match err { + VideoError::UnsupportedContainer { detected } => { + assert_eq!(detected, None, "nothing should have been recognised"); + } + other => panic!("expected UnsupportedContainer, got {other}"), + } + } + + #[test] + fn a_recognised_but_unimplemented_container_names_itself() { + // The distinction a caller acts on: "rebuild with the feature" or + // "rawshift will never read this", versus "not a video file". + let err = VideoFile::open(Cursor::new(b"RIFF\x00\x00\x00\x00AVI LIST".to_vec())) + .expect_err("AVI is named but not implemented"); + assert!(matches!(err, VideoError::UnsupportedContainer { .. })); + } + + #[test] + fn a_short_reader_does_not_break_detection() { + /// Returns one byte at a time, as a pipe or a slow socket would. + struct Dribble(Cursor>); + impl Read for Dribble { + fn read(&mut self, buf: &mut [u8]) -> std::io::Result { + if buf.is_empty() { + return Ok(0); + } + self.0.read(&mut buf[..1]) + } + } + impl Seek for Dribble { + fn seek(&mut self, pos: std::io::SeekFrom) -> std::io::Result { + self.0.seek(pos) + } + } + + let header = b"\x00\x00\x00\x18ftypisom\x00\x00\x02\x00mp41 padding padding"; + let mut source = Dribble(Cursor::new(header.to_vec())); + let mut buf = [0u8; SNIFF_BYTES]; + let read = read_prefix(&mut source, &mut buf).expect("prefix read"); + + assert_eq!( + read, + header.len(), + "a one-byte-at-a-time reader must still fill the buffer" + ); + assert_eq!( + &buf[..read], + &header[..], + "and fill it with the right bytes" + ); + + // Detection itself is feature-gated; the point above is that the + // prefix reaches it intact however the source dribbles it out. + #[cfg(feature = "mp4")] + assert_eq!(detect_container(&buf[..read]), Some(ContainerId::Mp4)); + } + + #[test] + fn reading_the_prefix_leaves_the_source_rewound() { + // The demuxer that follows starts from byte zero, so detection must + // not consume the header. + let mut source = Cursor::new(b"\x00\x00\x00\x18ftypisom".to_vec()); + let mut buf = [0u8; SNIFF_BYTES]; + read_prefix(&mut source, &mut buf).expect("prefix read"); + assert_eq!(source.position(), 0); + } + + #[test] + fn hw_decode_availability_is_answerable_for_every_codec() { + // Must never panic, whatever this machine has, and must be false for + // codecs rawshift has no decoder for at all. + for codec in [VideoCodecId::H264, VideoCodecId::Hevc] { + let _ = crate::hw_decode_available(codec); + } + assert!(!crate::hw_decode_available(VideoCodecId::ProRes)); + assert!(!crate::hw_decode_available(VideoCodecId::Vp9)); + } + + #[test] + fn the_decoder_registry_only_reports_decode_direction() { + for info in crate::available_decoders() { + assert_eq!(info.direction, rawshift_core::CodecDirection::Decode); + assert!(info.id.id.contains('/'), "ids are \"format/impl\""); + } + } +} diff --git a/crates/rawshift-video/src/lib.rs b/crates/rawshift-video/src/lib.rs index 2d68c6f..b276496 100644 --- a/crates/rawshift-video/src/lib.rs +++ b/crates/rawshift-video/src/lib.rs @@ -1,10 +1,141 @@ -//! Video format support for rawshift. +//! Video decoding and metadata for [rawshift]. //! -//! Video support is **planned but not yet implemented**. This crate is a -//! placeholder so the workspace, feature tree, and public API surface can be -//! laid out ahead of the implementation. See the "Video" section of the -//! project README for the format roadmap (XAVC HS, XAVC S, HEVC, H.264, -//! ProRes). +//! The video counterpart to `rawshift-image`: a feature facade over the +//! container and decoder crates, plus the entry points that tie them together. //! -//! No video decoding or container parsing is available yet. +//! ```no_run +//! use rawshift_video::{VideoFile, Track}; +//! +//! # fn main() -> Result<(), Box> { +//! let mut file = VideoFile::open(std::fs::File::open("clip.mp4")?)?; +//! +//! println!("{}, {:?}", file.container(), file.metadata().container.duration); +//! for track in file.video_tracks() { +//! println!("{} {}x{}", track.codec, track.dimensions.width, track.dimensions.height); +//! } +//! +//! // Decoding needs a hardware backend; everything above does not. +//! if let Some(frame) = file.thumbnail()? { +//! println!("{}x{}", frame.width(), frame.height()); +//! } +//! # Ok(()) } +//! ``` +//! +//! # What works without a decoder +//! +//! Container parsing, track enumeration, the timeline, packet access and +//! metadata work on **every** target in `docs/SUPPORT.md`, including `wasm32` +//! and `musl` where no hardware decode API exists. Only frame decode needs a +//! backend; without one it reports +//! [`VideoError::HwDecoderUnavailable`], which +//! [`hw_decode_available`] lets a caller check up front. +//! +//! # Features +//! +//! - **Bundles** — `video` (every container and codec), `full` (adds `hw`). +//! - **Containers** — `mp4` (MP4/M4V/QuickTime), `mkv` (Matroska/WebM). +//! - **Codecs** — `h264`, `hevc`. +//! - **Backends** — `hw` (portable), `hw-vaapi` / `hw-videotoolbox` / +//! `hw-mediacodec` (pinned; these fail to compile off-target by design). +//! - **Device aliases** — `xavc-hs`, `xavc-s`. +//! +//! A container feature without a codec feature is a valid +//! metadata-and-timeline build, the same shape as `heic`/`avif` without `hw` +//! on the image side. +//! +//! [rawshift]: https://github.com/visualcommons/rawshift + #![forbid(unsafe_code)] +#![warn(missing_docs)] + +mod file; + +pub mod prelude; + +pub use file::{VideoFile, VideoProbe, VideoSource, detect_container}; + +// The vocabulary is re-exported wholesale: callers work in these types, and +// making them reach for `rawshift-video-core` would be a papercut with no +// benefit. +pub use rawshift_video_core::{ + AudioCodecId, AudioTrack, ChromaSubsampling, CicpColor, CodecConfig, ColorRange, ContainerId, + ContainerMetadata, Demuxer, FrameDesc, FramePixelFormat, MatrixCoefficients, Packet, Plane, + Rotation, SeekMode, SeekTo, Track, TrackId, TrackKind, VideoCodecId, VideoDecoder, + VideoDecoderFactory, VideoEncoder, VideoError, VideoFrame, VideoMetadata, VideoResult, + VideoTrack, +}; + +use rawshift_core::{CodecDirection, CodecInfo}; +use rawshift_video_core::DecoderRegistry; + +/// The decoder backends compiled into this build, in priority order. +/// +/// Built once: the set is fixed at compile time, and each factory's +/// `supports()` answers the per-machine question on every call anyway. +fn registry() -> &'static DecoderRegistry { + use std::sync::OnceLock; + static REGISTRY: OnceLock = OnceLock::new(); + REGISTRY.get_or_init(|| { + #[allow(unused_mut)] + let mut registry = DecoderRegistry::new(); + // `dep:` optional dependencies create no implicit feature, so this + // keys off the direction features that pull the crate in. + #[cfg(any(feature = "h264-decode", feature = "hevc-decode"))] + rawshift_video_hwdec::register(&mut registry); + registry + }) +} + +/// Every decoder compiled into this build. +/// +/// Reports what was *compiled in*, not what this machine can run — ask +/// [`hw_decode_available`] for that. Mirrors `rawshift_image::available_decoders`. +#[must_use] +pub fn available_decoders() -> Vec { + registry() + .factories() + .iter() + .map(|f| CodecInfo { + id: f.id(), + // Hardware decoders report the platform API's version at runtime, + // which the seam does not surface; the crate version is the honest + // stand-in and matches how the image registry handles pure-Rust + // backends. + version: env!("CARGO_PKG_VERSION").to_string(), + direction: CodecDirection::Decode, + }) + .collect() +} + +/// Whether a decoder for `codec` can actually run on this machine right now. +/// +/// False when no decoder was compiled in **and** when one was but no backend +/// is usable — a caller that only needs to know "will decoding work" should +/// ask this rather than inspecting features. Mirrors the image side's +/// `heic_hw_decode_available()` / `avif_hw_decode_available()`. +#[must_use] +pub fn hw_decode_available(codec: VideoCodecId) -> bool { + registry() + .factories() + .iter() + .any(|f| f.codec() == codec && f.supports(&probe_track(codec))) +} + +/// A minimal track used only to ask a factory whether it could decode this +/// codec on this machine. +fn probe_track(codec: VideoCodecId) -> VideoTrack { + let mut track = VideoTrack::new( + TrackId::new(0), + codec, + rawshift_core::Dimensions { + width: 1920, + height: 1080, + }, + rawshift_core::metadata::URational::new(1, 1000), + ); + // Factories refuse a track with no configuration record, since no session + // could be opened for it. A non-empty placeholder asks the question this + // function actually means: "is there a backend for this codec". + track.codec_config = CodecConfig::new(vec![0u8; 32]); + track +} diff --git a/crates/rawshift-video/src/prelude.rs b/crates/rawshift-video/src/prelude.rs new file mode 100644 index 0000000..36f103c --- /dev/null +++ b/crates/rawshift-video/src/prelude.rs @@ -0,0 +1,12 @@ +//! The curated surface, for `use rawshift_video::prelude::*;`. +//! +//! Everything needed to open a file, inspect its tracks and metadata, and +//! decode frames — without naming a container or backend crate. + +pub use crate::{VideoFile, VideoProbe, available_decoders, detect_container, hw_decode_available}; + +pub use rawshift_video_core::{ + AudioTrack, ContainerId, Demuxer, Packet, Rotation, SeekMode, SeekTo, Track, TrackId, + TrackKind, VideoCodecId, VideoDecoder, VideoError, VideoFrame, VideoMetadata, VideoResult, + VideoTrack, +}; diff --git a/crates/rawshift/Cargo.toml b/crates/rawshift/Cargo.toml index ffe7b9b..ff19220 100644 --- a/crates/rawshift/Cargo.toml +++ b/crates/rawshift/Cargo.toml @@ -18,33 +18,34 @@ rustdoc-args = ["--cfg", "docsrs"] [dependencies] rawshift-image = { workspace = true, optional = true } -# `rawshift-video` is deliberately absent: it is parked and unpublished for v1. -# A facade that ships to crates.io cannot depend on an unpublished crate, and a -# `video` feature that gates zero code is a promise the workspace cannot keep. -# It is re-added when video has an implementation. +rawshift-video = { workspace = true, optional = true } # The `rawshift` facade exposes only coarse, high-level features. Cargo cannot -# auto-forward a child crate's features, so per-format flags (jpeg, png, arw, …) -# are intentionally NOT re-listed here — depend on `rawshift-image` directly for -# fine-grained control. See the README "Feature Flags" section. +# auto-forward a child crate's features, so per-format flags (jpeg, png, arw, … +# and mp4, mkv, h264, hevc) are intentionally NOT re-listed here — depend on +# `rawshift-image` or `rawshift-video` directly for fine-grained control. See +# the README "Feature Flags" section. [features] default = ["image"] # Enable still-image support with `rawshift-image`'s own default formats. image = ["dep:rawshift-image"] -# Serde derives for metadata/option types (no-op unless `image` is also on). -serde = ["rawshift-image?/serde"] +# Enable video support: containers, tracks, timelines and metadata always, and +# frame decode wherever a hardware backend exists (see docs/SUPPORT.md). +video = ["dep:rawshift-video", "rawshift-video/video"] +# Serde derives for metadata/option types. +serde = ["rawshift-image?/serde", "rawshift-video?/serde"] # Hardware still-frame decode (HEVC for HEIC, AV1 for AVIF) via rawshift-hwdec. # `hw` selects the native backend for the compile target; the `hw-*` flags pin # one explicit backend and fail the compile on any other target. All are # no-ops unless `image` is also on. See docs/SUPPORT.md for the permanent # target/API matrix. -hw = ["rawshift-image?/hw"] -hw-videotoolbox = ["rawshift-image?/hw-videotoolbox"] -hw-vaapi = ["rawshift-image?/hw-vaapi"] -hw-mediacodec = ["rawshift-image?/hw-mediacodec"] -# Everything: every image format, serde, and hardware decode (`hw` is listed -# explicitly so the facade's own flag is active, matching the design contract -# `full` = all formats + serde + experimental + `hw`; `rawshift-image/full` -# already enables it transitively). v1 is image-only — there is no `video` -# feature until `rawshift-video` has an implementation. -full = ["image", "rawshift-image/full", "serde", "hw"] +hw = ["rawshift-image?/hw", "rawshift-video?/hw"] +hw-videotoolbox = ["rawshift-image?/hw-videotoolbox", "rawshift-video?/hw-videotoolbox"] +hw-vaapi = ["rawshift-image?/hw-vaapi", "rawshift-video?/hw-vaapi"] +hw-mediacodec = ["rawshift-image?/hw-mediacodec", "rawshift-video?/hw-mediacodec"] +# Everything: every image format, every video container and codec, serde, and +# hardware decode (`hw` is listed explicitly so the facade's own flag is +# active, matching the design contract `full` = all formats + serde + +# experimental + `hw`; the child crates' `full` already enable it +# transitively). +full = ["image", "rawshift-image/full", "video", "serde", "hw"] diff --git a/crates/rawshift/src/lib.rs b/crates/rawshift/src/lib.rs index f0f8b7c..fe7a2ed 100644 --- a/crates/rawshift/src/lib.rs +++ b/crates/rawshift/src/lib.rs @@ -1,32 +1,22 @@ //! # rawshift //! -//! `rawshift` is a facade crate. It re-exports the workspace's image library -//! behind coarse feature flags so most consumers can depend on a single crate: +//! `rawshift` is a facade crate. It re-exports the workspace's image and +//! video libraries behind coarse feature flags so most consumers can depend on +//! a single crate: //! -//! - **`image`** (default) — re-exports [`rawshift-image`]: RAW decoding for -//! Sony/Canon/Nikon/Fujifilm/Adobe, standard formats (JPEG, PNG, WebP, JXL, -//! GIF, TIFF, AVIF, HEIC, SVG), the full RAW processing pipeline, and -//! encoding. Everything appears at the crate root, e.g. [`formats`], -//! [`core`], [`processing`], [`transforms`], [`prelude`]. +//! - **`image`** (default) — re-exports [`rawshift-image`] **at the crate +//! root**: RAW decoding for Sony/Canon/Nikon/Fujifilm/Adobe, standard +//! formats (JPEG, PNG, WebP, JXL, GIF, TIFF, AVIF, HEIC, SVG), the full RAW +//! processing pipeline, and encoding. Everything appears at the root, e.g. +//! [`formats`], [`core`], [`processing`], [`transforms`], [`prelude`]. +//! - **`video`** — re-exports [`rawshift-video`] under [`video`]: MP4, M4V, +//! QuickTime, Matroska and WebM containers, tracks, timelines and metadata, +//! plus hardware-backed frame decode. //! -//! ## Video is parked for v1 -//! -//! rawshift v1 ships **image only**. `rawshift-video` remains in the workspace -//! as an unpublished placeholder holding the roadmap for post-v1 work, but it -//! is not a dependency of this facade and there is no `video` feature. A -//! feature that gates zero code is a promise the workspace cannot keep, and a -//! published facade cannot depend on an unpublished crate. Both are re-added -//! when video has an implementation. -//! -//! ## Feature flags -//! -//! This facade exposes only `image`, `serde`, the hardware-decode flags -//! (`hw`, `hw-videotoolbox`, `hw-vaapi`, `hw-mediacodec`), and `full`. It does -//! **not** surface per-format flags — Cargo cannot auto-forward a child crate's -//! features, so re-listing them here would be duplicated, rot-prone state. For -//! fine-grained control (individual formats or directions) depend on -//! [`rawshift-image`] directly; its own feature tree is documented on that -//! crate. +//! Video lives under a module rather than at the root because the two +//! libraries share names — `prelude`, `Track`, and their error types — and +//! flattening both would collide. Image stays at the root for source +//! compatibility. //! //! ```no_run //! // Default `image` feature is enough for standard-format decoding. @@ -38,8 +28,46 @@ //! println!("{format:?}: {}x{}", image.width(), image.height()); //! ``` //! +//! ```no_run +//! # #[cfg(feature = "video")] { +//! use rawshift::video::prelude::*; +//! +//! let file = VideoFile::open(std::fs::File::open("clip.mp4").expect("open")).expect("read"); +//! for track in file.video_tracks() { +//! println!("{} {}x{}", track.codec, track.dimensions.width, track.dimensions.height); +//! } +//! # } +//! ``` +//! +//! ## What works without a hardware decoder +//! +//! Container parsing, track enumeration, timelines and metadata work on every +//! target in `docs/SUPPORT.md`, including those with no hardware decode API at +//! all. Only frame decode needs a backend, and its absence is reported as a +//! matchable capability gap rather than a failure. See `docs/SUPPORT.md`. +//! +//! ## Feature flags +//! +//! This facade exposes only `image`, `video`, `serde`, the hardware-decode +//! flags (`hw`, `hw-videotoolbox`, `hw-vaapi`, `hw-mediacodec`), and `full`. +//! It does **not** surface per-format flags — Cargo cannot auto-forward a +//! child crate's features, so re-listing them here would be duplicated, +//! rot-prone state. For fine-grained control (individual formats, containers, +//! codecs, or directions) depend on [`rawshift-image`] or [`rawshift-video`] +//! directly; their own feature trees are documented on those crates. +//! //! [`rawshift-image`]: https://docs.rs/rawshift-image +//! [`rawshift-video`]: https://docs.rs/rawshift-video #![forbid(unsafe_code)] #[cfg(feature = "image")] pub use rawshift_image::*; + +/// Video containers, tracks, metadata and decoding. +/// +/// The whole of [`rawshift-video`](https://docs.rs/rawshift-video); see that +/// crate for the full API. +#[cfg(feature = "video")] +pub mod video { + pub use rawshift_video::*; +} diff --git a/docs/SUPPORT.md b/docs/SUPPORT.md index 24768e7..57e9fe3 100644 --- a/docs/SUPPORT.md +++ b/docs/SUPPORT.md @@ -31,15 +31,28 @@ The list is intentionally minimal. ## Hardware decode APIs -rawshift decodes HEVC (HEIC) and AV1 (AVIF) still-frame codestreams through -platform hardware decoders in the `rawshift-hwdec` crate. The API set is -fixed: - -| API | Status | Platforms | HEVC | AV1 | Linking | -| --- | --- | --- | --- | --- | --- | -| VideoToolbox | ✅ in | macOS 11+, iOS 14+ | ✅ | ✅ runtime-probed (M3+ / A17 Pro+ hardware) | system framework | -| VAAPI (libva) | ✅ in | Linux (gnu) | ✅ Main / Main10 | ✅ AV1 Main (driver-dependent) | dlopen at runtime — absence degrades to "no decoder", never a link failure | -| MediaCodec (NDK) | ✅ in | Android | ✅ | ✅ (device codec; mandated on newer API levels) | NDK | +rawshift decodes codestreams through platform hardware decoders in the +`rawshift-hwdec` crate, on two seams: **still frames** (HEVC for HEIC, AV1 for +AVIF) and **coded video sequences** (HEVC and H.264, for `rawshift-video`). +The API set is fixed: + +| API | Status | Platforms | HEVC | AV1 | H.264 | Linking | +| --- | --- | --- | --- | --- | --- | --- | +| VideoToolbox | ✅ in | macOS 11+, iOS 14+ | ✅ | ✅ runtime-probed (M3+ / A17 Pro+ hardware) | still: n/a · sequence: pending | system framework | +| VAAPI (libva) | ✅ in | Linux (gnu) | ✅ Main / Main10 | ✅ AV1 Main (driver-dependent) | ✅ sequence only (Constrained Baseline / Main / High, 8-bit 4:2:0) | dlopen at runtime — absence degrades to "no decoder", never a link failure | +| MediaCodec (NDK) | ✅ in | Android | ✅ | ✅ (device codec; mandated on newer API levels) | still: n/a · sequence: pending | NDK | + +**Seam scope.** H.264 exists only on the sequence seam: `decoder(HwCodec::H264)` +returns `None` and `available_codecs()` never lists it, because no rawshift +still-image format uses H.264 and shipping an untested path would be dishonest. +Conversely AV1 exists only on the still seam today; AV1 video sequences are a +post-v1 follow-up over the same backend. + +**Progressive frames only.** The sequence seam rejects H.264 `field_pic_flag = 1` +(PAFF / interlaced field coding) with a clear error. Every camera-origin source +in rawshift's device list shoots progressive frames, and paired-field reference +management roughly doubles the decoder state machine for no covered use case. +MBAFF is accepted — it is frame coding and does not affect the picture buffer. VAAPI covers Intel and AMD natively, **and NVIDIA via the maintained [`nvidia-vaapi-driver`](https://github.com/elFarto/nvidia-vaapi-driver) @@ -63,13 +76,39 @@ translation layer over NVDEC**. - **D3D12 / Vulkan Video** — pre-1.0 API churn (Vulkan Video), the largest implementation surface of all options, and no coverage gain over the three chosen APIs on any tier-1 target. +- **FFmpeg / libav** — excluded as a dependency in any form, including as an + optional feature. (1) **License:** FFmpeg is LGPL-2.1+ (GPL once `x264`/`x265` + are enabled). rawshift is MPL-2.0, and static linking an LGPL library into an + iOS application binary leaves the end user no relinking path — the same + redistribution concern that excluded NVDEC above. (2) **Portability:** it + needs autotools plus `nasm`/`yasm` at build time, which cannot be satisfied on + `wasm32-unknown-unknown` at all and would require cross C toolchains in the + `aarch64-apple-ios` and `aarch64-linux-android` build lanes, breaking the + "builds anywhere `cargo` does" property the target list above depends on. + (3) **Coverage:** the codecs rawshift's device list actually produces + (H.264, HEVC, ProRes) are served by platform hardware decoders that the OS + already licenses. Demuxing is served by `symphonia`, which is pure Rust and + MPL-2.0. A `ffmpeg-sidecar`-style subprocess is likewise out of scope: it + moves the license and distribution problem onto the consuming application + rather than solving it. ### Software decode - **HEVC: never.** No acceptable pure-Rust implementation exists, and the - patent posture rules out shipping one. + patent posture rules out shipping one. This covers video sequences as well as + still frames. +- **H.264: never**, for the same reason. H.264 is patent-encumbered + independently of any implementation's own license, and the Cisco OpenH264 + royalty grant covers Cisco's *prebuilt binary*, not a from-source build and + not a clean-slate reimplementation. The pure-Rust `rusty_h264` and `rust_h265` + crates were evaluated in August 2026 and rejected on this ground and on + maturity (both are months old and pre-1.0). Hardware decoders are the answer: + the device OEM already holds the licence. - **AV1: post-v1.** gamut's planned pure-Rust AV1 still decoder will restore AVIF decode on Windows, musl, and wasm as a software fallback. +- **ProRes: undecided.** Unlike H.264/HEVC the bitstream is publicly specified + (SMPTE RDD 36) and intra-only, so a clean-slate safe-Rust decoder is + technically open. Not scoped for v1. ### Behavior without a hardware decoder @@ -78,3 +117,10 @@ every target. Pixel decode without a compiled-in, runtime-probed backend fails with the matchable `RawError::HwDecoderUnavailable`; capability is exposed honestly via `heic_hw_decode_available()` / `avif_hw_decode_available()` and the codec registry. + +The same contract holds for video: container parsing, track enumeration, +timeline, and metadata always work on every target, including `wasm32` and +`musl` where no hardware decoder can exist. Only frame decode requires a +backend, and its absence surfaces as the matchable +`VideoError::HwDecoderUnavailable` with `rawshift_video::hw_decode_available()` +reporting capability up front. diff --git a/justfile b/justfile index 6ff4ef2..697f582 100644 --- a/justfile +++ b/justfile @@ -31,10 +31,11 @@ build-features features: build-image: cargo build -p rawshift --no-default-features --features image -# Build the parked video crate directly — the facade has no `video` feature -# while rawshift-video is parked (see crates/rawshift/Cargo.toml) +# Build the video stack. `--all-features` is wrong here for the same reason as +# `test-all`: the hw-* backend pins are mutually exclusive and would +# compile_error!. `full` is the meaningful everything-build. build-video: - cargo build -p rawshift-video --all-features + cargo build -p rawshift-video --features full # Run tests for the whole workspace (default features) — fetches fixtures first test: setup-test-data @@ -72,6 +73,7 @@ publish-check: cargo doc --workspace --no-deps cargo publish --dry-run -p rawshift-core cargo publish --dry-run -p rawshift-image + cargo publish --dry-run -p rawshift-video-core cargo publish --dry-run -p rawshift-video # Install git hooks diff --git a/release-plz.toml b/release-plz.toml index e85898f..103b84a 100644 --- a/release-plz.toml +++ b/release-plz.toml @@ -10,12 +10,12 @@ # # Publish order is dependency order, which cargo resolves on its own: # -# rawshift-core -> rawshift-hwdec -> rawshift-image-core -> the format -# crates (rawshift-image-{ifd,ljpeg,metadata,arw,…}) -> rawshift-image -# -> rawshift -# -# `rawshift-video` is parked and unpublished for v1 (`publish = false`) and so -# is absent from that order. +# rawshift-core -> rawshift-hwdec +# -> rawshift-image-core -> the image format crates +# (rawshift-image-{ifd,ljpeg,metadata,arw,…}) -> rawshift-image +# -> rawshift-video-core -> rawshift-video-{symphonia,isobmff,matroska,hwdec} +# -> rawshift-video +# -> rawshift [workspace] # The lockstep defaults, applied to *every* member so a newly added crate is @@ -62,8 +62,4 @@ git_release_enable = true git_tag_name = "v{{ version }}" # e.g. v0.1.0 (not rawshift-v0.1.0) git_release_name = "v{{ version }}" -[[package]] -name = "rawshift-video" -# Parked for v1: the crate carries `publish = false`, so release-plz must not -# try to release it either. -publish = false +