From 4d4d823c665aefde06b2c95997155916b0446795 Mon Sep 17 00:00:00 2001 From: Justin Chung Date: Fri, 28 Aug 2026 21:38:26 -0400 Subject: [PATCH 01/13] docs: carve video out of the gamut upstream-first policy gamut has declared video permanently out of scope ("gamut will not grow video primitives"), and gamut-isobmff parses the HEIF still-image item model rather than the movie model, so it cannot back MP4/MOV video. The Upstream-First Policy read as though it governed video work too, which would have blocked rawshift-video on issues gamut would decline. Record the carve-out, and with it the decisions it implies: - Video containers and codecs take third-party dependencies judged by the PRINCIPLES.md maturity rule rather than the upstream-first rule. - FFmpeg/libav is excluded in any form, on license (LGPL reach into an MPL-2.0 library with no iOS relinking path) and portability grounds (autotools + nasm cannot satisfy the wasm32 and mobile build lanes). - Software H.264 joins software HEVC as never: both are patent-encumbered independently of an implementation's own license, and the OpenH264 royalty grant covers Cisco's prebuilt binary rather than a from-source or clean-slate build. Hardware decoders carry the OEM's licence. - The hardware decode matrix gains H.264 on the sequence seam only, and states that container parsing and metadata keep working on every target with no backend at all. Also fixes the safety boundaries for the crates this path introduces. Refs #39 --- AGENTS.md | 37 +++++++++++++++++++++++++++ PRINCIPLES.md | 6 ++++- docs/SUPPORT.md | 66 +++++++++++++++++++++++++++++++++++++++++-------- 3 files changed, 98 insertions(+), 11 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index d2368c9..909084c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -38,6 +38,43 @@ dependencies are not permitted because they prevent publishing rawshift. that only updates the version requirements and lockfile, a full test + benchmark run, and a CHANGELOG.md note for any behavioral change. +### Video is outside gamut's charter + +The policy above governs **image** work. It does not apply to video, because +gamut has declared video permanently out of scope. From gamut's README: + +> **gamut is image-first.** Even where a format's codec (AV1, AV2, VVC, HEVC) +> is fundamentally a video codec, gamut implements only the intra-frame, +> still-image subset those formats use — no inter-frame prediction, no motion +> compensation, no video sequences. […] gamut will not grow video primitives. + +`gamut-isobmff` says the same at the container level: *"Image sequences/tracks +and item protection are out of scope."* It parses the HEIF still-image item +model (`meta`/`iloc`/`iinf`/`iprp`), not the movie model (`moov`/`trak`/`stbl`), +so it cannot back MP4/MOV video however much the two formats share a box +grammar. + +Consequently, for `rawshift-video` and the `rawshift-video-*` crates: + +- Do **not** open gamut issues for video containers, video codecs, or + multi-frame primitives. They will be declined as out of charter, and + `blocked-upstream` is the wrong label for them. +- Third-party dependencies are permitted where a mature library exists — + judged by the `PRINCIPLES.md` rule ("reinvent the wheel only when + necessary"), not by the upstream-first rule. Current choices: + `symphonia-format-isomp4` and `symphonia-format-mkv` for demuxing (MPL-2.0, + matching rawshift's own license; pure Rust with no build script). +- Video **decode** stays in `rawshift-hwdec`. `docs/SUPPORT.md` fixes the + reasons: software HEVC is ruled out by the patent posture, and FFmpeg/libav + is ruled out on license and portability grounds (see "Excluded" there). +- Everything else in this file still applies to video: no git dependencies, + the fixed target list, the MSRV rule, and a CHANGELOG note for behavioral + change. + +Image work is unaffected — gamut remains the upstream home for image +primitives, color, metadata, still-image container parsing, and codecs, and the +upstream-first sequence above is still mandatory there. + ## Testing Methodology Try to unit test the bulk majority of the code but functions that take in external inputs such as image/video file(s) should use test fixtures derived from external sources (which may require human sourcing as prerequisite). Also extend example binaries in `examples/` as necessary to show that each feature actually works. diff --git a/PRINCIPLES.md b/PRINCIPLES.md index 0a55769..02769ff 100644 --- a/PRINCIPLES.md +++ b/PRINCIPLES.md @@ -2,7 +2,7 @@ - Stateless: The library should assume nothing about the state to support portability and parallelization. - Separation of IO and CPU: Writing good IO-heavy and CPU-heavy code can be tough in different ways so we separate it where possible to simplify benching. -- Reinvent the wheel only when necessary: We should aim to use existing mature libraries for functionality; but often there are libraries that are either: lacking low-level features, non-performant for specific use cases, or insufficiently mature. For image primitives, codecs, containers, and metadata, the mature library is [gamut](https://github.com/visualcommons/gamut) — improve it upstream rather than reimplementing here (see the Upstream-First Policy in AGENTS.md). Accepted exceptions: GIF (`gif`), SVG (`resvg`), PPM (`zune-ppm`). +- Reinvent the wheel only when necessary: We should aim to use existing mature libraries for functionality; but often there are libraries that are either: lacking low-level features, non-performant for specific use cases, or insufficiently mature. For image primitives, codecs, containers, and metadata, the mature library is [gamut](https://github.com/visualcommons/gamut) — improve it upstream rather than reimplementing here (see the Upstream-First Policy in AGENTS.md). Accepted exceptions: GIF (`gif`), SVG (`resvg`), PPM (`zune-ppm`). For **video** containers and codecs there is no upstream to improve — gamut's charter excludes video (see "Video is outside gamut's charter" in AGENTS.md) — so this principle applies directly: `symphonia` backs demuxing, and platform hardware decoders back frame decode. ## Safety Boundaries @@ -13,6 +13,10 @@ - `crates/rawshift-image/src/processing`: Unsafe Rust is acceptable as long as it is constrained to hot paths. - `crates/rawshift-image/src/transforms`: Unsafe Rust is acceptable as long as it is constrained to hot paths. - `crates/rawshift-hwdec`: Unsafe platform FFI is permitted — `#![deny(unsafe_op_in_unsafe_fn)]`, every public item is a safe wrapper, every unsafe block documents its invariants. + - `crates/rawshift-hwdec/src/codec`: Safe Rust is strictly required. Bitstream parsing, picture-order derivation, and reference-picture management are pure functions over bytes; they carry no platform types and must stay testable on every target, so they live outside the backend `cfg` and outside the FFI. +- `crates/rawshift-video-core`: `#![forbid(unsafe_code)]`. +- `crates/rawshift-video-*`: Safe Rust is strictly required. +- `crates/rawshift-video`: Safe Rust is strictly required. All platform FFI for video decode lives in `rawshift-hwdec`, never here. - `**/**`: TBD ## Testing Strategy diff --git a/docs/SUPPORT.md b/docs/SUPPORT.md index 24768e7..57e9fe3 100644 --- a/docs/SUPPORT.md +++ b/docs/SUPPORT.md @@ -31,15 +31,28 @@ The list is intentionally minimal. ## Hardware decode APIs -rawshift decodes HEVC (HEIC) and AV1 (AVIF) still-frame codestreams through -platform hardware decoders in the `rawshift-hwdec` crate. The API set is -fixed: - -| API | Status | Platforms | HEVC | AV1 | Linking | -| --- | --- | --- | --- | --- | --- | -| VideoToolbox | ✅ in | macOS 11+, iOS 14+ | ✅ | ✅ runtime-probed (M3+ / A17 Pro+ hardware) | system framework | -| VAAPI (libva) | ✅ in | Linux (gnu) | ✅ Main / Main10 | ✅ AV1 Main (driver-dependent) | dlopen at runtime — absence degrades to "no decoder", never a link failure | -| MediaCodec (NDK) | ✅ in | Android | ✅ | ✅ (device codec; mandated on newer API levels) | NDK | +rawshift decodes codestreams through platform hardware decoders in the +`rawshift-hwdec` crate, on two seams: **still frames** (HEVC for HEIC, AV1 for +AVIF) and **coded video sequences** (HEVC and H.264, for `rawshift-video`). +The API set is fixed: + +| API | Status | Platforms | HEVC | AV1 | H.264 | Linking | +| --- | --- | --- | --- | --- | --- | --- | +| VideoToolbox | ✅ in | macOS 11+, iOS 14+ | ✅ | ✅ runtime-probed (M3+ / A17 Pro+ hardware) | still: n/a · sequence: pending | system framework | +| VAAPI (libva) | ✅ in | Linux (gnu) | ✅ Main / Main10 | ✅ AV1 Main (driver-dependent) | ✅ sequence only (Constrained Baseline / Main / High, 8-bit 4:2:0) | dlopen at runtime — absence degrades to "no decoder", never a link failure | +| MediaCodec (NDK) | ✅ in | Android | ✅ | ✅ (device codec; mandated on newer API levels) | still: n/a · sequence: pending | NDK | + +**Seam scope.** H.264 exists only on the sequence seam: `decoder(HwCodec::H264)` +returns `None` and `available_codecs()` never lists it, because no rawshift +still-image format uses H.264 and shipping an untested path would be dishonest. +Conversely AV1 exists only on the still seam today; AV1 video sequences are a +post-v1 follow-up over the same backend. + +**Progressive frames only.** The sequence seam rejects H.264 `field_pic_flag = 1` +(PAFF / interlaced field coding) with a clear error. Every camera-origin source +in rawshift's device list shoots progressive frames, and paired-field reference +management roughly doubles the decoder state machine for no covered use case. +MBAFF is accepted — it is frame coding and does not affect the picture buffer. VAAPI covers Intel and AMD natively, **and NVIDIA via the maintained [`nvidia-vaapi-driver`](https://github.com/elFarto/nvidia-vaapi-driver) @@ -63,13 +76,39 @@ translation layer over NVDEC**. - **D3D12 / Vulkan Video** — pre-1.0 API churn (Vulkan Video), the largest implementation surface of all options, and no coverage gain over the three chosen APIs on any tier-1 target. +- **FFmpeg / libav** — excluded as a dependency in any form, including as an + optional feature. (1) **License:** FFmpeg is LGPL-2.1+ (GPL once `x264`/`x265` + are enabled). rawshift is MPL-2.0, and static linking an LGPL library into an + iOS application binary leaves the end user no relinking path — the same + redistribution concern that excluded NVDEC above. (2) **Portability:** it + needs autotools plus `nasm`/`yasm` at build time, which cannot be satisfied on + `wasm32-unknown-unknown` at all and would require cross C toolchains in the + `aarch64-apple-ios` and `aarch64-linux-android` build lanes, breaking the + "builds anywhere `cargo` does" property the target list above depends on. + (3) **Coverage:** the codecs rawshift's device list actually produces + (H.264, HEVC, ProRes) are served by platform hardware decoders that the OS + already licenses. Demuxing is served by `symphonia`, which is pure Rust and + MPL-2.0. A `ffmpeg-sidecar`-style subprocess is likewise out of scope: it + moves the license and distribution problem onto the consuming application + rather than solving it. ### Software decode - **HEVC: never.** No acceptable pure-Rust implementation exists, and the - patent posture rules out shipping one. + patent posture rules out shipping one. This covers video sequences as well as + still frames. +- **H.264: never**, for the same reason. H.264 is patent-encumbered + independently of any implementation's own license, and the Cisco OpenH264 + royalty grant covers Cisco's *prebuilt binary*, not a from-source build and + not a clean-slate reimplementation. The pure-Rust `rusty_h264` and `rust_h265` + crates were evaluated in August 2026 and rejected on this ground and on + maturity (both are months old and pre-1.0). Hardware decoders are the answer: + the device OEM already holds the licence. - **AV1: post-v1.** gamut's planned pure-Rust AV1 still decoder will restore AVIF decode on Windows, musl, and wasm as a software fallback. +- **ProRes: undecided.** Unlike H.264/HEVC the bitstream is publicly specified + (SMPTE RDD 36) and intra-only, so a clean-slate safe-Rust decoder is + technically open. Not scoped for v1. ### Behavior without a hardware decoder @@ -78,3 +117,10 @@ every target. Pixel decode without a compiled-in, runtime-probed backend fails with the matchable `RawError::HwDecoderUnavailable`; capability is exposed honestly via `heic_hw_decode_available()` / `avif_hw_decode_available()` and the codec registry. + +The same contract holds for video: container parsing, track enumeration, +timeline, and metadata always work on every target, including `wasm32` and +`musl` where no hardware decoder can exist. Only frame decode requires a +backend, and its absence surfaces as the matchable +`VideoError::HwDecoderUnavailable` with `rawshift_video::hw_decode_available()` +reporting capability up front. From de07053199441da8bf23c0fcbedcb345bfde10da Mon Sep 17 00:00:00 2001 From: Justin Chung Date: Fri, 28 Aug 2026 21:39:56 -0400 Subject: [PATCH 02/13] feat(core)!: add Quicktime and Matroska metadata namespaces MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit rawshift-core's module docs already claim ImageMetadata and its MetadataNamespace parts are "genuinely video-shared". rawshift-video is the first consumer of that claim, and it needs somewhere to put container-level facts: moov-tree values and the com.apple.quicktime.* udta/mdta keys iOS writes, and Matroska Tags/Info/TrackEntry elements. BREAKING CHANGE: MetadataNamespace is now #[non_exhaustive] and has two new variants. Adding a variant to an open enum is itself breaking, so both happen once, now, while the workspace is pre-1.0 — after which new namespaces are additive forever. Downstream matches need a `_` arm. No in-tree match on MetadataNamespace was exhaustive, so no other crate changes. Refs #39 --- crates/rawshift-core/src/lib.rs | 4 ++-- crates/rawshift-core/src/metadata.rs | 11 +++++++++++ 2 files changed, 13 insertions(+), 2 deletions(-) diff --git a/crates/rawshift-core/src/lib.rs b/crates/rawshift-core/src/lib.rs index 2595ef6..d55e673 100644 --- a/crates/rawshift-core/src/lib.rs +++ b/crates/rawshift-core/src/lib.rs @@ -25,8 +25,8 @@ //! now keeps the boundary from drifting into "everything shared-ish lives in //! core". The division is: //! -//! **Genuinely video-shared** — media-agnostic, and video will consume these -//! as-is: +//! **Genuinely video-shared** — media-agnostic, and consumed as-is by +//! `rawshift-video-core` and the `rawshift-video-*` crates: //! //! - Geometry — [`image::Dimensions`], [`image::Point`], [`image::Rect`]. //! - Codec descriptors — [`codec::CodecId`], [`codec::CodecInfo`], diff --git a/crates/rawshift-core/src/metadata.rs b/crates/rawshift-core/src/metadata.rs index e048093..ac22154 100644 --- a/crates/rawshift-core/src/metadata.rs +++ b/crates/rawshift-core/src/metadata.rs @@ -250,8 +250,13 @@ pub enum MetadataValue { } /// Namespace identifying the origin of a generic metadata tag. +/// +/// Marked `#[non_exhaustive]`: new namespaces arrive whenever a new container +/// or format grows container-level facts worth surfacing, and that must not be +/// a breaking change. Match with a `_` arm. #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] #[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))] +#[non_exhaustive] pub enum MetadataNamespace { /// Standard TIFF/EXIF IFD tags. Exif, @@ -267,6 +272,12 @@ pub enum MetadataNamespace { Heic, /// AVIF container-level facts. Avif, + /// QuickTime / ISOBMFF movie-level facts — `moov`-tree values and the + /// `com.apple.quicktime.*` `udta`/`mdta` keys iOS writes. + Quicktime, + /// Matroska / WebM segment-level facts — `Tags`, `Info`, and `TrackEntry` + /// elements. + Matroska, /// Vendor/format-specific, identified by the accompanying tag string. Other, } From 409bda922c6316db0f2fb6d0cbd6263a02ba3723 Mon Sep 17 00:00:00 2001 From: Justin Chung Date: Fri, 28 Aug 2026 21:51:04 -0400 Subject: [PATCH 03/13] feat(video-core): add the shared vocabulary for the video crates MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The video counterpart to rawshift-image-core: errors, format and codec identity, the track/packet/frame model, container metadata, and the demux/decode contracts. No I/O, no platform code, no container or codec dependencies, #![forbid(unsafe_code)]. It has to be its own crate rather than a reuse of rawshift-image-core because CI enforces that no rawshift-image* crate appears in rawshift-video's dependency tree; image and video share only rawshift-core. Design points worth calling out: - Identity enums name more than rawshift can decode (AVI, MXF, ProRes), so "recognised but unsupported" is expressible and adding support later is additive. All are #[non_exhaustive]. - VideoError separates a capability gap from a broken file, and distinguishes "no decoder compiled in" (UnsupportedCodec) from "no backend on this machine" (HwDecoderUnavailable) — callers act on those differently. Backend error types are stringified at the leaf so none crosses a public boundary. - Colour is stored as raw CICP code points rather than typed enums, so a code point no library names yet survives instead of collapsing to "unspecified", with resolved_matrix() applying the universal SD-means-BT.601 / HD-means-BT.709 fallback. - VideoDecoder splits send_packet from receive_frame because coded and display order differ whenever a stream reorders, and because that is the shape every hardware decode API natively has. - DecoderRegistry makes supports()==false the only signal that falls through to the next backend, so a backend that accepts a track and then faults is reported rather than silently masked by a later one. This mirrors gamut-codec-abi's documented fallback contract. The track structs deliberately do not derive serde: they embed upstream gamut geometry and colour types that have no serde support, and skipping those fields would serialize a track that lost its resolution and pixel format. Refs #39 --- Cargo.lock | 10 + Cargo.toml | 2 + crates/rawshift-video-core/Cargo.toml | 23 + crates/rawshift-video-core/README.md | 21 + crates/rawshift-video-core/src/decode.rs | 492 ++++++++++++++++++ crates/rawshift-video-core/src/demux.rs | 154 ++++++ crates/rawshift-video-core/src/error.rs | 218 ++++++++ crates/rawshift-video-core/src/frame.rs | 576 ++++++++++++++++++++ crates/rawshift-video-core/src/id.rs | 257 +++++++++ crates/rawshift-video-core/src/lib.rs | 64 +++ crates/rawshift-video-core/src/metadata.rs | 133 +++++ crates/rawshift-video-core/src/packet.rs | 156 ++++++ crates/rawshift-video-core/src/track.rs | 577 +++++++++++++++++++++ 13 files changed, 2683 insertions(+) create mode 100644 crates/rawshift-video-core/Cargo.toml create mode 100644 crates/rawshift-video-core/README.md create mode 100644 crates/rawshift-video-core/src/decode.rs create mode 100644 crates/rawshift-video-core/src/demux.rs create mode 100644 crates/rawshift-video-core/src/error.rs create mode 100644 crates/rawshift-video-core/src/frame.rs create mode 100644 crates/rawshift-video-core/src/id.rs create mode 100644 crates/rawshift-video-core/src/lib.rs create mode 100644 crates/rawshift-video-core/src/metadata.rs create mode 100644 crates/rawshift-video-core/src/packet.rs create mode 100644 crates/rawshift-video-core/src/track.rs diff --git a/Cargo.lock b/Cargo.lock index f33e310..4ed7deb 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1488,6 +1488,16 @@ dependencies = [ name = "rawshift-video" version = "0.1.1" +[[package]] +name = "rawshift-video-core" +version = "0.1.1" +dependencies = [ + "gamut-color", + "rawshift-core", + "serde", + "thiserror", +] + [[package]] name = "rayon" version = "1.11.0" diff --git a/Cargo.toml b/Cargo.toml index dd400d3..5263c40 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -27,6 +27,7 @@ members = [ "crates/rawshift-image-tiff", "crates/rawshift-image-webp", "crates/rawshift-video", + "crates/rawshift-video-core", ] [workspace.package] @@ -93,6 +94,7 @@ rawshift-image-svg = { path = "crates/rawshift-image-svg", version = "0.1.1", de rawshift-image-tiff = { path = "crates/rawshift-image-tiff", version = "0.1.1", default-features = false } rawshift-image-webp = { path = "crates/rawshift-image-webp", version = "0.1.1", default-features = false } rawshift-video = { path = "crates/rawshift-video", version = "0.1.1" } +rawshift-video-core = { path = "crates/rawshift-video-core", version = "0.1.1" } # Profiles must live at the workspace root — Cargo ignores profiles declared in # member manifests. diff --git a/crates/rawshift-video-core/Cargo.toml b/crates/rawshift-video-core/Cargo.toml new file mode 100644 index 0000000..e1dc8b0 --- /dev/null +++ b/crates/rawshift-video-core/Cargo.toml @@ -0,0 +1,23 @@ +[package] +name = "rawshift-video-core" +version.workspace = true +edition.workspace = true +rust-version.workspace = true +license.workspace = true +repository.workspace = true +homepage.workspace = true +description = "Shared vocabulary for rawshift's video crates: errors, identity, tracks, packets, frames, and the demux/decode contracts" +documentation = "https://docs.rs/rawshift-video-core" +keywords = ["video", "demux", "container", "codec"] +categories = ["multimedia::video"] +readme = "README.md" + +[dependencies] +rawshift-core = { workspace = true } +gamut-color = { workspace = true } +thiserror = { workspace = true } +serde = { workspace = true, optional = true } + +[features] +default = [] +serde = ["dep:serde", "rawshift-core/serde"] diff --git a/crates/rawshift-video-core/README.md b/crates/rawshift-video-core/README.md new file mode 100644 index 0000000..9c9e649 --- /dev/null +++ b/crates/rawshift-video-core/README.md @@ -0,0 +1,21 @@ +# rawshift-video-core + +Shared vocabulary for [rawshift](https://github.com/visualcommons/rawshift)'s +video crates — the video counterpart to `rawshift-image-core`. + +This crate holds the types every `rawshift-video-*` crate agrees on and no +implementation of its own: errors, format and codec identity, the track / +packet / frame model, container-level metadata, and the `Demuxer` / +`VideoDecoder` / `VideoEncoder` contracts. + +It exists because the video crates cannot reuse `rawshift-image-core`: the +workspace enforces (in CI) that no `rawshift-image*` crate appears in +`rawshift-video`'s dependency tree, so the two media share only +`rawshift-core`. + +`#![forbid(unsafe_code)]`. No I/O, no platform code, no container or codec +dependencies. + +## License + +Licensed under [MPL-2.0](../../LICENSE). diff --git a/crates/rawshift-video-core/src/decode.rs b/crates/rawshift-video-core/src/decode.rs new file mode 100644 index 0000000..0e26760 --- /dev/null +++ b/crates/rawshift-video-core/src/decode.rs @@ -0,0 +1,492 @@ +//! The decoding contracts and the backend registry. + +use rawshift_core::CodecId; + +use crate::error::{VideoError, VideoResult}; +use crate::frame::VideoFrame; +use crate::id::VideoCodecId; +use crate::packet::Packet; +use crate::track::VideoTrack; + +/// A decoder for one video track. +/// +/// **Send-packet / receive-frame**, not `decode(packet) -> frame`. Coded order +/// and display order differ whenever a stream reorders (B pictures, +/// hierarchical GOPs), so one packet may make zero, one, or several frames +/// available. The split is also the shape every hardware decode API natively +/// has. +/// +/// Drive it as: +/// +/// ```text +/// while let Some(packet) = demuxer.next_packet()? { +/// decoder.send_packet(&packet)?; +/// while let Some(frame) = decoder.receive_frame()? { sink(frame); } +/// } +/// decoder.flush()?; +/// while let Some(frame) = decoder.receive_frame()? { sink(frame); } +/// ``` +/// +/// Frames arrive in **display order**; a caller must not reorder them. +pub trait VideoDecoder: Send { + /// The codec this decoder decodes. + fn codec(&self) -> VideoCodecId; + + /// Submit one access unit, making zero or more frames available to + /// [`receive_frame`](Self::receive_frame). + /// + /// After [`reset`](Self::reset), packets before the next random access + /// point reference pictures that are gone; they are dropped and `Ok(())` + /// is returned, so seeking into the middle of a GOP is not an error. + /// + /// # Errors + /// + /// [`VideoError::Decode`] when the access unit is malformed or the backend + /// fails. + fn send_packet(&mut self, packet: &Packet) -> VideoResult<()>; + + /// Take the next frame in display order, or `None` when none is ready. + /// + /// `None` is the normal state while a reordering stream fills its buffer; + /// it means "not yet", not "end of stream". Only + /// [`flush`](Self::flush) establishes the latter. + /// + /// # Errors + /// + /// [`VideoError::Decode`] when reading the decoded picture back fails. + fn receive_frame(&mut self) -> VideoResult>; + + /// Signal end of stream: every buffered picture becomes available to + /// [`receive_frame`](Self::receive_frame). + /// + /// Without this the last few frames of every reordered stream are never + /// emitted. The decoder stays usable afterwards. + /// + /// # Errors + /// + /// As [`send_packet`](Self::send_packet). + fn flush(&mut self) -> VideoResult<()>; + + /// Discard all state **without** emitting buffered frames — the seek path. + fn reset(&mut self); +} + +/// An encoder for one video track. +/// +/// The seam only: rawshift ships no video encoder yet. It is defined now so +/// that adding one is additive, and so the shape is settled while the decode +/// side is fresh. +pub trait VideoEncoder: Send { + /// The codec this encoder produces. + fn codec(&self) -> VideoCodecId; + + /// Submit one raw frame for encoding. + /// + /// # Errors + /// + /// Backend-specific encode failures. + fn send_frame(&mut self, frame: &VideoFrame) -> VideoResult<()>; + + /// Take the next coded access unit, or `None` when none is ready. + /// + /// # Errors + /// + /// Backend-specific encode failures. + fn receive_packet(&mut self) -> VideoResult>; + + /// Signal end of stream and drain the encoder. + /// + /// # Errors + /// + /// Backend-specific encode failures. + fn flush(&mut self) -> VideoResult<()>; +} + +/// Opens decoders for tracks it supports. +/// +/// One factory per backend per codec. Registered in a [`DecoderRegistry`], +/// which picks between them. +pub trait VideoDecoderFactory: Send + Sync { + /// Stable `"{format}/{impl}"` identifier, e.g. `"h264/vaapi"`. + fn id(&self) -> CodecId; + + /// The codec this factory decodes. + fn codec(&self) -> VideoCodecId; + + /// Whether this backend can decode this specific track **right now**. + /// + /// Answers about the runtime too, not just the codec: a VAAPI factory + /// returns `false` with no libva, no device, or no driver support for the + /// track's profile. This is the **only** signal that lets the registry try + /// the next backend — a factory that accepts a track and then fails in + /// [`open`](Self::open) is reported to the caller rather than retried. + fn supports(&self, track: &VideoTrack) -> bool; + + /// Open a decoder for `track`. + /// + /// # Errors + /// + /// Any failure to start a session. Reaching here after `supports` returned + /// `true` is terminal for the whole open, by the contract above. + fn open(&self, track: &VideoTrack) -> VideoResult>; +} + +/// Backends to try, in order. +/// +/// **Push order is priority order**, and `supports() == false` is the only +/// thing that falls through to the next entry. A backend that accepts a track +/// and then fails produces an error rather than a silent retry, so a genuine +/// backend fault is never disguised as "unsupported". +/// +/// This mirrors the fallback contract `gamut-codec-abi` documents for image +/// codestream backends, adopted deliberately so both media behave the same. +#[derive(Default)] +pub struct DecoderRegistry { + factories: Vec>, +} + +impl DecoderRegistry { + /// An empty registry. + #[must_use] + pub fn new() -> Self { + Self::default() + } + + /// Append a factory, at lowest priority. + pub fn push(&mut self, factory: Box) { + self.factories.push(factory); + } + + /// Append a factory and return the registry, for chaining at startup. + #[must_use] + pub fn with(mut self, factory: Box) -> Self { + self.push(factory); + self + } + + /// Every registered factory, in priority order. + #[must_use] + pub fn factories(&self) -> &[Box] { + &self.factories + } + + /// Whether any registered backend can decode `track` right now. + #[must_use] + pub fn supports(&self, track: &VideoTrack) -> bool { + self.factories.iter().any(|f| f.supports(track)) + } + + /// Open a decoder for `track` using the first backend that supports it. + /// + /// # Errors + /// + /// [`VideoError::UnsupportedCodec`] when no factory for the codec is + /// registered at all, and + /// [`VideoError::HwDecoderUnavailable`] when factories for the codec exist + /// but none supports this track on this machine — the distinction between + /// "not built" and "no device", which callers act on differently. + pub fn open(&self, track: &VideoTrack) -> VideoResult> { + let mut saw_codec = false; + for factory in &self.factories { + if factory.codec() != track.codec { + continue; + } + saw_codec = true; + if factory.supports(track) { + return factory.open(track); + } + } + + if saw_codec { + Err(VideoError::hw_unavailable( + track.codec, + "a decoder is compiled in, but no backend on this machine supports this track", + )) + } else { + Err(VideoError::UnsupportedCodec { codec: track.codec }) + } + } +} + +impl std::fmt::Debug for DecoderRegistry { + /// Lists the registered ids: the factories themselves are trait objects + /// with nothing else printable, and the order is the interesting part. + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("DecoderRegistry") + .field( + "factories", + &self.factories.iter().map(|f| f.id().id).collect::>(), + ) + .finish() + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::track::{CicpColor, CodecConfig, Rotation}; + use gamut_color::ChromaSubsampling; + use rawshift_core::metadata::URational; + use rawshift_core::{BitDepth, Dimensions}; + use std::sync::atomic::{AtomicUsize, Ordering}; + + fn track(codec: VideoCodecId) -> VideoTrack { + VideoTrack { + id: crate::id::TrackId::new(1), + codec, + dimensions: Dimensions { + width: 1920, + height: 1080, + }, + color: CicpColor::UNSPECIFIED, + bit_depth: BitDepth::Eight, + chroma: ChromaSubsampling::Cs420, + frame_rate: None, + time_base: URational::new(1, 1000), + duration: None, + frame_count: None, + rotation: Rotation::None, + codec_config: CodecConfig::default(), + } + } + + struct StubDecoder(VideoCodecId); + impl VideoDecoder for StubDecoder { + fn codec(&self) -> VideoCodecId { + self.0 + } + fn send_packet(&mut self, _p: &Packet) -> VideoResult<()> { + Ok(()) + } + fn receive_frame(&mut self) -> VideoResult> { + Ok(None) + } + fn flush(&mut self) -> VideoResult<()> { + Ok(()) + } + fn reset(&mut self) {} + } + + struct StubFactory { + id: &'static str, + codec: VideoCodecId, + supports: bool, + opens: &'static AtomicUsize, + open_fails: bool, + } + + impl VideoDecoderFactory for StubFactory { + fn id(&self) -> CodecId { + CodecId::new(self.id) + } + fn codec(&self) -> VideoCodecId { + self.codec + } + fn supports(&self, _t: &VideoTrack) -> bool { + self.supports + } + fn open(&self, _t: &VideoTrack) -> VideoResult> { + self.opens.fetch_add(1, Ordering::SeqCst); + if self.open_fails { + Err(VideoError::decode(self.codec, "backend fault")) + } else { + Ok(Box::new(StubDecoder(self.codec))) + } + } + } + + /// `Box` is not `Debug`, so `expect_err` does not apply. + fn expect_err(result: VideoResult>, context: &str) -> VideoError { + match result { + Ok(_) => panic!("expected an error: {context}"), + Err(e) => e, + } + } + + fn expect_ok(result: VideoResult>, context: &str) { + if let Err(e) = result { + panic!("expected success ({context}), got {e}"); + } + } + + static FIRST: AtomicUsize = AtomicUsize::new(0); + static SECOND: AtomicUsize = AtomicUsize::new(0); + + fn factory( + id: &'static str, + codec: VideoCodecId, + supports: bool, + opens: &'static AtomicUsize, + open_fails: bool, + ) -> Box { + Box::new(StubFactory { + id, + codec, + supports, + opens, + open_fails, + }) + } + + #[test] + fn push_order_is_priority_order() { + FIRST.store(0, Ordering::SeqCst); + SECOND.store(0, Ordering::SeqCst); + let reg = DecoderRegistry::new() + .with(factory( + "h264/first", + VideoCodecId::H264, + true, + &FIRST, + false, + )) + .with(factory( + "h264/second", + VideoCodecId::H264, + true, + &SECOND, + false, + )); + + expect_ok(reg.open(&track(VideoCodecId::H264)), "a supporting backend"); + assert_eq!(FIRST.load(Ordering::SeqCst), 1, "first should have won"); + assert_eq!( + SECOND.load(Ordering::SeqCst), + 0, + "second should not be tried" + ); + } + + #[test] + fn an_unsupporting_backend_falls_through_to_the_next() { + FIRST.store(0, Ordering::SeqCst); + SECOND.store(0, Ordering::SeqCst); + let reg = DecoderRegistry::new() + .with(factory( + "h264/first", + VideoCodecId::H264, + false, + &FIRST, + false, + )) + .with(factory( + "h264/second", + VideoCodecId::H264, + true, + &SECOND, + false, + )); + + expect_ok(reg.open(&track(VideoCodecId::H264)), "a supporting backend"); + assert_eq!( + FIRST.load(Ordering::SeqCst), + 0, + "unsupporting must not open" + ); + assert_eq!(SECOND.load(Ordering::SeqCst), 1); + } + + #[test] + fn a_backend_that_accepts_then_fails_is_not_retried_elsewhere() { + // The core of the fallback contract: only supports()==false falls + // through, so a genuine backend fault surfaces instead of being + // disguised as "unsupported" by a later backend succeeding. + FIRST.store(0, Ordering::SeqCst); + SECOND.store(0, Ordering::SeqCst); + let reg = DecoderRegistry::new() + .with(factory( + "h264/first", + VideoCodecId::H264, + true, + &FIRST, + true, + )) + .with(factory( + "h264/second", + VideoCodecId::H264, + true, + &SECOND, + false, + )); + + let err = expect_err( + reg.open(&track(VideoCodecId::H264)), + "the fault must propagate", + ); + assert!(err.to_string().contains("backend fault"), "{err}"); + assert_eq!(SECOND.load(Ordering::SeqCst), 0, "must not fall through"); + } + + #[test] + fn no_factory_for_the_codec_is_unsupported_not_unavailable() { + let reg = DecoderRegistry::new().with(factory( + "h264/vaapi", + VideoCodecId::H264, + true, + &FIRST, + false, + )); + let err = expect_err(reg.open(&track(VideoCodecId::ProRes)), "no ProRes factory"); + assert!( + matches!(err, VideoError::UnsupportedCodec { .. }), + "expected UnsupportedCodec, got {err}" + ); + } + + #[test] + fn a_compiled_codec_with_no_usable_backend_is_unavailable_not_unsupported() { + // The distinction a caller acts on: "rebuild with the feature" versus + // "this machine has no GPU". + let reg = DecoderRegistry::new().with(factory( + "hevc/vaapi", + VideoCodecId::Hevc, + false, + &FIRST, + false, + )); + let err = expect_err( + reg.open(&track(VideoCodecId::Hevc)), + "no backend supports it", + ); + assert!( + matches!(err, VideoError::HwDecoderUnavailable { .. }), + "expected HwDecoderUnavailable, got {err}" + ); + assert!(err.is_capability_gap()); + } + + #[test] + fn an_empty_registry_reports_unsupported() { + let reg = DecoderRegistry::new(); + assert!(!reg.supports(&track(VideoCodecId::H264))); + assert!(matches!( + reg.open(&track(VideoCodecId::H264)), + Err(VideoError::UnsupportedCodec { .. }) + )); + } + + #[test] + fn debug_lists_ids_in_priority_order() { + let reg = DecoderRegistry::new() + .with(factory( + "hevc/vaapi", + VideoCodecId::Hevc, + true, + &FIRST, + false, + )) + .with(factory( + "h264/vaapi", + VideoCodecId::H264, + true, + &SECOND, + false, + )); + let printed = format!("{reg:?}"); + assert!(printed.contains("hevc/vaapi"), "{printed}"); + assert!( + printed.find("hevc/vaapi") < printed.find("h264/vaapi"), + "order must be visible: {printed}" + ); + } +} diff --git a/crates/rawshift-video-core/src/demux.rs b/crates/rawshift-video-core/src/demux.rs new file mode 100644 index 0000000..a1dcbc8 --- /dev/null +++ b/crates/rawshift-video-core/src/demux.rs @@ -0,0 +1,154 @@ +//! The demuxing contract. + +use crate::error::VideoResult; +use crate::id::{ContainerId, TrackId}; +use crate::metadata::VideoMetadata; +use crate::packet::{Packet, SeekMode, SeekTo}; +use crate::track::Track; + +/// Cheap signature detection for one container. +/// +/// Deliberately a static method on a marker type rather than a method on a +/// demuxer: detection must be answerable from a handful of leading bytes, +/// before anything is opened or allocated. +pub trait ContainerSniffer { + /// The container this sniffer recognises. + const CONTAINER: ContainerId; + + /// Whether `data` looks like this container. + /// + /// `data` may be short; return `false` rather than panicking. A `true` + /// here is a claim about the signature only, not that the file is valid. + fn matches(data: &[u8]) -> bool; +} + +/// A container reader: tracks, metadata, and a packet stream. +/// +/// Object-safe, and held behind `Box` by the aggregator, so the +/// set of compiled-in containers is a runtime property rather than a type +/// parameter that would leak into every caller's signature. +pub trait Demuxer: Send { + /// Which container this is reading. + fn container(&self) -> ContainerId; + + /// Every track in the file, in the container's own order. + /// + /// Available immediately after opening, on every target, with no decoder: + /// track enumeration never needs a codec. + fn tracks(&self) -> &[Track]; + + /// The file's container-level metadata. + fn metadata(&self) -> &VideoMetadata; + + /// The next packet from any track, in stored order, or `None` at end of + /// file. + /// + /// Packets from every track are interleaved as the container stores them. + /// Filter on [`Packet::track`] to follow one track; a caller that reads + /// only one track must still drain the others or they accumulate. + /// + /// # Errors + /// + /// Propagates I/O failures and container errors. An error is not + /// necessarily terminal, but a caller that cannot re-synchronise should + /// stop. + fn next_packet(&mut self) -> VideoResult>; + + /// Seek `track` to `to`, returning the timestamp actually landed on. + /// + /// With [`SeekMode::Precise`] the landing point is at or before the + /// request and always a random access point, so decoding can resume; + /// reaching an exact frame means decoding forward from there and + /// discarding. Any decoder consuming this demuxer must be + /// [`reset`](crate::decode::VideoDecoder::reset) afterwards. + /// + /// # Errors + /// + /// [`VideoError::NoSuchTrack`](crate::error::VideoError::NoSuchTrack) for + /// an unknown track, plus I/O and container errors. Seeking a + /// non-seekable source fails rather than silently reading forward. + fn seek(&mut self, track: TrackId, to: SeekTo, mode: SeekMode) -> VideoResult; + + /// The track with this identifier, if the file has one. + /// + /// Provided: a linear scan over [`tracks`](Self::tracks). Track counts are + /// single digits in practice, so no index is worth maintaining. + fn track(&self, id: TrackId) -> Option<&Track> { + self.tracks().iter().find(|t| t.id() == id) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::error::VideoError; + use crate::id::TrackKind; + use crate::track::OtherTrack; + use rawshift_core::metadata::URational; + + struct StubDemuxer { + tracks: Vec, + metadata: VideoMetadata, + } + + fn other_track(id: u32) -> Track { + Track::Other(OtherTrack { + id: TrackId::new(id), + kind: TrackKind::Data, + time_base: URational::new(1, 1000), + duration: None, + language: None, + }) + } + + impl Demuxer for StubDemuxer { + fn container(&self) -> ContainerId { + ContainerId::Mp4 + } + fn tracks(&self) -> &[Track] { + &self.tracks + } + fn metadata(&self) -> &VideoMetadata { + &self.metadata + } + fn next_packet(&mut self) -> VideoResult> { + Ok(None) + } + fn seek(&mut self, track: TrackId, _to: SeekTo, _mode: SeekMode) -> VideoResult { + Err(VideoError::NoSuchTrack { id: track }) + } + } + + #[test] + fn the_provided_track_lookup_matches_on_container_id_not_index() { + // Container-assigned ids are 1-based and need not be contiguous, so a + // lookup must never be an index into the slice. + let d = StubDemuxer { + tracks: vec![other_track(3), other_track(9)], + metadata: VideoMetadata::default(), + }; + + assert_eq!( + d.track(TrackId::new(3)).map(Track::id), + Some(TrackId::new(3)) + ); + assert_eq!( + d.track(TrackId::new(9)).map(Track::id), + Some(TrackId::new(9)) + ); + // Id 1 exists as neither a track id nor a valid index shortcut. + assert!(d.track(TrackId::new(1)).is_none()); + assert!(d.track(TrackId::new(0)).is_none()); + } + + #[test] + fn a_demuxer_is_object_safe() { + // The aggregator stores Box; this fails to compile if a + // signature above ever stops being object-safe. + let d: Box = Box::new(StubDemuxer { + tracks: Vec::new(), + metadata: VideoMetadata::default(), + }); + assert_eq!(d.container(), ContainerId::Mp4); + } +} diff --git a/crates/rawshift-video-core/src/error.rs b/crates/rawshift-video-core/src/error.rs new file mode 100644 index 0000000..1b0bed9 --- /dev/null +++ b/crates/rawshift-video-core/src/error.rs @@ -0,0 +1,218 @@ +//! The error type every rawshift video crate returns. +//! +//! Mirrors `rawshift-image-core`'s `RawError` in shape and in one rule: a +//! backend's own error type never crosses a rawshift boundary. Demuxer and +//! decoder backends are stringified into [`VideoError::Container`] or +//! [`VideoError::Decode`] at the leaf crate that owns them, so swapping a +//! backend is not a breaking change for callers. + +use std::io; + +use thiserror::Error; + +use crate::id::{ContainerId, TrackId, VideoCodecId}; + +/// Anything that can go wrong reading or decoding a video file. +/// +/// `#[non_exhaustive]`: new containers and codecs bring new failure modes, and +/// that must not be a breaking change. Match with a `_` arm. +#[derive(Debug, Error)] +#[non_exhaustive] +pub enum VideoError { + /// The underlying reader failed. + #[error("I/O error: {0}")] + Io(#[from] io::Error), + + /// The container is malformed, truncated, or uses a structure the demuxer + /// does not implement. + /// + /// `container` names the demuxer that rejected the file so the message can + /// be attributed; `message` is the backend's own text, flattened to a + /// string precisely so the backend's error type stays private. + #[error("{container} container error: {message}")] + Container { + /// The demuxer that produced this error. + container: &'static str, + /// Backend-supplied detail. + message: String, + }, + + /// The bytes are not a container rawshift can open. + /// + /// `detected` is `Some` when rawshift recognised the format but has no + /// demuxer compiled in for it (AVI and MXF are named but unimplemented; + /// MP4 and Matroska are feature-gated), and `None` when nothing matched. + #[error("unsupported container: {}", match .detected { + Some(c) => c.name(), + None => "unrecognised", + })] + UnsupportedContainer { + /// The container, when it was recognised. + detected: Option, + }, + + /// The track's codec has no decoder in this build. + /// + /// Distinct from [`HwDecoderUnavailable`](Self::HwDecoderUnavailable): + /// this one means no decoder for the codec was compiled in at all, not + /// that a compiled decoder found no backend at runtime. + #[error("unsupported codec: {codec}")] + UnsupportedCodec { + /// The codec the track carries. + codec: VideoCodecId, + }, + + /// A decoder was found, and decoding failed. + #[error("{codec} decode failed: {message}")] + Decode { + /// The codec being decoded. + codec: VideoCodecId, + /// Backend-supplied detail. + message: String, + }, + + /// A decoder for this codec exists in the build, but no hardware backend + /// could be found at runtime. + /// + /// This is the expected outcome on every target with no hardware decode + /// API — `wasm32`, `musl`, Windows — and on Linux hosts with no libva or + /// no capable driver. It is deliberately matchable so a caller can fall + /// back or degrade rather than treat it as corruption. Container parsing, + /// track enumeration, and metadata all still work: see `docs/SUPPORT.md`. + #[error("no hardware decoder available for {codec}: {reason}")] + HwDecoderUnavailable { + /// The codec no backend was found for. + codec: VideoCodecId, + /// Why — no backend compiled in, no device, or no driver support. + reason: String, + }, + + /// The file has no track with the requested identifier. + #[error("no such track: {id}")] + NoSuchTrack { + /// The identifier that did not match a track. + id: TrackId, + }, + + /// Container-level metadata was present but could not be interpreted. + /// + /// Metadata failures are never fatal to opening a file — a file with + /// unreadable metadata still demuxes — so this surfaces only from the + /// explicit metadata entry points. + #[error("metadata error: {message}")] + Metadata { + /// What could not be interpreted. + message: String, + }, +} + +impl VideoError { + /// Build a [`VideoError::Container`] from a backend error. + /// + /// The one place demuxer backends are flattened, so the rule "no backend + /// type crosses a public boundary" has a single obvious implementation. + pub fn container(container: &'static str, message: impl std::fmt::Display) -> Self { + Self::Container { + container, + message: message.to_string(), + } + } + + /// Build a [`VideoError::Decode`] from a backend error. + pub fn decode(codec: VideoCodecId, message: impl std::fmt::Display) -> Self { + Self::Decode { + codec, + message: message.to_string(), + } + } + + /// Build a [`VideoError::HwDecoderUnavailable`]. + pub fn hw_unavailable(codec: VideoCodecId, reason: impl std::fmt::Display) -> Self { + Self::HwDecoderUnavailable { + codec, + reason: reason.to_string(), + } + } + + /// Whether this error means "this build or machine cannot decode it", + /// rather than "this file is broken". + /// + /// True for [`UnsupportedContainer`](Self::UnsupportedContainer), + /// [`UnsupportedCodec`](Self::UnsupportedCodec) and + /// [`HwDecoderUnavailable`](Self::HwDecoderUnavailable). Lets a caller + /// distinguish a capability gap from corruption without matching every + /// variant of a `#[non_exhaustive]` enum. + #[must_use] + pub const fn is_capability_gap(&self) -> bool { + matches!( + self, + Self::UnsupportedContainer { .. } + | Self::UnsupportedCodec { .. } + | Self::HwDecoderUnavailable { .. } + ) + } +} + +/// `Result` alias used throughout rawshift's video crates. +pub type VideoResult = Result; + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn capability_gaps_are_distinguished_from_corruption() { + assert!( + VideoError::UnsupportedCodec { + codec: VideoCodecId::ProRes + } + .is_capability_gap() + ); + assert!(VideoError::hw_unavailable(VideoCodecId::Hevc, "no libva").is_capability_gap()); + assert!( + VideoError::UnsupportedContainer { + detected: Some(ContainerId::Mxf) + } + .is_capability_gap() + ); + + assert!(!VideoError::container("MP4", "truncated moov").is_capability_gap()); + assert!(!VideoError::decode(VideoCodecId::H264, "bad slice").is_capability_gap()); + assert!(!VideoError::Io(io::Error::from(io::ErrorKind::UnexpectedEof)).is_capability_gap()); + } + + #[test] + fn unsupported_container_reads_well_both_ways() { + let named = VideoError::UnsupportedContainer { + detected: Some(ContainerId::Avi), + }; + assert_eq!(named.to_string(), "unsupported container: AVI"); + + let unknown = VideoError::UnsupportedContainer { detected: None }; + assert_eq!(unknown.to_string(), "unsupported container: unrecognised"); + } + + #[test] + fn backend_detail_is_preserved_in_the_message() { + let e = VideoError::container("Matroska", "unexpected EBML id 0x1f43b675"); + assert!(e.to_string().contains("Matroska")); + assert!(e.to_string().contains("0x1f43b675")); + } + + #[test] + fn hw_unavailable_names_the_codec_and_reason() { + let e = VideoError::hw_unavailable(VideoCodecId::H264, "no VAAPI device"); + let s = e.to_string(); + assert!(s.contains("H.264"), "{s}"); + assert!(s.contains("no VAAPI device"), "{s}"); + } + + #[test] + fn io_errors_convert_with_the_question_mark_operator() { + fn inner() -> VideoResult<()> { + Err(io::Error::from(io::ErrorKind::PermissionDenied))?; + unreachable!() + } + assert!(matches!(inner(), Err(VideoError::Io(_)))); + } +} diff --git a/crates/rawshift-video-core/src/frame.rs b/crates/rawshift-video-core/src/frame.rs new file mode 100644 index 0000000..b0110b2 --- /dev/null +++ b/crates/rawshift-video-core/src/frame.rs @@ -0,0 +1,576 @@ +//! Decoded frames. +//! +//! A [`VideoFrame`] owns its pixel data. Hardware decoders hand back planes +//! that live in a driver-owned surface, so the decoder copies them out before +//! the surface is recycled; a frame therefore never pins a decoder resource +//! and can outlive the decoder that produced it. + +use gamut_color::{ColorRange, MatrixCoefficients}; +use rawshift_core::Dimensions; + +use crate::error::{VideoError, VideoResult}; +use crate::id::VideoCodecId; + +/// One plane of a decoded picture. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Plane { + /// Sample data, `stride` bytes per row. + pub data: Vec, + /// Bytes per row, which is at least the row's width in bytes and may + /// exceed it: hardware decoders align rows. + pub stride: usize, +} + +/// The sample layout of a decoded picture. +/// +/// These are the four 4:2:0 layouts hardware decoders produce. rawshift's own +/// type rather than the backend's, so swapping backends is not a breaking +/// change. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +#[non_exhaustive] +pub enum FramePixelFormat { + /// 8-bit, Y plane then interleaved CbCr. Two planes. + Nv12, + /// 10-bit in 16-bit words, Y plane then interleaved CbCr, value in the + /// **most** significant bits. Two planes. + P010, + /// 8-bit planar Y, Cb, Cr. Three planes. + I420, + /// 10-bit planar Y, Cb, Cr in 16-bit words, value in the **least** + /// significant bits. Three planes. + I010, +} + +impl FramePixelFormat { + /// How many planes this layout has. + #[must_use] + pub const fn plane_count(self) -> usize { + match self { + Self::Nv12 | Self::P010 => 2, + Self::I420 | Self::I010 => 3, + } + } + + /// Bytes per sample: 1 for the 8-bit layouts, 2 for the 10-bit ones. + #[must_use] + pub const fn bytes_per_sample(self) -> usize { + match self { + Self::Nv12 | Self::I420 => 1, + Self::P010 | Self::I010 => 2, + } + } + + /// Whether chroma is stored interleaved (`NV12`/`P010`) rather than as two + /// separate planes. + #[must_use] + pub const fn is_semi_planar(self) -> bool { + matches!(self, Self::Nv12 | Self::P010) + } + + /// How far to shift a stored 16-bit word right to recover the sample. + /// + /// `P010` parks the 10-bit value in the top of the word; `I010` stores it + /// at the bottom. Zero for the 8-bit layouts. + #[must_use] + pub const fn sample_shift(self) -> u32 { + match self { + Self::P010 => 6, + Self::Nv12 | Self::I420 | Self::I010 => 0, + } + } +} + +/// Everything about a decoded picture except its pixels. +/// +/// Exists so [`VideoFrame::new`] takes one descriptor rather than a row of +/// same-typed positional arguments, where transposing two is silent. +#[derive(Debug, Clone, Copy, PartialEq)] +#[non_exhaustive] +pub struct FrameDesc { + /// The sample layout. + pub format: FramePixelFormat, + /// Picture dimensions. + pub dimensions: Dimensions, + /// Bits per sample. + pub bit_depth: u8, + /// Whether samples occupy the full code range or the studio subset. + pub range: ColorRange, + /// The matrix mapping YCbCr to RGB. Must already be resolved — never + /// [`MatrixCoefficients::Unspecified`]. + pub matrix: MatrixCoefficients, + /// Presentation timestamp of the packet this picture was coded from. + pub pts: Option, + /// Picture order count derived from the bitstream. + pub poc: i32, +} + +impl FrameDesc { + /// A descriptor for a picture, with no timestamp and a zero picture order + /// count. + #[must_use] + pub const fn new( + format: FramePixelFormat, + dimensions: Dimensions, + bit_depth: u8, + range: ColorRange, + matrix: MatrixCoefficients, + ) -> Self { + Self { + format, + dimensions, + bit_depth, + range, + matrix, + pts: None, + poc: 0, + } + } + + /// Set the presentation timestamp and picture order count. + #[must_use] + pub const fn with_timing(mut self, pts: Option, poc: i32) -> Self { + self.pts = pts; + self.poc = poc; + self + } +} + +/// A decoded picture, in output (display) order. +/// +/// Construct with [`VideoFrame::new`], which validates that the planes match +/// the format and dimensions — so every accessor below can be infallible. +#[derive(Debug, Clone, PartialEq)] +pub struct VideoFrame { + format: FramePixelFormat, + dimensions: Dimensions, + bit_depth: u8, + range: ColorRange, + matrix: MatrixCoefficients, + planes: Vec, + pts: Option, + poc: i32, +} + +impl VideoFrame { + /// Assemble a frame, validating it against its stated format. + /// + /// # Errors + /// + /// [`VideoError::Decode`] when the plane count does not match the format, + /// or a plane's stride or length cannot hold the stated dimensions. This + /// catches a backend that returned a surface inconsistent with what it + /// claimed, which is otherwise an out-of-bounds read waiting to happen in + /// every consumer. + pub fn new(codec: VideoCodecId, desc: FrameDesc, planes: Vec) -> VideoResult { + let FrameDesc { + format, + dimensions, + bit_depth, + range, + matrix, + pts, + poc, + } = desc; + + if planes.len() != format.plane_count() { + return Err(VideoError::decode( + codec, + format!( + "{format:?} needs {} planes, got {}", + format.plane_count(), + planes.len() + ), + )); + } + + let bps = format.bytes_per_sample(); + let width = dimensions.width as usize; + let height = dimensions.height as usize; + // 4:2:0: chroma is half resolution on both axes, rounded up so odd + // dimensions keep their final half-populated row and column. + let chroma_width = width.div_ceil(2); + let chroma_height = height.div_ceil(2); + + for (index, plane) in planes.iter().enumerate() { + let (min_row_bytes, rows) = match (index, format.is_semi_planar()) { + (0, _) => (width * bps, height), + // Interleaved CbCr: one row holds both components. + (_, true) => (chroma_width * 2 * bps, chroma_height), + (_, false) => (chroma_width * bps, chroma_height), + }; + + if plane.stride < min_row_bytes { + return Err(VideoError::decode( + codec, + format!( + "plane {index} stride {} is below the {min_row_bytes} bytes a row needs", + plane.stride + ), + )); + } + // The last row need only be complete up to min_row_bytes; drivers + // do not always pad it out to a full stride. + let required = plane.stride * rows.saturating_sub(1) + min_row_bytes; + if plane.data.len() < required { + return Err(VideoError::decode( + codec, + format!( + "plane {index} holds {} bytes, needs {required} for {rows} rows", + plane.data.len() + ), + )); + } + } + + Ok(Self { + format, + dimensions, + bit_depth, + range, + matrix, + planes, + pts, + poc, + }) + } + + /// Picture width in pixels. + #[must_use] + pub const fn width(&self) -> u32 { + self.dimensions.width + } + + /// Picture height in pixels. + #[must_use] + pub const fn height(&self) -> u32 { + self.dimensions.height + } + + /// Picture dimensions. + #[must_use] + pub const fn dimensions(&self) -> Dimensions { + self.dimensions + } + + /// The sample layout. + #[must_use] + pub const fn format(&self) -> FramePixelFormat { + self.format + } + + /// Bits per sample — 8 or 10 for the layouts rawshift decodes. + #[must_use] + pub const fn bit_depth(&self) -> u8 { + self.bit_depth + } + + /// Whether samples occupy the full code range or the studio subset. + #[must_use] + pub const fn range(&self) -> ColorRange { + self.range + } + + /// The matrix that maps this frame's YCbCr samples to RGB. + /// + /// Already resolved: never `Unspecified`. + #[must_use] + pub const fn matrix(&self) -> MatrixCoefficients { + self.matrix + } + + /// The planes, in the order the format defines. + #[must_use] + pub fn planes(&self) -> &[Plane] { + &self.planes + } + + /// The presentation timestamp of the packet this picture was coded from, + /// in the track's `time_base` units. + #[must_use] + pub const fn pts(&self) -> Option { + self.pts + } + + /// The picture order count derived from the bitstream. + /// + /// Strictly increasing across the frames emitted between two random access + /// points, which makes it the reliable ordering key even where a container + /// carries absent or broken timestamps. + #[must_use] + pub const fn poc(&self) -> i32 { + self.poc + } + + /// Read one luma sample, or `None` if the coordinates are outside the + /// picture. + /// + /// Returned on a 0–65535 scale regardless of the source bit depth, so + /// callers need not branch on depth. + #[must_use] + pub fn luma(&self, x: u32, y: u32) -> Option { + if x >= self.dimensions.width || y >= self.dimensions.height { + return None; + } + let plane = self.planes.first()?; + let (x, y) = (x as usize, y as usize); + let sample = match self.format.bytes_per_sample() { + 1 => u16::from(*plane.data.get(y * plane.stride + x)?), + _ => { + let at = y * plane.stride + x * 2; + let lo = *plane.data.get(at)?; + let hi = *plane.data.get(at + 1)?; + u16::from_le_bytes([lo, hi]) >> self.format.sample_shift() + } + }; + Some(scale_to_u16(sample, self.bit_depth)) + } +} + +/// Scale a sample of `bit_depth` bits up to the full `u16` range. +/// +/// Replicates the sample's own high bits into the vacated low bits rather than +/// shifting in zeros. A plain `<< 8` would map 8-bit 255 to 65280, so every +/// white in the picture would come out slightly grey and the scaling would not +/// be idempotent across depths. +fn scale_to_u16(sample: u16, bit_depth: u8) -> u16 { + let depth = u32::from(bit_depth); + if depth == 0 || depth >= 16 { + return sample; + } + // Mask defensively: a backend that reports 10-bit while leaving junk in + // the top 6 bits would otherwise scale that junk into the output. + let value = u32::from(sample) & ((1u32 << depth) - 1); + + // Left-align, then tile the value downwards to fill the remaining bits. + let mut out = value << (32 - depth); + let mut filled = depth; + while filled < 32 { + out |= (value << (32 - depth)) >> filled; + filled += depth; + } + (out >> 16) as u16 +} + +#[cfg(test)] +mod tests { + use super::*; + + fn dims(w: u32, h: u32) -> Dimensions { + Dimensions { + width: w, + height: h, + } + } + + fn nv12(width: usize, height: usize) -> Vec { + vec![ + Plane { + data: vec![0u8; width * height], + stride: width, + }, + Plane { + data: vec![0u8; width * height.div_ceil(2)], + stride: width, + }, + ] + } + + fn frame_from( + planes: Vec, + format: FramePixelFormat, + w: u32, + h: u32, + ) -> VideoResult { + let desc = FrameDesc::new( + format, + dims(w, h), + 8, + ColorRange::Limited, + MatrixCoefficients::Bt709, + ) + .with_timing(Some(0), 0); + VideoFrame::new(VideoCodecId::Hevc, desc, planes) + } + + #[test] + fn plane_counts_match_the_layouts() { + assert_eq!(FramePixelFormat::Nv12.plane_count(), 2); + assert_eq!(FramePixelFormat::P010.plane_count(), 2); + assert_eq!(FramePixelFormat::I420.plane_count(), 3); + assert_eq!(FramePixelFormat::I010.plane_count(), 3); + } + + #[test] + fn only_p010_parks_its_value_in_the_high_bits() { + assert_eq!(FramePixelFormat::P010.sample_shift(), 6); + assert_eq!(FramePixelFormat::I010.sample_shift(), 0); + assert_eq!(FramePixelFormat::Nv12.sample_shift(), 0); + } + + #[test] + fn a_well_formed_frame_is_accepted() { + let f = frame_from(nv12(64, 48), FramePixelFormat::Nv12, 64, 48) + .expect("well-formed NV12 frame"); + assert_eq!(f.width(), 64); + assert_eq!(f.height(), 48); + assert_eq!(f.planes().len(), 2); + } + + #[test] + fn a_wrong_plane_count_is_rejected() { + // NV12 needs two planes; hand it three. + let mut planes = nv12(64, 48); + planes.push(Plane { + data: vec![0; 16], + stride: 4, + }); + let err = frame_from(planes, FramePixelFormat::Nv12, 64, 48) + .expect_err("three planes is not NV12"); + assert!(err.to_string().contains("needs 2 planes"), "{err}"); + } + + #[test] + fn a_stride_too_small_for_the_width_is_rejected() { + let planes = vec![ + Plane { + data: vec![0; 64 * 48], + stride: 32, // half the 64 bytes a row needs + }, + Plane { + data: vec![0; 64 * 24], + stride: 64, + }, + ]; + let err = + frame_from(planes, FramePixelFormat::Nv12, 64, 48).expect_err("understated stride"); + assert!(err.to_string().contains("stride"), "{err}"); + } + + #[test] + fn a_truncated_plane_is_rejected() { + let planes = vec![ + Plane { + data: vec![0; 64 * 10], // 10 rows where 48 are claimed + stride: 64, + }, + Plane { + data: vec![0; 64 * 24], + stride: 64, + }, + ]; + let err = + frame_from(planes, FramePixelFormat::Nv12, 64, 48).expect_err("truncated luma plane"); + assert!(err.to_string().contains("needs"), "{err}"); + } + + #[test] + fn an_aligned_stride_larger_than_the_width_is_fine() { + // Drivers align rows; a 64-wide picture in a 128-byte stride is normal. + let planes = vec![ + Plane { + data: vec![0; 128 * 48], + stride: 128, + }, + Plane { + data: vec![0; 128 * 24], + stride: 128, + }, + ]; + assert!(frame_from(planes, FramePixelFormat::Nv12, 64, 48).is_ok()); + } + + #[test] + fn odd_dimensions_round_chroma_up() { + // A 65x49 picture has 33x25 chroma, not 32x24 — rounding down would + // drop the last column and row. + let planes = vec![ + Plane { + data: vec![0; 65 * 49], + stride: 65, + }, + Plane { + data: vec![0; 66 * 25], + stride: 66, + }, + ]; + assert!(frame_from(planes, FramePixelFormat::Nv12, 65, 49).is_ok()); + } + + #[test] + fn the_final_row_need_not_be_padded_to_a_full_stride() { + // A driver may hand back exactly enough bytes to hold the last row's + // pixels, with no trailing alignment padding. + let stride = 128; + let planes = vec![ + Plane { + data: vec![0; stride * 47 + 64], + stride, + }, + Plane { + data: vec![0; stride * 23 + 64], + stride, + }, + ]; + assert!(frame_from(planes, FramePixelFormat::Nv12, 64, 48).is_ok()); + } + + #[test] + fn luma_reads_8_bit_samples_scaled_to_full_range() { + let mut planes = nv12(4, 2); + planes[0].data[0] = 255; + planes[0].data[1] = 0; + let f = frame_from(planes, FramePixelFormat::Nv12, 4, 2).expect("frame"); + // 8-bit 255 must reach u16::MAX, not 65280. + assert_eq!(f.luma(0, 0), Some(u16::MAX)); + assert_eq!(f.luma(1, 0), Some(0)); + } + + #[test] + fn luma_is_none_outside_the_picture() { + let f = frame_from(nv12(4, 2), FramePixelFormat::Nv12, 4, 2).expect("frame"); + assert_eq!(f.luma(4, 0), None); + assert_eq!(f.luma(0, 2), None); + } + + #[test] + fn scaling_maps_extremes_to_extremes_at_every_depth() { + for depth in [1u8, 4, 8, 10, 12, 15, 16] { + let max = ((1u32 << depth) - 1) as u16; + assert_eq!(scale_to_u16(0, depth), 0, "depth {depth} floor"); + assert_eq!(scale_to_u16(max, depth), u16::MAX, "depth {depth} ceiling"); + } + } + + #[test] + fn scaling_replicates_bits_rather_than_shifting_in_zeros() { + // The textbook cases: 8-bit replicates the byte, 10-bit borrows its + // own top 6 bits, 12-bit its own top 4. + assert_eq!(scale_to_u16(0xAB, 8), 0xABAB); + assert_eq!(scale_to_u16(0b11_1111_0000, 10), 0b1111_1100_0011_1111); + assert_eq!(scale_to_u16(0x800, 12), 0x8008); + } + + #[test] + fn scaling_is_monotonic() { + // Ordering must survive scaling, or gradients would band. + let mut previous = 0; + for value in 0..=1023u16 { + let scaled = scale_to_u16(value, 10); + assert!(scaled >= previous, "value {value} went backwards"); + previous = scaled; + } + } + + #[test] + fn scaling_ignores_junk_above_the_stated_depth() { + // A backend claiming 10 bits but leaving the top 6 set must not have + // that junk scaled into the result. + assert_eq!(scale_to_u16(0x03FF, 10), scale_to_u16(0xFFFF, 10)); + } + + #[test] + fn full_and_zero_depth_pass_through_untouched() { + assert_eq!(scale_to_u16(12345, 16), 12345); + assert_eq!(scale_to_u16(12345, 0), 12345); + } +} diff --git a/crates/rawshift-video-core/src/id.rs b/crates/rawshift-video-core/src/id.rs new file mode 100644 index 0000000..b2fc586 --- /dev/null +++ b/crates/rawshift-video-core/src/id.rs @@ -0,0 +1,257 @@ +//! Format and codec identity. +//! +//! Every enum here is `#[non_exhaustive]`. rawshift names containers and +//! codecs it does not yet decode, so that "we know what this is and cannot +//! handle it" is expressible — see [`crate::error::VideoError`] — and so that +//! growing support later is additive rather than breaking. + +use std::fmt; + +/// A container (file) format. +/// +/// A variant existing says only that rawshift can *name* the container. Ask +/// the aggregator whether a demuxer for it was compiled in; opening an +/// unsupported one fails with +/// [`VideoError::UnsupportedContainer`](crate::error::VideoError::UnsupportedContainer). +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +#[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))] +#[non_exhaustive] +pub enum ContainerId { + /// ISO Base Media File Format as MP4 / M4V (`isom`, `mp41`, `mp42`, …). + Mp4, + /// ISOBMFF in its QuickTime flavour (`qt `) — the `.mov` Apple writes. + Mov, + /// Matroska. + Matroska, + /// WebM, the Matroska subset restricted to royalty-free codecs. + WebM, + /// Audio Video Interleave (RIFF). Named, not implemented. + Avi, + /// Material Exchange Format, SMPTE 377M. Named, not implemented. + Mxf, +} + +impl ContainerId { + /// A short, stable, human-readable name (`"MP4"`, `"QuickTime"`, …). + /// + /// Stable across releases: it appears in error messages and in + /// [`CodecId`](rawshift_core::CodecId) strings. + #[must_use] + pub const fn name(self) -> &'static str { + match self { + Self::Mp4 => "MP4", + Self::Mov => "QuickTime", + Self::Matroska => "Matroska", + Self::WebM => "WebM", + Self::Avi => "AVI", + Self::Mxf => "MXF", + } + } + + /// Whether this container is an ISOBMFF flavour, and so shares a box + /// grammar and a demuxer with the others that are. + #[must_use] + pub const fn is_isobmff(self) -> bool { + matches!(self, Self::Mp4 | Self::Mov) + } + + /// Whether this container is a Matroska flavour. + #[must_use] + pub const fn is_matroska(self) -> bool { + matches!(self, Self::Matroska | Self::WebM) + } +} + +impl fmt::Display for ContainerId { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str(self.name()) + } +} + +/// A video codec. +/// +/// As with [`ContainerId`], a variant existing does not imply a decoder was +/// compiled in or that a backend can be found at runtime. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +#[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))] +#[non_exhaustive] +pub enum VideoCodecId { + /// H.264 / MPEG-4 AVC. + H264, + /// H.265 / HEVC. + Hevc, + /// AOMedia Video 1. + Av1, + /// VP9. + Vp9, + /// Apple ProRes. Named, not implemented — see the crate README. + ProRes, + /// A codec rawshift recognises the track as carrying but does not name. + Other, +} + +impl VideoCodecId { + /// A short, stable, human-readable name. + #[must_use] + pub const fn name(self) -> &'static str { + match self { + Self::H264 => "H.264", + Self::Hevc => "HEVC", + Self::Av1 => "AV1", + Self::Vp9 => "VP9", + Self::ProRes => "ProRes", + Self::Other => "unknown", + } + } +} + +impl fmt::Display for VideoCodecId { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str(self.name()) + } +} + +/// An audio codec. +/// +/// rawshift **enumerates** audio tracks and hands back their packets; it never +/// decodes audio. This enum exists so a caller can tell what an audio track +/// carries before routing those packets somewhere that does. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +#[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))] +#[non_exhaustive] +pub enum AudioCodecId { + /// Advanced Audio Coding. + Aac, + /// Linear PCM. + Pcm, + /// Opus. + Opus, + /// Vorbis. + Vorbis, + /// FLAC. + Flac, + /// Dolby AC-3 or E-AC-3. + Ac3, + /// A codec rawshift does not name. + Other, +} + +impl AudioCodecId { + /// A short, stable, human-readable name. + #[must_use] + pub const fn name(self) -> &'static str { + match self { + Self::Aac => "AAC", + Self::Pcm => "PCM", + Self::Opus => "Opus", + Self::Vorbis => "Vorbis", + Self::Flac => "FLAC", + Self::Ac3 => "AC-3", + Self::Other => "unknown", + } + } +} + +impl fmt::Display for AudioCodecId { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str(self.name()) + } +} + +/// What kind of media a track carries. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +#[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))] +#[non_exhaustive] +pub enum TrackKind { + /// Coded pictures. + Video, + /// Coded audio. + Audio, + /// Subtitles or captions. + Subtitle, + /// SMPTE timecode. + Timecode, + /// Anything else the container carries as a track. + Data, +} + +/// A track's identifier within one file. +/// +/// Opaque and container-assigned: for ISOBMFF this is the `tkhd` track ID, +/// which is 1-based and need not be contiguous, so never treat it as an index +/// into [`tracks`](crate::demux::Demuxer::tracks). +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)] +#[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))] +pub struct TrackId(u32); + +impl TrackId { + /// Wrap a container-assigned track identifier. + #[must_use] + pub const fn new(id: u32) -> Self { + Self(id) + } + + /// The underlying container-assigned value. + #[must_use] + pub const fn get(self) -> u32 { + self.0 + } +} + +impl fmt::Display for TrackId { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "{}", self.0) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn isobmff_flavours_are_grouped() { + assert!(ContainerId::Mp4.is_isobmff()); + assert!(ContainerId::Mov.is_isobmff()); + assert!(!ContainerId::Matroska.is_isobmff()); + assert!(!ContainerId::Avi.is_isobmff()); + } + + #[test] + fn matroska_flavours_are_grouped() { + assert!(ContainerId::Matroska.is_matroska()); + assert!(ContainerId::WebM.is_matroska()); + assert!(!ContainerId::Mp4.is_matroska()); + } + + #[test] + fn a_container_is_never_both_families() { + for c in [ + ContainerId::Mp4, + ContainerId::Mov, + ContainerId::Matroska, + ContainerId::WebM, + ContainerId::Avi, + ContainerId::Mxf, + ] { + assert!( + !(c.is_isobmff() && c.is_matroska()), + "{c} claims two container families" + ); + } + } + + #[test] + fn track_id_round_trips_and_displays() { + let id = TrackId::new(7); + assert_eq!(id.get(), 7); + assert_eq!(id.to_string(), "7"); + } + + #[test] + fn names_are_stable_and_non_empty() { + assert_eq!(ContainerId::Mov.name(), "QuickTime"); + assert_eq!(VideoCodecId::Hevc.name(), "HEVC"); + assert_eq!(AudioCodecId::Aac.name(), "AAC"); + assert!(!VideoCodecId::Other.name().is_empty()); + } +} diff --git a/crates/rawshift-video-core/src/lib.rs b/crates/rawshift-video-core/src/lib.rs new file mode 100644 index 0000000..03e155b --- /dev/null +++ b/crates/rawshift-video-core/src/lib.rs @@ -0,0 +1,64 @@ +//! Shared vocabulary for rawshift's video crates. +//! +//! This is the video counterpart to `rawshift-image-core`: the types every +//! `rawshift-video-*` crate agrees on, and no implementation of its own. No +//! I/O, no platform code, no container or codec dependencies. +//! +//! # Why this crate exists separately +//! +//! The workspace enforces in CI that no `rawshift-image*` crate appears in +//! `rawshift-video`'s dependency tree, so the video crates cannot reuse +//! `rawshift-image-core`'s error type or format traits. Image and video share +//! only [`rawshift_core`] — the media-agnostic geometry, codec descriptors, +//! metadata model, and colour vocabulary — and this crate adds the video +//! half on top of it. +//! +//! # The model +//! +//! - **Identity** — [`ContainerId`], [`VideoCodecId`], [`AudioCodecId`], +//! [`TrackKind`], [`TrackId`]. All `#[non_exhaustive]`, and all naming more +//! than rawshift can decode, so "recognised but unsupported" is expressible. +//! - **Errors** — [`VideoError`], which keeps every backend's own error type +//! private, and distinguishes a capability gap from a broken file via +//! [`VideoError::is_capability_gap`]. +//! - **Tracks** — [`Track`], [`VideoTrack`], [`AudioTrack`], plus +//! [`CicpColor`] and [`Rotation`]. All header-derived, so a full track list +//! is available on targets with no decoder at all. +//! - **Packets and frames** — [`Packet`] in, [`VideoFrame`] out. +//! - **Contracts** — [`Demuxer`], [`VideoDecoder`], [`VideoEncoder`], and the +//! [`DecoderRegistry`] that picks between decoder backends. +//! +//! # Decoding is a two-step conversation +//! +//! [`VideoDecoder`] splits submit from retrieve, because coded order and +//! display order differ whenever a stream reorders. See its documentation for +//! the loop, including the [`flush`](VideoDecoder::flush) without which the +//! last frames of every reordered stream are silently dropped. + +#![forbid(unsafe_code)] +#![warn(missing_docs)] + +pub mod decode; +pub mod demux; +pub mod error; +pub mod frame; +pub mod id; +pub mod metadata; +pub mod packet; +pub mod track; + +pub use decode::{DecoderRegistry, VideoDecoder, VideoDecoderFactory, VideoEncoder}; +pub use demux::{ContainerSniffer, Demuxer}; +pub use error::{VideoError, VideoResult}; +pub use frame::{FrameDesc, FramePixelFormat, Plane, VideoFrame}; +pub use id::{AudioCodecId, ContainerId, TrackId, TrackKind, VideoCodecId}; +pub use metadata::{ContainerMetadata, VideoMetadata}; +pub use packet::{Packet, SeekMode, SeekTo}; +pub use track::{AudioTrack, CicpColor, CodecConfig, OtherTrack, Rotation, Track, VideoTrack}; + +// The colour vocabulary is gamut's, re-exported so callers need not depend on +// gamut directly — the same arrangement `rawshift-core` already uses for the +// pixel and CICP types. +pub use gamut_color::{ + ChromaSubsampling, ColorRange, ColourPrimaries, MatrixCoefficients, TransferCharacteristics, +}; diff --git a/crates/rawshift-video-core/src/metadata.rs b/crates/rawshift-video-core/src/metadata.rs new file mode 100644 index 0000000..05b8bb7 --- /dev/null +++ b/crates/rawshift-video-core/src/metadata.rs @@ -0,0 +1,133 @@ +//! Container-level metadata. +//! +//! The EXIF/XMP/GPS core is [`rawshift_core::ImageMetadata`], reused as-is: +//! `rawshift-core` documents that model as "genuinely video-shared … EXIF/XMP +//! semantics are the same for a video file as for a still", and this is its +//! first video consumer. What video adds on top is the *movie*-level facts a +//! still has no analogue for, which is what [`ContainerMetadata`] holds. + +use std::time::Duration; + +use rawshift_core::{ImageMetadata, MetadataEntry, MetadataKey, MetadataNamespace, MetadataValue}; + +use crate::id::ContainerId; + +/// Everything rawshift read from a video file's headers. +#[derive(Debug, Clone, Default, PartialEq)] +#[non_exhaustive] +pub struct VideoMetadata { + /// The media-agnostic core: EXIF, XMP, GPS, camera identification, and the + /// generic `extra` table. Container-level facts are also projected into + /// `common.extra` under the [`MetadataNamespace::Quicktime`] and + /// [`MetadataNamespace::Matroska`] namespaces, so a caller that only knows + /// `ImageMetadata` still sees them. + pub common: ImageMetadata, + /// Movie-level facts with no still-image analogue. + pub container: ContainerMetadata, +} + +/// Movie-level facts: the parts of a container that describe the file as a +/// whole rather than any one track. +#[derive(Debug, Clone, Default, PartialEq)] +#[non_exhaustive] +pub struct ContainerMetadata { + /// Which container this came from. + pub container: Option, + /// Total presentation duration, when the container states one. + pub duration: Option, + /// The movie timescale, in ticks per second. + /// + /// The movie header's own unit, distinct from any track's `time_base`. + pub timescale: Option, + /// Creation time, as the container states it. + pub creation_time: Option, + /// Modification time, as the container states it. + pub modification_time: Option, + /// ISOBMFF `ftyp` major brand and compatible brands (`"qt "`, `"mp42"`, + /// `"isom"`, …). Empty for non-ISOBMFF containers. + pub brands: Vec, + /// The writing application or muxer, when stated. + pub writing_app: Option, +} + +impl VideoMetadata { + /// Record a container-level fact into the shared `extra` table. + /// + /// The single place container facts enter [`ImageMetadata`], so every + /// demuxer namespaces them the same way and a caller can find them all + /// without knowing which container produced them. + pub fn push_container_entry( + &mut self, + namespace: MetadataNamespace, + tag: impl Into, + value: MetadataValue, + ) { + self.common.extra.push(MetadataEntry { + key: MetadataKey::new(namespace, tag), + value, + }); + } + + /// Look up a container-level fact previously recorded. + #[must_use] + pub fn container_entry( + &self, + namespace: MetadataNamespace, + tag: &str, + ) -> Option<&MetadataValue> { + self.common + .extra + .iter() + .find(|e| e.key.namespace == namespace && e.key.tag == tag) + .map(|e| &e.value) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn container_entries_round_trip_through_the_shared_table() { + let mut md = VideoMetadata::default(); + md.push_container_entry( + MetadataNamespace::Quicktime, + "com.apple.quicktime.model", + MetadataValue::Text("iPhone 17 Pro Max".into()), + ); + + assert_eq!( + md.container_entry(MetadataNamespace::Quicktime, "com.apple.quicktime.model"), + Some(&MetadataValue::Text("iPhone 17 Pro Max".into())) + ); + // It really landed in the shared table, not a side channel. + assert_eq!(md.common.extra.len(), 1); + } + + #[test] + fn lookup_is_namespaced_not_just_tag_matched() { + let mut md = VideoMetadata::default(); + md.push_container_entry( + MetadataNamespace::Matroska, + "title", + MetadataValue::Text("mkv".into()), + ); + // Same tag, different namespace: must not match. + assert_eq!( + md.container_entry(MetadataNamespace::Quicktime, "title"), + None + ); + assert!( + md.container_entry(MetadataNamespace::Matroska, "title") + .is_some() + ); + } + + #[test] + fn a_default_is_empty_rather_than_absent() { + let md = VideoMetadata::default(); + assert!(md.common.extra.is_empty()); + assert_eq!(md.container.container, None); + assert!(md.container.brands.is_empty()); + } +} diff --git a/crates/rawshift-video-core/src/packet.rs b/crates/rawshift-video-core/src/packet.rs new file mode 100644 index 0000000..80fa3f3 --- /dev/null +++ b/crates/rawshift-video-core/src/packet.rs @@ -0,0 +1,156 @@ +//! Demuxed packets and the seek vocabulary. + +use std::time::Duration; + +use rawshift_core::metadata::URational; + +use crate::id::TrackId; + +/// One demuxed access unit — the coded data for a single presentation time on +/// one track. +/// +/// For H.264 and HEVC in MP4 or Matroska this is a **length-prefixed** NAL +/// stream, never Annex B; the prefix width comes from the track's +/// [`CodecConfig`](crate::track::CodecConfig). +#[derive(Debug, Clone, PartialEq, Eq)] +#[non_exhaustive] +pub struct Packet { + /// The track this packet belongs to. + pub track: TrackId, + /// Presentation timestamp, in the track's + /// [`time_base`](crate::track::VideoTrack::time_base) units. + pub pts: Option, + /// Decode timestamp, in the same units. + /// + /// Differs from `pts` exactly when the stream reorders. rawshift never + /// orders output by `dts` — the bitstream's picture order is + /// authoritative — so this is carried for diagnostics and for callers + /// remuxing rather than decoding. + pub dts: Option, + /// How long this packet occupies, in the same units, when stated. + pub duration: Option, + /// Whether the container marks this as a sync sample (a random access + /// point). + /// + /// Advisory. Containers get this wrong, and the bitstream is + /// authoritative; a decoder uses it to find a starting point, not to + /// decide how to decode. + pub is_keyframe: bool, + /// The coded bytes. + pub data: Vec, +} + +impl Packet { + /// Convert a timestamp in `time_base` units to a [`Duration`]. + /// + /// Returns `None` for an absent or negative timestamp: a `Duration` cannot + /// be negative, and negative presentation times are real (an edit list can + /// place a track's first samples before the presentation origin). + #[must_use] + pub fn timestamp_to_duration(timestamp: i64, time_base: URational) -> Option { + if timestamp < 0 || time_base.denominator == 0 { + return None; + } + let seconds = + timestamp as f64 * f64::from(time_base.numerator) / f64::from(time_base.denominator); + Duration::try_from_secs_f64(seconds).ok() + } + + /// This packet's presentation time, when it has one. + #[must_use] + pub fn presentation_time(&self, time_base: URational) -> Option { + self.pts + .and_then(|pts| Self::timestamp_to_duration(pts, time_base)) + } +} + +/// Where to seek to. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[non_exhaustive] +pub enum SeekTo { + /// A timestamp in the target track's `time_base` units. + Timestamp(i64), + /// A position from the start of the track. + Time(Duration), +} + +/// How precisely to seek. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] +#[non_exhaustive] +pub enum SeekMode { + /// Land at or before the request, on a random access point. + /// + /// The default, and the only mode that lets decoding resume: a decoder + /// cannot start mid-GOP. Reaching the exact frame means decoding forward + /// and discarding, which is the caller's choice to make. + #[default] + Precise, + /// Land near the request, wherever the container's index is coarsest. + /// + /// May land *after* the requested position. For scrubbing, where latency + /// matters more than landing before a specific frame. + Coarse, +} + +#[cfg(test)] +mod tests { + use super::*; + + const MS: URational = URational { + numerator: 1, + denominator: 1000, + }; + + #[test] + fn timestamps_convert_against_the_time_base() { + // 1500 ticks of 1/1000 s is 1.5 s. + assert_eq!( + Packet::timestamp_to_duration(1500, MS), + Some(Duration::from_millis(1500)) + ); + assert_eq!(Packet::timestamp_to_duration(0, MS), Some(Duration::ZERO)); + } + + #[test] + fn negative_timestamps_have_no_duration() { + // Edit lists legitimately place samples before the presentation + // origin; a Duration cannot express that, so it is None rather than a + // panic or a wrapped huge value. + assert_eq!(Packet::timestamp_to_duration(-1, MS), None); + } + + #[test] + fn a_zero_denominator_time_base_does_not_divide_by_zero() { + let bad = URational { + numerator: 1, + denominator: 0, + }; + assert_eq!(Packet::timestamp_to_duration(1000, bad), None); + } + + #[test] + fn presentation_time_uses_pts_and_tolerates_its_absence() { + let mut p = Packet { + track: TrackId::new(1), + pts: Some(90_000), + dts: None, + duration: None, + is_keyframe: true, + data: Vec::new(), + }; + // A 90 kHz time base: 90000 ticks is one second. + let hz90 = URational { + numerator: 1, + denominator: 90_000, + }; + assert_eq!(p.presentation_time(hz90), Some(Duration::from_secs(1))); + + p.pts = None; + assert_eq!(p.presentation_time(hz90), None); + } + + #[test] + fn precise_is_the_default_seek_mode() { + assert_eq!(SeekMode::default(), SeekMode::Precise); + } +} diff --git a/crates/rawshift-video-core/src/track.rs b/crates/rawshift-video-core/src/track.rs new file mode 100644 index 0000000..08c6f45 --- /dev/null +++ b/crates/rawshift-video-core/src/track.rs @@ -0,0 +1,577 @@ +//! The track model: what a file contains, before any of it is decoded. +//! +//! Every field here comes from the container's headers, so a full track list +//! is available on any target with no decoder at all — see the "Behavior +//! without a hardware decoder" section of `docs/SUPPORT.md`. + +use std::fmt; +use std::time::Duration; + +use gamut_color::{ + ChromaSubsampling, ColorRange, ColourPrimaries, MatrixCoefficients, TransferCharacteristics, +}; +use rawshift_core::metadata::URational; +use rawshift_core::{BitDepth, ColorDescription, Dimensions}; + +use crate::id::{AudioCodecId, TrackId, TrackKind, VideoCodecId}; + +/// The codec configuration record a track carries out of band. +/// +/// For ISOBMFF this is the `avcC` or `hvcC` box *body* from the sample entry; +/// for Matroska it is `CodecPrivate`, which holds the same records. It carries +/// the parameter sets (SPS/PPS, and VPS for HEVC) and the NAL length-prefix +/// width every packet of the track uses, so a decoder cannot be opened without +/// it. +/// +/// Deliberately opaque: the bytes are a codec-defined structure, and rawshift +/// exposes them rather than a parsed form so that the parsed form stays an +/// implementation detail of whichever decoder consumes it. +#[derive(Clone, PartialEq, Eq, Hash, Default)] +#[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))] +pub struct CodecConfig(Vec); + +impl CodecConfig { + /// Wrap a configuration record body. + #[must_use] + pub const fn new(bytes: Vec) -> Self { + Self(bytes) + } + + /// The record body. + #[must_use] + pub fn as_bytes(&self) -> &[u8] { + &self.0 + } + + /// Whether the track carried no configuration record. + /// + /// Legal in some containers and for some codecs, but H.264 and HEVC + /// tracks without one cannot be decoded: there are no parameter sets and + /// no length-prefix width. + #[must_use] + pub fn is_empty(&self) -> bool { + self.0.is_empty() + } +} + +impl fmt::Debug for CodecConfig { + /// Prints the length rather than the bytes — a configuration record is + /// tens to hundreds of opaque bytes and dumping it buries every other + /// field of a `{:?}`-printed track. + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "CodecConfig({} bytes)", self.0.len()) + } +} + +/// Display rotation the container asks for, in degrees clockwise. +/// +/// Phones record in the sensor's native orientation and store the correction +/// in the container rather than rotating pixels, so honouring this is not +/// optional: ignoring it shows most phone video sideways. Decoded frames are +/// **not** rotated by rawshift — this is the instruction, applied by whoever +/// presents the frames. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Default)] +#[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))] +pub enum Rotation { + /// No rotation. + #[default] + None, + /// 90° clockwise. + Clockwise90, + /// 180°. + Clockwise180, + /// 270° clockwise (90° counter-clockwise). + Clockwise270, +} + +impl Rotation { + /// Degrees clockwise: 0, 90, 180, or 270. + #[must_use] + pub const fn degrees(self) -> u16 { + match self { + Self::None => 0, + Self::Clockwise90 => 90, + Self::Clockwise180 => 180, + Self::Clockwise270 => 270, + } + } + + /// Whether applying this rotation swaps width and height. + #[must_use] + pub const fn swaps_axes(self) -> bool { + matches!(self, Self::Clockwise90 | Self::Clockwise270) + } + + /// Nearest quarter turn to `degrees` clockwise, normalising any multiple + /// of 360 and any negative value. + /// + /// Containers store rotation as a transform matrix rather than an angle, + /// so a demuxer recovers an angle that may be slightly off or expressed + /// as a negative; snapping to the nearest quarter turn is the standard + /// reading and keeps a 89.98°-from-matrix value from becoming `None`. + #[must_use] + pub fn from_degrees(degrees: f64) -> Self { + if !degrees.is_finite() { + return Self::None; + } + // rem_euclid keeps the result non-negative, so -90 becomes 270. + let normalised = degrees.rem_euclid(360.0); + match ((normalised / 90.0).round() as i64) % 4 { + 1 => Self::Clockwise90, + 2 => Self::Clockwise180, + 3 => Self::Clockwise270, + _ => Self::None, + } + } + + /// Recover the rotation from an ISOBMFF `tkhd` display matrix. + /// + /// The matrix is stored as nine fixed-point values `[a, b, u, c, d, v, x, + /// y, w]`; only `a`, `b`, `c`, `d` carry rotation. They are 16.16 fixed + /// point, so callers pass them already converted to `f64`. + #[must_use] + pub fn from_display_matrix(a: f64, b: f64, c: f64, d: f64) -> Self { + // atan2(b, a) is the angle the matrix rotates by, counter-clockwise in + // the mathematical convention; the container's convention is clockwise. + // `c` and `d` are accepted so a caller passes the whole rotation + // sub-matrix, and are used only to reject a degenerate matrix. + if a == 0.0 && b == 0.0 && c == 0.0 && d == 0.0 { + return Self::None; + } + Self::from_degrees(b.atan2(a).to_degrees()) + } +} + +impl fmt::Display for Rotation { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "{}°", self.degrees()) + } +} + +/// A video track's header-derived description. +/// # Serialization +/// +/// Not `Serialize`/`Deserialize` even under the `serde` feature. The track +/// types embed upstream geometry and colour vocabulary (`Dimensions`, +/// `BitDepth`, `ChromaSubsampling`) that gamut does not implement serde for, +/// and skipping those fields would serialize a track that silently lost its +/// resolution and pixel format — worse than not serializing at all. The leaf +/// types here ([`CicpColor`], [`Rotation`], [`CodecConfig`], and everything in +/// [`crate::id`]) do support serde, so a caller can build and serialize its +/// own projection. +#[derive(Debug, Clone, PartialEq)] +#[non_exhaustive] +pub struct VideoTrack { + /// Container-assigned identifier. + pub id: TrackId, + /// The codec the samples are coded with. + pub codec: VideoCodecId, + /// Coded picture size, before [`rotation`](Self::rotation) is applied. + pub dimensions: Dimensions, + /// The track's colour signalling. + pub color: CicpColor, + /// Bits per sample. + pub bit_depth: BitDepth, + /// Chroma subsampling. + pub chroma: ChromaSubsampling, + /// Average frame rate, when the container states or implies one. + /// + /// Absent for variable-frame-rate recordings that state no average. Not a + /// substitute for per-packet timestamps, which remain authoritative. + pub frame_rate: Option, + /// The unit of this track's timestamps, in seconds. + /// + /// Every `pts`/`dts` on a [`Packet`](crate::packet::Packet) of this track + /// is a count of these. + pub time_base: URational, + /// Track duration, when the container states one. + pub duration: Option, + /// Number of coded pictures, when the container states one. + pub frame_count: Option, + /// Display rotation the container asks for. + pub rotation: Rotation, + /// The out-of-band codec configuration record. + pub codec_config: CodecConfig, +} + +/// A track's colour signalling, as the container states it. +/// +/// Stored as raw CICP (ITU-T H.273) code points rather than as typed enums, +/// for two reasons: it is exactly what the container carries, and a code point +/// no library names yet survives round-tripping instead of collapsing to +/// "unspecified". The accessors below type them where a type exists. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Default)] +#[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))] +pub struct CicpColor { + /// CICP colour primaries code point. 2 means unspecified. + pub primaries: u16, + /// CICP transfer characteristics code point. 2 means unspecified. + pub transfer: u16, + /// CICP matrix coefficients code point. 2 means unspecified. + /// + /// Its own axis, separate from the primaries: this is what maps YCbCr to + /// RGB, and getting it wrong tints the entire picture. + pub matrix: u16, + /// Whether samples occupy the full code range or the studio subset. + pub full_range: bool, +} + +impl CicpColor { + /// The code points meaning "unspecified" on every axis, studio range — + /// what an untagged camera file effectively asserts. + pub const UNSPECIFIED: Self = Self { + primaries: 2, + transfer: 2, + matrix: 2, + full_range: false, + }; + + /// The typed colour primaries, when the code point is one gamut names. + #[must_use] + pub fn primaries(self) -> Option { + ColourPrimaries::from_code_point(self.primaries) + } + + /// The typed transfer characteristics, when the code point is one gamut + /// names. + #[must_use] + pub fn transfer(self) -> Option { + TransferCharacteristics::from_code_point(self.transfer) + } + + /// The typed matrix coefficients, when the code point is one gamut names. + #[must_use] + pub fn matrix(self) -> Option { + MatrixCoefficients::from_code_point(self.matrix) + } + + /// Whether samples occupy the full code range or the studio subset. + #[must_use] + pub fn range(self) -> ColorRange { + if self.full_range { + ColorRange::Full + } else { + ColorRange::Limited + } + } + + /// The primaries-and-transfer pair as rawshift's [`ColorDescription`], + /// falling back to its `UNSPECIFIED` value for code points gamut does not + /// name. + #[must_use] + pub fn description(self) -> ColorDescription { + match (self.primaries(), self.transfer()) { + (Some(primaries), Some(transfer)) => ColorDescription { + primaries, + transfer, + }, + _ => ColorDescription::UNSPECIFIED, + } + } + + /// The matrix to actually decode with, resolving "unspecified" against the + /// coded picture height. + /// + /// Containers frequently omit the matrix, and decoders in the field all + /// fall back the same way: standard definition means BT.601, anything + /// larger means BT.709. Defaulting to BT.2020 is never right — files that + /// use it say so, because HDR is unusable if mistagged. + #[must_use] + pub fn resolved_matrix(self, coded_height: u32) -> MatrixCoefficients { + match self.matrix() { + Some(MatrixCoefficients::Unspecified) | None => { + if coded_height <= 576 { + MatrixCoefficients::Bt601 + } else { + MatrixCoefficients::Bt709 + } + } + Some(explicit) => explicit, + } + } +} + +/// An audio track's header-derived description. +/// +/// rawshift enumerates audio and yields its packets; it never decodes audio. +/// # Serialization +/// +/// Not `Serialize`/`Deserialize` even under the `serde` feature. The track +/// types embed upstream geometry and colour vocabulary (`Dimensions`, +/// `BitDepth`, `ChromaSubsampling`) that gamut does not implement serde for, +/// and skipping those fields would serialize a track that silently lost its +/// resolution and pixel format — worse than not serializing at all. The leaf +/// types here ([`CicpColor`], [`Rotation`], [`CodecConfig`], and everything in +/// [`crate::id`]) do support serde, so a caller can build and serialize its +/// own projection. +#[derive(Debug, Clone, PartialEq)] +#[non_exhaustive] +pub struct AudioTrack { + /// Container-assigned identifier. + pub id: TrackId, + /// The codec the samples are coded with. + pub codec: AudioCodecId, + /// Sampling rate in Hz, when stated. + pub sample_rate: Option, + /// Channel count, when stated. + pub channels: Option, + /// The unit of this track's timestamps, in seconds. + pub time_base: URational, + /// Track duration, when the container states one. + pub duration: Option, + /// The out-of-band codec configuration record, when the track has one. + pub codec_config: CodecConfig, + /// BCP 47 / ISO 639 language tag, when the container states one. + pub language: Option, +} + +/// A track rawshift enumerates but models no further. +/// # Serialization +/// +/// Not `Serialize`/`Deserialize` even under the `serde` feature. The track +/// types embed upstream geometry and colour vocabulary (`Dimensions`, +/// `BitDepth`, `ChromaSubsampling`) that gamut does not implement serde for, +/// and skipping those fields would serialize a track that silently lost its +/// resolution and pixel format — worse than not serializing at all. The leaf +/// types here ([`CicpColor`], [`Rotation`], [`CodecConfig`], and everything in +/// [`crate::id`]) do support serde, so a caller can build and serialize its +/// own projection. +#[derive(Debug, Clone, PartialEq)] +#[non_exhaustive] +pub struct OtherTrack { + /// Container-assigned identifier. + pub id: TrackId, + /// What the track carries. + pub kind: TrackKind, + /// The unit of this track's timestamps, in seconds. + pub time_base: URational, + /// Track duration, when the container states one. + pub duration: Option, + /// BCP 47 / ISO 639 language tag, when the container states one. + pub language: Option, +} + +/// One track of any kind. +/// # Serialization +/// +/// Not `Serialize`/`Deserialize` even under the `serde` feature. The track +/// types embed upstream geometry and colour vocabulary (`Dimensions`, +/// `BitDepth`, `ChromaSubsampling`) that gamut does not implement serde for, +/// and skipping those fields would serialize a track that silently lost its +/// resolution and pixel format — worse than not serializing at all. The leaf +/// types here ([`CicpColor`], [`Rotation`], [`CodecConfig`], and everything in +/// [`crate::id`]) do support serde, so a caller can build and serialize its +/// own projection. +#[derive(Debug, Clone, PartialEq)] +#[non_exhaustive] +pub enum Track { + /// A video track. + Video(VideoTrack), + /// An audio track. + Audio(AudioTrack), + /// A subtitle, timecode, or data track. + Other(OtherTrack), +} + +impl Track { + /// The track's container-assigned identifier. + #[must_use] + pub fn id(&self) -> TrackId { + match self { + Self::Video(t) => t.id, + Self::Audio(t) => t.id, + Self::Other(t) => t.id, + } + } + + /// What the track carries. + #[must_use] + pub fn kind(&self) -> TrackKind { + match self { + Self::Video(_) => TrackKind::Video, + Self::Audio(_) => TrackKind::Audio, + Self::Other(t) => t.kind, + } + } + + /// The video track, if this is one. + #[must_use] + pub fn as_video(&self) -> Option<&VideoTrack> { + match self { + Self::Video(t) => Some(t), + _ => None, + } + } + + /// The audio track, if this is one. + #[must_use] + pub fn as_audio(&self) -> Option<&AudioTrack> { + match self { + Self::Audio(t) => Some(t), + _ => None, + } + } + + /// The unit of this track's timestamps, in seconds. + #[must_use] + pub fn time_base(&self) -> URational { + match self { + Self::Video(t) => t.time_base, + Self::Audio(t) => t.time_base, + Self::Other(t) => t.time_base, + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn rotation_degrees_round_trip() { + for r in [ + Rotation::None, + Rotation::Clockwise90, + Rotation::Clockwise180, + Rotation::Clockwise270, + ] { + assert_eq!(Rotation::from_degrees(f64::from(r.degrees())), r); + } + } + + #[test] + fn rotation_normalises_negative_and_wrapped_angles() { + assert_eq!(Rotation::from_degrees(-90.0), Rotation::Clockwise270); + assert_eq!(Rotation::from_degrees(450.0), Rotation::Clockwise90); + assert_eq!(Rotation::from_degrees(720.0), Rotation::None); + assert_eq!(Rotation::from_degrees(-360.0), Rotation::None); + } + + #[test] + fn rotation_snaps_near_misses_from_fixed_point_matrices() { + // A 16.16 matrix cannot represent 90° exactly; the recovered angle is + // a hair off and must not collapse to None. + assert_eq!(Rotation::from_degrees(89.98), Rotation::Clockwise90); + assert_eq!(Rotation::from_degrees(270.03), Rotation::Clockwise270); + } + + #[test] + fn rotation_rejects_non_finite_angles() { + assert_eq!(Rotation::from_degrees(f64::NAN), Rotation::None); + assert_eq!(Rotation::from_degrees(f64::INFINITY), Rotation::None); + } + + #[test] + fn display_matrix_recovers_the_quarter_turns_phones_write() { + // Identity: no rotation. + assert_eq!( + Rotation::from_display_matrix(1.0, 0.0, 0.0, 1.0), + Rotation::None + ); + // The matrix iOS writes for a portrait capture: 90° clockwise. + assert_eq!( + Rotation::from_display_matrix(0.0, 1.0, -1.0, 0.0), + Rotation::Clockwise90 + ); + // Upside-down landscape. + assert_eq!( + Rotation::from_display_matrix(-1.0, 0.0, 0.0, -1.0), + Rotation::Clockwise180 + ); + // 270°. + assert_eq!( + Rotation::from_display_matrix(0.0, -1.0, 1.0, 0.0), + Rotation::Clockwise270 + ); + } + + #[test] + fn degenerate_display_matrix_is_not_a_rotation() { + assert_eq!( + Rotation::from_display_matrix(0.0, 0.0, 0.0, 0.0), + Rotation::None + ); + } + + #[test] + fn only_quarter_turns_swap_axes() { + assert!(Rotation::Clockwise90.swaps_axes()); + assert!(Rotation::Clockwise270.swaps_axes()); + assert!(!Rotation::None.swaps_axes()); + assert!(!Rotation::Clockwise180.swaps_axes()); + } + + #[test] + fn unspecified_matrix_resolves_by_height() { + let c = CicpColor::UNSPECIFIED; + // SD heights take BT.601 ... + assert_eq!(c.resolved_matrix(480), MatrixCoefficients::Bt601); + assert_eq!(c.resolved_matrix(576), MatrixCoefficients::Bt601); + // ... anything larger takes BT.709. + assert_eq!(c.resolved_matrix(720), MatrixCoefficients::Bt709); + assert_eq!(c.resolved_matrix(2160), MatrixCoefficients::Bt709); + } + + #[test] + fn an_unnamed_matrix_code_point_still_resolves_rather_than_panicking() { + // Code point 7 (SMPTE 240M) is real but not modelled by gamut. It must + // fall back like "unspecified" rather than being lost or fatal. + let exotic = CicpColor { + matrix: 7, + ..CicpColor::UNSPECIFIED + }; + assert_eq!(exotic.matrix(), None); + assert_eq!(exotic.resolved_matrix(1080), MatrixCoefficients::Bt709); + // ... and the raw code point survives for a caller that does know it. + assert_eq!(exotic.matrix, 7); + } + + #[test] + fn resolving_never_overrides_an_explicit_matrix() { + // A 4K BT.601 file stays BT.601, and an SD BT.2020 file stays BT.2020. + let bt601 = CicpColor { + matrix: MatrixCoefficients::Bt601.code_point(), + ..CicpColor::UNSPECIFIED + }; + assert_eq!(bt601.resolved_matrix(2160), MatrixCoefficients::Bt601); + + let bt2020 = CicpColor { + matrix: MatrixCoefficients::Bt2020Ncl.code_point(), + ..CicpColor::UNSPECIFIED + }; + assert_eq!(bt2020.resolved_matrix(480), MatrixCoefficients::Bt2020Ncl); + } + + #[test] + fn range_defaults_to_studio_which_is_what_cameras_write() { + assert_eq!(CicpColor::UNSPECIFIED.range(), ColorRange::Limited); + assert_eq!(CicpColor::default().range(), ColorRange::Limited); + let full = CicpColor { + full_range: true, + ..CicpColor::UNSPECIFIED + }; + assert_eq!(full.range(), ColorRange::Full); + } + + #[test] + fn typed_accessors_round_trip_named_code_points() { + let rec709 = CicpColor { + primaries: ColourPrimaries::Bt709.code_point(), + transfer: TransferCharacteristics::Bt709.code_point(), + matrix: MatrixCoefficients::Bt709.code_point(), + full_range: false, + }; + assert_eq!(rec709.primaries(), Some(ColourPrimaries::Bt709)); + assert_eq!(rec709.matrix(), Some(MatrixCoefficients::Bt709)); + assert_eq!(rec709.description().primaries, ColourPrimaries::Bt709); + } + + #[test] + fn codec_config_debug_reports_length_not_bytes() { + let cfg = CodecConfig::new(vec![0xde, 0xad, 0xbe, 0xef]); + assert_eq!(format!("{cfg:?}"), "CodecConfig(4 bytes)"); + assert_eq!(cfg.as_bytes(), &[0xde, 0xad, 0xbe, 0xef]); + assert!(!cfg.is_empty()); + assert!(CodecConfig::default().is_empty()); + } +} From 7de9589c318a214989649a355d42cef1c58fad38 Mon Sep 17 00:00:00 2001 From: Justin Chung Date: Fri, 28 Aug 2026 21:53:01 -0400 Subject: [PATCH 04/13] feat(video-core): derive random-access points from the bitstream MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add NAL framing for the H.264/HEVC families: length-prefix width from the avcC/hvcC record, an iterator over an access unit's NAL units, and is_random_access_point. This exists because the chosen demux backend exposes no sync-sample flag at all — not on its packet type, not on its track type; MP4's stss table is parsed only for its internal seeking. Reading IDR (H.264 NAL type 5) and IRAP (HEVC NAL types 16..=23) out of the bitstream is the only source available, and is the better one regardless: container sync tables are a well-known source of wrong keyframe flags. Framing and NAL headers only — no slice headers, no parameter sets. It lives in the vocabulary crate because both the demuxers and the decoders walk this framing. Truncated prefixes and payloads terminate the iterator cleanly rather than yielding a short NAL, since containers are untrusted input and a truncated tail is what a damaged file looks like. Refs #39 --- crates/rawshift-video-core/src/lib.rs | 2 + crates/rawshift-video-core/src/nal.rs | 350 ++++++++++++++++++++++++++ 2 files changed, 352 insertions(+) create mode 100644 crates/rawshift-video-core/src/nal.rs diff --git a/crates/rawshift-video-core/src/lib.rs b/crates/rawshift-video-core/src/lib.rs index 03e155b..e9c7806 100644 --- a/crates/rawshift-video-core/src/lib.rs +++ b/crates/rawshift-video-core/src/lib.rs @@ -44,6 +44,7 @@ pub mod error; pub mod frame; pub mod id; pub mod metadata; +pub mod nal; pub mod packet; pub mod track; @@ -53,6 +54,7 @@ pub use error::{VideoError, VideoResult}; pub use frame::{FrameDesc, FramePixelFormat, Plane, VideoFrame}; pub use id::{AudioCodecId, ContainerId, TrackId, TrackKind, VideoCodecId}; pub use metadata::{ContainerMetadata, VideoMetadata}; +pub use nal::{NalLengthSize, is_random_access_point, split_length_prefixed}; pub use packet::{Packet, SeekMode, SeekTo}; pub use track::{AudioTrack, CicpColor, CodecConfig, OtherTrack, Rotation, Track, VideoTrack}; diff --git a/crates/rawshift-video-core/src/nal.rs b/crates/rawshift-video-core/src/nal.rs new file mode 100644 index 0000000..1775f96 --- /dev/null +++ b/crates/rawshift-video-core/src/nal.rs @@ -0,0 +1,350 @@ +//! NAL-unit framing for the H.264 and HEVC families. +//! +//! MP4 and Matroska store H.264/HEVC access units as **length-prefixed** NAL +//! units — never Annex B start codes — with the prefix width declared once, in +//! the track's `avcC`/`hvcC` configuration record. Both the demuxers and the +//! decoders need to walk that framing, so it lives here rather than in either. +//! +//! This module reads *framing and NAL headers only*. It never parses a slice +//! header, a parameter set, or anything else that would make it a decoder. + +use crate::id::VideoCodecId; +use crate::track::CodecConfig; + +/// The width, in bytes, of the length prefix before each NAL unit. +/// +/// Legal values are 1, 2 and 4. Three-byte prefixes are representable in the +/// configuration record's two-bit field but are not legal, and are rejected. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +pub struct NalLengthSize(u8); + +impl NalLengthSize { + /// Wrap a prefix width, rejecting anything but 1, 2, or 4. + #[must_use] + pub const fn new(bytes: u8) -> Option { + match bytes { + 1 | 2 | 4 => Some(Self(bytes)), + _ => None, + } + } + + /// The width in bytes. + #[must_use] + pub const fn get(self) -> usize { + self.0 as usize + } + + /// Read the prefix width from an `avcC` or `hvcC` record body. + /// + /// Both records store `lengthSizeMinusOne` in the low two bits of a byte: + /// offset 4 for `avcC` (ISO/IEC 14496-15 §5.3.3.1), offset 21 for `hvcC` + /// (§8.3.3.1). + /// + /// Returns `None` for a record that is too short, states an illegal + /// 3-byte prefix, or belongs to a codec with no such record. + #[must_use] + pub fn from_config(codec: VideoCodecId, config: &CodecConfig) -> Option { + let bytes = config.as_bytes(); + let offset = match codec { + VideoCodecId::H264 => 4, + VideoCodecId::Hevc => 21, + _ => return None, + }; + let field = bytes.get(offset)?; + Self::new((field & 0b11) + 1) + } +} + +/// Walk the length-prefixed NAL units of one access unit. +/// +/// Yields each NAL's payload, prefix excluded. Stops at the first prefix that +/// overruns the buffer rather than panicking or yielding a truncated unit: a +/// container is untrusted input, and a truncated tail is the common shape of +/// a damaged file. +pub fn split_length_prefixed(data: &[u8], length_size: NalLengthSize) -> NalUnits<'_> { + NalUnits { + data, + offset: 0, + length_size: length_size.get(), + } +} + +/// Iterator over the NAL units of one access unit. See +/// [`split_length_prefixed`]. +#[derive(Debug, Clone)] +pub struct NalUnits<'a> { + data: &'a [u8], + offset: usize, + length_size: usize, +} + +impl<'a> Iterator for NalUnits<'a> { + type Item = &'a [u8]; + + fn next(&mut self) -> Option<&'a [u8]> { + let header_end = self.offset.checked_add(self.length_size)?; + if header_end > self.data.len() { + return None; + } + + let mut length = 0usize; + for &byte in &self.data[self.offset..header_end] { + // At most 4 bytes, so this cannot overflow a usize on any target + // rawshift supports (all are 32-bit or wider). + length = (length << 8) | byte as usize; + } + + let payload_end = header_end.checked_add(length)?; + if payload_end > self.data.len() { + // Truncated: stop cleanly rather than yielding a short NAL. + return None; + } + + self.offset = payload_end; + Some(&self.data[header_end..payload_end]) + } +} + +/// Whether an access unit is a random access point — a picture a decoder can +/// start on with no earlier data. +/// +/// Read from the bitstream rather than from the container's sync-sample table, +/// because the bitstream is authoritative and container indices are a +/// well-known source of wrong keyframe flags. The chosen demux backend also +/// exposes no sync-sample flag at all, so this is the only source available. +/// +/// Returns `false` for codecs whose NAL framing this module does not model, +/// and for a malformed or empty access unit. +#[must_use] +pub fn is_random_access_point( + codec: VideoCodecId, + config: &CodecConfig, + access_unit: &[u8], +) -> bool { + let Some(length_size) = NalLengthSize::from_config(codec, config) else { + return false; + }; + + split_length_prefixed(access_unit, length_size).any(|nal| match codec { + // H.264 (ITU-T H.264 Table 7-1): the type is the low 5 bits of the + // one-byte header, and type 5 is an IDR slice. + VideoCodecId::H264 => nal.first().is_some_and(|h| h & 0x1f == 5), + // HEVC (ITU-T H.265 Table 7-1): the type is bits 1..6 of the + // two-byte header. Types 16..=23 are the IRAP range — BLA, IDR and + // CRA — every one of which is a random access point. + VideoCodecId::Hevc => nal + .first() + .is_some_and(|h| matches!((h >> 1) & 0x3f, 16..=23)), + _ => false, + }) +} + +#[cfg(test)] +mod tests { + use super::*; + + /// Frame `nals` with 4-byte length prefixes, as MP4 stores them. + fn framed(nals: &[&[u8]]) -> Vec { + let mut out = Vec::new(); + for nal in nals { + out.extend_from_slice(&(nal.len() as u32).to_be_bytes()); + out.extend_from_slice(nal); + } + out + } + + fn four() -> NalLengthSize { + NalLengthSize::new(4).expect("4 is legal") + } + + #[test] + fn only_legal_prefix_widths_are_accepted() { + assert!(NalLengthSize::new(1).is_some()); + assert!(NalLengthSize::new(2).is_some()); + assert!(NalLengthSize::new(4).is_some()); + // 3 is representable in the two-bit field but is not legal. + assert!(NalLengthSize::new(3).is_none()); + assert!(NalLengthSize::new(0).is_none()); + assert!(NalLengthSize::new(8).is_none()); + } + + #[test] + fn avcc_length_size_comes_from_byte_four() { + // lengthSizeMinusOne = 3 -> 4-byte prefixes. 0xff sets the reserved + // high bits too, as real records do. + let cfg = CodecConfig::new(vec![0x01, 0x64, 0x00, 0x28, 0xff]); + assert_eq!( + NalLengthSize::from_config(VideoCodecId::H264, &cfg).map(NalLengthSize::get), + Some(4) + ); + + // lengthSizeMinusOne = 1 -> 2-byte prefixes. + let cfg = CodecConfig::new(vec![0x01, 0x64, 0x00, 0x28, 0xfd]); + assert_eq!( + NalLengthSize::from_config(VideoCodecId::H264, &cfg).map(NalLengthSize::get), + Some(2) + ); + } + + #[test] + fn hvcc_length_size_comes_from_byte_twenty_one() { + let mut bytes = vec![0u8; 22]; + bytes[21] = 0xff; // lengthSizeMinusOne = 3 + let cfg = CodecConfig::new(bytes); + assert_eq!( + NalLengthSize::from_config(VideoCodecId::Hevc, &cfg).map(NalLengthSize::get), + Some(4) + ); + } + + #[test] + fn an_illegal_three_byte_prefix_is_rejected_not_rounded() { + // lengthSizeMinusOne = 2 means a 3-byte prefix, which is illegal. + let cfg = CodecConfig::new(vec![0x01, 0x64, 0x00, 0x28, 0xfe]); + assert_eq!(NalLengthSize::from_config(VideoCodecId::H264, &cfg), None); + } + + #[test] + fn a_truncated_config_record_yields_no_length_size() { + assert_eq!( + NalLengthSize::from_config(VideoCodecId::H264, &CodecConfig::new(vec![0x01, 0x64])), + None + ); + assert_eq!( + NalLengthSize::from_config(VideoCodecId::Hevc, &CodecConfig::new(vec![0u8; 10])), + None + ); + assert_eq!( + NalLengthSize::from_config(VideoCodecId::H264, &CodecConfig::default()), + None + ); + } + + #[test] + fn codecs_without_nal_framing_have_no_length_size() { + let cfg = CodecConfig::new(vec![0xff; 32]); + assert_eq!(NalLengthSize::from_config(VideoCodecId::Av1, &cfg), None); + assert_eq!(NalLengthSize::from_config(VideoCodecId::ProRes, &cfg), None); + } + + #[test] + fn splitting_yields_each_nal_without_its_prefix() { + let au = framed(&[&[0x65, 0xaa], &[0x41, 0xbb, 0xcc]]); + let nals: Vec<_> = split_length_prefixed(&au, four()).collect(); + assert_eq!(nals, vec![&[0x65, 0xaa][..], &[0x41, 0xbb, 0xcc][..]]); + } + + #[test] + fn splitting_handles_every_legal_prefix_width() { + for width in [1u8, 2, 4] { + let size = NalLengthSize::new(width).expect("legal"); + let mut au = Vec::new(); + let payload = [0x65, 0x01, 0x02]; + au.extend_from_slice(&(payload.len() as u32).to_be_bytes()[4 - size.get()..]); + au.extend_from_slice(&payload); + + let nals: Vec<_> = split_length_prefixed(&au, size).collect(); + assert_eq!(nals, vec![&payload[..]], "width {width}"); + } + } + + #[test] + fn a_truncated_trailing_nal_is_dropped_not_yielded_short() { + // A well-formed NAL, then a prefix claiming 100 bytes with 2 present. + let mut au = framed(&[&[0x65, 0xaa]]); + au.extend_from_slice(&100u32.to_be_bytes()); + au.extend_from_slice(&[0xde, 0xad]); + + let nals: Vec<_> = split_length_prefixed(&au, four()).collect(); + assert_eq!(nals, vec![&[0x65, 0xaa][..]], "the good NAL must survive"); + } + + #[test] + fn a_truncated_prefix_terminates_cleanly() { + let au = vec![0x00, 0x00]; // two bytes of a four-byte prefix + assert_eq!(split_length_prefixed(&au, four()).count(), 0); + } + + #[test] + fn empty_input_yields_nothing() { + assert_eq!(split_length_prefixed(&[], four()).count(), 0); + } + + #[test] + fn a_zero_length_nal_does_not_stall_the_iterator() { + // A zero-length NAL is degenerate but must not loop forever. + let au = framed(&[&[], &[0x65]]); + let nals: Vec<_> = split_length_prefixed(&au, four()).collect(); + assert_eq!(nals.len(), 2); + assert!(nals[0].is_empty()); + } + + fn avcc() -> CodecConfig { + CodecConfig::new(vec![0x01, 0x64, 0x00, 0x28, 0xff]) + } + + fn hvcc() -> CodecConfig { + let mut bytes = vec![0u8; 22]; + bytes[21] = 0xff; + CodecConfig::new(bytes) + } + + #[test] + fn h264_idr_slices_are_random_access_points() { + // NAL type 5 = IDR. + let idr = framed(&[&[0x65, 0x88]]); + assert!(is_random_access_point(VideoCodecId::H264, &avcc(), &idr)); + + // NAL type 1 = a non-IDR slice. + let inter = framed(&[&[0x41, 0x9a]]); + assert!(!is_random_access_point(VideoCodecId::H264, &avcc(), &inter)); + } + + #[test] + fn h264_parameter_sets_before_the_idr_do_not_hide_it() { + // Real keyframe access units lead with SPS (7) and PPS (8). + let au = framed(&[&[0x67, 0x64], &[0x68, 0xee], &[0x65, 0x88]]); + assert!(is_random_access_point(VideoCodecId::H264, &avcc(), &au)); + } + + #[test] + fn hevc_irap_types_are_random_access_points() { + // Types 16..=23 are the IRAP range; check the edges and the middle. + for nal_type in [16u8, 19, 20, 21, 23] { + let au = framed(&[&[nal_type << 1, 0x01]]); + assert!( + is_random_access_point(VideoCodecId::Hevc, &hvcc(), &au), + "type {nal_type} should be IRAP" + ); + } + } + + #[test] + fn hevc_trailing_pictures_are_not_random_access_points() { + // Types 0..=15 are the non-IRAP leading/trailing range; 24+ is + // reserved or non-VCL. + for nal_type in [0u8, 1, 15, 24, 32] { + let au = framed(&[&[nal_type << 1, 0x01]]); + assert!( + !is_random_access_point(VideoCodecId::Hevc, &hvcc(), &au), + "type {nal_type} should not be IRAP" + ); + } + } + + #[test] + fn a_codec_without_framing_is_never_a_random_access_point() { + let au = framed(&[&[0x65, 0x88]]); + assert!(!is_random_access_point(VideoCodecId::ProRes, &avcc(), &au)); + } + + #[test] + fn a_malformed_access_unit_is_not_a_random_access_point() { + assert!(!is_random_access_point(VideoCodecId::H264, &avcc(), &[])); + assert!(!is_random_access_point( + VideoCodecId::H264, + &avcc(), + &[0x00, 0x00] + )); + } +} From 1b27d186e56bdf0ca0d45a965bd664c1a0ca9d46 Mon Sep 17 00:00:00 2001 From: Justin Chung Date: Fri, 28 Aug 2026 21:59:22 -0400 Subject: [PATCH 05/13] feat(video-isobmff): demux MP4, M4V and QuickTime Wraps symphonia-format-isomp4 (MPL-2.0, pure Rust, no build script, so it builds on every target in docs/SUPPORT.md) and maps its tracks, packets and seeking onto rawshift's model. No symphonia type reaches the public API; its errors are flattened into VideoError::Container at the boundary. The backend leaves three gaps this crate fills: - Rotation. It does not expose the tkhd display matrix, so a focused header walk reads it. Without this most phone video presents sideways, since phones record in the sensor's orientation and correct in the container. - ftyp brands and mvhd timescale/times, needed for metadata and to tell MP4 from QuickTime. - Random access points, which come from the bitstream via rawshift-video-core rather than from a sync-sample table the backend does not expose. The header walk is deliberately not a demuxer and must not become one: it reads four small boxes, bounds-checks every field, and caps recursion depth and box size, because it parses untrusted bytes. Malformed structure returns what was recovered rather than failing the open, since these facts are supplementary and such a file should still demux. Two limitations are documented rather than papered over: ISOBMFF colour lives in a colr box the backend does not surface, so tracks report unspecified colour and resolve by picture height; and ProRes tracks are invisible because the backend does not recognise ap4h/apch/apcn sample entries. Adds constructors to the video-core track types, which are another crate. Refs #39 --- Cargo.lock | 76 +++ Cargo.toml | 13 + crates/rawshift-video-core/src/packet.rs | 17 + crates/rawshift-video-core/src/track.rs | 65 +++ crates/rawshift-video-isobmff/Cargo.toml | 23 + crates/rawshift-video-isobmff/README.md | 27 + crates/rawshift-video-isobmff/src/boxes.rs | 616 +++++++++++++++++++++ crates/rawshift-video-isobmff/src/demux.rs | 472 ++++++++++++++++ crates/rawshift-video-isobmff/src/lib.rs | 38 ++ crates/rawshift-video-isobmff/src/sniff.rs | 112 ++++ 10 files changed, 1459 insertions(+) create mode 100644 crates/rawshift-video-isobmff/Cargo.toml create mode 100644 crates/rawshift-video-isobmff/README.md create mode 100644 crates/rawshift-video-isobmff/src/boxes.rs create mode 100644 crates/rawshift-video-isobmff/src/demux.rs create mode 100644 crates/rawshift-video-isobmff/src/lib.rs create mode 100644 crates/rawshift-video-isobmff/src/sniff.rs diff --git a/Cargo.lock b/Cargo.lock index 4ed7deb..c992f7f 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1010,6 +1010,15 @@ dependencies = [ "windows-sys", ] +[[package]] +name = "num-complex" +version = "0.4.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "73f88a1307638156682bada9d7604135552957b7818057dcef22705b4d509495" +dependencies = [ + "num-traits", +] + [[package]] name = "num-derive" version = "0.4.2" @@ -1498,6 +1507,17 @@ dependencies = [ "thiserror", ] +[[package]] +name = "rawshift-video-isobmff" +version = "0.1.1" +dependencies = [ + "rawshift-core", + "rawshift-video-core", + "symphonia-core", + "symphonia-format-isomp4", + "tracing", +] + [[package]] name = "rayon" version = "1.11.0" @@ -1550,6 +1570,12 @@ dependencies = [ "regex-syntax", ] +[[package]] +name = "regex-lite" +version = "0.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cab834c73d247e67f4fae452806d17d3c7501756d98c8808d7c9c7aa7d18f973" + [[package]] name = "regex-syntax" version = "0.8.10" @@ -1766,6 +1792,56 @@ dependencies = [ "siphasher", ] +[[package]] +name = "symphonia-common" +version = "0.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2acc3fcc18ec9b8cdd48614e259c4cf0d27b71d41e5d9b120b42c5adab12d7c4" +dependencies = [ + "log", + "symphonia-core", + "symphonia-metadata", +] + +[[package]] +name = "symphonia-core" +version = "0.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "01c412864d599d4750d0c3d684d7e093ec05e5309681ef5252cc1096a437f6e0" +dependencies = [ + "bitflags 2.11.0", + "bytemuck", + "lazy_static", + "log", + "num-complex", + "smallvec", +] + +[[package]] +name = "symphonia-format-isomp4" +version = "0.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0e681a70e1870d34e02abf1dbc51e4267c3f1827801474e8870be8c689fc4dc3" +dependencies = [ + "log", + "symphonia-common", + "symphonia-core", + "symphonia-metadata", +] + +[[package]] +name = "symphonia-metadata" +version = "0.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "83713a97705d77bdef7cdbc0768fd6e5a54e4cd7e48d60a806ae85639e2c87c6" +dependencies = [ + "lazy_static", + "log", + "regex-lite", + "smallvec", + "symphonia-core", +] + [[package]] name = "syn" version = "2.0.117" diff --git a/Cargo.toml b/Cargo.toml index 5263c40..c4c0251 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -28,6 +28,7 @@ members = [ "crates/rawshift-image-webp", "crates/rawshift-video", "crates/rawshift-video-core", + "crates/rawshift-video-isobmff", ] [workspace.package] @@ -62,6 +63,17 @@ gamut-heic = "0.2.2" # backend-less HEIC enumeration/decode-pipeline tests. gamut-isobmff = "2.0.1" +# symphonia — demuxing for rawshift's video crates. MPL-2.0 (rawshift's own +# license), pure Rust with no build script, MSRV 1.85, so it builds on every +# target in docs/SUPPORT.md including wasm32, musl, iOS and Android. gamut's +# charter excludes video, so this is a direct dependency rather than an +# upstream ask — see "Video is outside gamut's charter" in AGENTS.md. +# default-features off keeps symphonia-core's optional rustfft (audio DSP) out +# of the tree; rawshift never decodes audio. +symphonia-core = { version = "0.6.1", default-features = false } +symphonia-format-isomp4 = "0.6.1" +symphonia-format-mkv = "0.6.1" + # Shared infrastructure dependencies (used across multiple workspace crates). thiserror = "2.0" tracing = "0.1" @@ -95,6 +107,7 @@ rawshift-image-tiff = { path = "crates/rawshift-image-tiff", version = "0.1.1", rawshift-image-webp = { path = "crates/rawshift-image-webp", version = "0.1.1", default-features = false } rawshift-video = { path = "crates/rawshift-video", version = "0.1.1" } rawshift-video-core = { path = "crates/rawshift-video-core", version = "0.1.1" } +rawshift-video-isobmff = { path = "crates/rawshift-video-isobmff", version = "0.1.1" } # Profiles must live at the workspace root — Cargo ignores profiles declared in # member manifests. diff --git a/crates/rawshift-video-core/src/packet.rs b/crates/rawshift-video-core/src/packet.rs index 80fa3f3..1100c9d 100644 --- a/crates/rawshift-video-core/src/packet.rs +++ b/crates/rawshift-video-core/src/packet.rs @@ -41,6 +41,23 @@ pub struct Packet { } impl Packet { + /// A packet carrying `data` for `track`, with no timing and not marked as + /// a random access point. + /// + /// [`Packet`] is `#[non_exhaustive]`, so demuxers in other crates build + /// one through this and then set the fields the container supplied. + #[must_use] + pub const fn new(track: TrackId, data: Vec) -> Self { + Self { + track, + pts: None, + dts: None, + duration: None, + is_keyframe: false, + data, + } + } + /// Convert a timestamp in `time_base` units to a [`Duration`]. /// /// Returns `None` for an absent or negative timestamp: a `Duration` cannot diff --git a/crates/rawshift-video-core/src/track.rs b/crates/rawshift-video-core/src/track.rs index 08c6f45..0979f81 100644 --- a/crates/rawshift-video-core/src/track.rs +++ b/crates/rawshift-video-core/src/track.rs @@ -194,6 +194,38 @@ pub struct VideoTrack { pub codec_config: CodecConfig, } +impl VideoTrack { + /// A video track with `codec`, `dimensions` and `time_base` set and every + /// other field at its neutral default. + /// + /// These types are `#[non_exhaustive]`, so demuxers in other crates build + /// one through this and then assign the fields their container supplied. + /// A new field therefore arrives with a default rather than breaking every + /// demuxer. + #[must_use] + pub fn new( + id: TrackId, + codec: VideoCodecId, + dimensions: Dimensions, + time_base: URational, + ) -> Self { + Self { + id, + codec, + dimensions, + color: CicpColor::UNSPECIFIED, + bit_depth: BitDepth::Eight, + chroma: ChromaSubsampling::Cs420, + frame_rate: None, + time_base, + duration: None, + frame_count: None, + rotation: Rotation::None, + codec_config: CodecConfig::default(), + } + } +} + /// A track's colour signalling, as the container states it. /// /// Stored as raw CICP (ITU-T H.273) code points rather than as typed enums, @@ -325,6 +357,24 @@ pub struct AudioTrack { pub language: Option, } +impl AudioTrack { + /// An audio track with `codec` and `time_base` set and every other field + /// at its neutral default. See [`VideoTrack::new`]. + #[must_use] + pub fn new(id: TrackId, codec: AudioCodecId, time_base: URational) -> Self { + Self { + id, + codec, + sample_rate: None, + channels: None, + time_base, + duration: None, + codec_config: CodecConfig::default(), + language: None, + } + } +} + /// A track rawshift enumerates but models no further. /// # Serialization /// @@ -351,6 +401,21 @@ pub struct OtherTrack { pub language: Option, } +impl OtherTrack { + /// A track of `kind` with `time_base` set and every other field at its + /// neutral default. See [`VideoTrack::new`]. + #[must_use] + pub fn new(id: TrackId, kind: TrackKind, time_base: URational) -> Self { + Self { + id, + kind, + time_base, + duration: None, + language: None, + } + } +} + /// One track of any kind. /// # Serialization /// diff --git a/crates/rawshift-video-isobmff/Cargo.toml b/crates/rawshift-video-isobmff/Cargo.toml new file mode 100644 index 0000000..c0e8a5f --- /dev/null +++ b/crates/rawshift-video-isobmff/Cargo.toml @@ -0,0 +1,23 @@ +[package] +name = "rawshift-video-isobmff" +version.workspace = true +edition.workspace = true +rust-version.workspace = true +license.workspace = true +repository.workspace = true +homepage.workspace = true +description = "MP4, M4V and QuickTime (MOV) demuxing and metadata for rawshift" +documentation = "https://docs.rs/rawshift-video-isobmff" +keywords = ["video", "mp4", "mov", "quicktime", "isobmff"] +categories = ["multimedia::video"] +readme = "README.md" + +[dependencies] +rawshift-core = { workspace = true } +rawshift-video-core = { workspace = true } +symphonia-core = { workspace = true } +symphonia-format-isomp4 = { workspace = true } +tracing = { workspace = true } + +[features] +default = [] diff --git a/crates/rawshift-video-isobmff/README.md b/crates/rawshift-video-isobmff/README.md new file mode 100644 index 0000000..985dc6a --- /dev/null +++ b/crates/rawshift-video-isobmff/README.md @@ -0,0 +1,27 @@ +# rawshift-video-isobmff + +MP4, M4V and QuickTime (MOV) support for +[rawshift](https://github.com/visualcommons/rawshift). + +Demuxing is backed by +[`symphonia-format-isomp4`](https://crates.io/crates/symphonia-format-isomp4) +(MPL-2.0, pure Rust, no build script), which parses the movie box tree, the +sample tables, fragments, and the `avc1`/`hvc1`/`hev1`/`av01` sample entries +rawshift needs. + +Two things symphonia does not surface, which this crate reads itself from a +focused header walk: + +- the `tkhd` display matrix, and so track rotation — without which most phone + video presents sideways; +- the `ftyp` brands and `mvhd` timescale and times. + +Sync samples are a third: symphonia exposes no keyframe flag, so random access +points are read out of the bitstream by `rawshift-video-core`, which is the +authoritative source anyway. + +No symphonia type appears in this crate's public API. + +## License + +Licensed under [MPL-2.0](../../LICENSE). diff --git a/crates/rawshift-video-isobmff/src/boxes.rs b/crates/rawshift-video-isobmff/src/boxes.rs new file mode 100644 index 0000000..53f09ab --- /dev/null +++ b/crates/rawshift-video-isobmff/src/boxes.rs @@ -0,0 +1,616 @@ +//! A focused ISOBMFF header walk for the facts the demux backend does not +//! surface. +//! +//! This is **not** a demuxer and must not grow into one. It reads box headers +//! and four small boxes — `ftyp`, `mvhd`, `tkhd`, `mdhd` — because +//! `symphonia-format-isomp4` exposes neither the track display matrix (and so +//! no rotation, without which most phone video presents sideways) nor the +//! `ftyp` brands. Everything else about the file comes from symphonia. +//! +//! Every read is bounds-checked and every container descent is depth- and +//! size-limited: this parses attacker-supplied bytes. + +use std::io::{self, Read, Seek, SeekFrom}; + +use rawshift_video_core::{ContainerId, Rotation, VideoResult}; + +/// How deep to descend into nested boxes. +/// +/// The tree this walks is `moov > trak > mdia > mdhd`, so four levels are +/// enough; the limit stops a crafted file from recursing without bound. +const MAX_DEPTH: u32 = 8; + +/// The largest box header this walker will honour, as a sanity bound against +/// a crafted 64-bit size. +const MAX_BOX_SIZE: u64 = 1 << 40; + +/// What the header walk recovered. +#[derive(Debug, Clone, Default, PartialEq)] +pub struct BoxSurvey { + /// `ftyp` major brand, then compatible brands. + pub brands: Vec, + /// `mvhd` timescale, in ticks per second. + pub timescale: Option, + /// `mvhd` duration, in timescale ticks. + pub duration: Option, + /// `mvhd` creation time, in seconds since 1904-01-01 UTC. + pub creation_time: Option, + /// `mvhd` modification time, in the same epoch. + pub modification_time: Option, + /// Per-track facts, keyed by `tkhd` track id. + pub tracks: Vec, +} + +/// Per-track facts from `tkhd`. +#[derive(Debug, Clone, PartialEq)] +pub struct TrackSurvey { + /// The `tkhd` track identifier. + pub id: u32, + /// Display rotation recovered from the `tkhd` matrix. + pub rotation: Rotation, +} + +impl BoxSurvey { + /// The rotation for a track id, or [`Rotation::None`] if not seen. + #[must_use] + pub fn rotation(&self, track_id: u32) -> Rotation { + self.tracks + .iter() + .find(|t| t.id == track_id) + .map_or(Rotation::None, |t| t.rotation) + } + + /// The container flavour implied by the `ftyp` brands. + /// + /// The QuickTime brand `qt ` means [`ContainerId::Mov`]; anything else + /// with a `ftyp` is treated as MP4. Files with no `ftyp` at all are + /// old-style QuickTime. + #[must_use] + pub fn container(&self) -> ContainerId { + if self.brands.is_empty() { + return ContainerId::Mov; + } + if self.brands.iter().any(|b| b == "qt ") { + ContainerId::Mov + } else { + ContainerId::Mp4 + } + } +} + +/// Walk the box tree of `reader`, recovering [`BoxSurvey`]. +/// +/// Leaves the reader's position undefined; the caller rewinds. +/// +/// # Errors +/// +/// Propagates I/O failures. A structurally odd file is *not* an error: the +/// walk stops and returns what it has, because these facts are supplementary +/// and a file whose `tkhd` cannot be read should still demux. +pub fn survey(reader: &mut R) -> VideoResult { + let end = reader.seek(SeekFrom::End(0))?; + reader.seek(SeekFrom::Start(0))?; + + let mut out = BoxSurvey::default(); + walk(reader, 0, end, 0, &mut out)?; + Ok(out) +} + +/// One box header: its payload range and type. +struct BoxHeader { + kind: [u8; 4], + payload_start: u64, + payload_end: u64, +} + +/// Read a box header at `offset`, or `None` at a clean end or on anything +/// malformed. +fn read_header( + reader: &mut R, + offset: u64, + limit: u64, +) -> io::Result> { + if offset.saturating_add(8) > limit { + return Ok(None); + } + reader.seek(SeekFrom::Start(offset))?; + + let mut header = [0u8; 8]; + if reader.read_exact(&mut header).is_err() { + return Ok(None); + } + let size32 = u32::from_be_bytes([header[0], header[1], header[2], header[3]]); + let kind = [header[4], header[5], header[6], header[7]]; + + let (size, payload_start) = match size32 { + // 1 means the real size is in a following 64-bit field. + 1 => { + let mut extended = [0u8; 8]; + if reader.read_exact(&mut extended).is_err() { + return Ok(None); + } + (u64::from_be_bytes(extended), offset + 16) + } + // 0 means "extends to the end of the enclosing container". + 0 => (limit - offset, offset + 8), + n => (u64::from(n), offset + 8), + }; + + // A box must at least contain its own header, and must not overrun its + // parent or exceed the sanity bound. + let header_len = payload_start - offset; + if size < header_len || size > MAX_BOX_SIZE { + return Ok(None); + } + let payload_end = offset.saturating_add(size).min(limit); + if payload_end < payload_start { + return Ok(None); + } + + Ok(Some(BoxHeader { + kind, + payload_start, + payload_end, + })) +} + +/// Descend through `[start, limit)`, collecting into `out`. +fn walk( + reader: &mut R, + depth: u32, + limit: u64, + start: u64, + out: &mut BoxSurvey, +) -> VideoResult<()> { + if depth > MAX_DEPTH { + return Ok(()); + } + + let mut offset = start; + while offset < limit { + let Some(header) = read_header(reader, offset, limit)? else { + return Ok(()); + }; + // A zero-length advance would spin forever on a crafted file. + if header.payload_end <= offset { + return Ok(()); + } + + match &header.kind { + b"ftyp" => read_ftyp(reader, &header, out)?, + b"mvhd" => read_mvhd(reader, &header, out)?, + b"tkhd" => read_tkhd(reader, &header, out)?, + // Containers worth descending into. `moof`/`traf` are skipped: + // fragments repeat no tkhd, and symphonia owns them. + b"moov" | b"trak" | b"mdia" => { + walk( + reader, + depth + 1, + header.payload_end, + header.payload_start, + out, + )?; + } + _ => {} + } + + offset = header.payload_end; + } + Ok(()) +} + +/// Read the whole payload of a small box, refusing anything implausible. +fn read_payload( + reader: &mut R, + header: &BoxHeader, + max: usize, +) -> io::Result>> { + let len = (header.payload_end - header.payload_start) as usize; + if len > max { + return Ok(None); + } + reader.seek(SeekFrom::Start(header.payload_start))?; + let mut buf = vec![0u8; len]; + if reader.read_exact(&mut buf).is_err() { + return Ok(None); + } + Ok(Some(buf)) +} + +fn read_ftyp( + reader: &mut R, + header: &BoxHeader, + out: &mut BoxSurvey, +) -> VideoResult<()> { + // major brand + minor version + compatible brands; a few hundred bytes at + // the very most. + let Some(payload) = read_payload(reader, header, 1024)? else { + return Ok(()); + }; + if payload.len() < 4 { + return Ok(()); + } + + let mut brands = vec![brand_string(&payload[0..4])]; + // Skip the 4-byte minor version, then read 4-byte compatible brands. + for chunk in payload.get(8..).unwrap_or_default().chunks_exact(4) { + brands.push(brand_string(chunk)); + } + out.brands = brands; + Ok(()) +} + +/// Render a four-character brand, replacing non-printable bytes so a crafted +/// brand cannot smuggle control characters into a log line or error message. +fn brand_string(bytes: &[u8]) -> String { + bytes + .iter() + .map(|&b| { + if b.is_ascii_graphic() || b == b' ' { + b as char + } else { + '?' + } + }) + .collect() +} + +fn read_mvhd( + reader: &mut R, + header: &BoxHeader, + out: &mut BoxSurvey, +) -> VideoResult<()> { + let Some(payload) = read_payload(reader, header, 256)? else { + return Ok(()); + }; + // version(1) + flags(3), then the version-dependent block. + let Some(&version) = payload.first() else { + return Ok(()); + }; + + let (creation, modification, timescale, duration) = match version { + 1 => { + if payload.len() < 4 + 28 { + return Ok(()); + } + ( + be64(&payload[4..12]), + be64(&payload[12..20]), + be32(&payload[20..24]), + be64(&payload[24..32]), + ) + } + _ => { + if payload.len() < 4 + 16 { + return Ok(()); + } + ( + u64::from(be32(&payload[4..8])), + u64::from(be32(&payload[8..12])), + be32(&payload[12..16]), + u64::from(be32(&payload[16..20])), + ) + } + }; + + out.creation_time = Some(creation); + out.modification_time = Some(modification); + out.timescale = (timescale != 0).then_some(timescale); + // 0xffff_ffff / u64::MAX is the "unknown duration" sentinel. + out.duration = (duration != u64::from(u32::MAX) && duration != u64::MAX).then_some(duration); + Ok(()) +} + +fn read_tkhd( + reader: &mut R, + header: &BoxHeader, + out: &mut BoxSurvey, +) -> VideoResult<()> { + let Some(payload) = read_payload(reader, header, 256)? else { + return Ok(()); + }; + let Some(&version) = payload.first() else { + return Ok(()); + }; + + // ISO/IEC 14496-12 §8.3.2. The track id follows version+flags and the two + // times, which are 8 bytes each in version 1 and 4 in version 0; the + // matrix then follows reserved, duration, reserved[2], layer, + // alternate_group, volume and one more reserved. + // + // v0: 4 +4+4 =12 (id) … +4+4 +8 +2+2 +2+2 = 40 (matrix) + // v1: 4 +8+8 =20 (id) … +4+8 +8 +2+2 +2+2 = 52 (matrix) + let (id_offset, matrix_offset) = if version == 1 { (20, 52) } else { (12, 40) }; + + let Some(id_bytes) = payload.get(id_offset..id_offset + 4) else { + return Ok(()); + }; + let id = be32(id_bytes); + + let rotation = payload + .get(matrix_offset..matrix_offset + 36) + .map_or(Rotation::None, rotation_from_matrix); + + out.tracks.push(TrackSurvey { id, rotation }); + Ok(()) +} + +/// Recover rotation from the nine 32-bit fixed-point values of a `tkhd` +/// matrix. +/// +/// Only `a`, `b`, `c`, `d` — indices 0, 1, 3, 4 — carry rotation. They are +/// 16.16 fixed point (`u`, `v`, `w` are 2.30, and unused here). +fn rotation_from_matrix(matrix: &[u8]) -> Rotation { + let fixed = |index: usize| -> f64 { + let at = index * 4; + f64::from(i32::from_be_bytes([ + matrix[at], + matrix[at + 1], + matrix[at + 2], + matrix[at + 3], + ])) / 65536.0 + }; + Rotation::from_display_matrix(fixed(0), fixed(1), fixed(3), fixed(4)) +} + +fn be32(bytes: &[u8]) -> u32 { + u32::from_be_bytes([bytes[0], bytes[1], bytes[2], bytes[3]]) +} + +fn be64(bytes: &[u8]) -> u64 { + let mut out = [0u8; 8]; + out.copy_from_slice(&bytes[..8]); + u64::from_be_bytes(out) +} + +#[cfg(test)] +mod tests { + use super::*; + use std::io::Cursor; + + /// Build a box: 4-byte size, 4-byte type, payload. + fn bx(kind: &[u8; 4], payload: &[u8]) -> Vec { + let mut out = ((payload.len() + 8) as u32).to_be_bytes().to_vec(); + out.extend_from_slice(kind); + out.extend_from_slice(payload); + out + } + + /// A 16.16 fixed-point matrix from its rotation sub-matrix. + fn matrix(a: f64, b: f64, c: f64, d: f64) -> Vec { + let fx = |v: f64| ((v * 65536.0) as i32).to_be_bytes().to_vec(); + let mut out = Vec::new(); + out.extend(fx(a)); + out.extend(fx(b)); + out.extend(vec![0u8; 4]); // u + out.extend(fx(c)); + out.extend(fx(d)); + out.extend(vec![0u8; 4]); // v + out.extend(vec![0u8; 12]); // x, y, w + out + } + + fn tkhd_v0(track_id: u32, m: &[u8]) -> Vec { + let mut p = vec![0u8; 4]; // version 0 + flags + p.extend_from_slice(&0u32.to_be_bytes()); // creation + p.extend_from_slice(&0u32.to_be_bytes()); // modification + p.extend_from_slice(&track_id.to_be_bytes()); // track id @ 12 + p.extend_from_slice(&0u32.to_be_bytes()); // reserved + p.extend_from_slice(&0u32.to_be_bytes()); // duration + p.extend(vec![0u8; 8]); // reserved + p.extend(vec![0u8; 8]); // layer, alt group, volume, reserved + assert_eq!(p.len(), 40, "matrix must land at offset 40 in a v0 tkhd"); + p.extend_from_slice(m); + p.extend(vec![0u8; 8]); // width, height + p + } + + fn mvhd_v0(timescale: u32, duration: u32) -> Vec { + let mut p = vec![0u8; 4]; + p.extend_from_slice(&1_000u32.to_be_bytes()); // creation + p.extend_from_slice(&2_000u32.to_be_bytes()); // modification + p.extend_from_slice(×cale.to_be_bytes()); + p.extend_from_slice(&duration.to_be_bytes()); + p.extend(vec![0u8; 80]); + p + } + + fn file(parts: &[Vec]) -> Cursor> { + Cursor::new(parts.concat()) + } + + #[test] + fn brands_come_from_ftyp() { + let mut ftyp = b"isom".to_vec(); + ftyp.extend_from_slice(&512u32.to_be_bytes()); + ftyp.extend_from_slice(b"isomiso2avc1mp41"); + let mut f = file(&[bx(b"ftyp", &ftyp)]); + + let s = survey(&mut f).expect("survey"); + assert_eq!(s.brands, ["isom", "isom", "iso2", "avc1", "mp41"]); + assert_eq!(s.container(), ContainerId::Mp4); + } + + #[test] + fn the_quicktime_brand_selects_mov() { + let mut ftyp = b"qt ".to_vec(); + ftyp.extend_from_slice(&0u32.to_be_bytes()); + ftyp.extend_from_slice(b"qt "); + let mut f = file(&[bx(b"ftyp", &ftyp)]); + + assert_eq!( + survey(&mut f).expect("survey").container(), + ContainerId::Mov + ); + } + + #[test] + fn a_file_with_no_ftyp_is_old_style_quicktime() { + let mut f = file(&[bx(b"moov", &[])]); + assert_eq!( + survey(&mut f).expect("survey").container(), + ContainerId::Mov + ); + } + + #[test] + fn a_brand_with_control_bytes_cannot_smuggle_them_into_a_message() { + let mut ftyp = vec![0x00, 0x1b, 0x5b, 0x41]; + ftyp.extend_from_slice(&0u32.to_be_bytes()); + let mut f = file(&[bx(b"ftyp", &ftyp)]); + + let s = survey(&mut f).expect("survey"); + // The NUL and the ESC are replaced; the printable bytes are kept, so + // no control character can reach a log line or error message. + assert_eq!(s.brands, ["??[A"]); + } + + #[test] + fn mvhd_yields_timescale_and_times() { + let moov = bx(b"mvhd", &mvhd_v0(600, 12_000)); + let mut f = file(&[bx(b"moov", &moov)]); + + let s = survey(&mut f).expect("survey"); + assert_eq!(s.timescale, Some(600)); + assert_eq!(s.duration, Some(12_000)); + assert_eq!(s.creation_time, Some(1_000)); + assert_eq!(s.modification_time, Some(2_000)); + } + + #[test] + fn an_unknown_duration_sentinel_is_reported_as_absent() { + let moov = bx(b"mvhd", &mvhd_v0(600, u32::MAX)); + let mut f = file(&[bx(b"moov", &moov)]); + assert_eq!(survey(&mut f).expect("survey").duration, None); + } + + #[test] + fn a_zero_timescale_is_reported_as_absent_rather_than_dividing_by_zero() { + let moov = bx(b"mvhd", &mvhd_v0(0, 100)); + let mut f = file(&[bx(b"moov", &moov)]); + assert_eq!(survey(&mut f).expect("survey").timescale, None); + } + + #[test] + fn rotation_is_recovered_per_track_from_the_tkhd_matrix() { + // Track 1 upright, track 2 rotated 90° as a portrait phone capture. + let trak1 = bx(b"tkhd", &tkhd_v0(1, &matrix(1.0, 0.0, 0.0, 1.0))); + let trak2 = bx(b"tkhd", &tkhd_v0(2, &matrix(0.0, 1.0, -1.0, 0.0))); + let moov = [bx(b"trak", &trak1), bx(b"trak", &trak2)].concat(); + let mut f = file(&[bx(b"moov", &moov)]); + + let s = survey(&mut f).expect("survey"); + assert_eq!(s.tracks.len(), 2); + assert_eq!(s.rotation(1), Rotation::None); + assert_eq!(s.rotation(2), Rotation::Clockwise90); + // A track the file does not have reports no rotation, not a panic. + assert_eq!(s.rotation(99), Rotation::None); + } + + fn tkhd_v1(track_id: u32, m: &[u8]) -> Vec { + let mut p = vec![1u8, 0, 0, 0]; // version 1 + flags + p.extend(vec![0u8; 8]); // creation (64-bit) + p.extend(vec![0u8; 8]); // modification (64-bit) + p.extend_from_slice(&track_id.to_be_bytes()); // track id @ 20 + p.extend_from_slice(&0u32.to_be_bytes()); // reserved + p.extend(vec![0u8; 8]); // duration (64-bit) + p.extend(vec![0u8; 8]); // reserved + p.extend(vec![0u8; 4]); // layer, alternate group + p.extend(vec![0u8; 4]); // volume, reserved + assert_eq!(p.len(), 52, "matrix must land at offset 52 in a v1 tkhd"); + p.extend_from_slice(m); + p.extend(vec![0u8; 8]); // width, height + p + } + + #[test] + fn a_version_1_tkhd_uses_its_own_field_offsets() { + // Version 1 widens the times to 64 bits, moving both the track id and + // the matrix. Getting either offset wrong silently yields a wrong + // track id and no rotation, which is why this is tested separately. + let trak = bx(b"tkhd", &tkhd_v1(7, &matrix(0.0, -1.0, 1.0, 0.0))); + let mut f = file(&[bx(b"moov", &bx(b"trak", &trak))]); + + let s = survey(&mut f).expect("survey"); + assert_eq!(s.tracks.len(), 1); + assert_eq!(s.tracks[0].id, 7, "track id offset"); + assert_eq!(s.rotation(7), Rotation::Clockwise270, "matrix offset"); + } + + #[test] + fn a_64_bit_box_size_is_honoured() { + // size==1 means the real size follows as a 64-bit field. + let payload = mvhd_v0(90_000, 180_000); + let mut mvhd = 1u32.to_be_bytes().to_vec(); + mvhd.extend_from_slice(b"mvhd"); + mvhd.extend_from_slice(&((payload.len() + 16) as u64).to_be_bytes()); + mvhd.extend_from_slice(&payload); + let mut f = file(&[bx(b"moov", &mvhd)]); + + assert_eq!(survey(&mut f).expect("survey").timescale, Some(90_000)); + } + + #[test] + fn a_zero_size_box_terminates_rather_than_spinning() { + // size==0 means "to the end of the container"; a following box would + // be unreachable, and the walk must simply stop. + let mut data = 0u32.to_be_bytes().to_vec(); + data.extend_from_slice(b"free"); + data.extend(vec![0u8; 16]); + let mut f = Cursor::new(data); + + assert!(survey(&mut f).is_ok()); + } + + #[test] + fn a_box_claiming_less_than_its_header_does_not_loop_forever() { + // A size of 4 is smaller than the 8-byte header: malformed. The walk + // must stop instead of advancing by zero forever. + let mut data = 4u32.to_be_bytes().to_vec(); + data.extend_from_slice(b"junk"); + data.extend(vec![0u8; 32]); + let mut f = Cursor::new(data); + + let s = survey(&mut f).expect("must terminate"); + assert!(s.brands.is_empty()); + } + + #[test] + fn a_box_overrunning_the_file_is_clamped_not_trusted() { + let mut data = 0xffff_ff00u32.to_be_bytes().to_vec(); + data.extend_from_slice(b"moov"); + data.extend(vec![0u8; 16]); + let mut f = Cursor::new(data); + + assert!(survey(&mut f).is_ok()); + } + + #[test] + fn a_truncated_tkhd_yields_no_rotation_rather_than_an_error() { + // Header present, matrix cut off. + let short = vec![0u8; 20]; + let moov = bx(b"trak", &bx(b"tkhd", &short)); + let mut f = file(&[bx(b"moov", &moov)]); + + let s = survey(&mut f).expect("survey"); + assert_eq!(s.rotation(0), Rotation::None); + } + + #[test] + fn an_empty_file_surveys_to_nothing() { + let mut f = Cursor::new(Vec::new()); + assert_eq!(survey(&mut f).expect("survey"), BoxSurvey::default()); + } + + #[test] + fn deeply_nested_containers_stop_at_the_depth_limit() { + // Nest moov > trak > mdia > trak > ... far past MAX_DEPTH. The walk + // must return rather than recurse until the stack gives out. + let mut inner = bx(b"mvhd", &mvhd_v0(600, 100)); + for _ in 0..64 { + inner = bx(b"trak", &inner); + } + let mut f = file(&[bx(b"moov", &inner)]); + assert!(survey(&mut f).is_ok()); + } +} diff --git a/crates/rawshift-video-isobmff/src/demux.rs b/crates/rawshift-video-isobmff/src/demux.rs new file mode 100644 index 0000000..9f17b39 --- /dev/null +++ b/crates/rawshift-video-isobmff/src/demux.rs @@ -0,0 +1,472 @@ +//! The MP4 / QuickTime demuxer. +//! +//! A thin bridge over `symphonia-format-isomp4`, plus the facts symphonia does +//! not surface (see [`crate::boxes`]). No symphonia type escapes this module. + +use std::io::{Read, Seek, SeekFrom}; +use std::time::Duration; + +use symphonia_core::codecs::CodecParameters; +use symphonia_core::codecs::video::well_known as vwk; +use symphonia_core::codecs::video::well_known::extra_data as vxd; +use symphonia_core::codecs::video::{VideoCodecParameters, VideoExtraData}; +use symphonia_core::formats::{ + FormatOptions, FormatReader, SeekMode as SymSeekMode, SeekTo as SymSeekTo, +}; +use symphonia_core::io::{MediaSource, MediaSourceStream, MediaSourceStreamOptions}; +use symphonia_core::units::{TimeBase, Timestamp}; +use symphonia_format_isomp4::IsoMp4Reader; + +use rawshift_core::metadata::URational; +use rawshift_core::{Dimensions, MetadataNamespace, MetadataValue}; +use rawshift_video_core::{ + AudioCodecId, AudioTrack, CodecConfig, ContainerId, Demuxer, OtherTrack, Packet, Rotation, + SeekMode, SeekTo, Track, TrackId, TrackKind, VideoCodecId, VideoError, VideoMetadata, + VideoResult, VideoTrack, nal, +}; + +use crate::boxes::{self, BoxSurvey}; + +/// The name that appears in [`VideoError::Container`] messages. +const CONTAINER_NAME: &str = "MP4"; + +/// A demuxer for MP4, M4V and QuickTime files. +pub struct IsoBmffDemuxer { + reader: IsoMp4Reader<'static>, + container: ContainerId, + tracks: Vec, + metadata: VideoMetadata, +} + +impl IsoBmffDemuxer { + /// Open an ISOBMFF file. + /// + /// Performs a short header walk for the rotation and brand facts the + /// backend does not expose, then rewinds and hands the source to it. + /// + /// # Errors + /// + /// [`VideoError::Container`] for a malformed or unsupported file, and + /// [`VideoError::Io`] for a reader failure. + pub fn open(mut source: R) -> VideoResult + where + R: Read + Seek + Send + Sync + 'static, + { + let survey = boxes::survey(&mut source).unwrap_or_default(); + source.seek(SeekFrom::Start(0))?; + + let stream = MediaSourceStream::new( + Box::new(SeekableSource(source)), + MediaSourceStreamOptions::default(), + ); + let reader = IsoMp4Reader::try_new(stream, FormatOptions::default()) + .map_err(|e| VideoError::container(CONTAINER_NAME, e))?; + + let container = survey.container(); + let tracks = map_tracks(&reader, &survey); + let metadata = build_metadata(&reader, &survey, container); + + Ok(Self { + reader, + container, + tracks, + metadata, + }) + } + + /// The track's codec and config, for deriving a packet's keyframe flag. + fn video_codec_of(&self, track: TrackId) -> Option<(VideoCodecId, &CodecConfig)> { + self.tracks + .iter() + .find(|t| t.id() == track) + .and_then(Track::as_video) + .map(|v| (v.codec, &v.codec_config)) + } +} + +impl Demuxer for IsoBmffDemuxer { + fn container(&self) -> ContainerId { + self.container + } + + fn tracks(&self) -> &[Track] { + &self.tracks + } + + fn metadata(&self) -> &VideoMetadata { + &self.metadata + } + + fn next_packet(&mut self) -> VideoResult> { + let Some(packet) = self + .reader + .next_packet() + .map_err(|e| VideoError::container(CONTAINER_NAME, e))? + else { + return Ok(None); + }; + + let track = TrackId::new(packet.track_id); + let data = packet.data.into_vec(); + + // The backend exposes no sync-sample flag, so read random access out + // of the bitstream, which is authoritative regardless. + let is_keyframe = self + .video_codec_of(track) + .is_some_and(|(codec, config)| nal::is_random_access_point(codec, config, &data)); + + let mut out = Packet::new(track, data); + out.pts = Some(packet.pts.get()); + out.dts = Some(packet.dts.get()); + out.duration = Some(packet.dur.get()); + out.is_keyframe = is_keyframe; + Ok(Some(out)) + } + + fn seek(&mut self, track: TrackId, to: SeekTo, mode: SeekMode) -> VideoResult { + if self.track(track).is_none() { + return Err(VideoError::NoSuchTrack { id: track }); + } + + let target = match to { + SeekTo::Timestamp(ts) => SymSeekTo::Timestamp { + ts: Timestamp::new(ts), + track_id: track.get(), + }, + SeekTo::Time(time) => SymSeekTo::Time { + time: symphonia_core::units::Time::try_from_nanos_u128(time.as_nanos()) + .unwrap_or(symphonia_core::units::Time::ZERO), + track_id: Some(track.get()), + }, + // SeekTo is #[non_exhaustive]; a target this backend cannot express + // is reported rather than silently approximated. + other => { + return Err(VideoError::container( + CONTAINER_NAME, + format!("unsupported seek target: {other:?}"), + )); + } + }; + let sym_mode = match mode { + // "Accurate" in the backend's vocabulary means "at or before the + // request", which is rawshift's Precise. + SeekMode::Precise => SymSeekMode::Accurate, + SeekMode::Coarse => SymSeekMode::Coarse, + // SeekMode is #[non_exhaustive]; an unknown mode takes the + // conservative option, which is always safe to decode from. + _ => SymSeekMode::Accurate, + }; + + self.reader + .seek(sym_mode, target) + .map(|landed| landed.actual_ts.get()) + .map_err(|e| VideoError::container(CONTAINER_NAME, e)) + } +} + +/// Adapts a `Read + Seek` source to the backend's source trait. +/// +/// Every source rawshift accepts is seekable — the public API requires `Seek` +/// — so `is_seekable` is unconditionally true, and the length is recovered by +/// seeking to the end and back. +struct SeekableSource(R); + +impl Read for SeekableSource { + fn read(&mut self, buf: &mut [u8]) -> std::io::Result { + self.0.read(buf) + } +} + +impl Seek for SeekableSource { + fn seek(&mut self, pos: SeekFrom) -> std::io::Result { + self.0.seek(pos) + } +} + +impl MediaSource for SeekableSource { + fn is_seekable(&self) -> bool { + true + } + + fn byte_len(&self) -> Option { + None + } +} + +/// Map the backend's tracks onto rawshift's model. +fn map_tracks(reader: &IsoMp4Reader<'_>, survey: &BoxSurvey) -> Vec { + reader + .tracks() + .iter() + .map(|t| { + let id = TrackId::new(t.id); + let time_base = time_base_of(t.time_base); + let duration = t.duration.and_then(|d| duration_of(d.get(), time_base)); + + match &t.codec_params { + Some(CodecParameters::Video(v)) => Track::Video(map_video_track( + id, + v, + time_base, + duration, + t.num_frames, + survey.rotation(t.id), + )), + Some(CodecParameters::Audio(a)) => { + let mut track = AudioTrack::new(id, AudioCodecId::Other, time_base); + track.sample_rate = a.sample_rate; + track.channels = a.channels.as_ref().map(|c| c.count() as u16); + track.duration = duration; + track.language = t.language.clone(); + Track::Audio(track) + } + _ => { + let mut track = OtherTrack::new(id, TrackKind::Data, time_base); + track.duration = duration; + track.language = t.language.clone(); + Track::Other(track) + } + } + }) + .collect() +} + +fn map_video_track( + id: TrackId, + params: &VideoCodecParameters, + time_base: URational, + duration: Option, + frame_count: Option, + rotation: Rotation, +) -> VideoTrack { + let codec = map_video_codec(params.codec); + let dimensions = Dimensions { + width: u32::from(params.width.unwrap_or(0)), + height: u32::from(params.height.unwrap_or(0)), + }; + + let mut track = VideoTrack::new(id, codec, dimensions, time_base); + // ISOBMFF carries colour in a `colr` box the backend does not surface, so + // the constructor's UNSPECIFIED stands: honest, and it resolves by picture + // height where it matters. + track.duration = duration; + track.frame_count = frame_count; + track.rotation = rotation; + track.codec_config = CodecConfig::new(config_record(codec, ¶ms.extra_data)); + track +} + +/// Pick the decoder configuration record matching the codec. +/// +/// A track may carry several extra-data blobs (Dolby Vision configurations +/// accompany an HEVC base layer, for instance), so this selects by id rather +/// than taking the first. +fn config_record(codec: VideoCodecId, extra: &[VideoExtraData]) -> Vec { + let wanted = match codec { + VideoCodecId::H264 => vxd::VIDEO_EXTRA_DATA_ID_AVC_DECODER_CONFIG, + VideoCodecId::Hevc => vxd::VIDEO_EXTRA_DATA_ID_HEVC_DECODER_CONFIG, + VideoCodecId::Av1 => vxd::VIDEO_EXTRA_DATA_ID_AV1_DECODER_CONFIG, + VideoCodecId::Vp9 => vxd::VIDEO_EXTRA_DATA_ID_VP9_DECODER_CONFIG, + _ => return Vec::new(), + }; + extra + .iter() + .find(|e| e.id == wanted) + .map(|e| e.data.to_vec()) + .unwrap_or_default() +} + +fn map_video_codec(codec: symphonia_core::codecs::video::VideoCodecId) -> VideoCodecId { + match codec { + vwk::CODEC_ID_H264 => VideoCodecId::H264, + vwk::CODEC_ID_HEVC => VideoCodecId::Hevc, + vwk::CODEC_ID_AV1 => VideoCodecId::Av1, + vwk::CODEC_ID_VP9 => VideoCodecId::Vp9, + _ => VideoCodecId::Other, + } +} + +/// The backend's time base, or the 1/1000 default when a track states none. +fn time_base_of(tb: Option) -> URational { + tb.map_or(URational::new(1, 1000), |tb| { + URational::new(tb.numer.get(), tb.denom.get()) + }) +} + +fn duration_of(ticks: u64, time_base: URational) -> Option { + if time_base.denominator == 0 { + return None; + } + let seconds = ticks as f64 * f64::from(time_base.numerator) / f64::from(time_base.denominator); + Duration::try_from_secs_f64(seconds).ok() +} + +fn build_metadata( + reader: &IsoMp4Reader<'_>, + survey: &BoxSurvey, + container: ContainerId, +) -> VideoMetadata { + let mut md = VideoMetadata::default(); + + md.container.container = Some(container); + md.container.brands = survey.brands.clone(); + md.container.timescale = survey.timescale; + md.container.duration = survey.timescale.and_then(|scale| { + survey + .duration + .and_then(|ticks| duration_of(ticks, URational::new(1, scale))) + }); + md.container.creation_time = survey.creation_time.map(format_iso_time); + md.container.modification_time = survey.modification_time.map(format_iso_time); + + // Fall back to the media-level duration when the movie header had none. + if md.container.duration.is_none() { + let info = reader.media_info(); + if let (Some(tb), Some(dur)) = (info.time_base, info.duration) { + md.container.duration = + duration_of(dur.get(), URational::new(tb.numer.get(), tb.denom.get())); + } + } + + for brand in &survey.brands { + md.push_container_entry( + MetadataNamespace::Quicktime, + "ftyp.brand", + MetadataValue::Text(brand.clone()), + ); + } + if let Some(scale) = survey.timescale { + md.push_container_entry( + MetadataNamespace::Quicktime, + "mvhd.timescale", + MetadataValue::U64(u64::from(scale)), + ); + } + + md +} + +/// Render an ISOBMFF timestamp as an ISO 8601 date-time in UTC. +/// +/// ISOBMFF counts seconds from 1904-01-01, not the Unix epoch. Formatting is +/// done here rather than pulling in a date library for one field. +fn format_iso_time(seconds_since_1904: u64) -> String { + /// Seconds between 1904-01-01 and 1970-01-01: 66 years with 17 leap days. + const EPOCH_OFFSET: u64 = 2_082_844_800; + + if seconds_since_1904 < EPOCH_OFFSET { + // Before the Unix epoch: report the raw value rather than a wrong date. + return format!("{seconds_since_1904} (seconds since 1904-01-01)"); + } + let unix = seconds_since_1904 - EPOCH_OFFSET; + + let (days, time) = (unix / 86_400, unix % 86_400); + let (hour, minute, second) = (time / 3_600, (time % 3_600) / 60, time % 60); + let (year, month, day) = civil_from_days(days as i64); + + format!("{year:04}-{month:02}-{day:02}T{hour:02}:{minute:02}:{second:02}Z") +} + +/// Convert days since 1970-01-01 to a proleptic Gregorian date. +/// +/// Howard Hinnant's `civil_from_days`, which is exact for the whole +/// representable range and needs no tables. +fn civil_from_days(days: i64) -> (i64, u32, u32) { + let z = days + 719_468; + let era = if z >= 0 { z } else { z - 146_096 } / 146_097; + let doe = (z - era * 146_097) as u64; + let yoe = (doe - doe / 1_460 + doe / 36_524 - doe / 146_096) / 365; + let y = yoe as i64 + era * 400; + let doy = doe - (365 * yoe + yoe / 4 - yoe / 100); + let mp = (5 * doy + 2) / 153; + let d = (doy - (153 * mp + 2) / 5 + 1) as u32; + let m = if mp < 10 { mp + 3 } else { mp - 9 } as u32; + (if m <= 2 { y + 1 } else { y }, m, d) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn the_isobmff_epoch_is_offset_from_unix_not_equal_to_it() { + // 1904-01-01T00:00:00Z is timestamp 0 in ISOBMFF. + assert_eq!(format_iso_time(2_082_844_800), "1970-01-01T00:00:00Z"); + // A real capture time: 2024-01-15T12:30:45Z. + assert_eq!( + format_iso_time(2_082_844_800 + 1_705_321_845), + "2024-01-15T12:30:45Z" + ); + } + + #[test] + fn a_timestamp_before_the_unix_epoch_reports_raw_rather_than_a_wrong_date() { + let s = format_iso_time(0); + assert!(s.contains("1904"), "{s}"); + assert!(!s.starts_with("1970"), "{s}"); + } + + #[test] + fn civil_from_days_handles_leap_years_and_epoch_edges() { + assert_eq!(civil_from_days(0), (1970, 1, 1)); + assert_eq!(civil_from_days(59), (1970, 3, 1)); + // 2000 was a leap year (divisible by 400); 1900 was not. + assert_eq!(civil_from_days(11_016), (2000, 2, 29)); + // 2024-02-29, a leap day in a normal leap year. + assert_eq!(civil_from_days(19_782), (2024, 2, 29)); + assert_eq!(civil_from_days(19_783), (2024, 3, 1)); + } + + #[test] + fn a_missing_time_base_falls_back_rather_than_dividing_by_zero() { + assert_eq!(time_base_of(None), URational::new(1, 1000)); + assert_eq!(duration_of(1000, URational::new(1, 0)), None); + } + + #[test] + fn durations_convert_against_the_time_base() { + // 90000 ticks of a 90 kHz base is one second. + assert_eq!( + duration_of(90_000, URational::new(1, 90_000)), + Some(Duration::from_secs(1)) + ); + } + + #[test] + fn the_config_record_is_selected_by_codec_not_position() { + // An HEVC track carrying a Dolby Vision blob first must still yield + // the hvcC, not whatever happens to be at index zero. + let extra = vec![ + VideoExtraData { + id: vxd::VIDEO_EXTRA_DATA_ID_DOLBY_VISION_CONFIG, + data: vec![0xDD; 4].into_boxed_slice(), + }, + VideoExtraData { + id: vxd::VIDEO_EXTRA_DATA_ID_HEVC_DECODER_CONFIG, + data: vec![0xAA; 4].into_boxed_slice(), + }, + ]; + assert_eq!(config_record(VideoCodecId::Hevc, &extra), vec![0xAA; 4]); + } + + #[test] + fn a_track_with_no_matching_config_record_yields_an_empty_one() { + let extra = vec![VideoExtraData { + id: vxd::VIDEO_EXTRA_DATA_ID_AVC_DECODER_CONFIG, + data: vec![0x01; 4].into_boxed_slice(), + }]; + // Asking for HEVC when only an avcC is present must not return the avcC. + assert!(config_record(VideoCodecId::Hevc, &extra).is_empty()); + assert!(config_record(VideoCodecId::ProRes, &extra).is_empty()); + } + + #[test] + fn known_codecs_map_and_unknown_ones_degrade_to_other() { + assert_eq!(map_video_codec(vwk::CODEC_ID_H264), VideoCodecId::H264); + assert_eq!(map_video_codec(vwk::CODEC_ID_HEVC), VideoCodecId::Hevc); + assert_eq!(map_video_codec(vwk::CODEC_ID_AV1), VideoCodecId::Av1); + assert_eq!(map_video_codec(vwk::CODEC_ID_VP9), VideoCodecId::Vp9); + // A codec rawshift does not model is named Other, not dropped. + assert_eq!(map_video_codec(vwk::CODEC_ID_MJPEG), VideoCodecId::Other); + } +} diff --git a/crates/rawshift-video-isobmff/src/lib.rs b/crates/rawshift-video-isobmff/src/lib.rs new file mode 100644 index 0000000..230f5b6 --- /dev/null +++ b/crates/rawshift-video-isobmff/src/lib.rs @@ -0,0 +1,38 @@ +//! MP4, M4V and QuickTime (MOV) support for rawshift. +//! +//! Demuxing is backed by `symphonia-format-isomp4` — MPL-2.0, pure Rust, no +//! build script, so it compiles on every target in `docs/SUPPORT.md`. gamut's +//! charter excludes video, so this is a direct dependency rather than an +//! upstream ask; see "Video is outside gamut's charter" in `AGENTS.md`. +//! +//! Three things the backend does not surface, which this crate supplies: +//! +//! - **Rotation**, from the `tkhd` display matrix. Phones record in the +//! sensor's orientation and correct in the container, so without this most +//! phone video presents sideways. +//! - **`ftyp` brands and `mvhd` timescale and times**, for metadata and for +//! telling MP4 from QuickTime. +//! - **Random access points.** The backend exposes no sync-sample flag at all, +//! so these are read from the bitstream by `rawshift_video_core::nal`, which +//! is the authoritative source regardless. +//! +//! No symphonia type appears in this crate's public API. +//! +//! # Not covered +//! +//! ISOBMFF carries colour signalling in a `colr` box the backend does not +//! expose, so tracks report [`CicpColor::UNSPECIFIED`] and colour resolves by +//! picture height. ProRes tracks are invisible: the backend does not recognise +//! the `ap4h`/`apch`/`apcn`/`apcs`/`apco` sample entries. +//! +//! [`CicpColor::UNSPECIFIED`]: rawshift_video_core::CicpColor::UNSPECIFIED + +#![forbid(unsafe_code)] +#![warn(missing_docs)] + +mod boxes; +mod demux; +mod sniff; + +pub use demux::IsoBmffDemuxer; +pub use sniff::{IsoBmff, detect}; diff --git a/crates/rawshift-video-isobmff/src/sniff.rs b/crates/rawshift-video-isobmff/src/sniff.rs new file mode 100644 index 0000000..00c9a81 --- /dev/null +++ b/crates/rawshift-video-isobmff/src/sniff.rs @@ -0,0 +1,112 @@ +//! Signature detection for the ISOBMFF family. + +use rawshift_video_core::{ContainerId, ContainerSniffer}; + +/// Marker for MP4 / M4V / QuickTime detection and demuxing. +/// +/// One type for the whole family: MP4 and QuickTime share a box grammar and a +/// demuxer, and differ only in the `ftyp` brand, so splitting them into two +/// sniffers would duplicate every byte of the logic. +pub struct IsoBmff; + +impl ContainerSniffer for IsoBmff { + // The family's representative. Use [`detect`] for the MP4/MOV + // distinction, which needs the brand rather than just the signature. + const CONTAINER: ContainerId = ContainerId::Mp4; + + fn matches(data: &[u8]) -> bool { + detect(data).is_some() + } +} + +/// Which ISOBMFF flavour `data` looks like, if any. +/// +/// Recognises a leading `ftyp` box, and the legacy QuickTime layout that +/// leads with `moov`, `mdat`, `wide`, `free`, or `skip` instead — old `.mov` +/// files have no `ftyp` at all. +#[must_use] +pub fn detect(data: &[u8]) -> Option { + let kind = data.get(4..8)?; + + match kind { + b"ftyp" => { + // The major brand follows the box header. QuickTime's is `qt `; + // every other brand in practice means MP4. + let brand = data.get(8..12); + if brand == Some(b"qt ") { + Some(ContainerId::Mov) + } else { + Some(ContainerId::Mp4) + } + } + // No ftyp: legacy QuickTime. + b"moov" | b"mdat" | b"wide" | b"skip" | b"free" => Some(ContainerId::Mov), + _ => None, + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn with_header(kind: &[u8; 4], rest: &[u8]) -> Vec { + let mut out = ((rest.len() + 8) as u32).to_be_bytes().to_vec(); + out.extend_from_slice(kind); + out.extend_from_slice(rest); + out + } + + #[test] + fn an_mp4_brand_is_mp4() { + for brand in [b"isom", b"mp42", b"avc1", b"iso5", b"M4V "] { + let data = with_header(b"ftyp", brand); + assert_eq!(detect(&data), Some(ContainerId::Mp4), "{brand:?}"); + } + } + + #[test] + fn the_quicktime_brand_is_mov() { + let data = with_header(b"ftyp", b"qt "); + assert_eq!(detect(&data), Some(ContainerId::Mov)); + } + + #[test] + fn legacy_quicktime_without_an_ftyp_is_still_recognised() { + for kind in [b"moov", b"mdat", b"wide", b"free", b"skip"] { + let data = with_header(kind, &[0u8; 8]); + assert_eq!(detect(&data), Some(ContainerId::Mov), "{kind:?}"); + } + } + + #[test] + fn unrelated_formats_are_not_claimed() { + // PNG, JPEG, Matroska, RIFF, and plain text must all be rejected. + assert_eq!(detect(b"\x89PNG\r\n\x1a\n\x00\x00\x00\x00"), None); + assert_eq!(detect(&[0xff, 0xd8, 0xff, 0xe0, 0, 0, 0, 0]), None); + assert_eq!(detect(&[0x1a, 0x45, 0xdf, 0xa3, 0, 0, 0, 0]), None); + assert_eq!(detect(b"RIFF____AVI LIST"), None); + assert_eq!(detect(b"this is not a video"), None); + } + + #[test] + fn short_input_is_rejected_rather_than_panicking() { + for len in 0..8 { + assert_eq!(detect(&vec![0u8; len]), None, "len {len}"); + } + } + + #[test] + fn an_ftyp_truncated_before_its_brand_still_reads_as_mp4() { + // The signature is present even though the brand is not; treating it + // as MP4 lets the demuxer produce the real error. + let data = with_header(b"ftyp", &[]); + assert_eq!(detect(&data), Some(ContainerId::Mp4)); + } + + #[test] + fn the_sniffer_trait_agrees_with_detect() { + let mp4 = with_header(b"ftyp", b"isom"); + assert!(IsoBmff::matches(&mp4)); + assert!(!IsoBmff::matches(b"\x89PNG\r\n\x1a\n\x00\x00\x00\x00")); + } +} From 52f1a8c47e6222430c6983faad657f9a2d22252d Mon Sep 17 00:00:00 2001 From: Justin Chung Date: Fri, 28 Aug 2026 22:05:08 -0400 Subject: [PATCH 06/13] feat(video-matroska): demux Matroska and WebM, on a shared symphonia bridge MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds MKV/WebM support and extracts rawshift-video-symphonia, the one translation of symphonia's tracks, packets, timestamps and seek vocabulary into rawshift's model. Both container crates now share it rather than carrying copies that drift; it follows the established "focused support crate" pattern of rawshift-image-{metadata,ifd,ljpeg}. No symphonia type leaves that crate, which is what lets the container crates keep the same guarantee. Fixes an end-of-stream bug found by running the demuxers against real ffmpeg-generated files: the two readers disagree about how a stream ends. The ISOBMFF reader returns Ok(None) while the Matroska reader raises an UnexpectedEof I/O error, so every Matroska file ended in a spurious "unexpected end of file". Both now map to rawshift's Ok(None), and only UnexpectedEof does — a permission error or a malformed atom still fails. Verified end to end against ffmpeg-generated fixtures via the new probe examples: H.264 in MP4, MOV and MKV, and HEVC in MKV, all yield correct tracks, dimensions, durations, config records and packet counts, with random access points matching the encoder's GOP length. Matroska states rotation in Video/Projection, which the backend does not surface, so tracks report Rotation::None. That costs nothing in practice: rotation is a phone-capture concern and phones write MP4 or QuickTime. Refs #39 --- Cargo.lock | 33 ++ Cargo.toml | 4 + crates/rawshift-video-isobmff/Cargo.toml | 1 + .../rawshift-video-isobmff/examples/probe.rs | 54 +++ crates/rawshift-video-isobmff/src/demux.rs | 301 ++----------- crates/rawshift-video-matroska/Cargo.toml | 23 + crates/rawshift-video-matroska/README.md | 17 + .../rawshift-video-matroska/examples/probe.rs | 60 +++ crates/rawshift-video-matroska/src/demux.rs | 130 ++++++ crates/rawshift-video-matroska/src/lib.rs | 28 ++ crates/rawshift-video-matroska/src/sniff.rs | 104 +++++ crates/rawshift-video-symphonia/Cargo.toml | 21 + crates/rawshift-video-symphonia/README.md | 16 + crates/rawshift-video-symphonia/src/lib.rs | 424 ++++++++++++++++++ 14 files changed, 949 insertions(+), 267 deletions(-) create mode 100644 crates/rawshift-video-isobmff/examples/probe.rs create mode 100644 crates/rawshift-video-matroska/Cargo.toml create mode 100644 crates/rawshift-video-matroska/README.md create mode 100644 crates/rawshift-video-matroska/examples/probe.rs create mode 100644 crates/rawshift-video-matroska/src/demux.rs create mode 100644 crates/rawshift-video-matroska/src/lib.rs create mode 100644 crates/rawshift-video-matroska/src/sniff.rs create mode 100644 crates/rawshift-video-symphonia/Cargo.toml create mode 100644 crates/rawshift-video-symphonia/README.md create mode 100644 crates/rawshift-video-symphonia/src/lib.rs diff --git a/Cargo.lock b/Cargo.lock index c992f7f..80a8d9b 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1513,11 +1513,32 @@ version = "0.1.1" dependencies = [ "rawshift-core", "rawshift-video-core", + "rawshift-video-symphonia", "symphonia-core", "symphonia-format-isomp4", "tracing", ] +[[package]] +name = "rawshift-video-matroska" +version = "0.1.1" +dependencies = [ + "rawshift-core", + "rawshift-video-core", + "rawshift-video-symphonia", + "symphonia-core", + "symphonia-format-mkv", +] + +[[package]] +name = "rawshift-video-symphonia" +version = "0.1.1" +dependencies = [ + "rawshift-core", + "rawshift-video-core", + "symphonia-core", +] + [[package]] name = "rayon" version = "1.11.0" @@ -1829,6 +1850,18 @@ dependencies = [ "symphonia-metadata", ] +[[package]] +name = "symphonia-format-mkv" +version = "0.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d015c5c0558864665894b3f4cbd95e10abb01b9c868e751c72670f326a56360e" +dependencies = [ + "lazy_static", + "log", + "symphonia-common", + "symphonia-core", +] + [[package]] name = "symphonia-metadata" version = "0.6.1" diff --git a/Cargo.toml b/Cargo.toml index c4c0251..4f99932 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -29,6 +29,8 @@ members = [ "crates/rawshift-video", "crates/rawshift-video-core", "crates/rawshift-video-isobmff", + "crates/rawshift-video-matroska", + "crates/rawshift-video-symphonia", ] [workspace.package] @@ -108,6 +110,8 @@ rawshift-image-webp = { path = "crates/rawshift-image-webp", version = "0.1.1", rawshift-video = { path = "crates/rawshift-video", version = "0.1.1" } rawshift-video-core = { path = "crates/rawshift-video-core", version = "0.1.1" } rawshift-video-isobmff = { path = "crates/rawshift-video-isobmff", version = "0.1.1" } +rawshift-video-matroska = { path = "crates/rawshift-video-matroska", version = "0.1.1" } +rawshift-video-symphonia = { path = "crates/rawshift-video-symphonia", version = "0.1.1" } # Profiles must live at the workspace root — Cargo ignores profiles declared in # member manifests. diff --git a/crates/rawshift-video-isobmff/Cargo.toml b/crates/rawshift-video-isobmff/Cargo.toml index c0e8a5f..0c6d310 100644 --- a/crates/rawshift-video-isobmff/Cargo.toml +++ b/crates/rawshift-video-isobmff/Cargo.toml @@ -15,6 +15,7 @@ readme = "README.md" [dependencies] rawshift-core = { workspace = true } rawshift-video-core = { workspace = true } +rawshift-video-symphonia = { workspace = true } symphonia-core = { workspace = true } symphonia-format-isomp4 = { workspace = true } tracing = { workspace = true } diff --git a/crates/rawshift-video-isobmff/examples/probe.rs b/crates/rawshift-video-isobmff/examples/probe.rs new file mode 100644 index 0000000..7145ad1 --- /dev/null +++ b/crates/rawshift-video-isobmff/examples/probe.rs @@ -0,0 +1,54 @@ +//! Print what the ISOBMFF demuxer sees in a file. +//! +//! `cargo run -p rawshift-video-isobmff --example probe -- FILE` + +use std::fs::File; + +use rawshift_video_core::{Demuxer, Track}; +use rawshift_video_isobmff::IsoBmffDemuxer; + +fn main() -> Result<(), Box> { + let path = std::env::args().nth(1).ok_or("usage: probe FILE")?; + let mut demuxer = IsoBmffDemuxer::open(File::open(&path)?)?; + + println!("container: {}", demuxer.container()); + let md = demuxer.metadata(); + println!("brands: {:?}", md.container.brands); + println!("timescale: {:?}", md.container.timescale); + println!("duration: {:?}", md.container.duration); + println!("created: {:?}", md.container.creation_time); + + for track in demuxer.tracks() { + match track { + Track::Video(v) => println!( + "video #{} {} {}x{} rot={} tb={}/{} frames={:?} dur={:?} cfg={}B", + v.id, + v.codec, + v.dimensions.width, + v.dimensions.height, + v.rotation, + v.time_base.numerator, + v.time_base.denominator, + v.frame_count, + v.duration, + v.codec_config.as_bytes().len(), + ), + Track::Audio(a) => println!("audio #{} {}", a.id, a.codec), + Track::Other(o) => println!("other #{} {:?}", o.id, o.kind), + other => println!("other #{}", other.id()), + } + } + + let (mut packets, mut keyframes, mut first_pts) = (0usize, 0usize, None); + while let Some(p) = demuxer.next_packet()? { + if first_pts.is_none() { + first_pts = p.pts; + } + packets += 1; + if p.is_keyframe { + keyframes += 1; + } + } + println!("packets: {packets} ({keyframes} random access points), first pts {first_pts:?}"); + Ok(()) +} diff --git a/crates/rawshift-video-isobmff/src/demux.rs b/crates/rawshift-video-isobmff/src/demux.rs index 9f17b39..f89af30 100644 --- a/crates/rawshift-video-isobmff/src/demux.rs +++ b/crates/rawshift-video-isobmff/src/demux.rs @@ -1,29 +1,23 @@ //! The MP4 / QuickTime demuxer. //! -//! A thin bridge over `symphonia-format-isomp4`, plus the facts symphonia does -//! not surface (see [`crate::boxes`]). No symphonia type escapes this module. +//! Track, packet and seek translation is shared with the Matroska crate via +//! `rawshift-video-symphonia`; what is specific to ISOBMFF is the header walk +//! in [`crate::boxes`] and the movie-level metadata built from it. No +//! symphonia type escapes this module. use std::io::{Read, Seek, SeekFrom}; -use std::time::Duration; -use symphonia_core::codecs::CodecParameters; -use symphonia_core::codecs::video::well_known as vwk; -use symphonia_core::codecs::video::well_known::extra_data as vxd; -use symphonia_core::codecs::video::{VideoCodecParameters, VideoExtraData}; -use symphonia_core::formats::{ - FormatOptions, FormatReader, SeekMode as SymSeekMode, SeekTo as SymSeekTo, -}; -use symphonia_core::io::{MediaSource, MediaSourceStream, MediaSourceStreamOptions}; -use symphonia_core::units::{TimeBase, Timestamp}; +use symphonia_core::formats::{FormatOptions, FormatReader}; +use symphonia_core::io::{MediaSourceStream, MediaSourceStreamOptions}; use symphonia_format_isomp4::IsoMp4Reader; use rawshift_core::metadata::URational; -use rawshift_core::{Dimensions, MetadataNamespace, MetadataValue}; +use rawshift_core::{MetadataNamespace, MetadataValue}; use rawshift_video_core::{ - AudioCodecId, AudioTrack, CodecConfig, ContainerId, Demuxer, OtherTrack, Packet, Rotation, - SeekMode, SeekTo, Track, TrackId, TrackKind, VideoCodecId, VideoError, VideoMetadata, - VideoResult, VideoTrack, nal, + CodecConfig, ContainerId, Demuxer, Packet, SeekMode, SeekTo, Track, TrackId, VideoCodecId, + VideoError, VideoMetadata, VideoResult, }; +use rawshift_video_symphonia as bridge; use crate::boxes::{self, BoxSurvey}; @@ -52,18 +46,24 @@ impl IsoBmffDemuxer { where R: Read + Seek + Send + Sync + 'static, { + // A structurally odd file should still demux: the survey supplies + // supplementary facts, so its failure degrades rather than propagates. let survey = boxes::survey(&mut source).unwrap_or_default(); source.seek(SeekFrom::Start(0))?; let stream = MediaSourceStream::new( - Box::new(SeekableSource(source)), + Box::new(bridge::SeekableSource::new(source)), MediaSourceStreamOptions::default(), ); let reader = IsoMp4Reader::try_new(stream, FormatOptions::default()) .map_err(|e| VideoError::container(CONTAINER_NAME, e))?; let container = survey.container(); - let tracks = map_tracks(&reader, &survey); + let tracks = reader + .tracks() + .iter() + .map(|t| bridge::map_track(t, survey.rotation(t.id))) + .collect(); let metadata = build_metadata(&reader, &survey, container); Ok(Self { @@ -98,64 +98,21 @@ impl Demuxer for IsoBmffDemuxer { } fn next_packet(&mut self) -> VideoResult> { - let Some(packet) = self - .reader - .next_packet() - .map_err(|e| VideoError::container(CONTAINER_NAME, e))? - else { - return Ok(None); + // Peek the track before consuming, so the keyframe flag can be derived + // from the right track's codec configuration. + let result = self.reader.next_packet(); + let video = match &result { + Ok(Some(p)) => self.video_codec_of(TrackId::new(p.track_id)), + _ => None, }; - - let track = TrackId::new(packet.track_id); - let data = packet.data.into_vec(); - - // The backend exposes no sync-sample flag, so read random access out - // of the bitstream, which is authoritative regardless. - let is_keyframe = self - .video_codec_of(track) - .is_some_and(|(codec, config)| nal::is_random_access_point(codec, config, &data)); - - let mut out = Packet::new(track, data); - out.pts = Some(packet.pts.get()); - out.dts = Some(packet.dts.get()); - out.duration = Some(packet.dur.get()); - out.is_keyframe = is_keyframe; - Ok(Some(out)) + bridge::map_packet_result(CONTAINER_NAME, result, video) } fn seek(&mut self, track: TrackId, to: SeekTo, mode: SeekMode) -> VideoResult { if self.track(track).is_none() { return Err(VideoError::NoSuchTrack { id: track }); } - - let target = match to { - SeekTo::Timestamp(ts) => SymSeekTo::Timestamp { - ts: Timestamp::new(ts), - track_id: track.get(), - }, - SeekTo::Time(time) => SymSeekTo::Time { - time: symphonia_core::units::Time::try_from_nanos_u128(time.as_nanos()) - .unwrap_or(symphonia_core::units::Time::ZERO), - track_id: Some(track.get()), - }, - // SeekTo is #[non_exhaustive]; a target this backend cannot express - // is reported rather than silently approximated. - other => { - return Err(VideoError::container( - CONTAINER_NAME, - format!("unsupported seek target: {other:?}"), - )); - } - }; - let sym_mode = match mode { - // "Accurate" in the backend's vocabulary means "at or before the - // request", which is rawshift's Precise. - SeekMode::Precise => SymSeekMode::Accurate, - SeekMode::Coarse => SymSeekMode::Coarse, - // SeekMode is #[non_exhaustive]; an unknown mode takes the - // conservative option, which is always safe to decode from. - _ => SymSeekMode::Accurate, - }; + let (sym_mode, target) = bridge::map_seek(CONTAINER_NAME, track, to, mode)?; self.reader .seek(sym_mode, target) @@ -164,143 +121,6 @@ impl Demuxer for IsoBmffDemuxer { } } -/// Adapts a `Read + Seek` source to the backend's source trait. -/// -/// Every source rawshift accepts is seekable — the public API requires `Seek` -/// — so `is_seekable` is unconditionally true, and the length is recovered by -/// seeking to the end and back. -struct SeekableSource(R); - -impl Read for SeekableSource { - fn read(&mut self, buf: &mut [u8]) -> std::io::Result { - self.0.read(buf) - } -} - -impl Seek for SeekableSource { - fn seek(&mut self, pos: SeekFrom) -> std::io::Result { - self.0.seek(pos) - } -} - -impl MediaSource for SeekableSource { - fn is_seekable(&self) -> bool { - true - } - - fn byte_len(&self) -> Option { - None - } -} - -/// Map the backend's tracks onto rawshift's model. -fn map_tracks(reader: &IsoMp4Reader<'_>, survey: &BoxSurvey) -> Vec { - reader - .tracks() - .iter() - .map(|t| { - let id = TrackId::new(t.id); - let time_base = time_base_of(t.time_base); - let duration = t.duration.and_then(|d| duration_of(d.get(), time_base)); - - match &t.codec_params { - Some(CodecParameters::Video(v)) => Track::Video(map_video_track( - id, - v, - time_base, - duration, - t.num_frames, - survey.rotation(t.id), - )), - Some(CodecParameters::Audio(a)) => { - let mut track = AudioTrack::new(id, AudioCodecId::Other, time_base); - track.sample_rate = a.sample_rate; - track.channels = a.channels.as_ref().map(|c| c.count() as u16); - track.duration = duration; - track.language = t.language.clone(); - Track::Audio(track) - } - _ => { - let mut track = OtherTrack::new(id, TrackKind::Data, time_base); - track.duration = duration; - track.language = t.language.clone(); - Track::Other(track) - } - } - }) - .collect() -} - -fn map_video_track( - id: TrackId, - params: &VideoCodecParameters, - time_base: URational, - duration: Option, - frame_count: Option, - rotation: Rotation, -) -> VideoTrack { - let codec = map_video_codec(params.codec); - let dimensions = Dimensions { - width: u32::from(params.width.unwrap_or(0)), - height: u32::from(params.height.unwrap_or(0)), - }; - - let mut track = VideoTrack::new(id, codec, dimensions, time_base); - // ISOBMFF carries colour in a `colr` box the backend does not surface, so - // the constructor's UNSPECIFIED stands: honest, and it resolves by picture - // height where it matters. - track.duration = duration; - track.frame_count = frame_count; - track.rotation = rotation; - track.codec_config = CodecConfig::new(config_record(codec, ¶ms.extra_data)); - track -} - -/// Pick the decoder configuration record matching the codec. -/// -/// A track may carry several extra-data blobs (Dolby Vision configurations -/// accompany an HEVC base layer, for instance), so this selects by id rather -/// than taking the first. -fn config_record(codec: VideoCodecId, extra: &[VideoExtraData]) -> Vec { - let wanted = match codec { - VideoCodecId::H264 => vxd::VIDEO_EXTRA_DATA_ID_AVC_DECODER_CONFIG, - VideoCodecId::Hevc => vxd::VIDEO_EXTRA_DATA_ID_HEVC_DECODER_CONFIG, - VideoCodecId::Av1 => vxd::VIDEO_EXTRA_DATA_ID_AV1_DECODER_CONFIG, - VideoCodecId::Vp9 => vxd::VIDEO_EXTRA_DATA_ID_VP9_DECODER_CONFIG, - _ => return Vec::new(), - }; - extra - .iter() - .find(|e| e.id == wanted) - .map(|e| e.data.to_vec()) - .unwrap_or_default() -} - -fn map_video_codec(codec: symphonia_core::codecs::video::VideoCodecId) -> VideoCodecId { - match codec { - vwk::CODEC_ID_H264 => VideoCodecId::H264, - vwk::CODEC_ID_HEVC => VideoCodecId::Hevc, - vwk::CODEC_ID_AV1 => VideoCodecId::Av1, - vwk::CODEC_ID_VP9 => VideoCodecId::Vp9, - _ => VideoCodecId::Other, - } -} - -/// The backend's time base, or the 1/1000 default when a track states none. -fn time_base_of(tb: Option) -> URational { - tb.map_or(URational::new(1, 1000), |tb| { - URational::new(tb.numer.get(), tb.denom.get()) - }) -} - -fn duration_of(ticks: u64, time_base: URational) -> Option { - if time_base.denominator == 0 { - return None; - } - let seconds = ticks as f64 * f64::from(time_base.numerator) / f64::from(time_base.denominator); - Duration::try_from_secs_f64(seconds).ok() -} - fn build_metadata( reader: &IsoMp4Reader<'_>, survey: &BoxSurvey, @@ -314,7 +134,7 @@ fn build_metadata( md.container.duration = survey.timescale.and_then(|scale| { survey .duration - .and_then(|ticks| duration_of(ticks, URational::new(1, scale))) + .and_then(|ticks| bridge::ticks_to_duration(ticks, URational::new(1, scale))) }); md.container.creation_time = survey.creation_time.map(format_iso_time); md.container.modification_time = survey.modification_time.map(format_iso_time); @@ -324,7 +144,7 @@ fn build_metadata( let info = reader.media_info(); if let (Some(tb), Some(dur)) = (info.time_base, info.duration) { md.container.duration = - duration_of(dur.get(), URational::new(tb.numer.get(), tb.denom.get())); + bridge::ticks_to_duration(dur.get(), bridge::map_time_base(Some(tb))); } } @@ -348,10 +168,10 @@ fn build_metadata( /// Render an ISOBMFF timestamp as an ISO 8601 date-time in UTC. /// -/// ISOBMFF counts seconds from 1904-01-01, not the Unix epoch. Formatting is -/// done here rather than pulling in a date library for one field. +/// ISOBMFF counts seconds from 1904-01-01, not the Unix epoch. Formatted here +/// rather than by pulling in a date library for one field. fn format_iso_time(seconds_since_1904: u64) -> String { - /// Seconds between 1904-01-01 and 1970-01-01: 66 years with 17 leap days. + /// Seconds between 1904-01-01 and 1970-01-01. const EPOCH_OFFSET: u64 = 2_082_844_800; if seconds_since_1904 < EPOCH_OFFSET { @@ -369,8 +189,8 @@ fn format_iso_time(seconds_since_1904: u64) -> String { /// Convert days since 1970-01-01 to a proleptic Gregorian date. /// -/// Howard Hinnant's `civil_from_days`, which is exact for the whole -/// representable range and needs no tables. +/// Howard Hinnant's `civil_from_days`: exact over the whole representable +/// range and table-free. fn civil_from_days(days: i64) -> (i64, u32, u32) { let z = days + 719_468; let era = if z >= 0 { z } else { z - 146_096 } / 146_097; @@ -412,61 +232,8 @@ mod tests { assert_eq!(civil_from_days(59), (1970, 3, 1)); // 2000 was a leap year (divisible by 400); 1900 was not. assert_eq!(civil_from_days(11_016), (2000, 2, 29)); - // 2024-02-29, a leap day in a normal leap year. + // 2024-02-29, a leap day in an ordinary leap year. assert_eq!(civil_from_days(19_782), (2024, 2, 29)); assert_eq!(civil_from_days(19_783), (2024, 3, 1)); } - - #[test] - fn a_missing_time_base_falls_back_rather_than_dividing_by_zero() { - assert_eq!(time_base_of(None), URational::new(1, 1000)); - assert_eq!(duration_of(1000, URational::new(1, 0)), None); - } - - #[test] - fn durations_convert_against_the_time_base() { - // 90000 ticks of a 90 kHz base is one second. - assert_eq!( - duration_of(90_000, URational::new(1, 90_000)), - Some(Duration::from_secs(1)) - ); - } - - #[test] - fn the_config_record_is_selected_by_codec_not_position() { - // An HEVC track carrying a Dolby Vision blob first must still yield - // the hvcC, not whatever happens to be at index zero. - let extra = vec![ - VideoExtraData { - id: vxd::VIDEO_EXTRA_DATA_ID_DOLBY_VISION_CONFIG, - data: vec![0xDD; 4].into_boxed_slice(), - }, - VideoExtraData { - id: vxd::VIDEO_EXTRA_DATA_ID_HEVC_DECODER_CONFIG, - data: vec![0xAA; 4].into_boxed_slice(), - }, - ]; - assert_eq!(config_record(VideoCodecId::Hevc, &extra), vec![0xAA; 4]); - } - - #[test] - fn a_track_with_no_matching_config_record_yields_an_empty_one() { - let extra = vec![VideoExtraData { - id: vxd::VIDEO_EXTRA_DATA_ID_AVC_DECODER_CONFIG, - data: vec![0x01; 4].into_boxed_slice(), - }]; - // Asking for HEVC when only an avcC is present must not return the avcC. - assert!(config_record(VideoCodecId::Hevc, &extra).is_empty()); - assert!(config_record(VideoCodecId::ProRes, &extra).is_empty()); - } - - #[test] - fn known_codecs_map_and_unknown_ones_degrade_to_other() { - assert_eq!(map_video_codec(vwk::CODEC_ID_H264), VideoCodecId::H264); - assert_eq!(map_video_codec(vwk::CODEC_ID_HEVC), VideoCodecId::Hevc); - assert_eq!(map_video_codec(vwk::CODEC_ID_AV1), VideoCodecId::Av1); - assert_eq!(map_video_codec(vwk::CODEC_ID_VP9), VideoCodecId::Vp9); - // A codec rawshift does not model is named Other, not dropped. - assert_eq!(map_video_codec(vwk::CODEC_ID_MJPEG), VideoCodecId::Other); - } } diff --git a/crates/rawshift-video-matroska/Cargo.toml b/crates/rawshift-video-matroska/Cargo.toml new file mode 100644 index 0000000..c9c710c --- /dev/null +++ b/crates/rawshift-video-matroska/Cargo.toml @@ -0,0 +1,23 @@ +[package] +name = "rawshift-video-matroska" +version.workspace = true +edition.workspace = true +rust-version.workspace = true +license.workspace = true +repository.workspace = true +homepage.workspace = true +description = "Matroska and WebM demuxing and metadata for rawshift" +documentation = "https://docs.rs/rawshift-video-matroska" +keywords = ["video", "matroska", "mkv", "webm", "demux"] +categories = ["multimedia::video"] +readme = "README.md" + +[dependencies] +rawshift-core = { workspace = true } +rawshift-video-core = { workspace = true } +rawshift-video-symphonia = { workspace = true } +symphonia-core = { workspace = true } +symphonia-format-mkv = { workspace = true } + +[features] +default = [] diff --git a/crates/rawshift-video-matroska/README.md b/crates/rawshift-video-matroska/README.md new file mode 100644 index 0000000..60942e8 --- /dev/null +++ b/crates/rawshift-video-matroska/README.md @@ -0,0 +1,17 @@ +# rawshift-video-matroska + +Matroska and WebM support for +[rawshift](https://github.com/visualcommons/rawshift). + +Backed by [`symphonia-format-mkv`](https://crates.io/crates/symphonia-format-mkv) +(MPL-2.0, pure Rust, no build script), which maps `V_MPEG4/ISO/AVC` and +`V_MPEGH/ISO/HEVC` tracks and surfaces their `CodecPrivate` — the same `avcC` +and `hvcC` records MP4 carries — as codec extra data. + +As with the ISOBMFF crate, random access points are read from the bitstream +rather than from a container flag, and no symphonia type appears in the public +API. + +## License + +Licensed under [MPL-2.0](../../LICENSE). diff --git a/crates/rawshift-video-matroska/examples/probe.rs b/crates/rawshift-video-matroska/examples/probe.rs new file mode 100644 index 0000000..c4a9ab5 --- /dev/null +++ b/crates/rawshift-video-matroska/examples/probe.rs @@ -0,0 +1,60 @@ +//! Print what the ISOBMFF demuxer sees in a file. +//! +//! `cargo run -p rawshift-video-isobmff --example probe -- FILE` + +use std::fs::File; + +use rawshift_video_core::{Demuxer, Track}; +use rawshift_video_matroska::MatroskaDemuxer; + +fn main() -> Result<(), Box> { + let path = std::env::args().nth(1).ok_or("usage: probe FILE")?; + let mut demuxer = MatroskaDemuxer::open( + File::open(&path)?, + rawshift_video_matroska::detect( + &std::fs::read(&path)?[..64.min(std::fs::metadata(&path)?.len() as usize)], + ) + .ok_or("not matroska")?, + )?; + + println!("container: {}", demuxer.container()); + let md = demuxer.metadata(); + println!("brands: {:?}", md.container.brands); + println!("timescale: {:?}", md.container.timescale); + println!("duration: {:?}", md.container.duration); + println!("created: {:?}", md.container.creation_time); + + for track in demuxer.tracks() { + match track { + Track::Video(v) => println!( + "video #{} {} {}x{} rot={} tb={}/{} frames={:?} dur={:?} cfg={}B", + v.id, + v.codec, + v.dimensions.width, + v.dimensions.height, + v.rotation, + v.time_base.numerator, + v.time_base.denominator, + v.frame_count, + v.duration, + v.codec_config.as_bytes().len(), + ), + Track::Audio(a) => println!("audio #{} {}", a.id, a.codec), + Track::Other(o) => println!("other #{} {:?}", o.id, o.kind), + other => println!("other #{}", other.id()), + } + } + + let (mut packets, mut keyframes, mut first_pts) = (0usize, 0usize, None); + while let Some(p) = demuxer.next_packet()? { + if first_pts.is_none() { + first_pts = p.pts; + } + packets += 1; + if p.is_keyframe { + keyframes += 1; + } + } + println!("packets: {packets} ({keyframes} random access points), first pts {first_pts:?}"); + Ok(()) +} diff --git a/crates/rawshift-video-matroska/src/demux.rs b/crates/rawshift-video-matroska/src/demux.rs new file mode 100644 index 0000000..2692e5a --- /dev/null +++ b/crates/rawshift-video-matroska/src/demux.rs @@ -0,0 +1,130 @@ +//! The Matroska / WebM demuxer. + +use std::io::{Read, Seek}; + +use symphonia_core::formats::{FormatOptions, FormatReader}; +use symphonia_core::io::{MediaSourceStream, MediaSourceStreamOptions}; +use symphonia_format_mkv::MkvReader; + +use rawshift_core::MetadataNamespace; +use rawshift_core::MetadataValue; +use rawshift_video_core::{ + CodecConfig, ContainerId, Demuxer, Packet, Rotation, SeekMode, SeekTo, Track, TrackId, + VideoCodecId, VideoError, VideoMetadata, VideoResult, +}; +use rawshift_video_symphonia as bridge; + +/// The name that appears in [`VideoError::Container`] messages. +const CONTAINER_NAME: &str = "Matroska"; + +/// A demuxer for Matroska and WebM files. +pub struct MatroskaDemuxer { + reader: MkvReader<'static>, + container: ContainerId, + tracks: Vec, + metadata: VideoMetadata, +} + +impl MatroskaDemuxer { + /// Open a Matroska or WebM file. + /// + /// `container` distinguishes the two flavours; use [`crate::detect`] on a + /// prefix of the file to obtain it. + /// + /// # Errors + /// + /// [`VideoError::Container`] for a malformed or unsupported file, and + /// [`VideoError::Io`] for a reader failure. + pub fn open(source: R, container: ContainerId) -> VideoResult + where + R: Read + Seek + Send + Sync + 'static, + { + let stream = MediaSourceStream::new( + Box::new(bridge::SeekableSource::new(source)), + MediaSourceStreamOptions::default(), + ); + let reader = MkvReader::try_new(stream, FormatOptions::default()) + .map_err(|e| VideoError::container(CONTAINER_NAME, e))?; + + // Matroska states rotation in Video/Projection, which the backend does + // not surface; see the crate docs for why that costs nothing here. + let tracks = reader + .tracks() + .iter() + .map(|t| bridge::map_track(t, Rotation::None)) + .collect(); + + let mut metadata = VideoMetadata::default(); + metadata.container.container = Some(container); + let info = reader.media_info(); + if let (Some(tb), Some(dur)) = (info.time_base, info.duration) { + let time_base = bridge::map_time_base(Some(tb)); + metadata.container.duration = bridge::ticks_to_duration(dur.get(), time_base); + metadata.container.timescale = Some(time_base.denominator); + } + metadata.push_container_entry( + MetadataNamespace::Matroska, + "DocType", + MetadataValue::Text( + if container == ContainerId::WebM { + "webm" + } else { + "matroska" + } + .to_string(), + ), + ); + + Ok(Self { + reader, + container, + tracks, + metadata, + }) + } + + fn video_codec_of(&self, track: TrackId) -> Option<(VideoCodecId, &CodecConfig)> { + self.tracks + .iter() + .find(|t| t.id() == track) + .and_then(Track::as_video) + .map(|v| (v.codec, &v.codec_config)) + } +} + +impl Demuxer for MatroskaDemuxer { + fn container(&self) -> ContainerId { + self.container + } + + fn tracks(&self) -> &[Track] { + &self.tracks + } + + fn metadata(&self) -> &VideoMetadata { + &self.metadata + } + + fn next_packet(&mut self) -> VideoResult> { + // Peek the track before consuming, so the keyframe flag can be derived + // from the right track's codec configuration. + let result = self.reader.next_packet(); + let video = match &result { + Ok(Some(p)) => self.video_codec_of(TrackId::new(p.track_id)), + _ => None, + }; + bridge::map_packet_result(CONTAINER_NAME, result, video) + } + + fn seek(&mut self, track: TrackId, to: SeekTo, mode: SeekMode) -> VideoResult { + if self.track(track).is_none() { + return Err(VideoError::NoSuchTrack { id: track }); + } + let (sym_mode, target) = bridge::map_seek(CONTAINER_NAME, track, to, mode)?; + + self.reader + .seek(sym_mode, target) + .map(|landed| landed.actual_ts.get()) + .map_err(|e| VideoError::container(CONTAINER_NAME, e)) + } +} diff --git a/crates/rawshift-video-matroska/src/lib.rs b/crates/rawshift-video-matroska/src/lib.rs new file mode 100644 index 0000000..f36db52 --- /dev/null +++ b/crates/rawshift-video-matroska/src/lib.rs @@ -0,0 +1,28 @@ +//! Matroska and WebM support for rawshift. +//! +//! Backed by `symphonia-format-mkv` — MPL-2.0, pure Rust, no build script — +//! which maps `V_MPEG4/ISO/AVC` and `V_MPEGH/ISO/HEVC` tracks and surfaces +//! their `CodecPrivate` as codec extra data. That `CodecPrivate` holds the +//! same `avcC`/`hvcC` records MP4 carries, so decoders take the same input +//! from either container. +//! +//! As in `rawshift-video-isobmff`, random access points are read from the +//! bitstream rather than from a container flag, and no symphonia type appears +//! in the public API. +//! +//! # Not covered +//! +//! Matroska expresses display rotation through `Video/Projection`, which the +//! backend does not surface; tracks therefore report +//! [`Rotation::None`](rawshift_video_core::Rotation::None). This costs nothing +//! in practice — rotation is a phone-capture concern, and phones write MP4 or +//! QuickTime, not Matroska. + +#![forbid(unsafe_code)] +#![warn(missing_docs)] + +mod demux; +mod sniff; + +pub use demux::MatroskaDemuxer; +pub use sniff::{Matroska, detect}; diff --git a/crates/rawshift-video-matroska/src/sniff.rs b/crates/rawshift-video-matroska/src/sniff.rs new file mode 100644 index 0000000..915611f --- /dev/null +++ b/crates/rawshift-video-matroska/src/sniff.rs @@ -0,0 +1,104 @@ +//! Signature detection for Matroska and WebM. + +use rawshift_video_core::{ContainerId, ContainerSniffer}; + +/// The EBML magic every Matroska and WebM file starts with. +const EBML_MAGIC: [u8; 4] = [0x1a, 0x45, 0xdf, 0xa3]; + +/// Marker for Matroska / WebM detection and demuxing. +pub struct Matroska; + +impl ContainerSniffer for Matroska { + const CONTAINER: ContainerId = ContainerId::Matroska; + + fn matches(data: &[u8]) -> bool { + detect(data).is_some() + } +} + +/// Which Matroska flavour `data` looks like, if any. +/// +/// WebM is Matroska restricted to royalty-free codecs and declares itself in +/// the `DocType` string of the EBML header. That header is small and sits at +/// the very start, so scanning a short prefix for the string is enough and +/// avoids pulling in an EBML parser just to name the flavour. +#[must_use] +pub fn detect(data: &[u8]) -> Option { + if data.len() < 4 || data[..4] != EBML_MAGIC { + return None; + } + + // The DocType lives within the first few dozen bytes; bound the scan so a + // crafted file cannot make this walk the whole buffer. + let horizon = &data[..data.len().min(64)]; + if horizon.windows(4).any(|w| w == b"webm") { + Some(ContainerId::WebM) + } else { + Some(ContainerId::Matroska) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn ebml(doctype: &[u8]) -> Vec { + let mut out = EBML_MAGIC.to_vec(); + out.extend_from_slice(&[0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x1f]); + out.extend_from_slice(&[0x42, 0x82]); // DocType element id + out.push(doctype.len() as u8); + out.extend_from_slice(doctype); + out + } + + #[test] + fn a_matroska_doctype_is_matroska() { + assert_eq!(detect(&ebml(b"matroska")), Some(ContainerId::Matroska)); + } + + #[test] + fn a_webm_doctype_is_webm() { + assert_eq!(detect(&ebml(b"webm")), Some(ContainerId::WebM)); + } + + #[test] + fn the_ebml_magic_is_required() { + // Right length, wrong magic. + let mut wrong = ebml(b"webm"); + wrong[0] = 0x00; + assert_eq!(detect(&wrong), None); + } + + #[test] + fn unrelated_formats_are_not_claimed() { + assert_eq!(detect(b"\x89PNG\r\n\x1a\n"), None); + assert_eq!(detect(b"\x00\x00\x00\x18ftypisom"), None); + assert_eq!(detect(b"RIFF____AVI "), None); + } + + #[test] + fn short_input_is_rejected_rather_than_panicking() { + for len in 0..4 { + assert_eq!(detect(&vec![0x1au8; len]), None, "len {len}"); + } + // Exactly the magic and nothing else: recognised, flavour unknown, so + // the more general one. + assert_eq!(detect(&EBML_MAGIC), Some(ContainerId::Matroska)); + } + + #[test] + fn the_doctype_scan_is_bounded_and_does_not_match_late_content() { + // "webm" appearing far into the file is track content, not a DocType, + // and must not reclassify the container. + let mut data = ebml(b"matroska"); + data.extend(vec![0u8; 4096]); + data.extend_from_slice(b"webm"); + assert_eq!(detect(&data), Some(ContainerId::Matroska)); + } + + #[test] + fn the_sniffer_trait_agrees_with_detect() { + assert!(Matroska::matches(&ebml(b"webm"))); + assert!(!Matroska::matches(b"\x89PNG\r\n\x1a\n")); + } +} diff --git a/crates/rawshift-video-symphonia/Cargo.toml b/crates/rawshift-video-symphonia/Cargo.toml new file mode 100644 index 0000000..f52eda7 --- /dev/null +++ b/crates/rawshift-video-symphonia/Cargo.toml @@ -0,0 +1,21 @@ +[package] +name = "rawshift-video-symphonia" +version.workspace = true +edition.workspace = true +rust-version.workspace = true +license.workspace = true +repository.workspace = true +homepage.workspace = true +description = "Internal symphonia bridge shared by rawshift's container crates" +documentation = "https://docs.rs/rawshift-video-symphonia" +keywords = ["video", "demux", "symphonia", "internal"] +categories = ["multimedia::video"] +readme = "README.md" + +[dependencies] +rawshift-core = { workspace = true } +rawshift-video-core = { workspace = true } +symphonia-core = { workspace = true } + +[features] +default = [] diff --git a/crates/rawshift-video-symphonia/README.md b/crates/rawshift-video-symphonia/README.md new file mode 100644 index 0000000..c0b3bdc --- /dev/null +++ b/crates/rawshift-video-symphonia/README.md @@ -0,0 +1,16 @@ +# rawshift-video-symphonia + +Internal bridge between [symphonia](https://github.com/pdeljanov/Symphonia) and +[rawshift](https://github.com/visualcommons/rawshift)'s video model. + +Not a container crate and not a public API. It exists so +`rawshift-video-isobmff` and `rawshift-video-matroska` share one translation of +symphonia's tracks, packets, timestamps and seek vocabulary instead of two +copies that drift. + +The translation is one-directional and total: no symphonia type leaves this +crate, which is what lets the container crates keep that guarantee too. + +## License + +Licensed under [MPL-2.0](../../LICENSE). diff --git a/crates/rawshift-video-symphonia/src/lib.rs b/crates/rawshift-video-symphonia/src/lib.rs new file mode 100644 index 0000000..e0c8459 --- /dev/null +++ b/crates/rawshift-video-symphonia/src/lib.rs @@ -0,0 +1,424 @@ +//! Internal bridge between symphonia and rawshift's video model. +//! +//! Not a container crate and not a public API: it exists so +//! `rawshift-video-isobmff` and `rawshift-video-matroska` share one translation +//! instead of two copies that drift apart. +//! +//! The translation is one-directional and total — **no symphonia type leaves +//! this crate**. That is what lets the container crates make the same promise, +//! and it is the property to preserve when editing here: every function takes +//! symphonia types and returns rawshift ones. + +#![forbid(unsafe_code)] +#![warn(missing_docs)] + +use std::io::{Read, Seek, SeekFrom}; +use std::time::Duration; + +use symphonia_core::codecs::CodecParameters; +use symphonia_core::codecs::video::well_known as vwk; +use symphonia_core::codecs::video::well_known::extra_data as vxd; +use symphonia_core::codecs::video::{VideoCodecParameters, VideoExtraData}; +use symphonia_core::errors::Error as SymError; +use symphonia_core::formats::{SeekMode as SymSeekMode, SeekTo as SymSeekTo, Track as SymTrack}; +use symphonia_core::io::MediaSource; +use symphonia_core::packet::Packet as SymPacket; +use symphonia_core::units::{Time, TimeBase, Timestamp}; + +use rawshift_core::Dimensions; +use rawshift_core::metadata::URational; +use rawshift_video_core::{ + AudioCodecId, AudioTrack, CodecConfig, OtherTrack, Packet, SeekMode, SeekTo, Track, TrackId, + TrackKind, VideoCodecId, VideoError, VideoResult, VideoTrack, nal, +}; + +/// The time base rawshift assumes when a track states none: milliseconds. +const DEFAULT_TIME_BASE: URational = URational { + numerator: 1, + denominator: 1000, +}; + +/// Adapts any `Read + Seek` source to symphonia's `MediaSource`. +/// +/// rawshift's public API requires `Seek`, so `is_seekable` is unconditionally +/// true. `byte_len` returns `None` rather than seeking to the end on every +/// call: symphonia treats an unknown length as "find out by reading", which is +/// correct, and probing it here would be a hidden syscall on a hot path. +pub struct SeekableSource(R); + +impl SeekableSource { + /// Wrap a source. + pub const fn new(source: R) -> Self { + Self(source) + } +} + +impl Read for SeekableSource { + fn read(&mut self, buf: &mut [u8]) -> std::io::Result { + self.0.read(buf) + } +} + +impl Seek for SeekableSource { + fn seek(&mut self, pos: SeekFrom) -> std::io::Result { + self.0.seek(pos) + } +} + +impl MediaSource for SeekableSource { + fn is_seekable(&self) -> bool { + true + } + + fn byte_len(&self) -> Option { + None + } +} + +/// Translate a symphonia track, using `rotation` for video tracks. +/// +/// Rotation is a parameter because no container surfaces it through +/// symphonia; each container crate recovers it its own way, or passes the +/// default. +#[must_use] +pub fn map_track(track: &SymTrack, rotation: rawshift_video_core::Rotation) -> Track { + let id = TrackId::new(track.id); + let time_base = map_time_base(track.time_base); + let duration = track + .duration + .and_then(|d| ticks_to_duration(d.get(), time_base)); + + match &track.codec_params { + Some(CodecParameters::Video(v)) => { + let mut out = map_video_track(id, v, time_base); + out.duration = duration; + out.frame_count = track.num_frames; + out.rotation = rotation; + Track::Video(out) + } + Some(CodecParameters::Audio(a)) => { + let mut out = AudioTrack::new(id, AudioCodecId::Other, time_base); + out.sample_rate = a.sample_rate; + out.channels = a.channels.as_ref().map(|c| c.count() as u16); + out.duration = duration; + out.language = track.language.clone(); + Track::Audio(out) + } + _ => { + let mut out = OtherTrack::new(id, TrackKind::Data, time_base); + out.duration = duration; + out.language = track.language.clone(); + Track::Other(out) + } + } +} + +fn map_video_track(id: TrackId, params: &VideoCodecParameters, time_base: URational) -> VideoTrack { + let codec = map_video_codec(params.codec); + let dimensions = Dimensions { + width: u32::from(params.width.unwrap_or(0)), + height: u32::from(params.height.unwrap_or(0)), + }; + + let mut track = VideoTrack::new(id, codec, dimensions, time_base); + track.codec_config = CodecConfig::new(config_record(codec, ¶ms.extra_data)); + track +} + +/// Pick the decoder configuration record matching `codec`. +/// +/// Selected by id rather than by position: a track may carry several extra +/// data blobs — a Dolby Vision configuration accompanies an HEVC base layer, +/// for instance — and taking the first would hand a decoder the wrong record. +#[must_use] +pub fn config_record(codec: VideoCodecId, extra: &[VideoExtraData]) -> Vec { + let wanted = match codec { + VideoCodecId::H264 => vxd::VIDEO_EXTRA_DATA_ID_AVC_DECODER_CONFIG, + VideoCodecId::Hevc => vxd::VIDEO_EXTRA_DATA_ID_HEVC_DECODER_CONFIG, + VideoCodecId::Av1 => vxd::VIDEO_EXTRA_DATA_ID_AV1_DECODER_CONFIG, + VideoCodecId::Vp9 => vxd::VIDEO_EXTRA_DATA_ID_VP9_DECODER_CONFIG, + _ => return Vec::new(), + }; + extra + .iter() + .find(|e| e.id == wanted) + .map(|e| e.data.to_vec()) + .unwrap_or_default() +} + +/// Translate a symphonia video codec id. +/// +/// A codec rawshift does not model becomes [`VideoCodecId::Other`] rather than +/// being dropped, so the track still appears in the track list. +#[must_use] +pub fn map_video_codec(codec: symphonia_core::codecs::video::VideoCodecId) -> VideoCodecId { + match codec { + vwk::CODEC_ID_H264 => VideoCodecId::H264, + vwk::CODEC_ID_HEVC => VideoCodecId::Hevc, + vwk::CODEC_ID_AV1 => VideoCodecId::Av1, + vwk::CODEC_ID_VP9 => VideoCodecId::Vp9, + _ => VideoCodecId::Other, + } +} + +/// Translate a symphonia time base, falling back to milliseconds. +#[must_use] +pub fn map_time_base(time_base: Option) -> URational { + time_base.map_or(DEFAULT_TIME_BASE, |tb| { + URational::new(tb.numer.get(), tb.denom.get()) + }) +} + +/// Convert a tick count in `time_base` units to a [`Duration`]. +/// +/// `None` for a zero denominator or a span no `Duration` can hold. +#[must_use] +pub fn ticks_to_duration(ticks: u64, time_base: URational) -> Option { + if time_base.denominator == 0 { + return None; + } + let seconds = ticks as f64 * f64::from(time_base.numerator) / f64::from(time_base.denominator); + Duration::try_from_secs_f64(seconds).ok() +} + +/// Translate a symphonia packet, deriving the keyframe flag from the +/// bitstream. +/// +/// `codec` and `config` describe the packet's track; pass `None` for a +/// non-video track, whose keyframe flag is left false. symphonia exposes no +/// sync-sample flag, and the bitstream is the authoritative source anyway. +#[must_use] +pub fn map_packet(packet: SymPacket, video: Option<(VideoCodecId, &CodecConfig)>) -> Packet { + let data = packet.data.into_vec(); + let is_keyframe = + video.is_some_and(|(codec, config)| nal::is_random_access_point(codec, config, &data)); + + let mut out = Packet::new(TrackId::new(packet.track_id), data); + out.pts = Some(packet.pts.get()); + out.dts = Some(packet.dts.get()); + out.duration = Some(packet.dur.get()); + out.is_keyframe = is_keyframe; + out +} + +/// Translate a `next_packet` result into rawshift's end-of-stream convention. +/// +/// The two demuxers disagree about how a stream ends: the ISOBMFF reader +/// returns `Ok(None)`, while the Matroska reader surfaces an +/// `UnexpectedEof` I/O error. Both mean the same thing, and a caller looping +/// until `Ok(None)` must not see one of them as a failure. +/// +/// # Errors +/// +/// Any genuine container or I/O failure, as [`VideoError`]. +pub fn map_packet_result( + container: &'static str, + result: Result, SymError>, + video: Option<(VideoCodecId, &CodecConfig)>, +) -> VideoResult> { + match result { + Ok(Some(packet)) => Ok(Some(map_packet(packet, video))), + Ok(None) => Ok(None), + // End of stream, spelled as an error by some readers. + Err(SymError::IoError(e)) if e.kind() == std::io::ErrorKind::UnexpectedEof => Ok(None), + Err(e) => Err(VideoError::container(container, e)), + } +} + +/// Translate a seek request. +/// +/// # Errors +/// +/// [`VideoError::Container`] for a `#[non_exhaustive]` target this bridge does +/// not know how to express, which is reported rather than silently +/// approximated to a different position. +pub fn map_seek( + container: &'static str, + track: TrackId, + to: SeekTo, + mode: SeekMode, +) -> VideoResult<(SymSeekMode, SymSeekTo)> { + let target = match to { + SeekTo::Timestamp(ts) => SymSeekTo::Timestamp { + ts: Timestamp::new(ts), + track_id: track.get(), + }, + SeekTo::Time(time) => SymSeekTo::Time { + time: Time::try_from_nanos_u128(time.as_nanos()).unwrap_or(Time::ZERO), + track_id: Some(track.get()), + }, + other => { + return Err(VideoError::container( + container, + format!("unsupported seek target: {other:?}"), + )); + } + }; + + let sym_mode = match mode { + // symphonia's "Accurate" means "at or before the request", which is + // what rawshift calls Precise. + SeekMode::Precise => SymSeekMode::Accurate, + SeekMode::Coarse => SymSeekMode::Coarse, + // SeekMode is #[non_exhaustive]; an unknown mode takes the + // conservative option, which is always safe to resume decoding from. + _ => SymSeekMode::Accurate, + }; + + Ok((sym_mode, target)) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn a_missing_time_base_falls_back_to_milliseconds() { + assert_eq!(map_time_base(None), URational::new(1, 1000)); + } + + #[test] + fn durations_convert_against_the_time_base() { + // 90000 ticks of a 90 kHz base is one second. + assert_eq!( + ticks_to_duration(90_000, URational::new(1, 90_000)), + Some(Duration::from_secs(1)) + ); + assert_eq!( + ticks_to_duration(1_500, URational::new(1, 1_000)), + Some(Duration::from_millis(1_500)) + ); + } + + #[test] + fn a_zero_denominator_does_not_divide_by_zero() { + assert_eq!(ticks_to_duration(1_000, URational::new(1, 0)), None); + } + + #[test] + fn the_config_record_is_selected_by_codec_not_position() { + // An HEVC track carrying a Dolby Vision blob first must still yield + // the hvcC, not whatever sits at index zero. + let extra = vec![ + VideoExtraData { + id: vxd::VIDEO_EXTRA_DATA_ID_DOLBY_VISION_CONFIG, + data: vec![0xDD; 4].into_boxed_slice(), + }, + VideoExtraData { + id: vxd::VIDEO_EXTRA_DATA_ID_HEVC_DECODER_CONFIG, + data: vec![0xAA; 4].into_boxed_slice(), + }, + ]; + assert_eq!(config_record(VideoCodecId::Hevc, &extra), vec![0xAA; 4]); + } + + #[test] + fn a_track_with_no_matching_record_yields_an_empty_one() { + let extra = vec![VideoExtraData { + id: vxd::VIDEO_EXTRA_DATA_ID_AVC_DECODER_CONFIG, + data: vec![0x01; 4].into_boxed_slice(), + }]; + // Asking for HEVC when only an avcC is present must not return it. + assert!(config_record(VideoCodecId::Hevc, &extra).is_empty()); + assert!(config_record(VideoCodecId::ProRes, &extra).is_empty()); + } + + #[test] + fn known_codecs_map_and_unknown_ones_degrade_to_other() { + assert_eq!(map_video_codec(vwk::CODEC_ID_H264), VideoCodecId::H264); + assert_eq!(map_video_codec(vwk::CODEC_ID_HEVC), VideoCodecId::Hevc); + assert_eq!(map_video_codec(vwk::CODEC_ID_AV1), VideoCodecId::Av1); + assert_eq!(map_video_codec(vwk::CODEC_ID_VP9), VideoCodecId::Vp9); + assert_eq!(map_video_codec(vwk::CODEC_ID_MJPEG), VideoCodecId::Other); + } + + #[test] + fn end_of_stream_is_ok_none_however_the_reader_spells_it() { + // The ISOBMFF reader says Ok(None); the Matroska reader raises + // UnexpectedEof. A caller looping until Ok(None) must not see the + // second as a failure. + let explicit: Result, SymError> = Ok(None); + assert!(matches!(map_packet_result("MP4", explicit, None), Ok(None))); + + let as_error: Result, SymError> = Err(SymError::IoError( + std::io::Error::from(std::io::ErrorKind::UnexpectedEof), + )); + assert!(matches!( + map_packet_result("Matroska", as_error, None), + Ok(None) + )); + } + + #[test] + fn a_genuine_io_failure_is_still_an_error() { + // Only UnexpectedEof means end of stream; a permission error must not + // be silently swallowed into a clean end. + let denied: Result, SymError> = Err(SymError::IoError( + std::io::Error::from(std::io::ErrorKind::PermissionDenied), + )); + assert!(map_packet_result("MP4", denied, None).is_err()); + + let malformed: Result, SymError> = Err(SymError::DecodeError("bad atom")); + assert!(map_packet_result("MP4", malformed, None).is_err()); + } + + #[test] + fn seek_modes_translate_with_precise_meaning_at_or_before() { + let (mode, _) = map_seek( + "MP4", + TrackId::new(1), + SeekTo::Timestamp(0), + SeekMode::Precise, + ) + .expect("precise"); + assert_eq!(mode, SymSeekMode::Accurate); + + let (mode, _) = map_seek( + "MP4", + TrackId::new(1), + SeekTo::Timestamp(0), + SeekMode::Coarse, + ) + .expect("coarse"); + assert_eq!(mode, SymSeekMode::Coarse); + } + + #[test] + fn a_time_seek_carries_sub_second_precision() { + let (_, target) = map_seek( + "MP4", + TrackId::new(3), + SeekTo::Time(Duration::from_millis(1_500)), + SeekMode::Precise, + ) + .expect("time seek"); + + match target { + SymSeekTo::Time { time, track_id } => { + assert_eq!(track_id, Some(3)); + // 1.5 s must survive as 1.5 s, not truncate to 1 s. + assert!((time.as_secs_f64() - 1.5).abs() < 1e-9, "{time:?}"); + } + _ => panic!("expected a time seek"), + } + } + + #[test] + fn a_timestamp_seek_is_relative_to_the_named_track() { + let (_, target) = map_seek( + "MKV", + TrackId::new(7), + SeekTo::Timestamp(4_242), + SeekMode::Precise, + ) + .expect("timestamp seek"); + + match target { + SymSeekTo::Timestamp { ts, track_id } => { + assert_eq!(ts.get(), 4_242); + assert_eq!(track_id, 7); + } + _ => panic!("expected a timestamp seek"), + } + } +} From 7719cd0b97812b773c6cbb77a81c08744c4a2e3a Mon Sep 17 00:00:00 2001 From: Justin Chung Date: Fri, 28 Aug 2026 22:31:21 -0400 Subject: [PATCH 07/13] feat(video-isobmff)!: own the sample index on mp4-atom MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replaces symphonia-format-isomp4 with mp4-atom plus a rawshift-owned sample index, track model and demuxer. The trigger was a hard blocker found by running the demuxer against real files: symphonia-format-isomp4 caps the hvcC configuration record at 1 KB ("It should not exceed 1 kB"), and real HEVC records are 2-3 KB — the plain x265 fixture here is 2440 bytes. The error aborts the open, so every HEVC MP4 and MOV failed outright. That is XAVC HS and default iPhone video, the two highest-priority formats on the roadmap, and it could not be worked around from outside. That reader is audio-first, and the same investigation found it hides four more things video needs: the tkhd matrix (rotation), the colr box (all colour signalling), the sync-sample table, and ProRes-shaped sample entries. mp4-atom is a box codec that does no interpretation, so rawshift reconstructs the sample index from the stbl tables itself — a few hundred lines that remove all five limitations at once. Also fixes timestamps. The edit list is now applied, so presentation starts at zero; without it every timestamp carried the encoder's reorder delay and disagreed with every other tool. rawshift's first PTS now matches ffprobe's on the same file. Verified against ffmpeg-generated fixtures: H.264 and HEVC in both MP4 and QuickTime now yield correct tracks, dimensions, durations, frame counts, config records and timestamps, with random access points matching the encoder's GOP length. The sample-table walk is total against untrusted input: an inconsistent or truncated table stops the walk and yields what was recovered, and no byte range can point outside the file. Fragmented MP4 is rejected with a clear error rather than presented as a track with no samples; camera files are not fragmented. BREAKING CHANGE: IsoBmffDemuxer is now generic over its source and no longer requires Send + Sync + 'static, since samples are read on demand rather than through a boxed media source. Refs #39 --- Cargo.lock | 169 ++++- Cargo.toml | 10 +- crates/rawshift-video-isobmff/Cargo.toml | 4 +- crates/rawshift-video-isobmff/README.md | 45 +- crates/rawshift-video-isobmff/src/boxes.rs | 616 ----------------- crates/rawshift-video-isobmff/src/demux.rs | 313 ++++----- crates/rawshift-video-isobmff/src/index.rs | 755 +++++++++++++++++++++ crates/rawshift-video-isobmff/src/lib.rs | 44 +- crates/rawshift-video-isobmff/src/movie.rs | 500 ++++++++++++++ 9 files changed, 1584 insertions(+), 872 deletions(-) delete mode 100644 crates/rawshift-video-isobmff/src/boxes.rs create mode 100644 crates/rawshift-video-isobmff/src/index.rs create mode 100644 crates/rawshift-video-isobmff/src/movie.rs diff --git a/Cargo.lock b/Cargo.lock index 80a8d9b..aedf809 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -365,6 +365,27 @@ version = "0.3.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "be1e0bca6c3637f992fc1cc7cbc52a78c1ef6db076dbf1059c4323d6a2048376" +[[package]] +name = "derive_more" +version = "2.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d751e9e49156b02b44f9c1815bcb94b984cdcc4396ecc32521c739452808b134" +dependencies = [ + "derive_more-impl", +] + +[[package]] +name = "derive_more-impl" +version = "2.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "799a97264921d8623a957f6c3b9011f3b5492f557bbb7a5a19b7fa6d06ba8dcb" +dependencies = [ + "proc-macro2", + "quote", + "rustc_version", + "syn", +] + [[package]] name = "digest" version = "0.10.7" @@ -539,7 +560,7 @@ version = "2.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b17c79c8672b675d538dceffda1dd5d00ba279cbcca4f75c7b4fa847741d5d31" dependencies = [ - "thiserror", + "thiserror 2.0.18", ] [[package]] @@ -575,7 +596,7 @@ checksum = "c9ce8735a1fc71d77b5361f4f126676abab8811cdd3797a6bacdb920cd20f85d" dependencies = [ "gamut-core", "gamut-ifd", - "thiserror", + "thiserror 2.0.18", ] [[package]] @@ -598,7 +619,7 @@ checksum = "09916f0aea3f9dbe1c5ac67e0e29956b9af43c8c86a0fbe6b47c597b63fd8cee" dependencies = [ "gamut-core", "md-5", - "thiserror", + "thiserror 2.0.18", ] [[package]] @@ -618,7 +639,7 @@ checksum = "99b7bb8412adf69f48657d207ff5358750ebe4efe81c2dd7b32091034f354d6b" dependencies = [ "gamut-core", "gamut-xmp", - "thiserror", + "thiserror 2.0.18", ] [[package]] @@ -684,7 +705,7 @@ dependencies = [ "gamut-icc", "gamut-iptc", "gamut-xmp", - "thiserror", + "thiserror 2.0.18", ] [[package]] @@ -730,7 +751,7 @@ checksum = "fd39a4ebd0d14800751e3ae64a0c86a07e00b77f3ac564570e0e8987d41945bb" dependencies = [ "gamut-core", "quick-xml", - "thiserror", + "thiserror 2.0.18", ] [[package]] @@ -862,7 +883,7 @@ dependencies = [ "jxl_transforms", "num-derive", "num-traits", - "thiserror", + "thiserror 2.0.18", ] [[package]] @@ -1001,6 +1022,19 @@ dependencies = [ "windows-sys", ] +[[package]] +name = "mp4-atom" +version = "0.15.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f3d7433df407a636db7047600422af07145adf6ee25e0622745b0131516dbc3c" +dependencies = [ + "derive_more", + "num", + "pastey", + "thiserror 1.0.69", + "tracing", +] + [[package]] name = "nu-ansi-term" version = "0.50.3" @@ -1010,6 +1044,30 @@ dependencies = [ "windows-sys", ] +[[package]] +name = "num" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "35bd024e8b2ff75562e5f34e7f4905839deb4b22955ef5e73d2fea1b9813cb23" +dependencies = [ + "num-bigint", + "num-complex", + "num-integer", + "num-iter", + "num-rational", + "num-traits", +] + +[[package]] +name = "num-bigint" +version = "0.4.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c89e69e7e0f03bea5ef08013795c25018e101932225a656383bd384495ecc367" +dependencies = [ + "num-integer", + "num-traits", +] + [[package]] name = "num-complex" version = "0.4.6" @@ -1030,6 +1088,36 @@ dependencies = [ "syn", ] +[[package]] +name = "num-integer" +version = "0.1.47" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7ce2d95d4b3734dc35aa2f45e1aa22cd416814592a4f9d9205e11affd5b8e10b" +dependencies = [ + "num-traits", +] + +[[package]] +name = "num-iter" +version = "0.1.46" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c92800bd69a1eac91786bcfe9da64a897eb72911b8dc3095decbd07429e8048b" +dependencies = [ + "num-integer", + "num-traits", +] + +[[package]] +name = "num-rational" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f83d14da390562dca69fc84082e73e548e1ad308d24accdedd2720017cb37824" +dependencies = [ + "num-bigint", + "num-integer", + "num-traits", +] + [[package]] name = "num-traits" version = "0.2.19" @@ -1080,6 +1168,12 @@ dependencies = [ "windows-link", ] +[[package]] +name = "pastey" +version = "0.2.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2ee67f1008b1ba2321834326597b8e186293b049a023cdef258527550b9935b4" + [[package]] name = "pico-args" version = "0.5.0" @@ -1211,7 +1305,7 @@ version = "0.1.1" dependencies = [ "gamut-color", "libloading", - "thiserror", + "thiserror 2.0.18", ] [[package]] @@ -1264,7 +1358,7 @@ dependencies = [ "resvg", "serde", "serde_json", - "thiserror", + "thiserror 2.0.18", "tiff", "tokio", "tracing", @@ -1306,7 +1400,7 @@ dependencies = [ "rawshift-core", "rawshift-hwdec", "serde", - "thiserror", + "thiserror 2.0.18", ] [[package]] @@ -1504,18 +1598,16 @@ dependencies = [ "gamut-color", "rawshift-core", "serde", - "thiserror", + "thiserror 2.0.18", ] [[package]] name = "rawshift-video-isobmff" version = "0.1.1" dependencies = [ + "mp4-atom", "rawshift-core", "rawshift-video-core", - "rawshift-video-symphonia", - "symphonia-core", - "symphonia-format-isomp4", "tracing", ] @@ -1635,6 +1727,15 @@ version = "0.20.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6c20b6793b5c2fa6553b250154b78d6d0db37e72700ae35fad9387a46f487c97" +[[package]] +name = "rustc_version" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92" +dependencies = [ + "semver", +] + [[package]] name = "rustversion" version = "1.0.22" @@ -1674,6 +1775,12 @@ version = "1.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" +[[package]] +name = "semver" +version = "1.0.28" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd" + [[package]] name = "serde" version = "1.0.228" @@ -1838,18 +1945,6 @@ dependencies = [ "smallvec", ] -[[package]] -name = "symphonia-format-isomp4" -version = "0.6.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0e681a70e1870d34e02abf1dbc51e4267c3f1827801474e8870be8c689fc4dc3" -dependencies = [ - "log", - "symphonia-common", - "symphonia-core", - "symphonia-metadata", -] - [[package]] name = "symphonia-format-mkv" version = "0.6.1" @@ -1886,13 +1981,33 @@ dependencies = [ "unicode-ident", ] +[[package]] +name = "thiserror" +version = "1.0.69" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6aaf5339b578ea85b50e080feb250a3e8ae8cfcdff9a461c9ec2904bc923f52" +dependencies = [ + "thiserror-impl 1.0.69", +] + [[package]] name = "thiserror" version = "2.0.18" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4288b5bcbc7920c07a1149a35cf9590a2aa808e0bc1eafaade0b80947865fbc4" dependencies = [ - "thiserror-impl", + "thiserror-impl 2.0.18", +] + +[[package]] +name = "thiserror-impl" +version = "1.0.69" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4fee6c4efc90059e10f81e6d42c60a18f76588c3d74cb83a0b242a2b6c7504c1" +dependencies = [ + "proc-macro2", + "quote", + "syn", ] [[package]] diff --git a/Cargo.toml b/Cargo.toml index 4f99932..a8a1f73 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -73,9 +73,17 @@ gamut-isobmff = "2.0.1" # default-features off keeps symphonia-core's optional rustfft (audio DSP) out # of the tree; rawshift never decodes audio. symphonia-core = { version = "0.6.1", default-features = false } -symphonia-format-isomp4 = "0.6.1" symphonia-format-mkv = "0.6.1" +# mp4-atom — the ISOBMFF box layer under rawshift-video-isobmff. MIT, pure +# Rust, no build script. It replaced symphonia-format-isomp4, which is +# audio-first and caps the hvcC record at 1 KB — real HEVC records are 2-3 KB, +# so every HEVC MP4 and MOV failed to open. mp4-atom additionally surfaces the +# tkhd matrix, the colr box and ProRes-shaped sample entries, all of which the +# audio-first reader hides. rawshift owns the sample-index and demux semantics +# on top; mp4-atom deliberately does no interpretation. +mp4-atom = "0.15.0" + # Shared infrastructure dependencies (used across multiple workspace crates). thiserror = "2.0" tracing = "0.1" diff --git a/crates/rawshift-video-isobmff/Cargo.toml b/crates/rawshift-video-isobmff/Cargo.toml index 0c6d310..dcbfef2 100644 --- a/crates/rawshift-video-isobmff/Cargo.toml +++ b/crates/rawshift-video-isobmff/Cargo.toml @@ -15,9 +15,7 @@ readme = "README.md" [dependencies] rawshift-core = { workspace = true } rawshift-video-core = { workspace = true } -rawshift-video-symphonia = { workspace = true } -symphonia-core = { workspace = true } -symphonia-format-isomp4 = { workspace = true } +mp4-atom = { workspace = true } tracing = { workspace = true } [features] diff --git a/crates/rawshift-video-isobmff/README.md b/crates/rawshift-video-isobmff/README.md index 985dc6a..511d663 100644 --- a/crates/rawshift-video-isobmff/README.md +++ b/crates/rawshift-video-isobmff/README.md @@ -3,24 +3,41 @@ MP4, M4V and QuickTime (MOV) support for [rawshift](https://github.com/visualcommons/rawshift). -Demuxing is backed by -[`symphonia-format-isomp4`](https://crates.io/crates/symphonia-format-isomp4) -(MPL-2.0, pure Rust, no build script), which parses the movie box tree, the -sample tables, fragments, and the `avc1`/`hvc1`/`hev1`/`av01` sample entries -rawshift needs. +The box layer is [`mp4-atom`](https://crates.io/crates/mp4-atom) (MIT, pure +Rust, no build script). It performs no interpretation, so this crate owns the +parts that give a container meaning: the sample index reconstructed from the +`stbl` tables, the track model, and the demux and seek semantics. -Two things symphonia does not surface, which this crate reads itself from a -focused header walk: +## Why not an off-the-shelf demuxer -- the `tkhd` display matrix, and so track rotation — without which most phone - video presents sideways; -- the `ftyp` brands and `mvhd` timescale and times. +This crate first used `symphonia-format-isomp4`, which is audio-first and caps +the `hvcC` configuration record at 1 KB. Real HEVC records are 2–3 KB, so +**every HEVC MP4 and MOV failed to open** — that is XAVC HS and default iPhone +video, the two highest-priority formats on rawshift's roadmap. It also hides +the `tkhd` matrix (and so rotation), the `colr` box (and so all colour +signalling), the sync-sample table, and ProRes-shaped sample entries. -Sync samples are a third: symphonia exposes no keyframe flag, so random access -points are read out of the bitstream by `rawshift-video-core`, which is the -authoritative source anyway. +Owning the sample index on a box-level library costs a few hundred lines and +removes all five limitations at once. Matroska keeps its symphonia backend, +which has none of these problems. -No symphonia type appears in this crate's public API. +## What it reads + +`ftyp` brands, the `mvhd` timescale and times, per-track `tkhd` rotation, +`mdhd` timescale and language, the `stsd` sample entry (including `colr` +colour signalling and the `avcC`/`hvcC` record a decoder needs), the full +`stbl` sample tables, and the `elst` presentation shift — without which every +timestamp is offset by the encoder's reorder delay and disagrees with every +other tool. + +Samples are read on demand, so opening a file costs only its headers however +large the media is. + +## Not covered + +Fragmented MP4 (`mvex`/`moof`) is rejected with a clear error rather than +presented as a track with no samples. Camera-origin files — rawshift's subject +— are not fragmented. ## License diff --git a/crates/rawshift-video-isobmff/src/boxes.rs b/crates/rawshift-video-isobmff/src/boxes.rs deleted file mode 100644 index 53f09ab..0000000 --- a/crates/rawshift-video-isobmff/src/boxes.rs +++ /dev/null @@ -1,616 +0,0 @@ -//! A focused ISOBMFF header walk for the facts the demux backend does not -//! surface. -//! -//! This is **not** a demuxer and must not grow into one. It reads box headers -//! and four small boxes — `ftyp`, `mvhd`, `tkhd`, `mdhd` — because -//! `symphonia-format-isomp4` exposes neither the track display matrix (and so -//! no rotation, without which most phone video presents sideways) nor the -//! `ftyp` brands. Everything else about the file comes from symphonia. -//! -//! Every read is bounds-checked and every container descent is depth- and -//! size-limited: this parses attacker-supplied bytes. - -use std::io::{self, Read, Seek, SeekFrom}; - -use rawshift_video_core::{ContainerId, Rotation, VideoResult}; - -/// How deep to descend into nested boxes. -/// -/// The tree this walks is `moov > trak > mdia > mdhd`, so four levels are -/// enough; the limit stops a crafted file from recursing without bound. -const MAX_DEPTH: u32 = 8; - -/// The largest box header this walker will honour, as a sanity bound against -/// a crafted 64-bit size. -const MAX_BOX_SIZE: u64 = 1 << 40; - -/// What the header walk recovered. -#[derive(Debug, Clone, Default, PartialEq)] -pub struct BoxSurvey { - /// `ftyp` major brand, then compatible brands. - pub brands: Vec, - /// `mvhd` timescale, in ticks per second. - pub timescale: Option, - /// `mvhd` duration, in timescale ticks. - pub duration: Option, - /// `mvhd` creation time, in seconds since 1904-01-01 UTC. - pub creation_time: Option, - /// `mvhd` modification time, in the same epoch. - pub modification_time: Option, - /// Per-track facts, keyed by `tkhd` track id. - pub tracks: Vec, -} - -/// Per-track facts from `tkhd`. -#[derive(Debug, Clone, PartialEq)] -pub struct TrackSurvey { - /// The `tkhd` track identifier. - pub id: u32, - /// Display rotation recovered from the `tkhd` matrix. - pub rotation: Rotation, -} - -impl BoxSurvey { - /// The rotation for a track id, or [`Rotation::None`] if not seen. - #[must_use] - pub fn rotation(&self, track_id: u32) -> Rotation { - self.tracks - .iter() - .find(|t| t.id == track_id) - .map_or(Rotation::None, |t| t.rotation) - } - - /// The container flavour implied by the `ftyp` brands. - /// - /// The QuickTime brand `qt ` means [`ContainerId::Mov`]; anything else - /// with a `ftyp` is treated as MP4. Files with no `ftyp` at all are - /// old-style QuickTime. - #[must_use] - pub fn container(&self) -> ContainerId { - if self.brands.is_empty() { - return ContainerId::Mov; - } - if self.brands.iter().any(|b| b == "qt ") { - ContainerId::Mov - } else { - ContainerId::Mp4 - } - } -} - -/// Walk the box tree of `reader`, recovering [`BoxSurvey`]. -/// -/// Leaves the reader's position undefined; the caller rewinds. -/// -/// # Errors -/// -/// Propagates I/O failures. A structurally odd file is *not* an error: the -/// walk stops and returns what it has, because these facts are supplementary -/// and a file whose `tkhd` cannot be read should still demux. -pub fn survey(reader: &mut R) -> VideoResult { - let end = reader.seek(SeekFrom::End(0))?; - reader.seek(SeekFrom::Start(0))?; - - let mut out = BoxSurvey::default(); - walk(reader, 0, end, 0, &mut out)?; - Ok(out) -} - -/// One box header: its payload range and type. -struct BoxHeader { - kind: [u8; 4], - payload_start: u64, - payload_end: u64, -} - -/// Read a box header at `offset`, or `None` at a clean end or on anything -/// malformed. -fn read_header( - reader: &mut R, - offset: u64, - limit: u64, -) -> io::Result> { - if offset.saturating_add(8) > limit { - return Ok(None); - } - reader.seek(SeekFrom::Start(offset))?; - - let mut header = [0u8; 8]; - if reader.read_exact(&mut header).is_err() { - return Ok(None); - } - let size32 = u32::from_be_bytes([header[0], header[1], header[2], header[3]]); - let kind = [header[4], header[5], header[6], header[7]]; - - let (size, payload_start) = match size32 { - // 1 means the real size is in a following 64-bit field. - 1 => { - let mut extended = [0u8; 8]; - if reader.read_exact(&mut extended).is_err() { - return Ok(None); - } - (u64::from_be_bytes(extended), offset + 16) - } - // 0 means "extends to the end of the enclosing container". - 0 => (limit - offset, offset + 8), - n => (u64::from(n), offset + 8), - }; - - // A box must at least contain its own header, and must not overrun its - // parent or exceed the sanity bound. - let header_len = payload_start - offset; - if size < header_len || size > MAX_BOX_SIZE { - return Ok(None); - } - let payload_end = offset.saturating_add(size).min(limit); - if payload_end < payload_start { - return Ok(None); - } - - Ok(Some(BoxHeader { - kind, - payload_start, - payload_end, - })) -} - -/// Descend through `[start, limit)`, collecting into `out`. -fn walk( - reader: &mut R, - depth: u32, - limit: u64, - start: u64, - out: &mut BoxSurvey, -) -> VideoResult<()> { - if depth > MAX_DEPTH { - return Ok(()); - } - - let mut offset = start; - while offset < limit { - let Some(header) = read_header(reader, offset, limit)? else { - return Ok(()); - }; - // A zero-length advance would spin forever on a crafted file. - if header.payload_end <= offset { - return Ok(()); - } - - match &header.kind { - b"ftyp" => read_ftyp(reader, &header, out)?, - b"mvhd" => read_mvhd(reader, &header, out)?, - b"tkhd" => read_tkhd(reader, &header, out)?, - // Containers worth descending into. `moof`/`traf` are skipped: - // fragments repeat no tkhd, and symphonia owns them. - b"moov" | b"trak" | b"mdia" => { - walk( - reader, - depth + 1, - header.payload_end, - header.payload_start, - out, - )?; - } - _ => {} - } - - offset = header.payload_end; - } - Ok(()) -} - -/// Read the whole payload of a small box, refusing anything implausible. -fn read_payload( - reader: &mut R, - header: &BoxHeader, - max: usize, -) -> io::Result>> { - let len = (header.payload_end - header.payload_start) as usize; - if len > max { - return Ok(None); - } - reader.seek(SeekFrom::Start(header.payload_start))?; - let mut buf = vec![0u8; len]; - if reader.read_exact(&mut buf).is_err() { - return Ok(None); - } - Ok(Some(buf)) -} - -fn read_ftyp( - reader: &mut R, - header: &BoxHeader, - out: &mut BoxSurvey, -) -> VideoResult<()> { - // major brand + minor version + compatible brands; a few hundred bytes at - // the very most. - let Some(payload) = read_payload(reader, header, 1024)? else { - return Ok(()); - }; - if payload.len() < 4 { - return Ok(()); - } - - let mut brands = vec![brand_string(&payload[0..4])]; - // Skip the 4-byte minor version, then read 4-byte compatible brands. - for chunk in payload.get(8..).unwrap_or_default().chunks_exact(4) { - brands.push(brand_string(chunk)); - } - out.brands = brands; - Ok(()) -} - -/// Render a four-character brand, replacing non-printable bytes so a crafted -/// brand cannot smuggle control characters into a log line or error message. -fn brand_string(bytes: &[u8]) -> String { - bytes - .iter() - .map(|&b| { - if b.is_ascii_graphic() || b == b' ' { - b as char - } else { - '?' - } - }) - .collect() -} - -fn read_mvhd( - reader: &mut R, - header: &BoxHeader, - out: &mut BoxSurvey, -) -> VideoResult<()> { - let Some(payload) = read_payload(reader, header, 256)? else { - return Ok(()); - }; - // version(1) + flags(3), then the version-dependent block. - let Some(&version) = payload.first() else { - return Ok(()); - }; - - let (creation, modification, timescale, duration) = match version { - 1 => { - if payload.len() < 4 + 28 { - return Ok(()); - } - ( - be64(&payload[4..12]), - be64(&payload[12..20]), - be32(&payload[20..24]), - be64(&payload[24..32]), - ) - } - _ => { - if payload.len() < 4 + 16 { - return Ok(()); - } - ( - u64::from(be32(&payload[4..8])), - u64::from(be32(&payload[8..12])), - be32(&payload[12..16]), - u64::from(be32(&payload[16..20])), - ) - } - }; - - out.creation_time = Some(creation); - out.modification_time = Some(modification); - out.timescale = (timescale != 0).then_some(timescale); - // 0xffff_ffff / u64::MAX is the "unknown duration" sentinel. - out.duration = (duration != u64::from(u32::MAX) && duration != u64::MAX).then_some(duration); - Ok(()) -} - -fn read_tkhd( - reader: &mut R, - header: &BoxHeader, - out: &mut BoxSurvey, -) -> VideoResult<()> { - let Some(payload) = read_payload(reader, header, 256)? else { - return Ok(()); - }; - let Some(&version) = payload.first() else { - return Ok(()); - }; - - // ISO/IEC 14496-12 §8.3.2. The track id follows version+flags and the two - // times, which are 8 bytes each in version 1 and 4 in version 0; the - // matrix then follows reserved, duration, reserved[2], layer, - // alternate_group, volume and one more reserved. - // - // v0: 4 +4+4 =12 (id) … +4+4 +8 +2+2 +2+2 = 40 (matrix) - // v1: 4 +8+8 =20 (id) … +4+8 +8 +2+2 +2+2 = 52 (matrix) - let (id_offset, matrix_offset) = if version == 1 { (20, 52) } else { (12, 40) }; - - let Some(id_bytes) = payload.get(id_offset..id_offset + 4) else { - return Ok(()); - }; - let id = be32(id_bytes); - - let rotation = payload - .get(matrix_offset..matrix_offset + 36) - .map_or(Rotation::None, rotation_from_matrix); - - out.tracks.push(TrackSurvey { id, rotation }); - Ok(()) -} - -/// Recover rotation from the nine 32-bit fixed-point values of a `tkhd` -/// matrix. -/// -/// Only `a`, `b`, `c`, `d` — indices 0, 1, 3, 4 — carry rotation. They are -/// 16.16 fixed point (`u`, `v`, `w` are 2.30, and unused here). -fn rotation_from_matrix(matrix: &[u8]) -> Rotation { - let fixed = |index: usize| -> f64 { - let at = index * 4; - f64::from(i32::from_be_bytes([ - matrix[at], - matrix[at + 1], - matrix[at + 2], - matrix[at + 3], - ])) / 65536.0 - }; - Rotation::from_display_matrix(fixed(0), fixed(1), fixed(3), fixed(4)) -} - -fn be32(bytes: &[u8]) -> u32 { - u32::from_be_bytes([bytes[0], bytes[1], bytes[2], bytes[3]]) -} - -fn be64(bytes: &[u8]) -> u64 { - let mut out = [0u8; 8]; - out.copy_from_slice(&bytes[..8]); - u64::from_be_bytes(out) -} - -#[cfg(test)] -mod tests { - use super::*; - use std::io::Cursor; - - /// Build a box: 4-byte size, 4-byte type, payload. - fn bx(kind: &[u8; 4], payload: &[u8]) -> Vec { - let mut out = ((payload.len() + 8) as u32).to_be_bytes().to_vec(); - out.extend_from_slice(kind); - out.extend_from_slice(payload); - out - } - - /// A 16.16 fixed-point matrix from its rotation sub-matrix. - fn matrix(a: f64, b: f64, c: f64, d: f64) -> Vec { - let fx = |v: f64| ((v * 65536.0) as i32).to_be_bytes().to_vec(); - let mut out = Vec::new(); - out.extend(fx(a)); - out.extend(fx(b)); - out.extend(vec![0u8; 4]); // u - out.extend(fx(c)); - out.extend(fx(d)); - out.extend(vec![0u8; 4]); // v - out.extend(vec![0u8; 12]); // x, y, w - out - } - - fn tkhd_v0(track_id: u32, m: &[u8]) -> Vec { - let mut p = vec![0u8; 4]; // version 0 + flags - p.extend_from_slice(&0u32.to_be_bytes()); // creation - p.extend_from_slice(&0u32.to_be_bytes()); // modification - p.extend_from_slice(&track_id.to_be_bytes()); // track id @ 12 - p.extend_from_slice(&0u32.to_be_bytes()); // reserved - p.extend_from_slice(&0u32.to_be_bytes()); // duration - p.extend(vec![0u8; 8]); // reserved - p.extend(vec![0u8; 8]); // layer, alt group, volume, reserved - assert_eq!(p.len(), 40, "matrix must land at offset 40 in a v0 tkhd"); - p.extend_from_slice(m); - p.extend(vec![0u8; 8]); // width, height - p - } - - fn mvhd_v0(timescale: u32, duration: u32) -> Vec { - let mut p = vec![0u8; 4]; - p.extend_from_slice(&1_000u32.to_be_bytes()); // creation - p.extend_from_slice(&2_000u32.to_be_bytes()); // modification - p.extend_from_slice(×cale.to_be_bytes()); - p.extend_from_slice(&duration.to_be_bytes()); - p.extend(vec![0u8; 80]); - p - } - - fn file(parts: &[Vec]) -> Cursor> { - Cursor::new(parts.concat()) - } - - #[test] - fn brands_come_from_ftyp() { - let mut ftyp = b"isom".to_vec(); - ftyp.extend_from_slice(&512u32.to_be_bytes()); - ftyp.extend_from_slice(b"isomiso2avc1mp41"); - let mut f = file(&[bx(b"ftyp", &ftyp)]); - - let s = survey(&mut f).expect("survey"); - assert_eq!(s.brands, ["isom", "isom", "iso2", "avc1", "mp41"]); - assert_eq!(s.container(), ContainerId::Mp4); - } - - #[test] - fn the_quicktime_brand_selects_mov() { - let mut ftyp = b"qt ".to_vec(); - ftyp.extend_from_slice(&0u32.to_be_bytes()); - ftyp.extend_from_slice(b"qt "); - let mut f = file(&[bx(b"ftyp", &ftyp)]); - - assert_eq!( - survey(&mut f).expect("survey").container(), - ContainerId::Mov - ); - } - - #[test] - fn a_file_with_no_ftyp_is_old_style_quicktime() { - let mut f = file(&[bx(b"moov", &[])]); - assert_eq!( - survey(&mut f).expect("survey").container(), - ContainerId::Mov - ); - } - - #[test] - fn a_brand_with_control_bytes_cannot_smuggle_them_into_a_message() { - let mut ftyp = vec![0x00, 0x1b, 0x5b, 0x41]; - ftyp.extend_from_slice(&0u32.to_be_bytes()); - let mut f = file(&[bx(b"ftyp", &ftyp)]); - - let s = survey(&mut f).expect("survey"); - // The NUL and the ESC are replaced; the printable bytes are kept, so - // no control character can reach a log line or error message. - assert_eq!(s.brands, ["??[A"]); - } - - #[test] - fn mvhd_yields_timescale_and_times() { - let moov = bx(b"mvhd", &mvhd_v0(600, 12_000)); - let mut f = file(&[bx(b"moov", &moov)]); - - let s = survey(&mut f).expect("survey"); - assert_eq!(s.timescale, Some(600)); - assert_eq!(s.duration, Some(12_000)); - assert_eq!(s.creation_time, Some(1_000)); - assert_eq!(s.modification_time, Some(2_000)); - } - - #[test] - fn an_unknown_duration_sentinel_is_reported_as_absent() { - let moov = bx(b"mvhd", &mvhd_v0(600, u32::MAX)); - let mut f = file(&[bx(b"moov", &moov)]); - assert_eq!(survey(&mut f).expect("survey").duration, None); - } - - #[test] - fn a_zero_timescale_is_reported_as_absent_rather_than_dividing_by_zero() { - let moov = bx(b"mvhd", &mvhd_v0(0, 100)); - let mut f = file(&[bx(b"moov", &moov)]); - assert_eq!(survey(&mut f).expect("survey").timescale, None); - } - - #[test] - fn rotation_is_recovered_per_track_from_the_tkhd_matrix() { - // Track 1 upright, track 2 rotated 90° as a portrait phone capture. - let trak1 = bx(b"tkhd", &tkhd_v0(1, &matrix(1.0, 0.0, 0.0, 1.0))); - let trak2 = bx(b"tkhd", &tkhd_v0(2, &matrix(0.0, 1.0, -1.0, 0.0))); - let moov = [bx(b"trak", &trak1), bx(b"trak", &trak2)].concat(); - let mut f = file(&[bx(b"moov", &moov)]); - - let s = survey(&mut f).expect("survey"); - assert_eq!(s.tracks.len(), 2); - assert_eq!(s.rotation(1), Rotation::None); - assert_eq!(s.rotation(2), Rotation::Clockwise90); - // A track the file does not have reports no rotation, not a panic. - assert_eq!(s.rotation(99), Rotation::None); - } - - fn tkhd_v1(track_id: u32, m: &[u8]) -> Vec { - let mut p = vec![1u8, 0, 0, 0]; // version 1 + flags - p.extend(vec![0u8; 8]); // creation (64-bit) - p.extend(vec![0u8; 8]); // modification (64-bit) - p.extend_from_slice(&track_id.to_be_bytes()); // track id @ 20 - p.extend_from_slice(&0u32.to_be_bytes()); // reserved - p.extend(vec![0u8; 8]); // duration (64-bit) - p.extend(vec![0u8; 8]); // reserved - p.extend(vec![0u8; 4]); // layer, alternate group - p.extend(vec![0u8; 4]); // volume, reserved - assert_eq!(p.len(), 52, "matrix must land at offset 52 in a v1 tkhd"); - p.extend_from_slice(m); - p.extend(vec![0u8; 8]); // width, height - p - } - - #[test] - fn a_version_1_tkhd_uses_its_own_field_offsets() { - // Version 1 widens the times to 64 bits, moving both the track id and - // the matrix. Getting either offset wrong silently yields a wrong - // track id and no rotation, which is why this is tested separately. - let trak = bx(b"tkhd", &tkhd_v1(7, &matrix(0.0, -1.0, 1.0, 0.0))); - let mut f = file(&[bx(b"moov", &bx(b"trak", &trak))]); - - let s = survey(&mut f).expect("survey"); - assert_eq!(s.tracks.len(), 1); - assert_eq!(s.tracks[0].id, 7, "track id offset"); - assert_eq!(s.rotation(7), Rotation::Clockwise270, "matrix offset"); - } - - #[test] - fn a_64_bit_box_size_is_honoured() { - // size==1 means the real size follows as a 64-bit field. - let payload = mvhd_v0(90_000, 180_000); - let mut mvhd = 1u32.to_be_bytes().to_vec(); - mvhd.extend_from_slice(b"mvhd"); - mvhd.extend_from_slice(&((payload.len() + 16) as u64).to_be_bytes()); - mvhd.extend_from_slice(&payload); - let mut f = file(&[bx(b"moov", &mvhd)]); - - assert_eq!(survey(&mut f).expect("survey").timescale, Some(90_000)); - } - - #[test] - fn a_zero_size_box_terminates_rather_than_spinning() { - // size==0 means "to the end of the container"; a following box would - // be unreachable, and the walk must simply stop. - let mut data = 0u32.to_be_bytes().to_vec(); - data.extend_from_slice(b"free"); - data.extend(vec![0u8; 16]); - let mut f = Cursor::new(data); - - assert!(survey(&mut f).is_ok()); - } - - #[test] - fn a_box_claiming_less_than_its_header_does_not_loop_forever() { - // A size of 4 is smaller than the 8-byte header: malformed. The walk - // must stop instead of advancing by zero forever. - let mut data = 4u32.to_be_bytes().to_vec(); - data.extend_from_slice(b"junk"); - data.extend(vec![0u8; 32]); - let mut f = Cursor::new(data); - - let s = survey(&mut f).expect("must terminate"); - assert!(s.brands.is_empty()); - } - - #[test] - fn a_box_overrunning_the_file_is_clamped_not_trusted() { - let mut data = 0xffff_ff00u32.to_be_bytes().to_vec(); - data.extend_from_slice(b"moov"); - data.extend(vec![0u8; 16]); - let mut f = Cursor::new(data); - - assert!(survey(&mut f).is_ok()); - } - - #[test] - fn a_truncated_tkhd_yields_no_rotation_rather_than_an_error() { - // Header present, matrix cut off. - let short = vec![0u8; 20]; - let moov = bx(b"trak", &bx(b"tkhd", &short)); - let mut f = file(&[bx(b"moov", &moov)]); - - let s = survey(&mut f).expect("survey"); - assert_eq!(s.rotation(0), Rotation::None); - } - - #[test] - fn an_empty_file_surveys_to_nothing() { - let mut f = Cursor::new(Vec::new()); - assert_eq!(survey(&mut f).expect("survey"), BoxSurvey::default()); - } - - #[test] - fn deeply_nested_containers_stop_at_the_depth_limit() { - // Nest moov > trak > mdia > trak > ... far past MAX_DEPTH. The walk - // must return rather than recurse until the stack gives out. - let mut inner = bx(b"mvhd", &mvhd_v0(600, 100)); - for _ in 0..64 { - inner = bx(b"trak", &inner); - } - let mut f = file(&[bx(b"moov", &inner)]); - assert!(survey(&mut f).is_ok()); - } -} diff --git a/crates/rawshift-video-isobmff/src/demux.rs b/crates/rawshift-video-isobmff/src/demux.rs index f89af30..6ff3c86 100644 --- a/crates/rawshift-video-isobmff/src/demux.rs +++ b/crates/rawshift-video-isobmff/src/demux.rs @@ -1,90 +1,91 @@ //! The MP4 / QuickTime demuxer. -//! -//! Track, packet and seek translation is shared with the Matroska crate via -//! `rawshift-video-symphonia`; what is specific to ISOBMFF is the header walk -//! in [`crate::boxes`] and the movie-level metadata built from it. No -//! symphonia type escapes this module. use std::io::{Read, Seek, SeekFrom}; -use symphonia_core::formats::{FormatOptions, FormatReader}; -use symphonia_core::io::{MediaSourceStream, MediaSourceStreamOptions}; -use symphonia_format_isomp4::IsoMp4Reader; - -use rawshift_core::metadata::URational; -use rawshift_core::{MetadataNamespace, MetadataValue}; use rawshift_video_core::{ - CodecConfig, ContainerId, Demuxer, Packet, SeekMode, SeekTo, Track, TrackId, VideoCodecId, - VideoError, VideoMetadata, VideoResult, + ContainerId, Demuxer, Packet, SeekMode, SeekTo, Track, TrackId, VideoError, VideoMetadata, + VideoResult, nal, }; -use rawshift_video_symphonia as bridge; - -use crate::boxes::{self, BoxSurvey}; -/// The name that appears in [`VideoError::Container`] messages. -const CONTAINER_NAME: &str = "MP4"; +use crate::index::SampleIndex; +use crate::movie::{CONTAINER_NAME, Movie}; /// A demuxer for MP4, M4V and QuickTime files. -pub struct IsoBmffDemuxer { - reader: IsoMp4Reader<'static>, - container: ContainerId, +/// +/// Samples are read on demand from the source rather than held in memory, so +/// opening a file costs only its headers however large the media is. +pub struct IsoBmffDemuxer { + source: R, tracks: Vec, + indices: Vec, + /// Read cursor into each track's sample index, parallel to `tracks`. + cursors: Vec, + container: ContainerId, metadata: VideoMetadata, } -impl IsoBmffDemuxer { +impl IsoBmffDemuxer { /// Open an ISOBMFF file. /// - /// Performs a short header walk for the rotation and brand facts the - /// backend does not expose, then rewinds and hands the source to it. - /// /// # Errors /// - /// [`VideoError::Container`] for a malformed or unsupported file, and - /// [`VideoError::Io`] for a reader failure. - pub fn open(mut source: R) -> VideoResult - where - R: Read + Seek + Send + Sync + 'static, - { - // A structurally odd file should still demux: the survey supplies - // supplementary facts, so its failure degrades rather than propagates. - let survey = boxes::survey(&mut source).unwrap_or_default(); - source.seek(SeekFrom::Start(0))?; - - let stream = MediaSourceStream::new( - Box::new(bridge::SeekableSource::new(source)), - MediaSourceStreamOptions::default(), - ); - let reader = IsoMp4Reader::try_new(stream, FormatOptions::default()) - .map_err(|e| VideoError::container(CONTAINER_NAME, e))?; - - let container = survey.container(); - let tracks = reader - .tracks() - .iter() - .map(|t| bridge::map_track(t, survey.rotation(t.id))) - .collect(); - let metadata = build_metadata(&reader, &survey, container); + /// [`VideoError::Container`] for a file with no `moov`, a malformed atom, + /// or a fragmented file, and [`VideoError::Io`] for a reader failure. + pub fn open(mut source: R) -> VideoResult { + let movie = Movie::read(&mut source)?; + let cursors = vec![0; movie.tracks.len()]; Ok(Self { - reader, - container, - tracks, - metadata, + source, + tracks: movie.tracks, + indices: movie.indices, + cursors, + container: movie.container, + metadata: movie.metadata, }) } - /// The track's codec and config, for deriving a packet's keyframe flag. - fn video_codec_of(&self, track: TrackId) -> Option<(VideoCodecId, &CodecConfig)> { - self.tracks + /// The position of a track in the parallel `tracks`/`indices`/`cursors` + /// vectors. + /// + /// Container-assigned track ids are 1-based and need not be contiguous, so + /// this is a search rather than an index. + fn slot_of(&self, id: TrackId) -> Option { + self.tracks.iter().position(|t| t.id() == id) + } + + /// The track whose next unread sample decodes earliest. + /// + /// Packets are delivered in the file's own interleaving so that a caller + /// reading every track streams the file in one forward pass. Ordering by + /// decode timestamp reproduces that interleaving from the per-track + /// indices. + fn next_slot(&self) -> Option { + self.cursors .iter() - .find(|t| t.id() == track) - .and_then(Track::as_video) - .map(|v| (v.codec, &v.codec_config)) + .enumerate() + .filter_map(|(slot, &cursor)| { + self.indices[slot] + .samples() + .get(cursor) + .map(|sample| (slot, sample.dts, sample.offset)) + }) + // Ties break on file offset, which keeps the order stable and + // matches how the samples are actually laid out. + .min_by_key(|&(_, dts, offset)| (dts, offset)) + .map(|(slot, _, _)| slot) + } + + /// Read one sample's bytes. + fn read_sample(&mut self, offset: u64, size: u32) -> VideoResult> { + self.source.seek(SeekFrom::Start(offset))?; + let mut data = vec![0u8; size as usize]; + self.source.read_exact(&mut data)?; + Ok(data) } } -impl Demuxer for IsoBmffDemuxer { +impl Demuxer for IsoBmffDemuxer { fn container(&self) -> ContainerId { self.container } @@ -98,142 +99,74 @@ impl Demuxer for IsoBmffDemuxer { } fn next_packet(&mut self) -> VideoResult> { - // Peek the track before consuming, so the keyframe flag can be derived - // from the right track's codec configuration. - let result = self.reader.next_packet(); - let video = match &result { - Ok(Some(p)) => self.video_codec_of(TrackId::new(p.track_id)), - _ => None, + let Some(slot) = self.next_slot() else { + return Ok(None); }; - bridge::map_packet_result(CONTAINER_NAME, result, video) + let sample = self.indices[slot].samples()[self.cursors[slot]]; + self.cursors[slot] += 1; + + let data = self.read_sample(sample.offset, sample.size)?; + let track = self.tracks[slot].id(); + + // The container's sync table is advisory; for the codecs rawshift + // decodes, the bitstream is authoritative and disagrees often enough + // to matter. Fall back to the table for anything else. + let is_keyframe = match self.tracks[slot].as_video() { + Some(video) if !video.codec_config.is_empty() => { + nal::is_random_access_point(video.codec, &video.codec_config, &data) + } + _ => sample.is_sync, + }; + + let mut packet = Packet::new(track, data); + packet.pts = Some(sample.pts); + packet.dts = Some(sample.dts); + packet.duration = Some(u64::from(sample.duration)); + packet.is_keyframe = is_keyframe; + Ok(Some(packet)) } fn seek(&mut self, track: TrackId, to: SeekTo, mode: SeekMode) -> VideoResult { - if self.track(track).is_none() { + let Some(slot) = self.slot_of(track) else { return Err(VideoError::NoSuchTrack { id: track }); - } - let (sym_mode, target) = bridge::map_seek(CONTAINER_NAME, track, to, mode)?; - - self.reader - .seek(sym_mode, target) - .map(|landed| landed.actual_ts.get()) - .map_err(|e| VideoError::container(CONTAINER_NAME, e)) - } -} - -fn build_metadata( - reader: &IsoMp4Reader<'_>, - survey: &BoxSurvey, - container: ContainerId, -) -> VideoMetadata { - let mut md = VideoMetadata::default(); - - md.container.container = Some(container); - md.container.brands = survey.brands.clone(); - md.container.timescale = survey.timescale; - md.container.duration = survey.timescale.and_then(|scale| { - survey - .duration - .and_then(|ticks| bridge::ticks_to_duration(ticks, URational::new(1, scale))) - }); - md.container.creation_time = survey.creation_time.map(format_iso_time); - md.container.modification_time = survey.modification_time.map(format_iso_time); - - // Fall back to the media-level duration when the movie header had none. - if md.container.duration.is_none() { - let info = reader.media_info(); - if let (Some(tb), Some(dur)) = (info.time_base, info.duration) { - md.container.duration = - bridge::ticks_to_duration(dur.get(), bridge::map_time_base(Some(tb))); - } - } - - for brand in &survey.brands { - md.push_container_entry( - MetadataNamespace::Quicktime, - "ftyp.brand", - MetadataValue::Text(brand.clone()), - ); - } - if let Some(scale) = survey.timescale { - md.push_container_entry( - MetadataNamespace::Quicktime, - "mvhd.timescale", - MetadataValue::U64(u64::from(scale)), - ); - } - - md -} - -/// Render an ISOBMFF timestamp as an ISO 8601 date-time in UTC. -/// -/// ISOBMFF counts seconds from 1904-01-01, not the Unix epoch. Formatted here -/// rather than by pulling in a date library for one field. -fn format_iso_time(seconds_since_1904: u64) -> String { - /// Seconds between 1904-01-01 and 1970-01-01. - const EPOCH_OFFSET: u64 = 2_082_844_800; - - if seconds_since_1904 < EPOCH_OFFSET { - // Before the Unix epoch: report the raw value rather than a wrong date. - return format!("{seconds_since_1904} (seconds since 1904-01-01)"); - } - let unix = seconds_since_1904 - EPOCH_OFFSET; - - let (days, time) = (unix / 86_400, unix % 86_400); - let (hour, minute, second) = (time / 3_600, (time % 3_600) / 60, time % 60); - let (year, month, day) = civil_from_days(days as i64); - - format!("{year:04}-{month:02}-{day:02}T{hour:02}:{minute:02}:{second:02}Z") -} - -/// Convert days since 1970-01-01 to a proleptic Gregorian date. -/// -/// Howard Hinnant's `civil_from_days`: exact over the whole representable -/// range and table-free. -fn civil_from_days(days: i64) -> (i64, u32, u32) { - let z = days + 719_468; - let era = if z >= 0 { z } else { z - 146_096 } / 146_097; - let doe = (z - era * 146_097) as u64; - let yoe = (doe - doe / 1_460 + doe / 36_524 - doe / 146_096) / 365; - let y = yoe as i64 + era * 400; - let doy = doe - (365 * yoe + yoe / 4 - yoe / 100); - let mp = (5 * doy + 2) / 153; - let d = (doy - (153 * mp + 2) / 5 + 1) as u32; - let m = if mp < 10 { mp + 3 } else { mp - 9 } as u32; - (if m <= 2 { y + 1 } else { y }, m, d) -} + }; -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn the_isobmff_epoch_is_offset_from_unix_not_equal_to_it() { - // 1904-01-01T00:00:00Z is timestamp 0 in ISOBMFF. - assert_eq!(format_iso_time(2_082_844_800), "1970-01-01T00:00:00Z"); - // A real capture time: 2024-01-15T12:30:45Z. - assert_eq!( - format_iso_time(2_082_844_800 + 1_705_321_845), - "2024-01-15T12:30:45Z" - ); - } + let time_base = self.tracks[slot].time_base(); + let timestamp = match to { + SeekTo::Timestamp(ts) => ts, + SeekTo::Time(time) => { + if time_base.numerator == 0 { + return Err(VideoError::container( + CONTAINER_NAME, + "track has a degenerate time base; seek by timestamp instead", + )); + } + // ticks = seconds * (denominator / numerator) + (time.as_secs_f64() * f64::from(time_base.denominator) + / f64::from(time_base.numerator)) as i64 + } + other => { + return Err(VideoError::container( + CONTAINER_NAME, + format!("unsupported seek target: {other:?}"), + )); + } + }; - #[test] - fn a_timestamp_before_the_unix_epoch_reports_raw_rather_than_a_wrong_date() { - let s = format_iso_time(0); - assert!(s.contains("1904"), "{s}"); - assert!(!s.starts_with("1970"), "{s}"); - } + // Both seek modes land on a random access point: a decoder cannot + // resume anywhere else. Coarse differs only in that the caller has said + // it will not decode forward to reach the exact frame, which changes + // nothing about where the demuxer can put the cursor. + let _ = mode; + let index = &self.indices[slot]; + let Some(landing) = index.sync_sample_at_or_before(timestamp) else { + return Err(VideoError::container( + CONTAINER_NAME, + "track has no samples to seek within", + )); + }; - #[test] - fn civil_from_days_handles_leap_years_and_epoch_edges() { - assert_eq!(civil_from_days(0), (1970, 1, 1)); - assert_eq!(civil_from_days(59), (1970, 3, 1)); - // 2000 was a leap year (divisible by 400); 1900 was not. - assert_eq!(civil_from_days(11_016), (2000, 2, 29)); - // 2024-02-29, a leap day in an ordinary leap year. - assert_eq!(civil_from_days(19_782), (2024, 2, 29)); - assert_eq!(civil_from_days(19_783), (2024, 3, 1)); + self.cursors[slot] = landing; + Ok(index.samples()[landing].pts) } } diff --git a/crates/rawshift-video-isobmff/src/index.rs b/crates/rawshift-video-isobmff/src/index.rs new file mode 100644 index 0000000..95ce818 --- /dev/null +++ b/crates/rawshift-video-isobmff/src/index.rs @@ -0,0 +1,755 @@ +//! The sample index: turning a track's sample tables into a flat, ordered list +//! of samples with byte ranges and timestamps. +//! +//! ISOBMFF does not store a sample table. It stores five or six *compressed* +//! tables — sizes, chunk offsets, samples-per-chunk runs, decode-time runs, +//! composition offsets, and sync sample numbers — that have to be walked +//! together to recover where each sample lives and when it is shown. That +//! reconstruction is what this module does, and it is the part `mp4-atom` +//! deliberately leaves to its caller. +//! +//! Every table is attacker-controlled, so the walk is total: an inconsistent +//! table truncates the index rather than panicking or producing a byte range +//! that points outside the file. + +use mp4_atom::{Edts, Stbl, StszSamples}; + +/// One sample: where it is, when it decodes, and when it is shown. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct Sample { + /// Byte offset from the start of the file. + pub offset: u64, + /// Size in bytes. + pub size: u32, + /// Decode timestamp, in the track's timescale. + pub dts: i64, + /// Presentation timestamp, in the track's timescale. + /// + /// Equal to `dts` unless a `ctts` composition offset applies, which is how + /// reordered streams say a picture is decoded before it is shown. + pub pts: i64, + /// How long the sample occupies, in the track's timescale. + pub duration: u32, + /// Whether the container marks this as a sync sample. + /// + /// Advisory only: rawshift derives random access from the bitstream, which + /// is authoritative. Kept because it is what seeking indexes against. + pub is_sync: bool, +} + +/// A track's samples, in decode order. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct SampleIndex { + samples: Vec, +} + +impl SampleIndex { + /// Reconstruct the index from a track's sample tables, applying . + /// + /// Never fails: a malformed or inconsistent table stops the walk and + /// yields the samples recovered so far, because a partly-readable track is + /// more useful than none and the caller cannot act on a parse error here + /// anyway. + #[must_use] + pub fn build(stbl: &Stbl, edts: Option<&Edts>) -> Self { + let shift = presentation_shift(edts); + let sizes = sample_sizes(&stbl.stsz.samples); + let sample_count = sizes.len(); + if sample_count == 0 { + return Self::default(); + } + + let chunk_offsets: Vec = match (&stbl.stco, &stbl.co64) { + // co64 wins when both are present: a file large enough to need + // 64-bit offsets has a stco that cannot address it. + (_, Some(co64)) => co64.entries.clone(), + (Some(stco), None) => stco.entries.iter().map(|&o| u64::from(o)).collect(), + (None, None) => return Self::default(), + }; + + let offsets = sample_offsets(&stbl.stsc.entries, &chunk_offsets, &sizes); + let times = sample_times(&stbl.stts.entries, sample_count); + let composition = composition_offsets(stbl.ctts.as_ref(), sample_count); + + // stss lists the sync samples by 1-based number. Its *absence* means + // every sample is a sync sample — the convention for all-intra tracks — + // which is the opposite of treating an absent table as "none". + let sync: Option> = stbl + .stss + .as_ref() + .map(|stss| stss.entries.iter().copied().collect()); + + let usable = offsets.len().min(times.len()).min(sample_count); + let samples = (0..usable) + .map(|i| { + let (dts, duration) = times[i]; + let composition_shift = composition.get(i).copied().unwrap_or(0); + Sample { + offset: offsets[i], + size: sizes[i], + dts: dts.saturating_sub(shift), + pts: dts.saturating_add(composition_shift).saturating_sub(shift), + duration, + is_sync: sync + .as_ref() + .is_none_or(|set| set.contains(&(i as u32 + 1))), + } + }) + .collect(); + + Self { samples } + } + + /// The samples, in decode order. + #[must_use] + pub fn samples(&self) -> &[Sample] { + &self.samples + } + + /// How many samples the track has. + #[must_use] + pub fn len(&self) -> usize { + self.samples.len() + } + + /// Whether the track has no samples. + #[must_use] + pub fn is_empty(&self) -> bool { + self.samples.is_empty() + } + + /// The index of the last sync sample at or before `timestamp` in + /// presentation order, or the first sample when none precedes it. + /// + /// This is the seek landing point: decoding can only resume at a random + /// access point, so a request between two of them lands on the earlier + /// one and the caller decodes forward. + #[must_use] + pub fn sync_sample_at_or_before(&self, timestamp: i64) -> Option { + if self.samples.is_empty() { + return None; + } + let landing = self + .samples + .iter() + .enumerate() + .rfind(|(_, s)| s.is_sync && s.pts <= timestamp) + .map(|(i, _)| i); + + // Before the first sync sample there is nothing to resume from, so the + // start of the track is the only honest answer. + Some(landing.unwrap_or(0)) + } +} + +/// How far to shift the timeline so presentation starts at zero. +/// +/// An `elst` maps presentation time onto media time. Encoders that reorder +/// write a single edit whose `media_time` equals the composition delay, which +/// says "presentation begins this far into the media" — the first frame is +/// shown at time zero even though its composition timestamp is not. Ignoring +/// it leaves every timestamp offset by the reorder delay, disagreeing with +/// every other tool that reads the file. +/// +/// Only the single-entry case is honoured, which is what encoders write. A +/// multi-entry edit list is a genuine cut list that would have to trim and +/// reorder samples; that is out of scope, and ignoring it yields untrimmed +/// media rather than wrong media. +fn presentation_shift(edts: Option<&Edts>) -> i64 { + let Some(elst) = edts.and_then(|e| e.elst.as_ref()) else { + return 0; + }; + let [entry] = elst.entries.as_slice() else { + return 0; + }; + // An empty edit (media_time == -1, spelled None here) is a dwell with no + // media and shifts nothing. + let Some(media_time) = entry.media_time else { + return 0; + }; + i64::try_from(media_time).unwrap_or(0) +} + +/// Expand `stsz` into a per-sample size list. +fn sample_sizes(samples: &StszSamples) -> Vec { + match samples { + StszSamples::Identical { count, size } => vec![*size; *count as usize], + StszSamples::Different { sizes } => sizes.clone(), + } +} + +/// Walk `stsc` and the chunk offsets to place every sample in the file. +/// +/// `stsc` stores runs: an entry `(first_chunk, samples_per_chunk, ..)` applies +/// from `first_chunk` until the next entry's `first_chunk`. Within a chunk, +/// samples are laid out back to back from the chunk's offset. +fn sample_offsets(stsc: &[mp4_atom::StscEntry], chunk_offsets: &[u64], sizes: &[u32]) -> Vec { + let mut offsets = Vec::with_capacity(sizes.len()); + let mut sample = 0usize; + + for (index, entry) in stsc.iter().enumerate() { + // Chunk numbers are 1-based; a zero would underflow the range below. + if entry.first_chunk == 0 { + break; + } + let first = entry.first_chunk as usize - 1; + let last = stsc + .get(index + 1) + .map_or(chunk_offsets.len(), |next| { + (next.first_chunk as usize).saturating_sub(1) + }) + .min(chunk_offsets.len()); + + for chunk in first..last { + let Some(&base) = chunk_offsets.get(chunk) else { + return offsets; + }; + let mut cursor = base; + for _ in 0..entry.samples_per_chunk { + let Some(&size) = sizes.get(sample) else { + return offsets; + }; + offsets.push(cursor); + // A crafted table can push the cursor past the file; the + // demuxer bounds-checks the read, so saturating here is enough + // to keep the walk total. + cursor = cursor.saturating_add(u64::from(size)); + sample += 1; + } + } + } + + offsets +} + +/// Expand the `stts` runs into per-sample `(dts, duration)`. +fn sample_times(stts: &[mp4_atom::SttsEntry], sample_count: usize) -> Vec<(i64, u32)> { + let mut times = Vec::with_capacity(sample_count); + let mut dts: i64 = 0; + + for entry in stts { + for _ in 0..entry.sample_count { + if times.len() >= sample_count { + return times; + } + times.push((dts, entry.sample_delta)); + dts = dts.saturating_add(i64::from(entry.sample_delta)); + } + } + + // A track whose stts covers fewer samples than stsz claims is malformed. + // Extend with zero-duration entries at the last timestamp rather than + // dropping the samples: their bytes are still readable. + while times.len() < sample_count { + times.push((dts, 0)); + } + times +} + +/// Expand the `ctts` runs into per-sample composition offsets. +fn composition_offsets(ctts: Option<&mp4_atom::Ctts>, sample_count: usize) -> Vec { + let Some(ctts) = ctts else { + return vec![0; sample_count]; + }; + + let mut offsets = Vec::with_capacity(sample_count); + for entry in &ctts.entries { + for _ in 0..entry.sample_count { + if offsets.len() >= sample_count { + return offsets; + } + offsets.push(entry.sample_offset); + } + } + offsets.resize(sample_count, 0); + offsets +} + +#[cfg(test)] +mod tests { + use super::*; + use mp4_atom::{ + Co64, Ctts, CttsEntry, Elst, ElstEntry, Stco, Stsc, StscEntry, Stsd, Stss, Stsz, Stts, + SttsEntry, + }; + + /// A minimal `stbl` from the parts a test cares about. + fn stbl( + sizes: StszSamples, + stsc: Vec, + chunks: Vec, + stts: Vec, + ctts: Option>, + stss: Option>, + ) -> Stbl { + Stbl { + stsd: Stsd { codecs: vec![] }, + stts: Stts { entries: stts }, + ctts: ctts.map(|entries| Ctts { entries }), + stss: stss.map(|entries| Stss { entries }), + stsc: Stsc { entries: stsc }, + stsz: Stsz { samples: sizes }, + stco: Some(Stco { entries: chunks }), + co64: None, + sbgp: vec![], + sgpd: vec![], + subs: vec![], + saiz: vec![], + saio: vec![], + cslg: None, + } + } + + fn run(count: u32, delta: u32) -> SttsEntry { + SttsEntry { + sample_count: count, + sample_delta: delta, + } + } + + #[test] + fn samples_are_laid_out_back_to_back_within_a_chunk() { + // Two chunks of two samples each, at offsets 100 and 500. + let index = SampleIndex::build( + &stbl( + StszSamples::Different { + sizes: vec![10, 20, 30, 40], + }, + vec![StscEntry { + first_chunk: 1, + samples_per_chunk: 2, + sample_description_index: 1, + }], + vec![100, 500], + vec![run(4, 512)], + None, + None, + ), + None, + ); + + let offsets: Vec<_> = index.samples().iter().map(|s| s.offset).collect(); + assert_eq!(offsets, vec![100, 110, 500, 530]); + let sizes: Vec<_> = index.samples().iter().map(|s| s.size).collect(); + assert_eq!(sizes, vec![10, 20, 30, 40]); + } + + #[test] + fn a_second_stsc_run_changes_the_samples_per_chunk() { + // Chunk 1 holds one sample; chunks 2 and 3 hold two each. + let index = SampleIndex::build( + &stbl( + StszSamples::Identical { count: 5, size: 10 }, + vec![ + StscEntry { + first_chunk: 1, + samples_per_chunk: 1, + sample_description_index: 1, + }, + StscEntry { + first_chunk: 2, + samples_per_chunk: 2, + sample_description_index: 1, + }, + ], + vec![1000, 2000, 3000], + vec![run(5, 100)], + None, + None, + ), + None, + ); + + let offsets: Vec<_> = index.samples().iter().map(|s| s.offset).collect(); + assert_eq!(offsets, vec![1000, 2000, 2010, 3000, 3010]); + } + + #[test] + fn decode_timestamps_accumulate_across_stts_runs() { + let index = SampleIndex::build( + &stbl( + StszSamples::Identical { count: 5, size: 1 }, + vec![StscEntry { + first_chunk: 1, + samples_per_chunk: 5, + sample_description_index: 1, + }], + vec![0], + // Three samples of 100, then two of 50. + vec![run(3, 100), run(2, 50)], + None, + None, + ), + None, + ); + + let dts: Vec<_> = index.samples().iter().map(|s| s.dts).collect(); + assert_eq!(dts, vec![0, 100, 200, 300, 350]); + let durations: Vec<_> = index.samples().iter().map(|s| s.duration).collect(); + assert_eq!(durations, vec![100, 100, 100, 50, 50]); + } + + #[test] + fn composition_offsets_separate_presentation_from_decode_order() { + // The classic reordered pattern: an I frame, a P frame shown last, and + // two B frames shown in between. + let index = SampleIndex::build( + &stbl( + StszSamples::Identical { count: 4, size: 1 }, + vec![StscEntry { + first_chunk: 1, + samples_per_chunk: 4, + sample_description_index: 1, + }], + vec![0], + vec![run(4, 100)], + Some(vec![ + CttsEntry { + sample_count: 1, + sample_offset: 0, + }, + CttsEntry { + sample_count: 1, + sample_offset: 300, + }, + CttsEntry { + sample_count: 2, + sample_offset: 0, + }, + ]), + None, + ), + None, + ); + + let dts: Vec<_> = index.samples().iter().map(|s| s.dts).collect(); + let pts: Vec<_> = index.samples().iter().map(|s| s.pts).collect(); + assert_eq!(dts, vec![0, 100, 200, 300]); + // The second sample decodes at 100 but is shown at 400, after the two + // B frames at 200 and 300. + assert_eq!(pts, vec![0, 400, 200, 300]); + } + + #[test] + fn a_negative_composition_offset_is_honoured() { + // Version 1 ctts allows negative offsets, which move presentation + // earlier than decode. + let index = SampleIndex::build( + &stbl( + StszSamples::Identical { count: 2, size: 1 }, + vec![StscEntry { + first_chunk: 1, + samples_per_chunk: 2, + sample_description_index: 1, + }], + vec![0], + vec![run(2, 100)], + Some(vec![CttsEntry { + sample_count: 2, + sample_offset: -100, + }]), + None, + ), + None, + ); + + let pts: Vec<_> = index.samples().iter().map(|s| s.pts).collect(); + assert_eq!(pts, vec![-100, 0]); + } + + #[test] + fn an_absent_stss_means_every_sample_is_a_sync_sample() { + // The all-intra convention. Reading an absent table as "no sync + // samples" would make such a track unseekable. + let index = SampleIndex::build( + &stbl( + StszSamples::Identical { count: 3, size: 1 }, + vec![StscEntry { + first_chunk: 1, + samples_per_chunk: 3, + sample_description_index: 1, + }], + vec![0], + vec![run(3, 100)], + None, + None, + ), + None, + ); + + assert!(index.samples().iter().all(|s| s.is_sync)); + } + + #[test] + fn stss_marks_sync_samples_by_one_based_number() { + let index = SampleIndex::build( + &stbl( + StszSamples::Identical { count: 5, size: 1 }, + vec![StscEntry { + first_chunk: 1, + samples_per_chunk: 5, + sample_description_index: 1, + }], + vec![0], + vec![run(5, 100)], + None, + // Samples 1 and 4, one-based: indices 0 and 3. + Some(vec![1, 4]), + ), + None, + ); + + let sync: Vec<_> = index.samples().iter().map(|s| s.is_sync).collect(); + assert_eq!(sync, vec![true, false, false, true, false]); + } + + #[test] + fn co64_wins_over_stco_when_both_are_present() { + // A file large enough to need 64-bit offsets has an stco that cannot + // address it, so preferring stco would place samples wrongly. + let mut tables = stbl( + StszSamples::Identical { count: 1, size: 4 }, + vec![StscEntry { + first_chunk: 1, + samples_per_chunk: 1, + sample_description_index: 1, + }], + vec![42], + vec![run(1, 100)], + None, + None, + ); + tables.co64 = Some(Co64 { + entries: vec![0x1_0000_0000], + }); + + let index = SampleIndex::build(&tables, None); + assert_eq!(index.samples()[0].offset, 0x1_0000_0000); + } + + #[test] + fn seeking_lands_on_the_sync_sample_at_or_before_the_request() { + let index = SampleIndex::build( + &stbl( + StszSamples::Identical { count: 6, size: 1 }, + vec![StscEntry { + first_chunk: 1, + samples_per_chunk: 6, + sample_description_index: 1, + }], + vec![0], + vec![run(6, 100)], + None, + Some(vec![1, 4]), // sync at pts 0 and 300 + ), + None, + ); + + // Exactly on a sync sample. + assert_eq!(index.sync_sample_at_or_before(300), Some(3)); + // Between two: must land on the earlier one, never the later. + assert_eq!(index.sync_sample_at_or_before(250), Some(0)); + assert_eq!(index.sync_sample_at_or_before(450), Some(3)); + // Past the end: the last sync sample. + assert_eq!(index.sync_sample_at_or_before(99_999), Some(3)); + // Before the first: the start of the track is the only resumable point. + assert_eq!(index.sync_sample_at_or_before(-100), Some(0)); + } + + fn edts(media_time: Option) -> Edts { + Edts { + elst: Some(Elst { + entries: vec![ElstEntry { + segment_duration: 1_000, + media_time, + media_rate: 1.into(), + }], + }), + } + } + + fn reordered_tables() -> Stbl { + // Two frames with a composition delay of 100, as a reordering encoder + // writes: composition timestamps start at 100, not 0. + stbl( + StszSamples::Identical { count: 2, size: 1 }, + vec![StscEntry { + first_chunk: 1, + samples_per_chunk: 2, + sample_description_index: 1, + }], + vec![0], + vec![run(2, 100)], + Some(vec![CttsEntry { + sample_count: 2, + sample_offset: 100, + }]), + None, + ) + } + + #[test] + fn an_edit_list_shifts_presentation_to_start_at_zero() { + // Without the edit list every timestamp is offset by the reorder + // delay, and rawshift would disagree with every other tool reading the + // same file. + let unshifted = SampleIndex::build(&reordered_tables(), None); + assert_eq!(unshifted.samples()[0].pts, 100); + + let shifted = SampleIndex::build(&reordered_tables(), Some(&edts(Some(100)))); + assert_eq!(shifted.samples()[0].pts, 0, "presentation must start at 0"); + // Decode time goes negative, which is correct: the frame is decoded + // before the presentation timeline begins. + assert_eq!(shifted.samples()[0].dts, -100); + } + + #[test] + fn an_empty_edit_shifts_nothing() { + // media_time == -1 is a dwell with no media, not an offset. + let index = SampleIndex::build(&reordered_tables(), Some(&edts(None))); + assert_eq!(index.samples()[0].pts, 100); + } + + #[test] + fn a_multi_entry_edit_list_is_ignored_rather_than_half_applied() { + // A real cut list would have to trim and reorder samples. Ignoring it + // yields untrimmed media, which is recoverable; applying only its + // first entry would yield wrong media, which is not. + let cut_list = Edts { + elst: Some(Elst { + entries: vec![ + ElstEntry { + segment_duration: 500, + media_time: Some(100), + media_rate: 1.into(), + }, + ElstEntry { + segment_duration: 500, + media_time: Some(900), + media_rate: 1.into(), + }, + ], + }), + }; + let index = SampleIndex::build(&reordered_tables(), Some(&cut_list)); + assert_eq!(index.samples()[0].pts, 100, "left untrimmed, not shifted"); + } + + #[test] + fn seeking_an_empty_index_yields_nothing() { + assert_eq!(SampleIndex::default().sync_sample_at_or_before(0), None); + assert!(SampleIndex::default().is_empty()); + } + + #[test] + fn a_chunk_offset_table_shorter_than_stsc_claims_truncates_cleanly() { + // stsc describes three chunks; stco lists one. The walk must stop at + // what it can place rather than indexing past the table. + let index = SampleIndex::build( + &stbl( + StszSamples::Identical { count: 6, size: 10 }, + vec![StscEntry { + first_chunk: 1, + samples_per_chunk: 2, + sample_description_index: 1, + }], + vec![100], + vec![run(6, 100)], + None, + None, + ), + None, + ); + + assert_eq!(index.len(), 2, "only the samples in the known chunk"); + } + + #[test] + fn a_zero_first_chunk_does_not_underflow() { + // first_chunk is 1-based; a zero is malformed and must not wrap. + let index = SampleIndex::build( + &stbl( + StszSamples::Identical { count: 2, size: 10 }, + vec![StscEntry { + first_chunk: 0, + samples_per_chunk: 2, + sample_description_index: 1, + }], + vec![100], + vec![run(2, 100)], + None, + None, + ), + None, + ); + assert!(index.is_empty()); + } + + #[test] + fn a_track_with_no_chunk_offsets_yields_no_samples() { + let mut tables = stbl( + StszSamples::Identical { count: 4, size: 10 }, + vec![StscEntry { + first_chunk: 1, + samples_per_chunk: 2, + sample_description_index: 1, + }], + vec![], + vec![run(4, 100)], + None, + None, + ); + tables.stco = None; + assert!(SampleIndex::build(&tables, None).is_empty()); + } + + #[test] + fn an_stts_shorter_than_stsz_still_yields_every_sample() { + // Malformed, but the sample bytes are readable, so they are kept with + // the last known timestamp rather than dropped. + let index = SampleIndex::build( + &stbl( + StszSamples::Identical { count: 4, size: 10 }, + vec![StscEntry { + first_chunk: 1, + samples_per_chunk: 4, + sample_description_index: 1, + }], + vec![0], + vec![run(2, 100)], + None, + None, + ), + None, + ); + + assert_eq!(index.len(), 4); + assert_eq!(index.samples()[3].dts, 200); + assert_eq!(index.samples()[3].duration, 0); + } + + #[test] + fn an_identical_size_table_expands_to_every_sample() { + let index = SampleIndex::build( + &stbl( + StszSamples::Identical { + count: 3, + size: 1234, + }, + vec![StscEntry { + first_chunk: 1, + samples_per_chunk: 3, + sample_description_index: 1, + }], + vec![0], + vec![run(3, 100)], + None, + None, + ), + None, + ); + + assert_eq!(index.len(), 3); + assert!(index.samples().iter().all(|s| s.size == 1234)); + } +} diff --git a/crates/rawshift-video-isobmff/src/lib.rs b/crates/rawshift-video-isobmff/src/lib.rs index 230f5b6..ad12118 100644 --- a/crates/rawshift-video-isobmff/src/lib.rs +++ b/crates/rawshift-video-isobmff/src/lib.rs @@ -1,38 +1,40 @@ //! MP4, M4V and QuickTime (MOV) support for rawshift. //! -//! Demuxing is backed by `symphonia-format-isomp4` — MPL-2.0, pure Rust, no -//! build script, so it compiles on every target in `docs/SUPPORT.md`. gamut's -//! charter excludes video, so this is a direct dependency rather than an -//! upstream ask; see "Video is outside gamut's charter" in `AGENTS.md`. +//! The box layer is [`mp4_atom`] — MIT, pure Rust, no build script, so it +//! compiles on every target in `docs/SUPPORT.md`. It performs no +//! interpretation, so rawshift owns the parts that give a container meaning: +//! the [sample index](index) reconstructed from the `stbl` tables, the track +//! model, and the demux and seek semantics. //! -//! Three things the backend does not surface, which this crate supplies: +//! gamut's charter excludes video, so this is a direct dependency rather than +//! an upstream ask; see "Video is outside gamut's charter" in `AGENTS.md`. //! -//! - **Rotation**, from the `tkhd` display matrix. Phones record in the -//! sensor's orientation and correct in the container, so without this most -//! phone video presents sideways. -//! - **`ftyp` brands and `mvhd` timescale and times**, for metadata and for -//! telling MP4 from QuickTime. -//! - **Random access points.** The backend exposes no sync-sample flag at all, -//! so these are read from the bitstream by `rawshift_video_core::nal`, which -//! is the authoritative source regardless. +//! # Why not an off-the-shelf demuxer //! -//! No symphonia type appears in this crate's public API. +//! This crate first used `symphonia-format-isomp4`, which is audio-first and +//! caps the `hvcC` configuration record at 1 KB. Real HEVC records are 2–3 KB, +//! so **every HEVC MP4 and MOV failed to open** — that is XAVC HS and default +//! iPhone video, the two highest-priority formats on rawshift's roadmap. It +//! also hides the `tkhd` matrix (and so rotation), the `colr` box (and so all +//! colour signalling), the sync-sample table, and ProRes-shaped sample +//! entries. Owning the sample index on a box-level library costs a few hundred +//! lines and removes all five limitations at once. //! //! # Not covered //! -//! ISOBMFF carries colour signalling in a `colr` box the backend does not -//! expose, so tracks report [`CicpColor::UNSPECIFIED`] and colour resolves by -//! picture height. ProRes tracks are invisible: the backend does not recognise -//! the `ap4h`/`apch`/`apcn`/`apcs`/`apco` sample entries. -//! -//! [`CicpColor::UNSPECIFIED`]: rawshift_video_core::CicpColor::UNSPECIFIED +//! Fragmented MP4 (`mvex`/`moof`) is rejected with a clear error rather than +//! presented as a track with no samples. Camera-origin files — rawshift's +//! subject — are not fragmented; streaming-shaped input is a follow-up. #![forbid(unsafe_code)] #![warn(missing_docs)] -mod boxes; +pub mod index; + mod demux; +mod movie; mod sniff; pub use demux::IsoBmffDemuxer; +pub use index::{Sample, SampleIndex}; pub use sniff::{IsoBmff, detect}; diff --git a/crates/rawshift-video-isobmff/src/movie.rs b/crates/rawshift-video-isobmff/src/movie.rs new file mode 100644 index 0000000..2183c8e --- /dev/null +++ b/crates/rawshift-video-isobmff/src/movie.rs @@ -0,0 +1,500 @@ +//! Reading the movie header, and mapping its tracks onto rawshift's model. + +use std::io::{Read, Seek, SeekFrom}; +use std::time::Duration; + +use mp4_atom::{Atom, Codec, Colr, Encode, Ftyp, Header, Moov, ReadAtom, ReadFrom, Trak}; + +use rawshift_core::metadata::URational; +use rawshift_core::{Dimensions, MetadataNamespace, MetadataValue}; +use rawshift_video_core::{ + AudioCodecId, AudioTrack, CicpColor, CodecConfig, ContainerId, OtherTrack, Rotation, Track, + TrackId, TrackKind, VideoCodecId, VideoError, VideoMetadata, VideoResult, VideoTrack, +}; + +use crate::index::SampleIndex; + +/// The name that appears in [`VideoError::Container`] messages. +pub const CONTAINER_NAME: &str = "MP4"; + +/// The movie header, and everything derived from it. +pub struct Movie { + /// One entry per track, in the file's own order. + pub tracks: Vec, + /// Sample indices, parallel to `tracks`. + pub indices: Vec, + /// The container flavour. + pub container: ContainerId, + /// File-level metadata. + pub metadata: VideoMetadata, +} + +impl Movie { + /// Read `ftyp` and `moov`, and build the tracks and their sample indices. + /// + /// Walks the top-level atoms by header so that `mdat` — which is the whole + /// media payload and routinely gigabytes — is skipped rather than read + /// into memory. + /// + /// # Errors + /// + /// [`VideoError::Container`] when no `moov` is present or an atom is + /// malformed, and [`VideoError::Io`] for a reader failure. + pub fn read(reader: &mut R) -> VideoResult { + let end = reader.seek(SeekFrom::End(0))?; + reader.seek(SeekFrom::Start(0))?; + + let mut ftyp: Option = None; + let mut moov: Option = None; + + while reader.stream_position()? < end { + let header = match Header::read_from(reader) { + Ok(header) => header, + // A trailing partial atom is where damaged files end; stop + // rather than failing a file whose moov we may already have. + Err(_) => break, + }; + + match header.kind { + Ftyp::KIND => { + ftyp = Ftyp::read_atom(&header, reader).ok(); + } + Moov::KIND => { + moov = Some( + Moov::read_atom(&header, reader) + .map_err(|e| VideoError::container(CONTAINER_NAME, e))?, + ); + } + _ => { + // Skip the body. An atom with no stated size runs to the + // end of the file, so there is nothing after it. + match header.size { + Some(size) => { + reader.seek(SeekFrom::Current(size as i64))?; + } + None => break, + } + } + } + + if ftyp.is_some() && moov.is_some() { + break; + } + } + + let Some(moov) = moov else { + return Err(VideoError::container( + CONTAINER_NAME, + "no moov atom: not a playable ISOBMFF file", + )); + }; + + // Fragmented files carry their sample tables in moof boxes rather than + // in stbl, so the index built here would be empty. Say so plainly + // instead of presenting a track with no samples. + if moov.mvex.is_some() { + return Err(VideoError::container( + CONTAINER_NAME, + "fragmented MP4 (mvex present) is not supported yet", + )); + } + + let container = container_of(ftyp.as_ref()); + let mut tracks = Vec::with_capacity(moov.trak.len()); + let mut indices = Vec::with_capacity(moov.trak.len()); + + for trak in &moov.trak { + tracks.push(map_track(trak)); + indices.push(SampleIndex::build(&trak.mdia.minf.stbl, trak.edts.as_ref())); + } + + let metadata = build_metadata(&moov, ftyp.as_ref(), container); + + Ok(Self { + tracks, + indices, + container, + metadata, + }) + } +} + +/// The container flavour implied by the `ftyp` brands. +/// +/// The QuickTime brand `qt ` means [`ContainerId::Mov`]; anything else with a +/// `ftyp` is MP4. A file with no `ftyp` is old-style QuickTime. +fn container_of(ftyp: Option<&Ftyp>) -> ContainerId { + let Some(ftyp) = ftyp else { + return ContainerId::Mov; + }; + let quicktime = ftyp.major_brand.as_ref() == b"qt " + || ftyp.compatible_brands.iter().any(|b| b.as_ref() == b"qt "); + if quicktime { + ContainerId::Mov + } else { + ContainerId::Mp4 + } +} + +fn map_track(trak: &Trak) -> Track { + let id = TrackId::new(trak.tkhd.track_id); + let mdhd = &trak.mdia.mdhd; + // The media timescale is the unit of this track's timestamps. + let time_base = URational::new(1, mdhd.timescale.max(1)); + let duration = ticks_to_duration(mdhd.duration, time_base); + let language = (!mdhd.language.is_empty()).then(|| mdhd.language.clone()); + + let stbl = &trak.mdia.minf.stbl; + let Some(codec) = stbl.stsd.codecs.first() else { + let mut other = OtherTrack::new(id, TrackKind::Data, time_base); + other.duration = duration; + other.language = language; + return Track::Other(other); + }; + + match codec { + Codec::Avc1(_) | Codec::Hvc1(_) | Codec::Hev1(_) | Codec::Av01(_) | Codec::Vp09(_) => { + let mut video = map_video_track(id, codec, time_base); + video.duration = duration; + video.frame_count = Some(stbl_sample_count(stbl)); + video.rotation = rotation_of(trak); + Track::Video(video) + } + Codec::Mp4a(_) | Codec::Opus(_) | Codec::Flac(_) | Codec::Ac3(_) | Codec::Eac3(_) => { + let mut audio = AudioTrack::new(id, audio_codec_of(codec), time_base); + audio.duration = duration; + audio.language = language; + Track::Audio(audio) + } + _ => { + let mut other = OtherTrack::new(id, TrackKind::Data, time_base); + other.duration = duration; + other.language = language; + Track::Other(other) + } + } +} + +fn map_video_track(id: TrackId, codec: &Codec, time_base: URational) -> VideoTrack { + let (video_codec, visual, colr, config) = match codec { + Codec::Avc1(a) => ( + VideoCodecId::H264, + &a.visual, + a.colr.as_ref(), + encode_body(&a.avcc), + ), + Codec::Hvc1(h) => ( + VideoCodecId::Hevc, + &h.visual, + h.colr.as_ref(), + encode_body(&h.hvcc), + ), + Codec::Hev1(h) => ( + VideoCodecId::Hevc, + &h.visual, + h.colr.as_ref(), + encode_body(&h.hvcc), + ), + Codec::Av01(a) => ( + VideoCodecId::Av1, + &a.visual, + a.colr.as_ref(), + encode_body(&a.av1c), + ), + Codec::Vp09(v) => (VideoCodecId::Vp9, &v.visual, None, Vec::new()), + // map_track only routes video sample entries here. + _ => unreachable!("map_video_track called with a non-video sample entry"), + }; + + let mut track = VideoTrack::new( + id, + video_codec, + Dimensions { + width: u32::from(visual.width), + height: u32::from(visual.height), + }, + time_base, + ); + track.color = colour_of(colr); + track.codec_config = CodecConfig::new(config); + track +} + +/// Re-encode a configuration atom back to its record body. +/// +/// The box layer parses `avcC`/`hvcC` into fields, but a decoder wants the +/// record exactly as the file stored it — that is what a hardware decoder's +/// configuration input is defined in terms of. Encoding the body back is the +/// faithful round trip, and it keeps rawshift from having to re-serialise the +/// record itself. +fn encode_body(atom: &A) -> Vec { + let mut buf = Vec::new(); + // Encoding into a Vec cannot fail for these small, fully-owned atoms. + if atom.encode(&mut buf).is_err() { + return Vec::new(); + } + // `encode` writes the full box; the record body is what follows the + // 8-byte size and type header. + buf.split_off(8.min(buf.len())) +} + +/// Translate a `colr` box into rawshift's colour signalling. +/// +/// `nclx` carries the full CICP triple plus the range flag; the older `nclc` +/// has no range flag, and studio range is the correct reading for it. An +/// absent or ICC-profile-only `colr` leaves everything unspecified, which +/// resolves by picture height at decode time. +fn colour_of(colr: Option<&Colr>) -> CicpColor { + match colr { + Some(Colr::Nclx { + colour_primaries, + transfer_characteristics, + matrix_coefficients, + full_range_flag, + }) => CicpColor { + primaries: *colour_primaries, + transfer: *transfer_characteristics, + matrix: *matrix_coefficients, + full_range: *full_range_flag, + }, + Some(Colr::Nclc { + colour_primaries, + transfer_characteristics, + matrix_coefficients, + }) => CicpColor { + primaries: *colour_primaries, + transfer: *transfer_characteristics, + matrix: *matrix_coefficients, + full_range: false, + }, + _ => CicpColor::UNSPECIFIED, + } +} + +/// Recover display rotation from the `tkhd` matrix. +/// +/// The matrix values are 16.16 fixed point; only `a`, `b`, `c`, `d` carry +/// rotation. +fn rotation_of(trak: &Trak) -> Rotation { + let m = &trak.tkhd.matrix; + let fixed = |v: i32| f64::from(v) / 65536.0; + Rotation::from_display_matrix(fixed(m.a), fixed(m.b), fixed(m.c), fixed(m.d)) +} + +fn audio_codec_of(codec: &Codec) -> AudioCodecId { + match codec { + Codec::Mp4a(_) => AudioCodecId::Aac, + Codec::Opus(_) => AudioCodecId::Opus, + Codec::Flac(_) => AudioCodecId::Flac, + Codec::Ac3(_) | Codec::Eac3(_) => AudioCodecId::Ac3, + _ => AudioCodecId::Other, + } +} + +fn stbl_sample_count(stbl: &mp4_atom::Stbl) -> u64 { + match &stbl.stsz.samples { + mp4_atom::StszSamples::Identical { count, .. } => u64::from(*count), + mp4_atom::StszSamples::Different { sizes } => sizes.len() as u64, + } +} + +/// Convert a tick count in `time_base` units to a [`Duration`]. +pub fn ticks_to_duration(ticks: u64, time_base: URational) -> Option { + if time_base.denominator == 0 { + return None; + } + let seconds = ticks as f64 * f64::from(time_base.numerator) / f64::from(time_base.denominator); + Duration::try_from_secs_f64(seconds).ok() +} + +fn build_metadata(moov: &Moov, ftyp: Option<&Ftyp>, container: ContainerId) -> VideoMetadata { + let mut md = VideoMetadata::default(); + let mvhd = &moov.mvhd; + + md.container.container = Some(container); + md.container.timescale = Some(mvhd.timescale); + md.container.duration = + ticks_to_duration(mvhd.duration, URational::new(1, mvhd.timescale.max(1))); + md.container.creation_time = Some(format_iso_time(mvhd.creation_time)); + md.container.modification_time = Some(format_iso_time(mvhd.modification_time)); + + if let Some(ftyp) = ftyp { + md.container.brands = std::iter::once(&ftyp.major_brand) + .chain(ftyp.compatible_brands.iter()) + .map(|b| brand_string(b.as_ref())) + .collect(); + } + + for brand in md.container.brands.clone() { + md.push_container_entry( + MetadataNamespace::Quicktime, + "ftyp.brand", + MetadataValue::Text(brand), + ); + } + md.push_container_entry( + MetadataNamespace::Quicktime, + "mvhd.timescale", + MetadataValue::U64(u64::from(mvhd.timescale)), + ); + + md +} + +/// Render a four-character brand, replacing non-printable bytes. +/// +/// A crafted brand must not smuggle control characters into a log line or an +/// error message. +fn brand_string(bytes: &[u8]) -> String { + bytes + .iter() + .map(|&b| { + if b.is_ascii_graphic() || b == b' ' { + b as char + } else { + '?' + } + }) + .collect() +} + +/// Render an ISOBMFF timestamp as an ISO 8601 date-time in UTC. +/// +/// ISOBMFF counts seconds from 1904-01-01, not the Unix epoch. Formatted here +/// rather than by pulling in a date library for one field. +pub fn format_iso_time(seconds_since_1904: u64) -> String { + /// Seconds between 1904-01-01 and 1970-01-01. + const EPOCH_OFFSET: u64 = 2_082_844_800; + + if seconds_since_1904 < EPOCH_OFFSET { + // Before the Unix epoch: report the raw value rather than a wrong date. + return format!("{seconds_since_1904} (seconds since 1904-01-01)"); + } + let unix = seconds_since_1904 - EPOCH_OFFSET; + + let (days, time) = (unix / 86_400, unix % 86_400); + let (hour, minute, second) = (time / 3_600, (time % 3_600) / 60, time % 60); + let (year, month, day) = civil_from_days(days as i64); + + format!("{year:04}-{month:02}-{day:02}T{hour:02}:{minute:02}:{second:02}Z") +} + +/// Convert days since 1970-01-01 to a proleptic Gregorian date. +/// +/// Howard Hinnant's `civil_from_days`: exact over the whole representable +/// range and table-free. +fn civil_from_days(days: i64) -> (i64, u32, u32) { + let z = days + 719_468; + let era = if z >= 0 { z } else { z - 146_096 } / 146_097; + let doe = (z - era * 146_097) as u64; + let yoe = (doe - doe / 1_460 + doe / 36_524 - doe / 146_096) / 365; + let y = yoe as i64 + era * 400; + let doy = doe - (365 * yoe + yoe / 4 - yoe / 100); + let mp = (5 * doy + 2) / 153; + let d = (doy - (153 * mp + 2) / 5 + 1) as u32; + let m = if mp < 10 { mp + 3 } else { mp - 9 } as u32; + (if m <= 2 { y + 1 } else { y }, m, d) +} + +#[cfg(test)] +mod tests { + use super::*; + use mp4_atom::FourCC; + + #[test] + fn the_isobmff_epoch_is_offset_from_unix_not_equal_to_it() { + assert_eq!(format_iso_time(2_082_844_800), "1970-01-01T00:00:00Z"); + assert_eq!( + format_iso_time(2_082_844_800 + 1_705_321_845), + "2024-01-15T12:30:45Z" + ); + } + + #[test] + fn a_timestamp_before_the_unix_epoch_reports_raw_rather_than_a_wrong_date() { + let s = format_iso_time(0); + assert!(s.contains("1904"), "{s}"); + assert!(!s.starts_with("1970"), "{s}"); + } + + #[test] + fn civil_from_days_handles_leap_years_and_epoch_edges() { + assert_eq!(civil_from_days(0), (1970, 1, 1)); + assert_eq!(civil_from_days(59), (1970, 3, 1)); + assert_eq!(civil_from_days(11_016), (2000, 2, 29)); + assert_eq!(civil_from_days(19_782), (2024, 2, 29)); + assert_eq!(civil_from_days(19_783), (2024, 3, 1)); + } + + #[test] + fn the_quicktime_brand_selects_mov_from_either_position() { + let major = Ftyp { + major_brand: FourCC::new(b"qt "), + minor_version: 0, + compatible_brands: vec![], + }; + assert_eq!(container_of(Some(&major)), ContainerId::Mov); + + let compatible = Ftyp { + major_brand: FourCC::new(b"isom"), + minor_version: 0, + compatible_brands: vec![FourCC::new(b"qt ")], + }; + assert_eq!(container_of(Some(&compatible)), ContainerId::Mov); + } + + #[test] + fn other_brands_are_mp4_and_a_missing_ftyp_is_legacy_quicktime() { + let mp4 = Ftyp { + major_brand: FourCC::new(b"isom"), + minor_version: 512, + compatible_brands: vec![FourCC::new(b"mp41")], + }; + assert_eq!(container_of(Some(&mp4)), ContainerId::Mp4); + assert_eq!(container_of(None), ContainerId::Mov); + } + + #[test] + fn nclx_colour_carries_the_full_cicp_triple_and_range() { + let colr = Colr::Nclx { + colour_primaries: 9, + transfer_characteristics: 16, + matrix_coefficients: 9, + full_range_flag: true, + }; + let c = colour_of(Some(&colr)); + assert_eq!((c.primaries, c.transfer, c.matrix), (9, 16, 9)); + assert!(c.full_range, "the range flag must survive"); + } + + #[test] + fn nclc_colour_has_no_range_flag_and_reads_as_studio() { + let colr = Colr::Nclc { + colour_primaries: 1, + transfer_characteristics: 1, + matrix_coefficients: 1, + }; + let c = colour_of(Some(&colr)); + assert_eq!(c.primaries, 1); + assert!( + !c.full_range, + "nclc has no range flag; studio is the reading" + ); + } + + #[test] + fn an_absent_or_icc_only_colr_leaves_colour_unspecified() { + assert_eq!(colour_of(None), CicpColor::UNSPECIFIED); + let icc = Colr::Prof { + profile: vec![0u8; 16], + }; + assert_eq!(colour_of(Some(&icc)), CicpColor::UNSPECIFIED); + } + + #[test] + fn brands_are_sanitised_of_control_bytes() { + assert_eq!(brand_string(b"isom"), "isom"); + assert_eq!(brand_string(b"qt "), "qt "); + assert_eq!(brand_string(&[0x00, 0x1b, b'[', b'A']), "??[A"); + } +} From 31a4650dfc15b9fdae56dab8e08e633d477c1e46 Mon Sep 17 00:00:00 2001 From: Justin Chung Date: Fri, 28 Aug 2026 22:33:32 -0400 Subject: [PATCH 08/13] test(video-isobmff): check demuxing against ffmpeg-generated files MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Locks in the behaviours that synthetic-table unit tests cannot reach: that rawshift agrees with the rest of the world about a file ffmpeg wrote. Covers track and dimension recovery, complete sample delivery, decode ordering, composition offsets actually being applied, seeks landing on a resumable point, and rejection of non-ISOBMFF and truncated input. Two are regression tests for bugs found by running against real files rather than by reading code: HEVC opening despite a multi-kilobyte hvcC, and the first PTS matching ffprobe's once the edit list is applied. Fixtures are generated at test time and cached in a temp directory rather than committed, per TEST_FIXTURES.md. Generation stages to a unique name and renames into place: tests run in parallel and share the cache, so writing directly to the final path made it exist — and so look ready — while ffmpeg was still filling it, and other tests opened a half-written file. Verified that the suite passes with no ffmpeg on PATH, skipping rather than failing. Refs #39 --- .../tests/common/mod.rs | 152 +++++++++++ .../tests/demux_fixtures.rs | 235 ++++++++++++++++++ 2 files changed, 387 insertions(+) create mode 100644 crates/rawshift-video-isobmff/tests/common/mod.rs create mode 100644 crates/rawshift-video-isobmff/tests/demux_fixtures.rs diff --git a/crates/rawshift-video-isobmff/tests/common/mod.rs b/crates/rawshift-video-isobmff/tests/common/mod.rs new file mode 100644 index 0000000..fa429f0 --- /dev/null +++ b/crates/rawshift-video-isobmff/tests/common/mod.rs @@ -0,0 +1,152 @@ +//! Shared fixture generation for the container integration tests. +//! +//! Fixtures are generated with `ffmpeg` at test time rather than committed, +//! following `TEST_FIXTURES.md`: generated inputs stay out of the repository, +//! and **tests skip gracefully when the tooling is absent**, so `cargo test` +//! passes on a machine with no ffmpeg. + +use std::path::{Path, PathBuf}; +use std::process::Command; + +/// Whether ffmpeg is available to generate fixtures. +pub fn ffmpeg_available() -> bool { + Command::new("ffmpeg") + .arg("-version") + .stdout(std::process::Stdio::null()) + .stderr(std::process::Stdio::null()) + .status() + .is_ok_and(|s| s.success()) +} + +/// Where generated fixtures live: a temp directory, never the repository. +fn fixture_dir() -> PathBuf { + let dir = std::env::temp_dir().join("rawshift-video-fixtures"); + let _ = std::fs::create_dir_all(&dir); + dir +} + +/// Generate a test clip, returning its path. +/// +/// Cached across runs: regenerating clips for every test would dominate the +/// suite's runtime. Returns `None` when ffmpeg is missing or lacks the +/// encoder, which callers treat as a skip — the tests assert rawshift's +/// behaviour, not the machine's codec inventory. +pub fn clip(name: &str, args: &[&str]) -> Option { + if !ffmpeg_available() { + return None; + } + let path = fixture_dir().join(name); + if path.exists() { + return Some(path); + } + + // Generate to a unique name and rename into place. Tests run in parallel + // and share this cache; writing directly to `path` would make it exist — + // and so look ready — while ffmpeg is still filling it, and the other + // tests would open a half-written file. + let staging = path.with_extension(format!( + "tmp.{}.{}", + std::process::id(), + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map_or(0, |d| d.as_nanos()) + )); + + let ok = Command::new("ffmpeg") + .args(["-y", "-loglevel", "error"]) + .args(args) + .arg(&staging) + .status() + .is_ok_and(|s| s.success()); + + if !ok || !staging.exists() { + let _ = std::fs::remove_file(&staging); + return None; + } + // Rename is atomic within a filesystem, so `path` goes from absent to + // complete with nothing observable in between. A concurrent generator + // renaming its own copy over ours is harmless: both are complete. + if std::fs::rename(&staging, &path).is_err() { + let _ = std::fs::remove_file(&staging); + return None; + } + Some(path) +} + +/// A short H.264 clip with B-frames and a 12-frame GOP. +pub fn h264_mp4() -> Option { + clip( + "h264.mp4", + &[ + "-f", + "lavfi", + "-i", + "testsrc2=size=128x96:duration=2:rate=25", + "-c:v", + "libx264", + "-bf", + "3", + "-g", + "12", + "-profile:v", + "high", + "-pix_fmt", + "yuv420p", + ], + ) +} + +/// The same content as HEVC — the case that motivated owning the sample index. +pub fn hevc_mp4() -> Option { + clip( + "hevc.mp4", + &[ + "-f", + "lavfi", + "-i", + "testsrc2=size=128x96:duration=2:rate=25", + "-c:v", + "libx265", + "-x265-params", + "log-level=error", + "-g", + "12", + "-pix_fmt", + "yuv420p", + ], + ) +} + +/// H.264 in a QuickTime container. +pub fn h264_mov() -> Option { + clip( + "h264.mov", + &[ + "-f", + "lavfi", + "-i", + "testsrc2=size=128x96:duration=1:rate=25", + "-c:v", + "libx264", + "-g", + "12", + "-pix_fmt", + "yuv420p", + ], + ) +} + +/// Ask ffprobe for values, as ground truth to compare rawshift against. +pub fn ffprobe(path: &Path, entries: &str, extra: &[&str]) -> Option { + let out = Command::new("ffprobe") + .args(["-v", "error", "-select_streams", "v:0", "-show_entries"]) + .arg(entries) + .args(extra) + .args(["-of", "default=noprint_wrappers=1:nokey=1"]) + .arg(path) + .output() + .ok()?; + out.status + .success() + .then(|| String::from_utf8_lossy(&out.stdout).trim().to_string()) +} diff --git a/crates/rawshift-video-isobmff/tests/demux_fixtures.rs b/crates/rawshift-video-isobmff/tests/demux_fixtures.rs new file mode 100644 index 0000000..83fe1e3 --- /dev/null +++ b/crates/rawshift-video-isobmff/tests/demux_fixtures.rs @@ -0,0 +1,235 @@ +//! End-to-end demuxing of real files, checked against ffprobe. +//! +//! These assert the behaviours that unit tests over synthetic tables cannot: +//! that rawshift agrees with the rest of the world about a file ffmpeg wrote. +//! Every test skips cleanly when ffmpeg is unavailable. + +mod common; + +use std::fs::File; + +use rawshift_video_core::{ContainerId, Demuxer, Track, VideoCodecId}; +use rawshift_video_isobmff::IsoBmffDemuxer; + +/// Skip the test when the fixture could not be generated. +macro_rules! fixture { + ($maker:expr, $what:literal) => { + match $maker { + Some(path) => path, + None => { + eprintln!("skipping: cannot generate the {} fixture (ffmpeg missing or without the encoder)", $what); + return; + } + } + }; +} + +fn open(path: &std::path::Path) -> IsoBmffDemuxer { + IsoBmffDemuxer::open(File::open(path).expect("fixture is readable")).expect("file opens") +} + +fn only_video_track(demuxer: &IsoBmffDemuxer) -> &rawshift_video_core::VideoTrack { + demuxer + .tracks() + .iter() + .find_map(Track::as_video) + .expect("the fixture has a video track") +} + +#[test] +fn h264_in_mp4_reads_its_track_and_every_sample() { + let path = fixture!(common::h264_mp4(), "H.264 MP4"); + let mut demuxer = open(&path); + + assert_eq!(demuxer.container(), ContainerId::Mp4); + + let video = only_video_track(&demuxer); + assert_eq!(video.codec, VideoCodecId::H264); + assert_eq!(video.dimensions.width, 128); + assert_eq!(video.dimensions.height, 96); + // An avcC must be present, or no decoder could be opened for the track. + assert!( + !video.codec_config.is_empty(), + "the track must carry its avcC" + ); + + // 2 seconds at 25 fps. + let mut packets = 0; + while demuxer.next_packet().expect("packets read").is_some() { + packets += 1; + } + assert_eq!(packets, 50, "every sample must be delivered exactly once"); +} + +#[test] +fn hevc_in_mp4_opens_despite_a_multi_kilobyte_hvcc() { + // The regression this crate's rewrite exists for: the previous backend + // capped hvcC at 1 KB, and every real HEVC file failed to open. + let path = fixture!(common::hevc_mp4(), "HEVC MP4"); + let mut demuxer = open(&path); + + let video = only_video_track(&demuxer); + assert_eq!(video.codec, VideoCodecId::Hevc); + assert!( + video.codec_config.as_bytes().len() > 1024, + "this fixture's hvcC should exceed the 1 KB that used to be fatal, got {} bytes", + video.codec_config.as_bytes().len() + ); + + let mut packets = 0; + while demuxer.next_packet().expect("packets read").is_some() { + packets += 1; + } + assert_eq!(packets, 50); +} + +#[test] +fn quicktime_is_distinguished_from_mp4_by_its_brand() { + let path = fixture!(common::h264_mov(), "H.264 MOV"); + let demuxer = open(&path); + assert_eq!(demuxer.container(), ContainerId::Mov); +} + +#[test] +fn presentation_timestamps_agree_with_ffprobe() { + // The edit list must be applied: without it every timestamp carries the + // encoder's reorder delay and rawshift disagrees with every other tool. + let path = fixture!(common::h264_mp4(), "H.264 MP4"); + let mut demuxer = open(&path); + + let first = demuxer + .next_packet() + .expect("first packet") + .expect("the file is not empty"); + + let Some(expected) = common::ffprobe(&path, "packet=pts", &["-read_intervals", "%+#1"]) else { + eprintln!("skipping the ffprobe comparison: ffprobe is unavailable"); + return; + }; + let expected: i64 = expected.trim().parse().expect("ffprobe prints an integer"); + assert_eq!( + first.pts, + Some(expected), + "rawshift's first PTS must match ffprobe's" + ); +} + +#[test] +fn random_access_points_match_the_encoders_gop_length() { + // Derived from the bitstream, not from the container's sync table. A + // 50-frame clip with -g 12 has keyframes at 0, 12, 24, 36 and 48. + let path = fixture!(common::h264_mp4(), "H.264 MP4"); + let mut demuxer = open(&path); + + let mut keyframes = 0; + while let Some(packet) = demuxer.next_packet().expect("packets read") { + if packet.is_keyframe { + keyframes += 1; + } + } + assert_eq!(keyframes, 5, "one per 12-frame GOP across 50 frames"); +} + +#[test] +fn packets_are_delivered_in_decode_order() { + // Decode order is what a decoder must be fed; presentation order is + // recovered afterwards from the timestamps. + let path = fixture!(common::h264_mp4(), "H.264 MP4"); + let mut demuxer = open(&path); + + let mut previous = i64::MIN; + while let Some(packet) = demuxer.next_packet().expect("packets read") { + let dts = packet.dts.expect("MP4 samples always carry a decode time"); + assert!(dts >= previous, "decode timestamps went backwards: {dts}"); + previous = dts; + } +} + +#[test] +fn a_reordered_stream_has_presentation_diverge_from_decode() { + // With -bf 3 the encoder reorders, so at least one packet must be shown + // later than it is decoded. If this ever stops holding, the ctts handling + // has silently become a no-op. + let path = fixture!(common::h264_mp4(), "H.264 MP4"); + let mut demuxer = open(&path); + + let mut reordered = false; + while let Some(packet) = demuxer.next_packet().expect("packets read") { + if packet.pts != packet.dts { + reordered = true; + } + } + assert!( + reordered, + "a B-frame stream must have composition offsets applied" + ); +} + +#[test] +fn seeking_lands_on_a_random_access_point_at_or_before_the_request() { + use rawshift_video_core::{SeekMode, SeekTo}; + + let path = fixture!(common::h264_mp4(), "H.264 MP4"); + let mut demuxer = open(&path); + let track = only_video_track(&demuxer).id; + + // Ask for a point mid-GOP; the demuxer must land at or before it. + let requested = 20_000; + let landed = demuxer + .seek(track, SeekTo::Timestamp(requested), SeekMode::Precise) + .expect("seek succeeds"); + assert!( + landed <= requested, + "landed at {landed}, after the requested {requested}" + ); + + // And the next packet must be decodable from cold, i.e. a keyframe. + let packet = demuxer + .next_packet() + .expect("packet after seek") + .expect("not at end of file"); + assert!( + packet.is_keyframe, + "a seek must land somewhere decoding can resume" + ); +} + +#[test] +fn seeking_an_unknown_track_is_reported_rather_than_ignored() { + use rawshift_video_core::{SeekMode, SeekTo, TrackId, VideoError}; + + let path = fixture!(common::h264_mp4(), "H.264 MP4"); + let mut demuxer = open(&path); + + let err = demuxer + .seek(TrackId::new(999), SeekTo::Timestamp(0), SeekMode::Precise) + .expect_err("track 999 does not exist"); + assert!(matches!(err, VideoError::NoSuchTrack { .. }), "{err}"); +} + +#[test] +fn a_non_isobmff_file_is_rejected_rather_than_half_read() { + use std::io::Cursor; + + let not_a_movie = Cursor::new(b"\x89PNG\r\n\x1a\n and then some bytes".to_vec()); + assert!( + IsoBmffDemuxer::open(not_a_movie).is_err(), + "a PNG must not open as a movie" + ); +} + +#[test] +fn a_truncated_file_fails_to_open_rather_than_panicking() { + use std::io::Cursor; + + let path = fixture!(common::h264_mp4(), "H.264 MP4"); + let whole = std::fs::read(&path).expect("fixture is readable"); + + // Truncating mid-header is the shape of a partial download. + for fraction in [1, 2, 8] { + let cut = whole.len() / fraction / 2; + let truncated = Cursor::new(whole[..cut].to_vec()); + // Either it opens with what it found or it errors; it must not panic. + let _ = IsoBmffDemuxer::open(truncated); + } +} From 0dad0dbb0176e0c14b4c357bfe9db796b1c2082d Mon Sep 17 00:00:00 2001 From: Justin Chung Date: Fri, 28 Aug 2026 22:37:01 -0400 Subject: [PATCH 09/13] feat(hwdec)!: add the video-sequence decode seam MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds the API for decoding a coded video sequence alongside the existing still-frame one: VideoConfig, VideoPacket, VideoFrame, the HwVideoDecoder trait, video_decoder() and available_video_codecs(). The two seams stay separate rather than one being rewritten as the other. The still path is a shipping API that HEIC and AVIF depend on and that has device tests behind it; re-expressing it on the sequence path would risk regressions in the image crates for no user-visible gain. What they share is the VAAPI plumbing, not the policy. Decoding is send-packet / receive-frame rather than decode(packet) -> frame, because coded order and output order differ whenever a stream reorders, so one packet may make zero, one, or several frames available. It is also the shape every hardware decode API natively has. The decoder owns the picture buffer and emits display order: VAAPI is a slice-level API where the caller supplies the reference lists, so the buffer and the ordering cannot live above it, and pushing them onto callers would mean exposing picture order counts and buffer fullness across the API. video_decoder returns a Result rather than an Option, unlike decoder(): the caller supplies a real configuration record, so there is a specific reportable reason for failure that a consumer needs for its own errors. BREAKING CHANGE: HwCodec gains H264. The enum is documented as deliberately exhaustive, so this is that decision taken explicitly. H.264 is reachable only through the sequence seam — decoder(HwCodec::H264) returns None and available_codecs() never lists it — because no rawshift still format uses H.264 and offering an untested still path would misreport what the crate can do. Refs #39 --- crates/rawshift-hwdec/src/lib.rs | 257 ++++++++++++++++++++++++- crates/rawshift-hwdec/src/vaapi/mod.rs | 4 + 2 files changed, 255 insertions(+), 6 deletions(-) diff --git a/crates/rawshift-hwdec/src/lib.rs b/crates/rawshift-hwdec/src/lib.rs index 5d77541..52bd807 100644 --- a/crates/rawshift-hwdec/src/lib.rs +++ b/crates/rawshift-hwdec/src/lib.rs @@ -89,26 +89,43 @@ pub use gamut_color::{ChromaSubsampling, ColorRange}; // ── Codec / backend identity ──────────────────────────────────────────────── -/// A codec this crate can decode still frames of. +/// A codec this crate can decode. /// -/// The set is fixed at v1 (see `docs/SUPPORT.md`): HEVC for HEIC and AV1 for -/// AVIF. It is deliberately exhaustive — a new codec is a deliberate, -/// breaking decision, not an additive one. +/// The set is fixed by `docs/SUPPORT.md` and is deliberately exhaustive — a +/// new codec is a deliberate, breaking decision, not an additive one. +/// +/// # Seams +/// +/// Not every codec is decodable on both seams, and the set differs by design +/// rather than by omission: +/// +/// - [`decoder`] (still frames) answers for [`Hevc`](Self::Hevc) (HEIC) and +/// [`Av1`](Self::Av1) (AVIF). +/// - [`video_decoder`] (coded sequences) answers for [`Hevc`](Self::Hevc) and +/// [`H264`](Self::H264). +/// +/// [`H264`](Self::H264) exists only on the sequence seam: no rawshift still +/// format uses H.264, and offering an untested still path for it would +/// misreport what this crate can do. [`Av1`](Self::Av1) is still-only for now; +/// AV1 video sequences are a follow-up over the same backend. #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] pub enum HwCodec { - /// HEVC / H.265 intra (HEIC still images). + /// HEVC / H.265 — intra stills (HEIC) and coded video sequences. Hevc, /// AV1 intra (AVIF still images). Av1, + /// H.264 / AVC coded video sequences. + H264, } impl HwCodec { - /// The conventional display name of the codec (`"HEVC"` / `"AV1"`). + /// The conventional display name of the codec. #[must_use] pub fn name(self) -> &'static str { match self { HwCodec::Hevc => "HEVC", HwCodec::Av1 => "AV1", + HwCodec::H264 => "H.264", } } } @@ -470,6 +487,188 @@ pub enum HwDecodeError { }, } +// ── Video sequence seam ───────────────────────────────────────────────────── + +/// The codec configuration record of a video-sequence decode session. +/// +/// **The variant is the codec**, so a mismatched codec/config pair cannot be +/// constructed. Separate from [`CodecConfig`] deliberately: adding an `Avcc` +/// variant there would make "H.264 still image" representable, which is not a +/// rawshift use case and would ship an untested path. +/// +/// The bytes are the record **body** as the container stores it, with no box +/// header — an `AVCDecoderConfigurationRecord` (ISO/IEC 14496-15 §5.3.3.1) or +/// an `HEVCDecoderConfigurationRecord` (§8.3.3.1). Both carry the parameter +/// sets and the NAL length-prefix width every [`VideoPacket::data`] of the +/// track uses, so a session cannot be opened without one. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum VideoConfig<'a> { + /// An `avcC` record body; the codec is [`HwCodec::H264`]. + Avcc(&'a [u8]), + /// An `hvcC` record body; the codec is [`HwCodec::Hevc`]. + Hvcc(&'a [u8]), +} + +impl<'a> VideoConfig<'a> { + /// The codec this configuration selects. + #[must_use] + pub fn codec(&self) -> HwCodec { + match self { + VideoConfig::Avcc(_) => HwCodec::H264, + VideoConfig::Hvcc(_) => HwCodec::Hevc, + } + } + + /// The configuration record body. + #[must_use] + pub fn bytes(&self) -> &'a [u8] { + match self { + VideoConfig::Avcc(b) | VideoConfig::Hvcc(b) => b, + } + } +} + +/// One access unit: the coded picture for a single presentation time. +/// +/// `data` is the sample payload exactly as the container stores it — a +/// **length-prefixed** NAL stream, never Annex B — with the prefix width taken +/// from the session's [`VideoConfig`]. Exactly one coded picture per packet, +/// as ISO/IEC 14496-15 requires of a sample. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct VideoPacket<'a> { + /// The access unit's length-prefixed NAL units. + pub data: &'a [u8], + /// Container presentation timestamp, carried through to the frame this + /// picture becomes and never used for ordering. + pub pts: Option, + /// Container decode timestamp. Diagnostics only: output order comes from + /// the bitstream's picture order count, which is authoritative where a + /// container's timestamps are absent or wrong. + pub dts: Option, + /// Whether the container marks this as a sync sample. Advisory; the + /// bitstream decides. + pub is_sync: bool, +} + +/// A decoded picture in output (display) order, with the timestamp of the +/// packet it was coded from. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct VideoFrame { + frame: DecodedFrame, + pts: Option, + poc: i32, +} + +impl VideoFrame { + /// Wrap a decoded picture with its timing. + #[must_use] + pub fn new(frame: DecodedFrame, pts: Option, poc: i32) -> Self { + Self { frame, pts, poc } + } + + /// The decoded picture. + #[must_use] + pub fn frame(&self) -> &DecodedFrame { + &self.frame + } + + /// Take ownership of the decoded picture. + #[must_use] + pub fn into_frame(self) -> DecodedFrame { + self.frame + } + + /// The `pts` of the packet this picture was coded from. + #[must_use] + pub fn pts(&self) -> Option { + self.pts + } + + /// The picture order count derived from the bitstream. + /// + /// Strictly increasing across the frames emitted between two random access + /// points, which makes it the reliable ordering key even for a container + /// whose timestamps are missing or wrong. + #[must_use] + pub fn poc(&self) -> i32 { + self.poc + } +} + +/// A hardware decoder for one coded video sequence, from [`video_decoder`]. +/// +/// **Send-packet / receive-frame**, because coded order and output order +/// differ whenever a stream reorders (B pictures, hierarchical GOPs): one +/// [`send_packet`](Self::send_packet) may make zero, one, or several frames +/// available. Drive it as: +/// +/// ```text +/// for packet in access_units { +/// decoder.send_packet(&packet)?; +/// while let Some(frame) = decoder.receive_frame()? { sink(frame); } +/// } +/// decoder.flush()?; +/// while let Some(frame) = decoder.receive_frame()? { sink(frame); } +/// ``` +/// +/// **This decoder owns the picture buffer and emits frames in output order**, +/// because VAAPI is a slice-level API: the caller must hand it fully-formed +/// picture parameters including the reference lists, so the buffer and the +/// ordering cannot live above it. Callers must not reorder the output; +/// [`VideoFrame::poc`] is already monotonic within a coded video sequence. +pub trait HwVideoDecoder: Send { + /// The codec this session decodes. + fn codec(&self) -> HwCodec; + + /// Submit one access unit, making zero or more frames available to + /// [`receive_frame`](Self::receive_frame). + /// + /// After [`reset`](Self::reset), packets before the next random access + /// point reference pictures that no longer exist; they are dropped and + /// `Ok(())` returned, so seeking into the middle of a group of pictures is + /// not an error. + /// + /// # Errors + /// + /// [`HwDecodeError::Decode`] when the access unit is malformed or uses a + /// coding tool outside the backend's scope, and + /// [`HwDecodeError::Unavailable`] when in-band parameter sets change the + /// stream to something the driver cannot decode. + fn send_packet(&mut self, packet: &VideoPacket<'_>) -> Result<(), HwDecodeError>; + + /// Take the next frame in output order, or `None` when none is ready. + /// + /// `None` is the normal state while a reordering stream fills its buffer: + /// it means "not yet", not "end of stream". Only [`flush`](Self::flush) + /// establishes the latter. + /// + /// # Errors + /// + /// [`HwDecodeError::Decode`] when reading a decoded surface back fails, + /// or [`HwDecodeError::InvalidFrame`] when the readback is inconsistent. + fn receive_frame(&mut self) -> Result, HwDecodeError>; + + /// Signal end of stream: every picture still held for output becomes + /// available to [`receive_frame`](Self::receive_frame). + /// + /// Without this the last frames of every reordering stream are never + /// emitted. The session stays usable — a later + /// [`send_packet`](Self::send_packet) starts a new coded video sequence + /// and must begin at a random access point. + /// + /// # Errors + /// + /// As [`send_packet`](Self::send_packet). + fn flush(&mut self) -> Result<(), HwDecodeError>; + + /// Discard all decoder state **without** emitting pending frames — the + /// seek path. + /// + /// Reference pictures and queued output are dropped; the parameter sets + /// from the session's [`VideoConfig`] are retained. + fn reset(&mut self); +} + // ── Backend discovery ─────────────────────────────────────────────────────── /// Returns a decoder for `codec`, or `None` when no compiled-in backend can @@ -494,6 +693,52 @@ pub fn decoder(codec: HwCodec) -> Option> { } } +/// Opens a video-sequence decoder for `config`. +/// +/// The configuration record is parsed and validated up front — parameter sets, +/// profile, bit depth, chroma format — and the driver is probed for the +/// matching profile, so an unsupported stream fails here rather than on the +/// first packet. +/// +/// Unlike [`decoder`] this returns a `Result`: the caller supplied a real +/// configuration record, so there is a specific, reportable reason for +/// failure, and a consumer needs it for its own error surface. +/// +/// # Errors +/// +/// [`HwDecodeError::Unavailable`] when no backend is compiled in or the driver +/// cannot decode the stream's profile, and [`HwDecodeError::Decode`] when the +/// configuration record is malformed or out of scope. +pub fn video_decoder(config: VideoConfig<'_>) -> Result, HwDecodeError> { + #[cfg(hwdec_backend = "vaapi")] + { + vaapi::video_decoder(config) + } + #[cfg(not(hwdec_backend = "vaapi"))] + { + Err(HwDecodeError::Unavailable { + codec: config.codec(), + reason: "no hardware decode backend is compiled in for this target".to_string(), + }) + } +} + +/// The codecs [`video_decoder`] can open a session for on this machine. +/// +/// Empty when no backend is compiled in, or when the runtime probe finds no +/// usable driver — never a link or startup failure. +#[must_use] +pub fn available_video_codecs() -> &'static [HwCodec] { + #[cfg(hwdec_backend = "vaapi")] + { + vaapi::available_video_codecs() + } + #[cfg(not(hwdec_backend = "vaapi"))] + { + &[] + } +} + /// The platform backend compiled into this build and usable at runtime, or /// `None`. /// diff --git a/crates/rawshift-hwdec/src/vaapi/mod.rs b/crates/rawshift-hwdec/src/vaapi/mod.rs index faf0aed..f09be1a 100644 --- a/crates/rawshift-hwdec/src/vaapi/mod.rs +++ b/crates/rawshift-hwdec/src/vaapi/mod.rs @@ -59,6 +59,10 @@ pub fn decoder(codec: HwCodec) -> Option> { let supported = match codec { HwCodec::Hevc => caps.hevc_main || caps.hevc_main10, HwCodec::Av1 => caps.av1_profile0, + // H.264 exists only on the sequence seam: no rawshift still format + // uses it, so offering a still decoder would advertise an untested + // path. See `HwCodec`'s "Seams" documentation. + HwCodec::H264 => false, }; if !supported { return None; From d5717bf11762e6c63ee134b4ac211502a9e2ecda Mon Sep 17 00:00:00 2001 From: Justin Chung Date: Fri, 28 Aug 2026 22:42:45 -0400 Subject: [PATCH 10/13] feat(hwdec): decode video sequences through VAAPI MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implements the sequence seam on the VAAPI backend: configuration records validated at open, H.264 profiles added to the runtime probe, and available_video_codecs() answering from what the driver actually exposes. Scope is random access points — every IRAP (HEVC) and IDR (H.264) access unit decodes; anything referencing other pictures is refused with a clear error rather than decoded into a plausible-looking wrong picture. That is a real capability rather than a stub: keyframe extraction, poster frames, timeline thumbnails and scrubbing all work, and All-Intra camera modes (Sony XAVC S-I, XAVC HS All-I) decode completely because every access unit in them is a random access point. Inter-frame decode is a purely internal change behind this unchanged API — it adds a decoded picture buffer, picture order counts, reference marking and reference lists, and drops the rejection. No part of the public seam moves, so it needs no second breaking release. Verified on an AMD RX 7900 XT (radeonsi, VA-API 1.23): a libx265 keyframe decodes to correct dimensions with non-blank pixels and its timestamp carried through, receive_frame reports "not ready" rather than failing, flush invents nothing, and reset discards pending output while leaving the session usable. Also extracts the device-test helpers into tests/common. Writing a second copy of the hvcC builder from memory put numOfArrays one byte off, which yields a record that parses but carries no parameter sets — the failure was "stream carries no SPS" on a stream that plainly had one. One copy, in one place, with a note saying why. Refs #39 --- crates/rawshift-hwdec/src/vaapi/mod.rs | 47 +++- crates/rawshift-hwdec/src/vaapi/sys.rs | 6 + crates/rawshift-hwdec/src/vaapi/video.rs | 202 ++++++++++++++++ crates/rawshift-hwdec/tests/common/mod.rs | 116 +++++++++ crates/rawshift-hwdec/tests/vaapi_device.rs | 109 +-------- .../tests/vaapi_video_device.rs | 224 ++++++++++++++++++ 6 files changed, 597 insertions(+), 107 deletions(-) create mode 100644 crates/rawshift-hwdec/src/vaapi/video.rs create mode 100644 crates/rawshift-hwdec/tests/common/mod.rs create mode 100644 crates/rawshift-hwdec/tests/vaapi_video_device.rs diff --git a/crates/rawshift-hwdec/src/vaapi/mod.rs b/crates/rawshift-hwdec/src/vaapi/mod.rs index f09be1a..a735da2 100644 --- a/crates/rawshift-hwdec/src/vaapi/mod.rs +++ b/crates/rawshift-hwdec/src/vaapi/mod.rs @@ -40,6 +40,7 @@ mod av1; mod bits; mod hevc; mod sys; +mod video; use std::ffi::c_int; use std::fs::File; @@ -78,7 +79,8 @@ pub fn decoder(codec: HwCodec) -> Option> { /// Backend hook for [`crate::backend`]. pub fn backend() -> Option { let caps = probe(); - (caps.hevc_main || caps.hevc_main10 || caps.av1_profile0).then_some(HwBackend::Vaapi) + (caps.hevc_main || caps.hevc_main10 || caps.av1_profile0 || caps.h264_any()) + .then_some(HwBackend::Vaapi) } /// Backend hook for [`crate::available_codecs`]. @@ -96,6 +98,29 @@ pub fn available_codecs() -> &'static [HwCodec] { } } +/// Backend hook for [`crate::video_decoder`]. +pub fn video_decoder( + config: crate::VideoConfig<'_>, +) -> Result, HwDecodeError> { + video::open(config) +} + +/// Backend hook for [`crate::available_video_codecs`]. +pub fn available_video_codecs() -> &'static [HwCodec] { + const NONE: &[HwCodec] = &[]; + const HEVC_ONLY: &[HwCodec] = &[HwCodec::Hevc]; + const H264_ONLY: &[HwCodec] = &[HwCodec::H264]; + const BOTH: &[HwCodec] = &[HwCodec::Hevc, HwCodec::H264]; + + let caps = probe(); + match (caps.hevc_main || caps.hevc_main10, caps.h264_any()) { + (true, true) => BOTH, + (true, false) => HEVC_ONLY, + (false, true) => H264_ONLY, + (false, false) => NONE, + } +} + // ── Runtime probe ─────────────────────────────────────────────────────────── /// What the driver actually decodes, per `vaQueryConfigProfiles` + @@ -105,6 +130,16 @@ struct Caps { hevc_main: bool, hevc_main10: bool, av1_profile0: bool, + h264_constrained_baseline: bool, + h264_main: bool, + h264_high: bool, +} + +impl Caps { + /// Whether the driver decodes H.264 at any profile rawshift accepts. + fn h264_any(&self) -> bool { + self.h264_constrained_baseline || self.h264_main || self.h264_high + } } /// Probe once per process: dlopen, open a render node, init a display, @@ -141,7 +176,12 @@ fn probe_uncached() -> Option { for &profile in &profiles { let interesting = matches!( profile, - sys::VA_PROFILE_HEVC_MAIN | sys::VA_PROFILE_HEVC_MAIN10 | sys::VA_PROFILE_AV1_PROFILE0 + sys::VA_PROFILE_HEVC_MAIN + | sys::VA_PROFILE_HEVC_MAIN10 + | sys::VA_PROFILE_AV1_PROFILE0 + | sys::VA_PROFILE_H264_CONSTRAINED_BASELINE + | sys::VA_PROFILE_H264_MAIN + | sys::VA_PROFILE_H264_HIGH ); if !interesting { continue; @@ -153,6 +193,9 @@ fn probe_uncached() -> Option { sys::VA_PROFILE_HEVC_MAIN => caps.hevc_main = true, sys::VA_PROFILE_HEVC_MAIN10 => caps.hevc_main10 = true, sys::VA_PROFILE_AV1_PROFILE0 => caps.av1_profile0 = true, + sys::VA_PROFILE_H264_CONSTRAINED_BASELINE => caps.h264_constrained_baseline = true, + sys::VA_PROFILE_H264_MAIN => caps.h264_main = true, + sys::VA_PROFILE_H264_HIGH => caps.h264_high = true, _ => {} } } diff --git a/crates/rawshift-hwdec/src/vaapi/sys.rs b/crates/rawshift-hwdec/src/vaapi/sys.rs index 97bb3db..ed08b74 100644 --- a/crates/rawshift-hwdec/src/vaapi/sys.rs +++ b/crates/rawshift-hwdec/src/vaapi/sys.rs @@ -42,6 +42,12 @@ pub const VA_INVALID_ID: u32 = 0xffff_ffff; pub const VA_INVALID_SURFACE: VASurfaceID = VA_INVALID_ID; // VAProfile values (va.h). +/// `VAProfileH264ConstrainedBaseline` (va.h). +pub const VA_PROFILE_H264_CONSTRAINED_BASELINE: VAProfile = 13; +/// `VAProfileH264Main` (va.h). +pub const VA_PROFILE_H264_MAIN: VAProfile = 6; +/// `VAProfileH264High` (va.h). +pub const VA_PROFILE_H264_HIGH: VAProfile = 7; pub const VA_PROFILE_HEVC_MAIN: VAProfile = 17; pub const VA_PROFILE_HEVC_MAIN10: VAProfile = 18; pub const VA_PROFILE_AV1_PROFILE0: VAProfile = 32; diff --git a/crates/rawshift-hwdec/src/vaapi/video.rs b/crates/rawshift-hwdec/src/vaapi/video.rs new file mode 100644 index 0000000..8f767e5 --- /dev/null +++ b/crates/rawshift-hwdec/src/vaapi/video.rs @@ -0,0 +1,202 @@ +//! VAAPI decoding of coded video sequences. +//! +//! Where [`super::still`]-shaped decoding handles one independent picture, +//! this owns a *session*: parameter sets that persist across access units, a +//! picture buffer, and the output-ordering that reordered streams require. +//! +//! # Current scope +//! +//! **Random access points only.** Every IRAP (HEVC) and IDR (H.264) access +//! unit decodes; an access unit that references other pictures is reported as +//! [`HwDecodeError::Decode`] rather than decoded incorrectly. +//! +//! That is a real capability, not a stub: it covers keyframe extraction, +//! poster frames, timeline thumbnails and scrubbing, and it decodes All-Intra +//! camera modes (Sony XAVC S-I and XAVC HS All-I) completely, since every +//! access unit in those is a random access point. +//! +//! Inter-frame decoding is a purely internal change behind this unchanged +//! API: it adds a decoded picture buffer, picture order counts, reference +//! marking and reference lists, and removes the rejection below. No part of +//! the public seam moves, so it needs no second breaking release. + +use crate::{ + CodecConfig, DecodedFrame, HwCodec, HwDecodeError, HwVideoDecoder, StillDecodeRequest, + VideoConfig, VideoFrame, VideoPacket, +}; + +use super::hevc; + +/// Open a sequence decoder for `config`. +pub fn open(config: VideoConfig<'_>) -> Result, HwDecodeError> { + let codec = config.codec(); + let caps = super::probe(); + + let supported = match codec { + HwCodec::Hevc => caps.hevc_main || caps.hevc_main10, + HwCodec::H264 => caps.h264_any(), + HwCodec::Av1 => false, + }; + if !supported { + return Err(HwDecodeError::Unavailable { + codec, + reason: "the VAAPI driver exposes no decode entry point for this codec".to_string(), + }); + } + + // Validate the configuration record now, so a stream that cannot work + // fails at open rather than on the first packet. + let record = config.bytes().to_vec(); + let nal_length_size = match config { + VideoConfig::Hvcc(bytes) => { + hevc::parse_hvcc(bytes) + .map_err(|e| HwDecodeError::Decode { + codec, + message: format!("hvcC: {e}"), + })? + .nal_length_size + } + VideoConfig::Avcc(bytes) => { + avcc_nal_length_size(bytes).ok_or_else(|| HwDecodeError::Decode { + codec, + message: "avcC: truncated or illegal length-prefix width".to_string(), + })? + } + }; + + let still = + super::decoder(codec_for_still(codec)).ok_or_else(|| HwDecodeError::Unavailable { + codec, + reason: "the VAAPI driver exposes no decode session for this codec".to_string(), + })?; + + Ok(Box::new(VaapiVideoDecoder { + codec, + record, + nal_length_size, + still, + ready: Vec::new(), + next_poc: 0, + })) +} + +/// The still-seam codec used to decode one random access point. +/// +/// Only HEVC has a still path today; H.264 sessions are rejected at +/// [`open`] before reaching here. +fn codec_for_still(codec: HwCodec) -> HwCodec { + codec +} + +/// Read `lengthSizeMinusOne` from an `avcC` record (ISO/IEC 14496-15 +/// §5.3.3.1, byte 4, low two bits). +/// +/// `None` for a truncated record or the illegal 3-byte width. +fn avcc_nal_length_size(record: &[u8]) -> Option { + let field = record.get(4)?; + match (field & 0b11) + 1 { + n @ (1 | 2 | 4) => Some(n as usize), + _ => None, + } +} + +struct VaapiVideoDecoder { + codec: HwCodec, + /// The configuration record, kept because every decode submits it and it + /// must survive [`HwVideoDecoder::reset`]. + record: Vec, + nal_length_size: usize, + still: Box, + /// Frames decoded and awaiting collection, in output order. + ready: Vec, + /// Picture order count assigned to the next frame. + /// + /// With random access points only there is no reordering, so output order + /// is submission order and a monotonic counter is the correct picture + /// order. Real picture order counts arrive with inter-frame support. + next_poc: i32, +} + +impl VaapiVideoDecoder { + /// Whether `access_unit` is a random access point this build can decode. + fn is_random_access_point(&self, access_unit: &[u8]) -> bool { + let Ok(nals) = hevc::split_length_prefixed(access_unit, self.nal_length_size) else { + return false; + }; + nals.iter().any(|nal| match self.codec { + // HEVC: types 16..=23 are the IRAP range (BLA, IDR, CRA). + HwCodec::Hevc => hevc::nal_type(nal).is_ok_and(hevc::is_irap), + // H.264: type 5 is an IDR slice. + HwCodec::H264 => nal.first().is_some_and(|h| h & 0x1f == 5), + HwCodec::Av1 => false, + }) + } +} + +impl HwVideoDecoder for VaapiVideoDecoder { + fn codec(&self) -> HwCodec { + self.codec + } + + fn send_packet(&mut self, packet: &VideoPacket<'_>) -> Result<(), HwDecodeError> { + if !self.is_random_access_point(packet.data) { + return Err(HwDecodeError::Decode { + codec: self.codec, + message: "this build decodes random access points only; inter-frame decoding \ + needs the decoded picture buffer that is not implemented yet" + .to_string(), + }); + } + + let frame = self.decode_random_access_point(packet.data)?; + let poc = self.next_poc; + self.next_poc = self.next_poc.saturating_add(1); + self.ready.push(VideoFrame::new(frame, packet.pts, poc)); + Ok(()) + } + + fn receive_frame(&mut self) -> Result, HwDecodeError> { + if self.ready.is_empty() { + return Ok(None); + } + Ok(Some(self.ready.remove(0))) + } + + fn flush(&mut self) -> Result<(), HwDecodeError> { + // Nothing is held back: with no reordering every decoded picture is + // already in `ready`. Inter-frame support drains the picture buffer + // here instead. + Ok(()) + } + + fn reset(&mut self) { + self.ready.clear(); + self.next_poc = 0; + } +} + +impl VaapiVideoDecoder { + fn decode_random_access_point(&mut self, data: &[u8]) -> Result { + let config = match self.codec { + HwCodec::Hevc => CodecConfig::Hvcc(&self.record), + codec => { + return Err(HwDecodeError::Decode { + codec, + message: "no still-decode path exists for this codec".to_string(), + }); + } + }; + + let request = StillDecodeRequest { + config, + payload: data, + // Advisory only; the still path takes the real geometry from the + // stream's own parameter sets. + width: 0, + height: 0, + bit_depth: 0, + chroma: gamut_color::ChromaSubsampling::Cs420, + }; + self.still.decode_still(&request) + } +} diff --git a/crates/rawshift-hwdec/tests/common/mod.rs b/crates/rawshift-hwdec/tests/common/mod.rs new file mode 100644 index 0000000..3e0ef75 --- /dev/null +++ b/crates/rawshift-hwdec/tests/common/mod.rs @@ -0,0 +1,116 @@ +//! Helpers shared by the VAAPI device tests. +//! +//! These live in one place because they encode fiddly, easy-to-get-wrong +//! details — the `hvcC` record layout in particular, where putting +//! `numOfArrays` one byte off yields a record that parses but carries no +//! parameter sets. Two copies of that drift, and the second copy is where the +//! bug hides. + +#![allow(dead_code)] + +use rawshift_hwdec::DecodedFrame; +use std::process::Command; + +/// Whether any DRM render node exists (cheap pre-check for skip messages). +pub fn has_render_node() -> bool { + (128..136).any(|minor| std::path::Path::new(&format!("/dev/dri/renderD{minor}")).exists()) +} + +/// Run ffmpeg, returning false (→ skip) when it is missing or fails. +pub fn ffmpeg(args: &[&str]) -> bool { + match Command::new("ffmpeg") + .args(["-y", "-hide_banner", "-loglevel", "error"]) + .args(args) + .status() + { + Ok(status) => status.success(), + Err(_) => false, + } +} + +/// Per-plane sample variance must be non-zero for a real test pattern — +/// catches "decode succeeded but wrote a blank surface". +pub fn luma_variance(frame: &DecodedFrame) -> f64 { + let plane = &frame.planes()[0]; + let bps = frame.format().bytes_per_sample(); + let width = frame.width() as usize; + let mut samples: Vec = Vec::new(); + for row in 0..frame.height() as usize { + let start = row * plane.stride; + for px in 0..width { + let value = if bps == 1 { + f64::from(plane.data[start + px]) + } else { + f64::from(u16::from_le_bytes([ + plane.data[start + px * 2], + plane.data[start + px * 2 + 1], + ])) + }; + samples.push(value); + } + } + let mean = samples.iter().sum::() / samples.len() as f64; + samples.iter().map(|s| (s - mean).powi(2)).sum::() / samples.len() as f64 +} + +/// Split an Annex-B stream into NAL units (3- or 4-byte start codes). +pub fn split_annex_b(data: &[u8]) -> Vec> { + let mut starts = Vec::new(); + let mut i = 0; + while i + 3 <= data.len() { + if data[i] == 0 && data[i + 1] == 0 && data[i + 2] == 1 { + starts.push(i); + i += 3; + } else { + i += 1; + } + } + let mut nals = Vec::new(); + for (n, &start) in starts.iter().enumerate() { + let end = starts.get(n + 1).copied().unwrap_or(data.len()); + let mut nal = &data[start + 3..end]; + // Trim the trailing zero(s) that belong to the next 4-byte start + // code / trailing_zero_8bits. + while let Some((&0, rest)) = nal.split_last() { + nal = rest; + } + if !nal.is_empty() { + nals.push(nal.to_vec()); + } + } + nals +} + +/// Build an hvcC record (4-byte length prefixes) carrying `param_sets` and +/// a length-prefixed payload from `slices`. +pub fn build_hvcc_and_payload(nals: &[Vec]) -> (Vec, Vec) { + let mut hvcc = vec![0u8; 23]; + hvcc[0] = 1; // configurationVersion + hvcc[21] = 0x03; // lengthSizeMinusOne = 3 + let mut arrays: Vec<(u8, Vec<&[u8]>)> = Vec::new(); + let mut payload = Vec::new(); + for nal in nals { + let nal_type = (nal[0] >> 1) & 0x3f; + match nal_type { + 32..=34 => match arrays.iter_mut().find(|(t, _)| *t == nal_type) { + Some((_, list)) => list.push(nal), + None => arrays.push((nal_type, vec![nal])), + }, + _ if nal_type < 32 => { + payload.extend_from_slice(&(nal.len() as u32).to_be_bytes()); + payload.extend_from_slice(nal); + } + _ => {} // SEI etc. + } + } + hvcc[22] = arrays.len() as u8; + for (nal_type, list) in &arrays { + hvcc.push(0x80 | nal_type); + hvcc.extend_from_slice(&(list.len() as u16).to_be_bytes()); + for nal in list { + hvcc.extend_from_slice(&(nal.len() as u16).to_be_bytes()); + hvcc.extend_from_slice(nal); + } + } + (hvcc, payload) +} diff --git a/crates/rawshift-hwdec/tests/vaapi_device.rs b/crates/rawshift-hwdec/tests/vaapi_device.rs index 10f80f8..21b4195 100644 --- a/crates/rawshift-hwdec/tests/vaapi_device.rs +++ b/crates/rawshift-hwdec/tests/vaapi_device.rs @@ -14,19 +14,17 @@ #![cfg(hwdec_backend = "vaapi")] +mod common; + +use common::{build_hvcc_and_payload, ffmpeg, has_render_node, luma_variance, split_annex_b}; + use rawshift_hwdec::{ ChromaSubsampling, CodecConfig, DecodedFrame, HwBackend, HwCodec, PixelFormat, StillDecodeRequest, available_codecs, backend, decoder, }; -use std::process::Command; // ── gating helpers ────────────────────────────────────────────────────────── -/// Whether any DRM render node exists (cheap pre-check for skip messages). -fn has_render_node() -> bool { - (128..136).any(|minor| std::path::Path::new(&format!("/dev/dri/renderD{minor}")).exists()) -} - macro_rules! device_or_skip { ($codec:expr) => { match decoder($codec) { @@ -44,107 +42,8 @@ macro_rules! device_or_skip { }; } -/// Run ffmpeg, returning false (→ skip) when it is missing or fails. -fn ffmpeg(args: &[&str]) -> bool { - match Command::new("ffmpeg") - .args(["-y", "-hide_banner", "-loglevel", "error"]) - .args(args) - .status() - { - Ok(status) => status.success(), - Err(_) => false, - } -} - -/// Per-plane sample variance must be non-zero for a real test pattern — -/// catches "decode succeeded but wrote a blank surface". -fn luma_variance(frame: &DecodedFrame) -> f64 { - let plane = &frame.planes()[0]; - let bps = frame.format().bytes_per_sample(); - let width = frame.width() as usize; - let mut samples: Vec = Vec::new(); - for row in 0..frame.height() as usize { - let start = row * plane.stride; - for px in 0..width { - let value = if bps == 1 { - f64::from(plane.data[start + px]) - } else { - f64::from(u16::from_le_bytes([ - plane.data[start + px * 2], - plane.data[start + px * 2 + 1], - ])) - }; - samples.push(value); - } - } - let mean = samples.iter().sum::() / samples.len() as f64; - samples.iter().map(|s| (s - mean).powi(2)).sum::() / samples.len() as f64 -} - // ── Annex-B → hvcC + length-prefixed payload (test-side container glue) ───── -/// Split an Annex-B stream into NAL units (3- or 4-byte start codes). -fn split_annex_b(data: &[u8]) -> Vec> { - let mut starts = Vec::new(); - let mut i = 0; - while i + 3 <= data.len() { - if data[i] == 0 && data[i + 1] == 0 && data[i + 2] == 1 { - starts.push(i); - i += 3; - } else { - i += 1; - } - } - let mut nals = Vec::new(); - for (n, &start) in starts.iter().enumerate() { - let end = starts.get(n + 1).copied().unwrap_or(data.len()); - let mut nal = &data[start + 3..end]; - // Trim the trailing zero(s) that belong to the next 4-byte start - // code / trailing_zero_8bits. - while let Some((&0, rest)) = nal.split_last() { - nal = rest; - } - if !nal.is_empty() { - nals.push(nal.to_vec()); - } - } - nals -} - -/// Build an hvcC record (4-byte length prefixes) carrying `param_sets` and -/// a length-prefixed payload from `slices`. -fn build_hvcc_and_payload(nals: &[Vec]) -> (Vec, Vec) { - let mut hvcc = vec![0u8; 23]; - hvcc[0] = 1; // configurationVersion - hvcc[21] = 0x03; // lengthSizeMinusOne = 3 - let mut arrays: Vec<(u8, Vec<&[u8]>)> = Vec::new(); - let mut payload = Vec::new(); - for nal in nals { - let nal_type = (nal[0] >> 1) & 0x3f; - match nal_type { - 32..=34 => match arrays.iter_mut().find(|(t, _)| *t == nal_type) { - Some((_, list)) => list.push(nal), - None => arrays.push((nal_type, vec![nal])), - }, - _ if nal_type < 32 => { - payload.extend_from_slice(&(nal.len() as u32).to_be_bytes()); - payload.extend_from_slice(nal); - } - _ => {} // SEI etc. - } - } - hvcc[22] = arrays.len() as u8; - for (nal_type, list) in &arrays { - hvcc.push(0x80 | nal_type); - hvcc.extend_from_slice(&(list.len() as u16).to_be_bytes()); - for nal in list { - hvcc.extend_from_slice(&(nal.len() as u16).to_be_bytes()); - hvcc.extend_from_slice(nal); - } - } - (hvcc, payload) -} - /// Generate one intra HEVC frame with ffmpeg/libx265 and decode it through /// the hardware; returns `None` when the encoder is unavailable (→ skip). fn decode_generated_hevc(pix_fmt: &str, tag: &str) -> Option { diff --git a/crates/rawshift-hwdec/tests/vaapi_video_device.rs b/crates/rawshift-hwdec/tests/vaapi_video_device.rs new file mode 100644 index 0000000..df21a8f --- /dev/null +++ b/crates/rawshift-hwdec/tests/vaapi_video_device.rs @@ -0,0 +1,224 @@ +//! Device-gated VAAPI sequence-decode tests. +//! +//! Skips gracefully (eprintln + return) when the machine has no render node, +//! libva cannot be dlopen'd, the driver lacks the codec, or ffmpeg is +//! unavailable — CI without a GPU stays green. On a machine with a working +//! driver these exercise the real path: an ffmpeg-generated bitstream through +//! `video_decoder()` to `VideoFrame` pixels. + +#![cfg(hwdec_backend = "vaapi")] + +mod common; + +use common::{build_hvcc_and_payload, ffmpeg, has_render_node, luma_variance, split_annex_b}; +use rawshift_hwdec::{HwCodec, VideoConfig, VideoPacket, available_video_codecs, video_decoder}; + +// ── gating helpers ────────────────────────────────────────────────────────── + +/// Open a sequence decoder, or skip when the driver cannot provide one. +macro_rules! decoder_or_skip { + ($config:expr) => { + match video_decoder($config) { + Ok(decoder) => decoder, + Err(e) => { + eprintln!( + "Skipping VAAPI sequence test: {e} (render node present: {})", + has_render_node() + ); + return; + } + } + }; +} + +/// A one-frame all-intra HEVC bitstream, or `None` to skip. +fn hevc_keyframe() -> Option> { + let path = std::env::temp_dir().join("rawshift-hwdec-seq-intra.h265"); + if !path.exists() { + let ok = ffmpeg(&[ + "-f", + "lavfi", + "-i", + "testsrc2=size=128x96:duration=1:rate=1", + "-c:v", + "libx265", + "-x265-params", + "log-level=error:keyint=1", + "-pix_fmt", + "yuv420p", + "-f", + "hevc", + path.to_str()?, + ]); + if !ok { + eprintln!("Skipping: ffmpeg could not produce an HEVC fixture (libx265 missing?)"); + return None; + } + } + std::fs::read(&path).ok() +} + +// ── tests ─────────────────────────────────────────────────────────────────── + +#[test] +fn a_keyframe_decodes_to_real_pixels() { + let Some(stream) = hevc_keyframe() else { + return; + }; + let nals = split_annex_b(&stream); + let (hvcc, payload) = build_hvcc_and_payload(&nals); + + let mut decoder = decoder_or_skip!(VideoConfig::Hvcc(&hvcc)); + assert_eq!(decoder.codec(), HwCodec::Hevc); + + decoder + .send_packet(&VideoPacket { + data: &payload, + pts: Some(4_242), + dts: Some(4_242), + is_sync: true, + }) + .expect("a random access point must decode"); + + let frame = decoder + .receive_frame() + .expect("receive succeeds") + .expect("a frame must be ready after a keyframe"); + + assert_eq!(frame.pts(), Some(4_242), "the timestamp must be carried"); + assert_eq!(frame.frame().width(), 128); + assert_eq!(frame.frame().height(), 96); + assert!( + luma_variance(frame.frame()) > 1.0, + "decoded a blank surface, not the test pattern" + ); +} + +#[test] +fn frames_arrive_only_after_a_packet_and_flush_leaves_nothing_behind() { + let Some(stream) = hevc_keyframe() else { + return; + }; + let nals = split_annex_b(&stream); + let (hvcc, payload) = build_hvcc_and_payload(&nals); + + let mut decoder = decoder_or_skip!(VideoConfig::Hvcc(&hvcc)); + + // Nothing has been submitted: "not ready" is None, not an error. + assert!(decoder.receive_frame().expect("receive succeeds").is_none()); + + decoder + .send_packet(&VideoPacket { + data: &payload, + pts: Some(0), + dts: Some(0), + is_sync: true, + }) + .expect("decodes"); + assert!(decoder.receive_frame().expect("receive").is_some()); + + // Drained: flush must not invent a frame, and the decoder stays usable. + decoder.flush().expect("flush succeeds"); + assert!(decoder.receive_frame().expect("receive").is_none()); +} + +#[test] +fn reset_discards_pending_output_rather_than_emitting_it() { + let Some(stream) = hevc_keyframe() else { + return; + }; + let nals = split_annex_b(&stream); + let (hvcc, payload) = build_hvcc_and_payload(&nals); + + let mut decoder = decoder_or_skip!(VideoConfig::Hvcc(&hvcc)); + decoder + .send_packet(&VideoPacket { + data: &payload, + pts: Some(0), + dts: Some(0), + is_sync: true, + }) + .expect("decodes"); + + // A seek must drop what was decoded, not play it out. + decoder.reset(); + assert!( + decoder.receive_frame().expect("receive").is_none(), + "reset must discard pending output" + ); + + // ... and the session must still work afterwards. + decoder + .send_packet(&VideoPacket { + data: &payload, + pts: Some(99), + dts: Some(99), + is_sync: true, + }) + .expect("decodes after reset"); + let frame = decoder.receive_frame().expect("receive").expect("a frame"); + assert_eq!(frame.pts(), Some(99)); +} + +#[test] +fn an_inter_access_unit_is_refused_rather_than_decoded_wrongly() { + // Until the picture buffer lands, a packet that references other pictures + // must produce a clear error, never a plausible-looking wrong picture. + let Some(stream) = hevc_keyframe() else { + return; + }; + let nals = split_annex_b(&stream); + let (hvcc, _) = build_hvcc_and_payload(&nals); + + let mut decoder = decoder_or_skip!(VideoConfig::Hvcc(&hvcc)); + + // NAL type 1 (TRAIL_R) is a non-IRAP coded slice. + let mut inter = Vec::new(); + let nal = [0x02u8, 0x01, 0xd0, 0x09]; + inter.extend_from_slice(&(nal.len() as u32).to_be_bytes()); + inter.extend_from_slice(&nal); + + let err = decoder + .send_packet(&VideoPacket { + data: &inter, + pts: Some(0), + dts: Some(0), + is_sync: false, + }) + .expect_err("an inter picture must be refused"); + assert!( + err.to_string().contains("random access"), + "the error must say why: {err}" + ); +} + +#[test] +fn a_malformed_configuration_record_fails_at_open() { + // Opening validates the record, so a stream that cannot work fails here + // rather than on the first packet. + // `Box` is not Debug, so expect_err does not apply. + match video_decoder(VideoConfig::Hvcc(&[0x01, 0x02])) { + Ok(_) => panic!("a truncated hvcC must not open"), + Err(e) => assert!(!e.to_string().is_empty()), + } + + // lengthSizeMinusOne = 2 means an illegal 3-byte prefix. + let mut bad = vec![0u8; 23]; + bad[0] = 1; + bad[21] = 0x02; + assert!(video_decoder(VideoConfig::Hvcc(&bad)).is_err()); +} + +#[test] +fn discovery_agrees_with_what_can_actually_be_opened() { + // available_video_codecs() must not promise a codec that fails to open. + for &codec in available_video_codecs() { + assert_ne!( + codec, + HwCodec::Av1, + "AV1 has no sequence path and must not be advertised" + ); + } + // H.264 is sequence-only; it must never appear on the still seam. + assert!(!rawshift_hwdec::available_codecs().contains(&HwCodec::H264)); +} From 4789e62c01ab31dbd6260118467a37ae4ea64cf0 Mon Sep 17 00:00:00 2001 From: Justin Chung Date: Fri, 28 Aug 2026 22:47:52 -0400 Subject: [PATCH 11/13] feat(video): add the public API and hardware-backed decoders MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds rawshift-video-hwdec, which binds the video decoder contract to rawshift-hwdec's sequence seam, and fills in rawshift-video: VideoFile, container detection, probe, the decoder registry, available_decoders, hw_decode_available, the prelude and the feature tree. One crate covers both codecs rather than two leaves. The image side splits per format because each wraps a different codec library; here both go through the same seam and the same backend and would differ only in which configuration record they carry, so the h264 and hevc features give a build the same narrowing without the duplicate crate. The feature tree separates container from codec, which the placeholder's tiers conflated. xavc-hs and xavc-s survive as device-oriented aliases so the documented roadmap names still resolve. A container feature with no codec feature is a valid metadata-and-timeline build, the same shape as heic/avif without hw on the image side. VideoFile is not generic over its reader — the containers box their sources internally, so carrying R would constrain callers for nothing — and its bound hides behind a sealed VideoSource trait, because the bounds are the Matroska backend's rather than rawshift's and should be relaxable without a break. Narrows what the VAAPI backend advertises. The probe finds this machine's driver decodes H.264, and the earlier code advertised it, but the crate has no H.264 picture-parameter path yet, so opening a session failed after discovery had promised it would work. available_video_codecs() now reports only HEVC, open() explains that H.264 is unimplemented rather than absent, and the device test proves the promise by opening a session for every codec discovery advertises. Verified end to end on an AMD RX 7900 XT: an HEVC MP4 goes from file to container to track to a decoded 128x96 NV12 frame through the public API alone, while H.264 and backend-less targets report a capability gap that callers can act on and keep full container, track and metadata access. Refs #39 --- Cargo.lock | 16 + Cargo.toml | 2 + crates/rawshift-hwdec/src/vaapi/mod.rs | 17 +- crates/rawshift-hwdec/src/vaapi/video.rs | 15 +- .../tests/vaapi_video_device.rs | 43 ++- crates/rawshift-video-hwdec/Cargo.toml | 32 ++ crates/rawshift-video-hwdec/README.md | 27 ++ crates/rawshift-video-hwdec/src/lib.rs | 298 +++++++++++++++ crates/rawshift-video/Cargo.toml | 80 ++-- .../rawshift-video/examples/video_inspect.rs | 83 ++++ crates/rawshift-video/src/file.rs | 361 ++++++++++++++++++ crates/rawshift-video/src/lib.rs | 145 ++++++- crates/rawshift-video/src/prelude.rs | 12 + 13 files changed, 1071 insertions(+), 60 deletions(-) create mode 100644 crates/rawshift-video-hwdec/Cargo.toml create mode 100644 crates/rawshift-video-hwdec/README.md create mode 100644 crates/rawshift-video-hwdec/src/lib.rs create mode 100644 crates/rawshift-video/examples/video_inspect.rs create mode 100644 crates/rawshift-video/src/file.rs create mode 100644 crates/rawshift-video/src/prelude.rs diff --git a/Cargo.lock b/Cargo.lock index aedf809..6281f77 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1590,6 +1590,13 @@ dependencies = [ [[package]] name = "rawshift-video" version = "0.1.1" +dependencies = [ + "rawshift-core", + "rawshift-video-core", + "rawshift-video-hwdec", + "rawshift-video-isobmff", + "rawshift-video-matroska", +] [[package]] name = "rawshift-video-core" @@ -1601,6 +1608,15 @@ dependencies = [ "thiserror 2.0.18", ] +[[package]] +name = "rawshift-video-hwdec" +version = "0.1.1" +dependencies = [ + "rawshift-core", + "rawshift-hwdec", + "rawshift-video-core", +] + [[package]] name = "rawshift-video-isobmff" version = "0.1.1" diff --git a/Cargo.toml b/Cargo.toml index a8a1f73..66b9869 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -31,6 +31,7 @@ members = [ "crates/rawshift-video-isobmff", "crates/rawshift-video-matroska", "crates/rawshift-video-symphonia", + "crates/rawshift-video-hwdec", ] [workspace.package] @@ -120,6 +121,7 @@ rawshift-video-core = { path = "crates/rawshift-video-core", version = "0.1.1" } rawshift-video-isobmff = { path = "crates/rawshift-video-isobmff", version = "0.1.1" } rawshift-video-matroska = { path = "crates/rawshift-video-matroska", version = "0.1.1" } rawshift-video-symphonia = { path = "crates/rawshift-video-symphonia", version = "0.1.1" } +rawshift-video-hwdec = { path = "crates/rawshift-video-hwdec", version = "0.1.1", default-features = false } # Profiles must live at the workspace root — Cargo ignores profiles declared in # member manifests. diff --git a/crates/rawshift-hwdec/src/vaapi/mod.rs b/crates/rawshift-hwdec/src/vaapi/mod.rs index a735da2..89418b6 100644 --- a/crates/rawshift-hwdec/src/vaapi/mod.rs +++ b/crates/rawshift-hwdec/src/vaapi/mod.rs @@ -106,18 +106,21 @@ pub fn video_decoder( } /// Backend hook for [`crate::available_video_codecs`]. +/// +/// Reports only what [`video_decoder`] can actually open. The driver on this +/// machine may well decode H.264 — the probe tracks that in +/// [`Caps::h264_any`] — but this crate has no H.264 picture-parameter path +/// yet, so advertising it would promise a session that fails to open. +/// Discovery that lies is worse than discovery that is narrow. pub fn available_video_codecs() -> &'static [HwCodec] { const NONE: &[HwCodec] = &[]; const HEVC_ONLY: &[HwCodec] = &[HwCodec::Hevc]; - const H264_ONLY: &[HwCodec] = &[HwCodec::H264]; - const BOTH: &[HwCodec] = &[HwCodec::Hevc, HwCodec::H264]; let caps = probe(); - match (caps.hevc_main || caps.hevc_main10, caps.h264_any()) { - (true, true) => BOTH, - (true, false) => HEVC_ONLY, - (false, true) => H264_ONLY, - (false, false) => NONE, + if caps.hevc_main || caps.hevc_main10 { + HEVC_ONLY + } else { + NONE } } diff --git a/crates/rawshift-hwdec/src/vaapi/video.rs b/crates/rawshift-hwdec/src/vaapi/video.rs index 8f767e5..a68886c 100644 --- a/crates/rawshift-hwdec/src/vaapi/video.rs +++ b/crates/rawshift-hwdec/src/vaapi/video.rs @@ -34,13 +34,22 @@ pub fn open(config: VideoConfig<'_>) -> Result, HwDecode let supported = match codec { HwCodec::Hevc => caps.hevc_main || caps.hevc_main10, - HwCodec::H264 => caps.h264_any(), - HwCodec::Av1 => false, + // The driver may well decode H.264, but this crate has no H.264 + // picture-parameter path yet. Refusing here keeps `open` consistent + // with `available_video_codecs`, which does not advertise it. + HwCodec::H264 | HwCodec::Av1 => false, }; if !supported { + let reason = match codec { + HwCodec::H264 if caps.h264_any() => { + "the driver decodes H.264, but this build has no H.264 sequence path yet" + } + HwCodec::Av1 => "AV1 video sequences are not implemented; AV1 stills use `decoder()`", + _ => "the VAAPI driver exposes no decode entry point for this codec", + }; return Err(HwDecodeError::Unavailable { codec, - reason: "the VAAPI driver exposes no decode entry point for this codec".to_string(), + reason: reason.to_string(), }); } diff --git a/crates/rawshift-hwdec/tests/vaapi_video_device.rs b/crates/rawshift-hwdec/tests/vaapi_video_device.rs index df21a8f..8762050 100644 --- a/crates/rawshift-hwdec/tests/vaapi_video_device.rs +++ b/crates/rawshift-hwdec/tests/vaapi_video_device.rs @@ -211,14 +211,45 @@ fn a_malformed_configuration_record_fails_at_open() { #[test] fn discovery_agrees_with_what_can_actually_be_opened() { - // available_video_codecs() must not promise a codec that fails to open. + // Every advertised codec must genuinely open a session. Advertising one + // the driver supports but this crate cannot drive would make callers + // choose a path that then fails. + let Some(stream) = hevc_keyframe() else { + return; + }; + let nals = split_annex_b(&stream); + let (hvcc, _) = build_hvcc_and_payload(&nals); + for &codec in available_video_codecs() { - assert_ne!( - codec, - HwCodec::Av1, - "AV1 has no sequence path and must not be advertised" + let config = match codec { + HwCodec::Hevc => VideoConfig::Hvcc(&hvcc), + other => panic!("{other} is advertised but this test cannot build a config for it"), + }; + assert!( + video_decoder(config).is_ok(), + "{codec} is advertised yet fails to open" ); } - // H.264 is sequence-only; it must never appear on the still seam. +} + +#[test] +fn an_unimplemented_codec_is_refused_with_a_reason_not_advertised() { + // H.264 must be absent from discovery, and asking anyway must explain + // itself rather than failing opaquely. + assert!(!available_video_codecs().contains(&HwCodec::H264)); + + let mut avcc = vec![0u8; 8]; + avcc[0] = 1; + avcc[4] = 0xff; // lengthSizeMinusOne = 3 + match video_decoder(VideoConfig::Avcc(&avcc)) { + Ok(_) => panic!("H.264 has no sequence path yet and must not open"), + Err(e) => assert!( + e.to_string().contains("H.264 sequence path") + || e.to_string().contains("no decode entry point"), + "the error must say why: {e}" + ), + } + + // And H.264 must never appear on the still seam either. assert!(!rawshift_hwdec::available_codecs().contains(&HwCodec::H264)); } diff --git a/crates/rawshift-video-hwdec/Cargo.toml b/crates/rawshift-video-hwdec/Cargo.toml new file mode 100644 index 0000000..812aac8 --- /dev/null +++ b/crates/rawshift-video-hwdec/Cargo.toml @@ -0,0 +1,32 @@ +[package] +name = "rawshift-video-hwdec" +version.workspace = true +edition.workspace = true +rust-version.workspace = true +license.workspace = true +repository.workspace = true +homepage.workspace = true +description = "Hardware-backed H.264 and HEVC video decoders for rawshift" +documentation = "https://docs.rs/rawshift-video-hwdec" +keywords = ["video", "h264", "hevc", "decode", "hardware"] +categories = ["multimedia::video"] +readme = "README.md" + +[dependencies] +rawshift-core = { workspace = true } +rawshift-video-core = { workspace = true } +rawshift-hwdec = { workspace = true } + +[features] +default = [] +# One per codec, so a build takes only the decoders it needs. +h264 = [] +hevc = [] + +# Hardware backend selection, forwarded to rawshift-hwdec. These are verified +# flags: `hw` is portable and picks the native backend for the target, while +# the pinned ones fail to compile off-target on purpose. +hw = ["rawshift-hwdec/hw"] +hw-vaapi = ["rawshift-hwdec/vaapi"] +hw-videotoolbox = ["rawshift-hwdec/videotoolbox"] +hw-mediacodec = ["rawshift-hwdec/mediacodec"] diff --git a/crates/rawshift-video-hwdec/README.md b/crates/rawshift-video-hwdec/README.md new file mode 100644 index 0000000..330fa73 --- /dev/null +++ b/crates/rawshift-video-hwdec/README.md @@ -0,0 +1,27 @@ +# rawshift-video-hwdec + +H.264 and HEVC video decoders for +[rawshift](https://github.com/visualcommons/rawshift), backed by the platform +hardware decoders in `rawshift-hwdec`. + +## Why one crate for two codecs + +The image side has a leaf crate per format because each wraps a different +codec library. Here both codecs go through the same seam and the same backend, +and the two implementations differ only in which configuration record they +carry — two crates would be near-identical copies. The `h264` and `hevc` +features give a build the same narrowing a split would, without the +duplication. + +## Software decode + +There is none, deliberately. `docs/SUPPORT.md` rules out shipping software +H.264 or HEVC: both are patent-encumbered independently of any +implementation's own license. Hardware decoders carry the device OEM's +licence, so that is the only path rawshift offers. Where no backend exists the +decoders report `VideoError::HwDecoderUnavailable`, and container parsing, +track enumeration and metadata keep working regardless. + +## License + +Licensed under [MPL-2.0](../../LICENSE). diff --git a/crates/rawshift-video-hwdec/src/lib.rs b/crates/rawshift-video-hwdec/src/lib.rs new file mode 100644 index 0000000..0bb887d --- /dev/null +++ b/crates/rawshift-video-hwdec/src/lib.rs @@ -0,0 +1,298 @@ +//! H.264 and HEVC video decoders backed by platform hardware. +//! +//! Binds `rawshift-video-core`'s [`VideoDecoder`] contract to +//! `rawshift-hwdec`'s sequence seam, translating between rawshift's video +//! vocabulary and the hardware layer's. No `rawshift_hwdec` type appears in +//! this crate's public API. +//! +//! # No software fallback, deliberately +//! +//! `docs/SUPPORT.md` rules out shipping software H.264 or HEVC: both are +//! patent-encumbered independently of any implementation's own license, and +//! the OpenH264 royalty grant covers Cisco's prebuilt binary rather than a +//! from-source or clean-slate build. Hardware decoders carry the device OEM's +//! licence, so that is the only path offered. +//! +//! Where no backend exists — `wasm32`, `musl`, Windows, or a Linux host with +//! no libva — [`VideoError::HwDecoderUnavailable`] is reported, and container +//! parsing, track enumeration and metadata keep working regardless. + +#![forbid(unsafe_code)] +#![warn(missing_docs)] + +use rawshift_core::CodecId; +use rawshift_hwdec as hw; +use rawshift_video_core::{ + ColorRange, FrameDesc, FramePixelFormat, MatrixCoefficients, Packet, Plane, VideoCodecId, + VideoDecoder, VideoDecoderFactory, VideoError, VideoFrame, VideoResult, VideoTrack, +}; + +/// Register every hardware-backed decoder compiled into this build. +/// +/// Push order is priority order; within a codec there is only one backend, so +/// the order between codecs is irrelevant. +pub fn register(registry: &mut rawshift_video_core::DecoderRegistry) { + #[cfg(feature = "hevc")] + registry.push(Box::new(HwDecoderFactory::new(VideoCodecId::Hevc))); + #[cfg(feature = "h264")] + registry.push(Box::new(HwDecoderFactory::new(VideoCodecId::H264))); + let _ = registry; +} + +/// Opens hardware-backed decoders for one codec. +#[derive(Debug, Clone, Copy)] +pub struct HwDecoderFactory { + codec: VideoCodecId, +} + +impl HwDecoderFactory { + /// A factory for `codec`. + #[must_use] + pub const fn new(codec: VideoCodecId) -> Self { + Self { codec } + } + + /// The hardware layer's name for this codec, or `None` if it has no + /// sequence path. + fn hw_codec(self) -> Option { + match self.codec { + VideoCodecId::H264 => Some(hw::HwCodec::H264), + VideoCodecId::Hevc => Some(hw::HwCodec::Hevc), + _ => None, + } + } +} + +/// Build the hardware configuration for a track. +/// +/// The variant is the codec on both sides, so this cannot pair a record with +/// the wrong codec. +fn hw_config<'a>(codec: VideoCodecId, record: &'a [u8]) -> Option> { + match codec { + VideoCodecId::H264 => Some(hw::VideoConfig::Avcc(record)), + VideoCodecId::Hevc => Some(hw::VideoConfig::Hvcc(record)), + _ => None, + } +} + +impl VideoDecoderFactory for HwDecoderFactory { + fn id(&self) -> CodecId { + match self.codec { + VideoCodecId::H264 => CodecId::new("h264/hwdec"), + VideoCodecId::Hevc => CodecId::new("hevc/hwdec"), + _ => CodecId::new("unknown/hwdec"), + } + } + + fn codec(&self) -> VideoCodecId { + self.codec + } + + fn supports(&self, track: &VideoTrack) -> bool { + if track.codec != self.codec { + return false; + } + // A track with no configuration record has no parameter sets and no + // NAL length-prefix width, so no session could be opened for it. + if track.codec_config.is_empty() { + return false; + } + // Answer about this machine, not just the codec: the registry treats + // `false` as "try the next backend", so claiming support without a + // usable driver would strand the track here. + self.hw_codec() + .is_some_and(|codec| hw::available_video_codecs().contains(&codec)) + } + + fn open(&self, track: &VideoTrack) -> VideoResult> { + let record = track.codec_config.as_bytes(); + let config = hw_config(self.codec, record) + .ok_or(VideoError::UnsupportedCodec { codec: self.codec })?; + + let inner = hw::video_decoder(config).map_err(|e| match e { + hw::HwDecodeError::Unavailable { reason, .. } => { + VideoError::hw_unavailable(self.codec, reason) + } + other => VideoError::decode(self.codec, other), + })?; + + Ok(Box::new(HwVideoDecoder { + codec: self.codec, + inner, + // Resolved once: the matrix a frame is decoded with cannot depend + // on which frame it is. + matrix: track.color.resolved_matrix(track.dimensions.height), + range: track.color.range(), + })) + } +} + +/// A decoder driving one hardware sequence session. +struct HwVideoDecoder { + codec: VideoCodecId, + inner: Box, + matrix: MatrixCoefficients, + range: ColorRange, +} + +impl VideoDecoder for HwVideoDecoder { + fn codec(&self) -> VideoCodecId { + self.codec + } + + fn send_packet(&mut self, packet: &Packet) -> VideoResult<()> { + self.inner + .send_packet(&hw::VideoPacket { + data: &packet.data, + pts: packet.pts, + dts: packet.dts, + is_sync: packet.is_keyframe, + }) + .map_err(|e| VideoError::decode(self.codec, e)) + } + + fn receive_frame(&mut self) -> VideoResult> { + let Some(frame) = self + .inner + .receive_frame() + .map_err(|e| VideoError::decode(self.codec, e))? + else { + return Ok(None); + }; + self.lift(frame).map(Some) + } + + fn flush(&mut self) -> VideoResult<()> { + self.inner + .flush() + .map_err(|e| VideoError::decode(self.codec, e)) + } + + fn reset(&mut self) { + self.inner.reset(); + } +} + +impl HwVideoDecoder { + /// Translate a hardware frame into rawshift's own, so no backend type + /// crosses the boundary. + fn lift(&self, frame: hw::VideoFrame) -> VideoResult { + let (pts, poc) = (frame.pts(), frame.poc()); + let decoded = frame.into_frame(); + + let format = match decoded.format() { + hw::PixelFormat::Nv12 => FramePixelFormat::Nv12, + hw::PixelFormat::P010 => FramePixelFormat::P010, + hw::PixelFormat::I420 => FramePixelFormat::I420, + hw::PixelFormat::I010 => FramePixelFormat::I010, + }; + + let dimensions = rawshift_core::Dimensions { + width: decoded.width(), + height: decoded.height(), + }; + // The decoded frame's own range wins over the container's: the + // bitstream's VUI is what the samples were actually coded against, + // and a container that disagrees is mistagged. + let range = match decoded.range() { + hw::ColorRange::Full => ColorRange::Full, + _ => self.range, + }; + + let planes = decoded + .planes() + .iter() + .map(|p| Plane { + data: p.data.clone(), + stride: p.stride, + }) + .collect(); + + let desc = FrameDesc::new(format, dimensions, decoded.bit_depth(), range, self.matrix) + .with_timing(pts, poc); + + VideoFrame::new(self.codec, desc, planes) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use rawshift_video_core::{CicpColor, CodecConfig, DecoderRegistry, TrackId}; + + fn track(codec: VideoCodecId, config: Vec) -> VideoTrack { + let mut t = VideoTrack::new( + TrackId::new(1), + codec, + rawshift_core::Dimensions { + width: 1920, + height: 1080, + }, + rawshift_core::metadata::URational::new(1, 1000), + ); + t.codec_config = CodecConfig::new(config); + t.color = CicpColor::UNSPECIFIED; + t + } + + #[test] + fn a_factory_only_claims_its_own_codec() { + let hevc = HwDecoderFactory::new(VideoCodecId::Hevc); + assert!(!hevc.supports(&track(VideoCodecId::H264, vec![0u8; 32]))); + } + + #[test] + fn a_track_without_a_configuration_record_is_never_claimed() { + // No parameter sets and no NAL length-prefix width means no session + // could be opened, so claiming it would strand the track. + let hevc = HwDecoderFactory::new(VideoCodecId::Hevc); + assert!(!hevc.supports(&track(VideoCodecId::Hevc, Vec::new()))); + } + + #[test] + fn support_tracks_what_the_machine_can_actually_do() { + // supports() must answer about this machine, since the registry reads + // `false` as "try the next backend". + let hevc = HwDecoderFactory::new(VideoCodecId::Hevc); + let claimed = hevc.supports(&track(VideoCodecId::Hevc, vec![0u8; 32])); + let available = hw::available_video_codecs().contains(&hw::HwCodec::Hevc); + assert_eq!(claimed, available); + } + + #[test] + fn codec_ids_are_stable_and_namespaced() { + assert_eq!( + HwDecoderFactory::new(VideoCodecId::H264).id().id, + "h264/hwdec" + ); + assert_eq!( + HwDecoderFactory::new(VideoCodecId::Hevc).id().id, + "hevc/hwdec" + ); + } + + #[test] + fn registration_matches_the_compiled_features() { + let mut registry = DecoderRegistry::new(); + register(&mut registry); + + let expected = usize::from(cfg!(feature = "hevc")) + usize::from(cfg!(feature = "h264")); + assert_eq!(registry.factories().len(), expected); + } + + #[test] + fn opening_without_a_backend_reports_unavailable_not_unsupported() { + // On a machine or build with no hardware decoder this must be a + // capability gap a caller can act on, never a claim the codec is + // unknown. + let hevc = HwDecoderFactory::new(VideoCodecId::Hevc); + if hw::available_video_codecs().contains(&hw::HwCodec::Hevc) { + eprintln!("skipping: this machine has an HEVC decoder"); + return; + } + match hevc.open(&track(VideoCodecId::Hevc, vec![0u8; 32])) { + Ok(_) => panic!("no backend, yet a decoder opened"), + Err(e) => assert!(e.is_capability_gap(), "{e}"), + } + } +} diff --git a/crates/rawshift-video/Cargo.toml b/crates/rawshift-video/Cargo.toml index 26f5319..1ad0d05 100644 --- a/crates/rawshift-video/Cargo.toml +++ b/crates/rawshift-video/Cargo.toml @@ -6,49 +6,55 @@ rust-version.workspace = true license.workspace = true repository.workspace = true homepage.workspace = true -description = "Video format support for rawshift (not yet implemented)" +description = "Video decoding and metadata for rawshift: containers, tracks, timelines, and hardware-backed frame decode" documentation = "https://docs.rs/rawshift-video" -keywords = ["video", "raw", "metadata"] +keywords = ["video", "mp4", "matroska", "metadata", "decode"] categories = ["multimedia::video"] readme = "README.md" -# Parked for v1: rawshift v1 ships image only. This crate is an unpublished -# placeholder that holds the workspace slot and roadmap for post-v1 video work. -# It is excluded from the publish set and from the `rawshift` facade until it -# has an implementation to publish. See README.md and release-plz.toml. -publish = false [dependencies] +rawshift-core = { workspace = true } +rawshift-video-core = { workspace = true } +rawshift-video-isobmff = { workspace = true, optional = true } +rawshift-video-matroska = { workspace = true, optional = true } +rawshift-video-hwdec = { workspace = true, optional = true } -# Feature flags mirror the tier structure of `rawshift-image`, so the two -# libraries present a consistent surface. Video is not yet implemented: there -# are no tier-4 implementation features and no dependencies — the tiers below -# exist so the feature tree, public API, and `rawshift` facade can be laid out -# ahead of the decoder work. See the README "Video" section for the roadmap. -# -# tier 1 bundle video, full -# tier 2 format xavc-hs, xavc-s, hevc, h264, prores -# tier 3 direction xavc-hs-decode, … (decode-only for now) [features] default = [] -# ── Tier 1: bundle features ─────────────────────────────────────────────────── -full = ["video"] -video = ["xavc-hs", "xavc-s", "hevc", "h264", "prores"] - -# ── Tier 2: format features ─────────────────────────────────────────────────── -# One per roadmap format. Each is the codec+container pairing produced by a -# supported device (see the device support table in the README). -xavc-hs = ["xavc-hs-decode"] # Sony XAVC HS — H.265/HEVC in MP4 -xavc-s = ["xavc-s-decode"] # Sony XAVC S — H.264/AVC in MP4 -hevc = ["hevc-decode"] # HEVC (H.265) in QuickTime — default iPhone video -h264 = ["h264-decode"] # H.264 (AVC) in QuickTime — legacy/compatibility -prores = ["prores-decode"] # Apple ProRes in QuickTime - -# ── Tier 3: direction features ──────────────────────────────────────────────── -# Decode-only for now. Encode directions and tier-4 implementation features -# will be added when concrete decoder backends land. -xavc-hs-decode = [] -xavc-s-decode = [] -hevc-decode = [] -h264-decode = [] -prores-decode = [] +# ── Tier 1: bundles ────────────────────────────────────────────────────────── +full = ["video", "hw"] +# Every container and codec rawshift implements. +video = ["mp4", "mkv", "h264", "hevc"] + +# ── Tier 2: containers ─────────────────────────────────────────────────────── +# MP4 / M4V / QuickTime. One feature: they share a box grammar and a demuxer. +mp4 = ["dep:rawshift-video-isobmff"] +# Matroska / WebM. +mkv = ["dep:rawshift-video-matroska"] + +# ── Tier 2: codecs ─────────────────────────────────────────────────────────── +h264 = ["h264-decode"] +hevc = ["hevc-decode"] + +# ── Tier 3: directions ─────────────────────────────────────────────────────── +# Decode only. Encode is a finished trait seam with no implementations, so +# there is nothing for an `-encode` feature to gate yet. +h264-decode = ["dep:rawshift-video-hwdec", "rawshift-video-hwdec/h264"] +hevc-decode = ["dep:rawshift-video-hwdec", "rawshift-video-hwdec/hevc"] + +# ── Tier 4: hardware backends ──────────────────────────────────────────────── +# Verified flags, forwarded to rawshift-hwdec: `hw` is portable and selects the +# native backend for the target, the pinned ones fail to compile off-target. +hw = ["rawshift-video-hwdec?/hw"] +hw-vaapi = ["rawshift-video-hwdec?/hw-vaapi"] +hw-videotoolbox = ["rawshift-video-hwdec?/hw-videotoolbox"] +hw-mediacodec = ["rawshift-video-hwdec?/hw-mediacodec"] + +# ── Device-oriented aliases ────────────────────────────────────────────────── +# The roadmap's names, kept so the documented surface survives now that +# container and codec are independent axes. +xavc-hs = ["mp4", "hevc-decode"] # Sony XAVC HS — HEVC in MP4 +xavc-s = ["mp4", "h264-decode"] # Sony XAVC S — H.264 in MP4 + +serde = ["rawshift-video-core/serde"] diff --git a/crates/rawshift-video/examples/video_inspect.rs b/crates/rawshift-video/examples/video_inspect.rs new file mode 100644 index 0000000..853ccfb --- /dev/null +++ b/crates/rawshift-video/examples/video_inspect.rs @@ -0,0 +1,83 @@ +//! Inspect a video file and decode its first frame, through the public API. +//! +//! ```text +//! cargo run -p rawshift-video --features full --example video_inspect -- FILE +//! ``` +//! +//! Everything except the final decode works on every supported target with no +//! hardware decoder at all; the decode step reports why when none is +//! available rather than failing the run. + +use std::fs::File; + +use rawshift_video::prelude::*; + +fn main() -> Result<(), Box> { + let path = std::env::args().nth(1).ok_or("usage: video_inspect FILE")?; + + let mut video = VideoFile::open(File::open(&path)?)?; + + println!("container : {}", video.container()); + let md = video.metadata(); + if !md.container.brands.is_empty() { + println!("brands : {}", md.container.brands.join(", ")); + } + println!("duration : {:?}", md.container.duration); + if let Some(created) = &md.container.creation_time { + println!("created : {created}"); + } + + println!("\ntracks"); + for track in video.tracks() { + match track { + Track::Video(v) => println!( + " #{} video {} {}x{} rotation {} {:?} frames, config {} bytes", + v.id, + v.codec, + v.dimensions.width, + v.dimensions.height, + v.rotation, + v.frame_count, + v.codec_config.as_bytes().len(), + ), + Track::Audio(a) => println!( + " #{} audio {} {:?} Hz, {:?} channels", + a.id, a.codec, a.sample_rate, a.channels + ), + other => println!(" #{} {:?}", other.id(), other.kind()), + } + } + + println!("\ndecoders compiled in"); + for codec in available_decoders() { + println!(" {} ({})", codec.id, codec.version); + } + + let Some(track) = video.tracks().iter().find_map(Track::as_video) else { + println!("\nno video track to decode"); + return Ok(()); + }; + println!( + "\n{} decode available here: {}", + track.codec, + hw_decode_available(track.codec) + ); + + match video.thumbnail() { + Ok(Some(frame)) => println!( + "first frame: {}x{} {:?} {}-bit, pts {:?}", + frame.width(), + frame.height(), + frame.format(), + frame.bit_depth(), + frame.pts(), + ), + Ok(None) => println!("first frame: none decodable"), + // A missing backend is the expected outcome on most targets, and is + // reported rather than treated as a failure of the file. + Err(e) if e.is_capability_gap() => println!("first frame: unavailable — {e}"), + Err(e) => return Err(e.into()), + } + + Ok(()) +} diff --git a/crates/rawshift-video/src/file.rs b/crates/rawshift-video/src/file.rs new file mode 100644 index 0000000..4ec9b5c --- /dev/null +++ b/crates/rawshift-video/src/file.rs @@ -0,0 +1,361 @@ +//! [`VideoFile`]: opening a file and working with what is in it. + +use std::io::{Read, Seek}; + +use rawshift_video_core::{ + ContainerId, Demuxer, Packet, SeekMode, SeekTo, Track, TrackId, VideoDecoder, VideoError, + VideoFrame, VideoMetadata, VideoResult, VideoTrack, +}; + +/// A source [`VideoFile`] can read. +/// +/// Blanket-implemented for every `Read + Seek + Send + Sync + 'static`, which +/// covers `File`, `&[u8]` and `Cursor>`. +/// +/// It exists as a **sealed** trait rather than as a bare set of bounds because +/// the bounds are not rawshift's own: the Matroska backend needs `Sync` and +/// `'static`, while the ISOBMFF one does not. Naming the requirement once +/// keeps that backend detail out of rawshift's public contract, so relaxing it +/// later is not a breaking change. +pub trait VideoSource: sealed::Sealed + Read + Seek + Send + Sync + 'static {} + +impl VideoSource for T {} + +mod sealed { + /// Prevents outside implementations, so the supertrait bounds above stay + /// an implementation detail rather than a promise. + pub trait Sealed {} + impl Sealed for T {} +} + +/// How many leading bytes container detection needs. +/// +/// Enough for an ISOBMFF `ftyp` header and brand, and for the EBML header a +/// Matroska `DocType` sits in. +const SNIFF_BYTES: usize = 64; + +/// Identify the container from a prefix of a file. +/// +/// Returns `None` when nothing matches, and may name a container this build +/// cannot open — [`ContainerId`] is a vocabulary of what rawshift can +/// *recognise*, which is deliberately wider than what it can demux. +#[must_use] +pub fn detect_container(data: &[u8]) -> Option { + #[cfg(feature = "mp4")] + if let Some(container) = rawshift_video_isobmff::detect(data) { + return Some(container); + } + #[cfg(feature = "mkv")] + if let Some(container) = rawshift_video_matroska::detect(data) { + return Some(container); + } + let _ = data; + None +} + +/// What [`VideoFile::probe`] reports without committing to a full open. +#[derive(Debug, Clone, PartialEq)] +#[non_exhaustive] +pub struct VideoProbe { + /// The container. + pub container: ContainerId, + /// Every track in the file. + pub tracks: Vec, + /// File-level metadata. + pub metadata: VideoMetadata, +} + +/// An open video file. +/// +/// Not generic over the reader: the containers box their sources internally, +/// so carrying `R` in the type would constrain callers without buying +/// anything. Sources need `Read + Seek + Send`, which `File`, `&[u8]` and +/// `Cursor>` all satisfy. +pub struct VideoFile { + demuxer: Box, +} + +impl std::fmt::Debug for VideoFile { + /// Summarises the file rather than dumping it: the demuxer is a trait + /// object with nothing printable, and the track list can be long. + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("VideoFile") + .field("container", &self.container()) + .field("tracks", &self.tracks().len()) + .field( + "video_tracks", + &self.video_tracks().map(|t| t.codec).collect::>(), + ) + .finish() + } +} + +impl VideoFile { + /// Open a video file. + /// + /// The container is detected from the file's leading bytes, so the caller + /// does not name it. + /// + /// # Errors + /// + /// [`VideoError::UnsupportedContainer`] when the bytes are not a container + /// this build can open — with `detected` naming it when rawshift + /// recognised the format but has no demuxer compiled in, which is the + /// difference between "rebuild with the feature" and "not a video file". + /// [`VideoError::Container`] for a malformed file, and + /// [`VideoError::Io`] for a reader failure. + pub fn open(mut source: R) -> VideoResult { + let mut header = [0u8; SNIFF_BYTES]; + let read = read_prefix(&mut source, &mut header)?; + let detected = detect_container(&header[..read]); + + match detected { + #[cfg(feature = "mp4")] + Some(container) if container.is_isobmff() => { + let demuxer = rawshift_video_isobmff::IsoBmffDemuxer::open(source)?; + Ok(Self { + demuxer: Box::new(demuxer), + }) + } + #[cfg(feature = "mkv")] + Some(container) if container.is_matroska() => { + let demuxer = rawshift_video_matroska::MatroskaDemuxer::open(source, container)?; + Ok(Self { + demuxer: Box::new(demuxer), + }) + } + // Recognised, but this build has no demuxer for it — or rawshift + // names it and implements nothing, as with AVI and MXF. + other => Err(VideoError::UnsupportedContainer { detected: other }), + } + } + + /// Open a file just far enough to report its tracks and metadata. + /// + /// # Errors + /// + /// As [`open`](Self::open). + pub fn probe(source: R) -> VideoResult { + let file = Self::open(source)?; + Ok(VideoProbe { + container: file.container(), + tracks: file.tracks().to_vec(), + metadata: file.metadata().clone(), + }) + } + + /// The container this file is in. + #[must_use] + pub fn container(&self) -> ContainerId { + self.demuxer.container() + } + + /// Every track, in the container's own order. + #[must_use] + pub fn tracks(&self) -> &[Track] { + self.demuxer.tracks() + } + + /// The video tracks. + pub fn video_tracks(&self) -> impl Iterator { + self.tracks().iter().filter_map(Track::as_video) + } + + /// The first video track, which is what "the video" means for the + /// single-stream files cameras produce. + #[must_use] + pub fn primary_video_track(&self) -> Option<&VideoTrack> { + self.video_tracks().next() + } + + /// File-level metadata. + #[must_use] + pub fn metadata(&self) -> &VideoMetadata { + self.demuxer.metadata() + } + + /// The next packet from any track, in stored order. + /// + /// # Errors + /// + /// Container and I/O failures. + pub fn next_packet(&mut self) -> VideoResult> { + self.demuxer.next_packet() + } + + /// Seek `track` to `to`, returning the timestamp landed on. + /// + /// # Errors + /// + /// [`VideoError::NoSuchTrack`] for an unknown track, plus container and + /// I/O failures. + pub fn seek(&mut self, track: TrackId, to: SeekTo, mode: SeekMode) -> VideoResult { + self.demuxer.seek(track, to, mode) + } + + /// Open a decoder for `track`. + /// + /// # Errors + /// + /// [`VideoError::NoSuchTrack`] when the track does not exist or is not + /// video, [`VideoError::UnsupportedCodec`] when no decoder for its codec + /// was compiled in, and [`VideoError::HwDecoderUnavailable`] when one was + /// but no backend on this machine can drive it. + pub fn decoder(&self, track: TrackId) -> VideoResult> { + let video = self + .tracks() + .iter() + .find(|t| t.id() == track) + .and_then(Track::as_video) + .ok_or(VideoError::NoSuchTrack { id: track })?; + crate::registry().open(video) + } + + /// Decode the file's first frame. + /// + /// The video analogue of a still's embedded thumbnail: the cheapest + /// representative image, decoded from the first random access point rather + /// than read from a preview the container may not carry. + /// + /// `Ok(None)` when the file has no video track or no decodable frame. + /// + /// # Errors + /// + /// As [`decoder`](Self::decoder), plus decode failures. + pub fn thumbnail(&mut self) -> VideoResult> { + let Some(track) = self.primary_video_track().map(|t| t.id) else { + return Ok(None); + }; + let mut decoder = self.decoder(track)?; + + while let Some(packet) = self.demuxer.next_packet()? { + if packet.track != track || !packet.is_keyframe { + continue; + } + decoder.send_packet(&packet)?; + if let Some(frame) = decoder.receive_frame()? { + return Ok(Some(frame)); + } + // A decoder that buffered the first picture has nothing more + // coming without an end-of-stream signal. + decoder.flush()?; + if let Some(frame) = decoder.receive_frame()? { + return Ok(Some(frame)); + } + } + Ok(None) + } +} + +/// Read up to `buf.len()` bytes, then rewind. +/// +/// `read` may return fewer bytes than asked for without being at end of file, +/// so this loops; a single call would misdetect a container on a reader that +/// returns short reads. +fn read_prefix(source: &mut R, buf: &mut [u8]) -> VideoResult { + use std::io::SeekFrom; + + let mut filled = 0; + while filled < buf.len() { + match source.read(&mut buf[filled..])? { + 0 => break, + n => filled += n, + } + } + source.seek(SeekFrom::Start(0))?; + Ok(filled) +} + +#[cfg(test)] +mod tests { + use super::*; + use rawshift_video_core::VideoCodecId; + use std::io::Cursor; + + #[test] + fn unrelated_bytes_are_not_a_container() { + assert_eq!(detect_container(b"\x89PNG\r\n\x1a\n\x00\x00\x00\x00"), None); + assert_eq!(detect_container(b"not a video at all"), None); + assert_eq!(detect_container(&[]), None); + } + + #[test] + fn opening_a_non_container_reports_unsupported_with_nothing_detected() { + let err = VideoFile::open(Cursor::new(b"\x89PNG\r\n\x1a\n----".to_vec())) + .expect_err("a PNG is not a video"); + match err { + VideoError::UnsupportedContainer { detected } => { + assert_eq!(detected, None, "nothing should have been recognised"); + } + other => panic!("expected UnsupportedContainer, got {other}"), + } + } + + #[test] + fn a_recognised_but_unimplemented_container_names_itself() { + // The distinction a caller acts on: "rebuild with the feature" or + // "rawshift will never read this", versus "not a video file". + let err = VideoFile::open(Cursor::new(b"RIFF\x00\x00\x00\x00AVI LIST".to_vec())) + .expect_err("AVI is named but not implemented"); + assert!(matches!(err, VideoError::UnsupportedContainer { .. })); + } + + #[test] + fn a_short_reader_does_not_break_detection() { + /// Returns one byte at a time, as a pipe or a slow socket would. + struct Dribble(Cursor>); + impl Read for Dribble { + fn read(&mut self, buf: &mut [u8]) -> std::io::Result { + if buf.is_empty() { + return Ok(0); + } + self.0.read(&mut buf[..1]) + } + } + impl Seek for Dribble { + fn seek(&mut self, pos: std::io::SeekFrom) -> std::io::Result { + self.0.seek(pos) + } + } + + let mut source = Dribble(Cursor::new( + b"\x00\x00\x00\x18ftypisom\x00\x00\x02\x00mp41 padding padding".to_vec(), + )); + let mut buf = [0u8; SNIFF_BYTES]; + let read = read_prefix(&mut source, &mut buf).expect("prefix read"); + assert!( + read > 12, + "a one-byte-at-a-time reader must still fill the buffer" + ); + assert_eq!(detect_container(&buf[..read]), Some(ContainerId::Mp4)); + } + + #[test] + fn reading_the_prefix_leaves_the_source_rewound() { + // The demuxer that follows starts from byte zero, so detection must + // not consume the header. + let mut source = Cursor::new(b"\x00\x00\x00\x18ftypisom".to_vec()); + let mut buf = [0u8; SNIFF_BYTES]; + read_prefix(&mut source, &mut buf).expect("prefix read"); + assert_eq!(source.position(), 0); + } + + #[test] + fn hw_decode_availability_is_answerable_for_every_codec() { + // Must never panic, whatever this machine has, and must be false for + // codecs rawshift has no decoder for at all. + for codec in [VideoCodecId::H264, VideoCodecId::Hevc] { + let _ = crate::hw_decode_available(codec); + } + assert!(!crate::hw_decode_available(VideoCodecId::ProRes)); + assert!(!crate::hw_decode_available(VideoCodecId::Vp9)); + } + + #[test] + fn the_decoder_registry_only_reports_decode_direction() { + for info in crate::available_decoders() { + assert_eq!(info.direction, rawshift_core::CodecDirection::Decode); + assert!(info.id.id.contains('/'), "ids are \"format/impl\""); + } + } +} diff --git a/crates/rawshift-video/src/lib.rs b/crates/rawshift-video/src/lib.rs index 2d68c6f..b276496 100644 --- a/crates/rawshift-video/src/lib.rs +++ b/crates/rawshift-video/src/lib.rs @@ -1,10 +1,141 @@ -//! Video format support for rawshift. +//! Video decoding and metadata for [rawshift]. //! -//! Video support is **planned but not yet implemented**. This crate is a -//! placeholder so the workspace, feature tree, and public API surface can be -//! laid out ahead of the implementation. See the "Video" section of the -//! project README for the format roadmap (XAVC HS, XAVC S, HEVC, H.264, -//! ProRes). +//! The video counterpart to `rawshift-image`: a feature facade over the +//! container and decoder crates, plus the entry points that tie them together. //! -//! No video decoding or container parsing is available yet. +//! ```no_run +//! use rawshift_video::{VideoFile, Track}; +//! +//! # fn main() -> Result<(), Box> { +//! let mut file = VideoFile::open(std::fs::File::open("clip.mp4")?)?; +//! +//! println!("{}, {:?}", file.container(), file.metadata().container.duration); +//! for track in file.video_tracks() { +//! println!("{} {}x{}", track.codec, track.dimensions.width, track.dimensions.height); +//! } +//! +//! // Decoding needs a hardware backend; everything above does not. +//! if let Some(frame) = file.thumbnail()? { +//! println!("{}x{}", frame.width(), frame.height()); +//! } +//! # Ok(()) } +//! ``` +//! +//! # What works without a decoder +//! +//! Container parsing, track enumeration, the timeline, packet access and +//! metadata work on **every** target in `docs/SUPPORT.md`, including `wasm32` +//! and `musl` where no hardware decode API exists. Only frame decode needs a +//! backend; without one it reports +//! [`VideoError::HwDecoderUnavailable`], which +//! [`hw_decode_available`] lets a caller check up front. +//! +//! # Features +//! +//! - **Bundles** — `video` (every container and codec), `full` (adds `hw`). +//! - **Containers** — `mp4` (MP4/M4V/QuickTime), `mkv` (Matroska/WebM). +//! - **Codecs** — `h264`, `hevc`. +//! - **Backends** — `hw` (portable), `hw-vaapi` / `hw-videotoolbox` / +//! `hw-mediacodec` (pinned; these fail to compile off-target by design). +//! - **Device aliases** — `xavc-hs`, `xavc-s`. +//! +//! A container feature without a codec feature is a valid +//! metadata-and-timeline build, the same shape as `heic`/`avif` without `hw` +//! on the image side. +//! +//! [rawshift]: https://github.com/visualcommons/rawshift + #![forbid(unsafe_code)] +#![warn(missing_docs)] + +mod file; + +pub mod prelude; + +pub use file::{VideoFile, VideoProbe, VideoSource, detect_container}; + +// The vocabulary is re-exported wholesale: callers work in these types, and +// making them reach for `rawshift-video-core` would be a papercut with no +// benefit. +pub use rawshift_video_core::{ + AudioCodecId, AudioTrack, ChromaSubsampling, CicpColor, CodecConfig, ColorRange, ContainerId, + ContainerMetadata, Demuxer, FrameDesc, FramePixelFormat, MatrixCoefficients, Packet, Plane, + Rotation, SeekMode, SeekTo, Track, TrackId, TrackKind, VideoCodecId, VideoDecoder, + VideoDecoderFactory, VideoEncoder, VideoError, VideoFrame, VideoMetadata, VideoResult, + VideoTrack, +}; + +use rawshift_core::{CodecDirection, CodecInfo}; +use rawshift_video_core::DecoderRegistry; + +/// The decoder backends compiled into this build, in priority order. +/// +/// Built once: the set is fixed at compile time, and each factory's +/// `supports()` answers the per-machine question on every call anyway. +fn registry() -> &'static DecoderRegistry { + use std::sync::OnceLock; + static REGISTRY: OnceLock = OnceLock::new(); + REGISTRY.get_or_init(|| { + #[allow(unused_mut)] + let mut registry = DecoderRegistry::new(); + // `dep:` optional dependencies create no implicit feature, so this + // keys off the direction features that pull the crate in. + #[cfg(any(feature = "h264-decode", feature = "hevc-decode"))] + rawshift_video_hwdec::register(&mut registry); + registry + }) +} + +/// Every decoder compiled into this build. +/// +/// Reports what was *compiled in*, not what this machine can run — ask +/// [`hw_decode_available`] for that. Mirrors `rawshift_image::available_decoders`. +#[must_use] +pub fn available_decoders() -> Vec { + registry() + .factories() + .iter() + .map(|f| CodecInfo { + id: f.id(), + // Hardware decoders report the platform API's version at runtime, + // which the seam does not surface; the crate version is the honest + // stand-in and matches how the image registry handles pure-Rust + // backends. + version: env!("CARGO_PKG_VERSION").to_string(), + direction: CodecDirection::Decode, + }) + .collect() +} + +/// Whether a decoder for `codec` can actually run on this machine right now. +/// +/// False when no decoder was compiled in **and** when one was but no backend +/// is usable — a caller that only needs to know "will decoding work" should +/// ask this rather than inspecting features. Mirrors the image side's +/// `heic_hw_decode_available()` / `avif_hw_decode_available()`. +#[must_use] +pub fn hw_decode_available(codec: VideoCodecId) -> bool { + registry() + .factories() + .iter() + .any(|f| f.codec() == codec && f.supports(&probe_track(codec))) +} + +/// A minimal track used only to ask a factory whether it could decode this +/// codec on this machine. +fn probe_track(codec: VideoCodecId) -> VideoTrack { + let mut track = VideoTrack::new( + TrackId::new(0), + codec, + rawshift_core::Dimensions { + width: 1920, + height: 1080, + }, + rawshift_core::metadata::URational::new(1, 1000), + ); + // Factories refuse a track with no configuration record, since no session + // could be opened for it. A non-empty placeholder asks the question this + // function actually means: "is there a backend for this codec". + track.codec_config = CodecConfig::new(vec![0u8; 32]); + track +} diff --git a/crates/rawshift-video/src/prelude.rs b/crates/rawshift-video/src/prelude.rs new file mode 100644 index 0000000..36f103c --- /dev/null +++ b/crates/rawshift-video/src/prelude.rs @@ -0,0 +1,12 @@ +//! The curated surface, for `use rawshift_video::prelude::*;`. +//! +//! Everything needed to open a file, inspect its tracks and metadata, and +//! decode frames — without naming a container or backend crate. + +pub use crate::{VideoFile, VideoProbe, available_decoders, detect_container, hw_decode_available}; + +pub use rawshift_video_core::{ + AudioTrack, ContainerId, Demuxer, Packet, Rotation, SeekMode, SeekTo, Track, TrackId, + TrackKind, VideoCodecId, VideoDecoder, VideoError, VideoFrame, VideoMetadata, VideoResult, + VideoTrack, +}; From 43538a1eca3d52f53b9c88c3658f6545e5ad9ea6 Mon Sep 17 00:00:00 2001 From: Justin Chung Date: Fri, 28 Aug 2026 22:51:51 -0400 Subject: [PATCH 12/13] feat(rawshift): un-park video and wire it into the facade MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit rawshift-video has an implementation, so it rejoins the published set. The facade gains a `video` feature, release-plz gains the six video crates and the corrected publish order, and the justfile and README stop describing a parked crate. Video re-exports under `rawshift::video` rather than at the root: the two libraries share names — `prelude`, `Track`, and their error types — so flattening both would collide, and image stays at the root for source compatibility. The video-only CI job is rewritten for a crate that now ships. It keeps the assertion the workspace split exists for (no image crate in the video dependency tree) and adds three more: - No backend type may appear in a public signature. The crates wrap symphonia, mp4-atom and rawshift-hwdec rather than re-exporting them, so swapping a backend is not a breaking change for callers — this is what keeps that true, and it is not a property that survives on goodwill. - The facade's `video` feature pulls the crate in, and an image-only build still does not. - The containers build with no codec features at all, which is what every backend-less target gets. All four assertions were run locally before landing. Pre-existing and unchanged: `cargo publish --dry-run` cannot verify any workspace leaf, because the internal crates are not on crates.io yet and dry-run resolves them from the registry. It fails identically on master for rawshift-image-png. release-plz publishes in dependency order, which is the path that actually works. Closes #36's parking decision. Refs #39 --- .github/workflows/ci.yml | 48 ++++++++++---- CHANGELOG.md | 42 ++++++++++++ Cargo.lock | 1 + Cargo.toml | 2 +- crates/rawshift-video/README.md | 104 ++++++++++++++++++++---------- crates/rawshift-video/src/file.rs | 20 ++++-- crates/rawshift/Cargo.toml | 39 +++++------ crates/rawshift/src/lib.rs | 78 +++++++++++++++------- justfile | 8 ++- release-plz.toml | 18 ++---- 10 files changed, 248 insertions(+), 112 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 348149c..8971761 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -108,30 +108,52 @@ jobs: with: toolchain: "1.92.0" - uses: Swatinem/rust-cache@v2 - # `rawshift-video` is parked and unpublished for v1, and the facade no - # longer has a `video` feature, so this builds the crate directly. - - run: cargo build -p rawshift-video --all-features - # Assert the video build pulls in zero image crates — this is the contract - # the image/video workspace split exists to guarantee, and it must keep - # holding while the crate is parked. + - run: cargo build -p rawshift-video --features full + # The contract the image/video workspace split exists to guarantee: + # video shares only rawshift-core and rawshift-hwdec with image, and + # pulls in none of the image codec stack. - name: Assert no image crates in the video dependency tree run: | - tree=$(cargo tree -p rawshift-video --all-features --prefix none --no-dedupe) + tree=$(cargo tree -p rawshift-video --features full --prefix none --no-dedupe) echo "$tree" if echo "$tree" | grep -qiE 'rawshift-image|zune-|libheif|libwebp|img-parts|little_exif|resvg|ravif|jxl-oxide|^image '; then echo "::error::video build pulled in image crates" exit 1 fi echo "OK: video build is free of image crates" - # Assert the parked crate stays out of the facade entirely. - - name: Assert the facade does not depend on rawshift-video + # Opacity: no backend type may appear in a public signature. The crates + # wrap every backend rather than re-exporting it, so swapping one is not + # a breaking change for callers — this keeps that true. + - name: Assert backend types stay out of the public API run: | - tree=$(cargo tree -p rawshift --all-features --prefix none --no-dedupe) - if echo "$tree" | grep -q 'rawshift-video'; then - echo "::error::rawshift facade depends on the parked rawshift-video crate" + set -o pipefail + leaks=$(grep -rn --include='*.rs' \ + -E '^\s*pub (fn|struct|enum|trait|type|const)[^;{]*\b(symphonia_[a-z_]*|mp4_atom|rawshift_hwdec)::' \ + crates/rawshift-video/src crates/rawshift-video-core/src \ + crates/rawshift-video-isobmff/src crates/rawshift-video-matroska/src \ + crates/rawshift-video-hwdec/src || true) + if [ -n "$leaks" ]; then + echo "$leaks" + echo "::error::a backend type appears in a public signature" exit 1 fi - echo "OK: facade is free of rawshift-video" + echo "OK: no backend type in a public signature" + # The facade must expose video, and must still build without it. + - name: Assert the facade wires video behind its own feature + run: | + cargo tree -p rawshift --features video --prefix none --no-dedupe \ + | grep -q 'rawshift-video' \ + || { echo "::error::the facade's video feature does not pull in rawshift-video"; exit 1; } + if cargo tree -p rawshift --no-default-features --features image --prefix none --no-dedupe \ + | grep -q 'rawshift-video'; then + echo "::error::an image-only facade build pulled in rawshift-video" + exit 1 + fi + echo "OK: video is behind its own facade feature" + # Container parsing and metadata must work with no decoder at all, which + # is what every backend-less target gets. + - name: Build the containers with no codec features + run: cargo build -p rawshift-video --no-default-features --features mp4,mkv # ── Compile boundaries (issue #34) ────────────────────────────────────────── # The `hw-*` backend pins are *verified* feature flags: rawshift-hwdec holds diff --git a/CHANGELOG.md b/CHANGELOG.md index 8728a8b..140ae5e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,10 +14,52 @@ from the published [gamut](https://github.com/visualcommons/gamut) crates. dependency is a registry package. 0.x source compatibility is a non-goal (see `docs/V1_DESIGN.md`). +### Added + +#### Video (`rawshift-video`) + +`rawshift-video` is no longer a parked placeholder: it ships containers, +tracks, timelines, metadata, and hardware-backed frame decode, and is wired +into the `rawshift` facade behind a new `video` feature. Six crates make it up +— `rawshift-video-core` (the shared vocabulary), `-isobmff`, `-matroska`, +`-symphonia` (an internal bridge), `-hwdec`, and the `rawshift-video` facade. + +- **Containers** — MP4, M4V and QuickTime on `mp4-atom` with a rawshift-owned + sample index; Matroska and WebM on `symphonia-format-mkv`. Both read tracks, + timelines, rotation, colour signalling and metadata. AVI and MXF are named + in the API and return a matchable `UnsupportedContainer`. +- **Decode** — HEVC random access points through `rawshift-hwdec`, which + covers keyframe extraction, thumbnails, scrubbing, and All-Intra camera + modes end to end. H.264's seam is final and its backend path is a follow-up. +- **Everything but decode works with no hardware decoder**, on every target + in `docs/SUPPORT.md` including `wasm32` and `musl`. A missing backend + surfaces as `VideoError::HwDecoderUnavailable`, and + `hw_decode_available()` reports capability up front. + +No software H.264 or HEVC decoder ships, and none will: `docs/SUPPORT.md` +records why. FFmpeg/libav is excluded as a dependency in any form, on license +and portability grounds. + +Video is outside gamut's charter — gamut states it "will not grow video +primitives" — so `AGENTS.md` now carves video out of the Upstream-First +Policy: video containers and codecs take third-party dependencies judged by +the `PRINCIPLES.md` maturity rule instead. + ### Changed All entries below are **breaking**, grouped by area. +#### Video and hardware decode + +- `HwCodec` gains `H264`. The enum is documented as deliberately exhaustive, + so this is that decision taken explicitly; matches on it need a new arm. + H.264 is reachable only through the new sequence seam — + `decoder(HwCodec::H264)` returns `None` and `available_codecs()` never lists + it — because no rawshift still format uses H.264. +- `MetadataNamespace` is now `#[non_exhaustive]` and gains `Quicktime` and + `Matroska`. Matches on it need a `_` arm. Both changes land together while + the workspace is pre-1.0, after which new namespaces are additive forever. + #### Package boundaries - Image formats now live in 17 independently publishable diff --git a/Cargo.lock b/Cargo.lock index 6281f77..23c432b 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1287,6 +1287,7 @@ name = "rawshift" version = "0.1.1" dependencies = [ "rawshift-image", + "rawshift-video", ] [[package]] diff --git a/Cargo.toml b/Cargo.toml index 66b9869..8e70e41 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -116,7 +116,7 @@ rawshift-image-ppm = { path = "crates/rawshift-image-ppm", version = "0.1.1", de rawshift-image-svg = { path = "crates/rawshift-image-svg", version = "0.1.1", default-features = false } rawshift-image-tiff = { path = "crates/rawshift-image-tiff", version = "0.1.1", default-features = false } rawshift-image-webp = { path = "crates/rawshift-image-webp", version = "0.1.1", default-features = false } -rawshift-video = { path = "crates/rawshift-video", version = "0.1.1" } +rawshift-video = { path = "crates/rawshift-video", version = "0.1.1", default-features = false } rawshift-video-core = { path = "crates/rawshift-video-core", version = "0.1.1" } rawshift-video-isobmff = { path = "crates/rawshift-video-isobmff", version = "0.1.1" } rawshift-video-matroska = { path = "crates/rawshift-video-matroska", version = "0.1.1" } diff --git a/crates/rawshift-video/README.md b/crates/rawshift-video/README.md index 23f7b96..d931334 100644 --- a/crates/rawshift-video/README.md +++ b/crates/rawshift-video/README.md @@ -1,44 +1,78 @@ # rawshift-video -Video format support for [rawshift](https://github.com/visualcommons/rawshift). - -> **Status: parked for v1, unpublished.** rawshift v1 ships image only. No -> video code ships today, this crate is marked `publish = false`, and it is -> **not** a dependency of the [`rawshift`](https://crates.io/crates/rawshift) -> facade — there is no `video` feature to enable. The crate remains in the -> workspace to hold the roadmap below and the workspace slot for post-v1 work. -> -> It is re-added to the publish set and to the facade when it has an -> implementation to publish. Until then the feature flags below gate no code and -> should be treated as a design sketch, not a supported surface. - -## Roadmap - -The formats below are prioritised by the cameras in rawshift's supported device -list: - -| Format / Codec | Container | Status | Notes | -| -------------------- | --------------- | ------- | ---------------------------------------------- | -| XAVC HS (H.265/HEVC) | MP4 | Planned | Sony mirrorless video. | -| XAVC S (H.264/AVC) | MP4 | Planned | Sony mirrorless video. | -| Apple ProRes | QuickTime (MOV) | Planned | iPhone Pro and professional editing workflows. | -| HEVC (H.265) | QuickTime (MOV) | Planned | Default iPhone video. | -| H.264 (AVC) | QuickTime (MOV) | Planned | Legacy and compatibility video. | - -Initial work will focus on container parsing and metadata extraction, reusing -the in-repo ISOBMFF parser already used for Canon CR3 (both MP4 and QuickTime -are ISOBMFF-based). Codec-level decoding is a later milestone. +Video support for [rawshift](https://github.com/visualcommons/rawshift): +containers, tracks, timelines, metadata, and hardware-backed frame decode. + +The video counterpart to `rawshift-image` — a feature facade over the +container and decoder crates, plus the entry points that tie them together. + +```rust,no_run +use rawshift_video::prelude::*; + +let mut video = VideoFile::open(std::fs::File::open("clip.mp4")?)?; +println!("{} {:?}", video.container(), video.metadata().container.duration); + +for track in video.video_tracks() { + println!("{} {}x{} rotation {}", track.codec, track.dimensions.width, + track.dimensions.height, track.rotation); +} + +if let Some(frame) = video.thumbnail()? { + println!("{}x{}", frame.width(), frame.height()); +} +# Ok::<(), Box>(()) +``` + +## What works without a hardware decoder + +Container parsing, track enumeration, timelines, packet access and metadata +work on **every** target in `docs/SUPPORT.md`, including `wasm32` and `musl` +where no hardware decode API can exist. Only frame decode needs a backend; +without one it reports `VideoError::HwDecoderUnavailable`, and +`hw_decode_available()` lets a caller check before committing. + +## Support + +| Container | Status | Backing | +| --- | --- | --- | +| MP4 / M4V | ✅ | `mp4-atom` boxes, rawshift sample index | +| QuickTime (MOV) | ✅ | the same | +| Matroska / WebM | ✅ | `symphonia-format-mkv` | +| AVI | named, not implemented | no viable Rust crate | +| MXF | named, not implemented | the only `mxf` crate dates from 2017 | + +| Codec | Decode | Notes | +| --- | --- | --- | +| HEVC / H.265 | ✅ random access points | via `rawshift-hwdec`; see below | +| H.264 / AVC | seam ready, no backend path yet | the API is final; the VAAPI picture-parameter path is a follow-up | +| ProRes | named, not implemented | | +| AV1, VP9 | named, not implemented | AV1 stills decode through `rawshift-image` | + +**Decode scope.** Every random access point decodes, which covers keyframe +extraction, poster frames, thumbnails and scrubbing, and decodes All-Intra +camera modes (Sony XAVC S-I, XAVC HS All-I) completely. Inter-frame decoding +is a purely internal change behind the same public API — it adds a decoded +picture buffer and reference handling — so it needs no second breaking +release. + +**No software decode**, deliberately. `docs/SUPPORT.md` rules out shipping +software H.264 or HEVC: both are patent-encumbered independently of any +implementation's own license. Hardware decoders carry the device OEM's +licence. ## Feature Flags -Video features mirror the `rawshift-image` tier structure but currently gate no -code or dependencies. They are a design sketch for post-v1 work and are not -reachable through the `rawshift` facade — see the status note above. +- **Bundles** — `video` (every container and codec), `full` (adds `hw`). +- **Containers** — `mp4`, `mkv`. +- **Codecs** — `h264`, `hevc`. +- **Directions** — `h264-decode`, `hevc-decode`. Encode is a finished trait + seam with no implementations, so no `-encode` flag gates anything yet. +- **Backends** — `hw` (portable), `hw-vaapi` / `hw-videotoolbox` / + `hw-mediacodec` (pinned; these fail to compile off-target by design). +- **Device aliases** — `xavc-hs` (HEVC in MP4), `xavc-s` (H.264 in MP4). -- **Bundles** — `video` (all formats), `full`. -- **Formats** — `xavc-hs`, `xavc-s`, `hevc`, `h264`, `prores`. -- **Directions** — `xavc-hs-decode`, `xavc-s-decode`, `hevc-decode`, - `h264-decode`, `prores-decode` (decode-only for now). +A container feature without a codec feature is a valid metadata-and-timeline +build — the same shape as `heic`/`avif` without `hw` on the image side. ## License diff --git a/crates/rawshift-video/src/file.rs b/crates/rawshift-video/src/file.rs index 4ec9b5c..977d6dc 100644 --- a/crates/rawshift-video/src/file.rs +++ b/crates/rawshift-video/src/file.rs @@ -318,15 +318,25 @@ mod tests { } } - let mut source = Dribble(Cursor::new( - b"\x00\x00\x00\x18ftypisom\x00\x00\x02\x00mp41 padding padding".to_vec(), - )); + let header = b"\x00\x00\x00\x18ftypisom\x00\x00\x02\x00mp41 padding padding"; + let mut source = Dribble(Cursor::new(header.to_vec())); let mut buf = [0u8; SNIFF_BYTES]; let read = read_prefix(&mut source, &mut buf).expect("prefix read"); - assert!( - read > 12, + + assert_eq!( + read, + header.len(), "a one-byte-at-a-time reader must still fill the buffer" ); + assert_eq!( + &buf[..read], + &header[..], + "and fill it with the right bytes" + ); + + // Detection itself is feature-gated; the point above is that the + // prefix reaches it intact however the source dribbles it out. + #[cfg(feature = "mp4")] assert_eq!(detect_container(&buf[..read]), Some(ContainerId::Mp4)); } diff --git a/crates/rawshift/Cargo.toml b/crates/rawshift/Cargo.toml index ffe7b9b..ff19220 100644 --- a/crates/rawshift/Cargo.toml +++ b/crates/rawshift/Cargo.toml @@ -18,33 +18,34 @@ rustdoc-args = ["--cfg", "docsrs"] [dependencies] rawshift-image = { workspace = true, optional = true } -# `rawshift-video` is deliberately absent: it is parked and unpublished for v1. -# A facade that ships to crates.io cannot depend on an unpublished crate, and a -# `video` feature that gates zero code is a promise the workspace cannot keep. -# It is re-added when video has an implementation. +rawshift-video = { workspace = true, optional = true } # The `rawshift` facade exposes only coarse, high-level features. Cargo cannot -# auto-forward a child crate's features, so per-format flags (jpeg, png, arw, …) -# are intentionally NOT re-listed here — depend on `rawshift-image` directly for -# fine-grained control. See the README "Feature Flags" section. +# auto-forward a child crate's features, so per-format flags (jpeg, png, arw, … +# and mp4, mkv, h264, hevc) are intentionally NOT re-listed here — depend on +# `rawshift-image` or `rawshift-video` directly for fine-grained control. See +# the README "Feature Flags" section. [features] default = ["image"] # Enable still-image support with `rawshift-image`'s own default formats. image = ["dep:rawshift-image"] -# Serde derives for metadata/option types (no-op unless `image` is also on). -serde = ["rawshift-image?/serde"] +# Enable video support: containers, tracks, timelines and metadata always, and +# frame decode wherever a hardware backend exists (see docs/SUPPORT.md). +video = ["dep:rawshift-video", "rawshift-video/video"] +# Serde derives for metadata/option types. +serde = ["rawshift-image?/serde", "rawshift-video?/serde"] # Hardware still-frame decode (HEVC for HEIC, AV1 for AVIF) via rawshift-hwdec. # `hw` selects the native backend for the compile target; the `hw-*` flags pin # one explicit backend and fail the compile on any other target. All are # no-ops unless `image` is also on. See docs/SUPPORT.md for the permanent # target/API matrix. -hw = ["rawshift-image?/hw"] -hw-videotoolbox = ["rawshift-image?/hw-videotoolbox"] -hw-vaapi = ["rawshift-image?/hw-vaapi"] -hw-mediacodec = ["rawshift-image?/hw-mediacodec"] -# Everything: every image format, serde, and hardware decode (`hw` is listed -# explicitly so the facade's own flag is active, matching the design contract -# `full` = all formats + serde + experimental + `hw`; `rawshift-image/full` -# already enables it transitively). v1 is image-only — there is no `video` -# feature until `rawshift-video` has an implementation. -full = ["image", "rawshift-image/full", "serde", "hw"] +hw = ["rawshift-image?/hw", "rawshift-video?/hw"] +hw-videotoolbox = ["rawshift-image?/hw-videotoolbox", "rawshift-video?/hw-videotoolbox"] +hw-vaapi = ["rawshift-image?/hw-vaapi", "rawshift-video?/hw-vaapi"] +hw-mediacodec = ["rawshift-image?/hw-mediacodec", "rawshift-video?/hw-mediacodec"] +# Everything: every image format, every video container and codec, serde, and +# hardware decode (`hw` is listed explicitly so the facade's own flag is +# active, matching the design contract `full` = all formats + serde + +# experimental + `hw`; the child crates' `full` already enable it +# transitively). +full = ["image", "rawshift-image/full", "video", "serde", "hw"] diff --git a/crates/rawshift/src/lib.rs b/crates/rawshift/src/lib.rs index f0f8b7c..fe7a2ed 100644 --- a/crates/rawshift/src/lib.rs +++ b/crates/rawshift/src/lib.rs @@ -1,32 +1,22 @@ //! # rawshift //! -//! `rawshift` is a facade crate. It re-exports the workspace's image library -//! behind coarse feature flags so most consumers can depend on a single crate: +//! `rawshift` is a facade crate. It re-exports the workspace's image and +//! video libraries behind coarse feature flags so most consumers can depend on +//! a single crate: //! -//! - **`image`** (default) — re-exports [`rawshift-image`]: RAW decoding for -//! Sony/Canon/Nikon/Fujifilm/Adobe, standard formats (JPEG, PNG, WebP, JXL, -//! GIF, TIFF, AVIF, HEIC, SVG), the full RAW processing pipeline, and -//! encoding. Everything appears at the crate root, e.g. [`formats`], -//! [`core`], [`processing`], [`transforms`], [`prelude`]. +//! - **`image`** (default) — re-exports [`rawshift-image`] **at the crate +//! root**: RAW decoding for Sony/Canon/Nikon/Fujifilm/Adobe, standard +//! formats (JPEG, PNG, WebP, JXL, GIF, TIFF, AVIF, HEIC, SVG), the full RAW +//! processing pipeline, and encoding. Everything appears at the root, e.g. +//! [`formats`], [`core`], [`processing`], [`transforms`], [`prelude`]. +//! - **`video`** — re-exports [`rawshift-video`] under [`video`]: MP4, M4V, +//! QuickTime, Matroska and WebM containers, tracks, timelines and metadata, +//! plus hardware-backed frame decode. //! -//! ## Video is parked for v1 -//! -//! rawshift v1 ships **image only**. `rawshift-video` remains in the workspace -//! as an unpublished placeholder holding the roadmap for post-v1 work, but it -//! is not a dependency of this facade and there is no `video` feature. A -//! feature that gates zero code is a promise the workspace cannot keep, and a -//! published facade cannot depend on an unpublished crate. Both are re-added -//! when video has an implementation. -//! -//! ## Feature flags -//! -//! This facade exposes only `image`, `serde`, the hardware-decode flags -//! (`hw`, `hw-videotoolbox`, `hw-vaapi`, `hw-mediacodec`), and `full`. It does -//! **not** surface per-format flags — Cargo cannot auto-forward a child crate's -//! features, so re-listing them here would be duplicated, rot-prone state. For -//! fine-grained control (individual formats or directions) depend on -//! [`rawshift-image`] directly; its own feature tree is documented on that -//! crate. +//! Video lives under a module rather than at the root because the two +//! libraries share names — `prelude`, `Track`, and their error types — and +//! flattening both would collide. Image stays at the root for source +//! compatibility. //! //! ```no_run //! // Default `image` feature is enough for standard-format decoding. @@ -38,8 +28,46 @@ //! println!("{format:?}: {}x{}", image.width(), image.height()); //! ``` //! +//! ```no_run +//! # #[cfg(feature = "video")] { +//! use rawshift::video::prelude::*; +//! +//! let file = VideoFile::open(std::fs::File::open("clip.mp4").expect("open")).expect("read"); +//! for track in file.video_tracks() { +//! println!("{} {}x{}", track.codec, track.dimensions.width, track.dimensions.height); +//! } +//! # } +//! ``` +//! +//! ## What works without a hardware decoder +//! +//! Container parsing, track enumeration, timelines and metadata work on every +//! target in `docs/SUPPORT.md`, including those with no hardware decode API at +//! all. Only frame decode needs a backend, and its absence is reported as a +//! matchable capability gap rather than a failure. See `docs/SUPPORT.md`. +//! +//! ## Feature flags +//! +//! This facade exposes only `image`, `video`, `serde`, the hardware-decode +//! flags (`hw`, `hw-videotoolbox`, `hw-vaapi`, `hw-mediacodec`), and `full`. +//! It does **not** surface per-format flags — Cargo cannot auto-forward a +//! child crate's features, so re-listing them here would be duplicated, +//! rot-prone state. For fine-grained control (individual formats, containers, +//! codecs, or directions) depend on [`rawshift-image`] or [`rawshift-video`] +//! directly; their own feature trees are documented on those crates. +//! //! [`rawshift-image`]: https://docs.rs/rawshift-image +//! [`rawshift-video`]: https://docs.rs/rawshift-video #![forbid(unsafe_code)] #[cfg(feature = "image")] pub use rawshift_image::*; + +/// Video containers, tracks, metadata and decoding. +/// +/// The whole of [`rawshift-video`](https://docs.rs/rawshift-video); see that +/// crate for the full API. +#[cfg(feature = "video")] +pub mod video { + pub use rawshift_video::*; +} diff --git a/justfile b/justfile index 6ff4ef2..697f582 100644 --- a/justfile +++ b/justfile @@ -31,10 +31,11 @@ build-features features: build-image: cargo build -p rawshift --no-default-features --features image -# Build the parked video crate directly — the facade has no `video` feature -# while rawshift-video is parked (see crates/rawshift/Cargo.toml) +# Build the video stack. `--all-features` is wrong here for the same reason as +# `test-all`: the hw-* backend pins are mutually exclusive and would +# compile_error!. `full` is the meaningful everything-build. build-video: - cargo build -p rawshift-video --all-features + cargo build -p rawshift-video --features full # Run tests for the whole workspace (default features) — fetches fixtures first test: setup-test-data @@ -72,6 +73,7 @@ publish-check: cargo doc --workspace --no-deps cargo publish --dry-run -p rawshift-core cargo publish --dry-run -p rawshift-image + cargo publish --dry-run -p rawshift-video-core cargo publish --dry-run -p rawshift-video # Install git hooks diff --git a/release-plz.toml b/release-plz.toml index e85898f..103b84a 100644 --- a/release-plz.toml +++ b/release-plz.toml @@ -10,12 +10,12 @@ # # Publish order is dependency order, which cargo resolves on its own: # -# rawshift-core -> rawshift-hwdec -> rawshift-image-core -> the format -# crates (rawshift-image-{ifd,ljpeg,metadata,arw,…}) -> rawshift-image -# -> rawshift -# -# `rawshift-video` is parked and unpublished for v1 (`publish = false`) and so -# is absent from that order. +# rawshift-core -> rawshift-hwdec +# -> rawshift-image-core -> the image format crates +# (rawshift-image-{ifd,ljpeg,metadata,arw,…}) -> rawshift-image +# -> rawshift-video-core -> rawshift-video-{symphonia,isobmff,matroska,hwdec} +# -> rawshift-video +# -> rawshift [workspace] # The lockstep defaults, applied to *every* member so a newly added crate is @@ -62,8 +62,4 @@ git_release_enable = true git_tag_name = "v{{ version }}" # e.g. v0.1.0 (not rawshift-v0.1.0) git_release_name = "v{{ version }}" -[[package]] -name = "rawshift-video" -# Parked for v1: the crate carries `publish = false`, so release-plz must not -# try to release it either. -publish = false + From 22913572b494b42b964e9221f3892a4bd3170aa1 Mon Sep 17 00:00:00 2001 From: Justin Chung Date: Fri, 28 Aug 2026 23:05:34 -0400 Subject: [PATCH 13/13] fix(video-isobmff): bound sample-table allocations by the file length MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two denial-of-service vectors, both reachable from a malformed file and both found by reviewing the diff rather than by a failing test: - An stsz claiming `Identical { count: u32::MAX }` expanded to a four-billion-entry vector — roughly 17 GB — during open, before a byte of media was read. - A sample size of u32::MAX allocated 4 GB per read_sample call. The sample tables are attacker-controlled and a sample occupies at least one byte, so the file length is the honest ceiling on both the sample count and any single sample's extent. Samples that would run past the end of the file are dropped at index time, since indexing them could only produce a failing read later. Also fixes a doc comment that an earlier unquoted heredoc had eaten. Refs #39 --- crates/rawshift-video-isobmff/src/index.rs | 170 +++++++++++++++++++-- crates/rawshift-video-isobmff/src/movie.rs | 6 +- 2 files changed, 162 insertions(+), 14 deletions(-) diff --git a/crates/rawshift-video-isobmff/src/index.rs b/crates/rawshift-video-isobmff/src/index.rs index 95ce818..3a3a871 100644 --- a/crates/rawshift-video-isobmff/src/index.rs +++ b/crates/rawshift-video-isobmff/src/index.rs @@ -44,16 +44,23 @@ pub struct SampleIndex { } impl SampleIndex { - /// Reconstruct the index from a track's sample tables, applying . + /// Reconstruct the index from a track's sample tables, applying `edts`. + /// + /// `file_len` bounds every table against reality. The sample tables are + /// attacker-controlled, and a sample occupies at least one byte, so no + /// honest table can claim more samples than the file has bytes, and no + /// sample can extend past the end. Without that bound an `stsz` claiming + /// `u32::MAX` samples allocates tens of gigabytes before a single byte of + /// media is read. /// /// Never fails: a malformed or inconsistent table stops the walk and /// yields the samples recovered so far, because a partly-readable track is /// more useful than none and the caller cannot act on a parse error here /// anyway. #[must_use] - pub fn build(stbl: &Stbl, edts: Option<&Edts>) -> Self { + pub fn build(stbl: &Stbl, edts: Option<&Edts>, file_len: u64) -> Self { let shift = presentation_shift(edts); - let sizes = sample_sizes(&stbl.stsz.samples); + let sizes = sample_sizes(&stbl.stsz.samples, file_len); let sample_count = sizes.len(); if sample_count == 0 { return Self::default(); @@ -81,6 +88,13 @@ impl SampleIndex { let usable = offsets.len().min(times.len()).min(sample_count); let samples = (0..usable) + // A sample that runs past the end of the file is not readable, so + // indexing it would only produce a failing read later. + .filter(|&i| { + offsets[i] + .checked_add(u64::from(sizes[i])) + .is_some_and(|end| end <= file_len) + }) .map(|i| { let (dts, duration) = times[i]; let composition_shift = composition.get(i).copied().unwrap_or(0); @@ -170,11 +184,22 @@ fn presentation_shift(edts: Option<&Edts>) -> i64 { i64::try_from(media_time).unwrap_or(0) } -/// Expand `stsz` into a per-sample size list. -fn sample_sizes(samples: &StszSamples) -> Vec { +/// Expand `stsz` into a per-sample size list, bounded by the file length. +/// +/// The count is clamped because it is attacker-controlled and drives an +/// allocation: a sample occupies at least one byte, so a file cannot hold more +/// samples than it has bytes. +fn sample_sizes(samples: &StszSamples, file_len: u64) -> Vec { + let ceiling = usize::try_from(file_len).unwrap_or(usize::MAX); match samples { - StszSamples::Identical { count, size } => vec![*size; *count as usize], - StszSamples::Different { sizes } => sizes.clone(), + StszSamples::Identical { count, size } => { + vec![*size; (*count as usize).min(ceiling)] + } + StszSamples::Different { sizes } => { + let mut sizes = sizes.clone(); + sizes.truncate(ceiling); + sizes + } } } @@ -268,6 +293,9 @@ fn composition_offsets(ctts: Option<&mp4_atom::Ctts>, sample_count: usize) -> Ve #[cfg(test)] mod tests { use super::*; + /// Generous bound for tests that are not about the file-length clamp. + const TEST_FILE_LEN: u64 = 1 << 20; + use mp4_atom::{ Co64, Ctts, CttsEntry, Elst, ElstEntry, Stco, Stsc, StscEntry, Stsd, Stss, Stsz, Stts, SttsEntry, @@ -326,6 +354,7 @@ mod tests { None, ), None, + TEST_FILE_LEN, ); let offsets: Vec<_> = index.samples().iter().map(|s| s.offset).collect(); @@ -358,6 +387,7 @@ mod tests { None, ), None, + TEST_FILE_LEN, ); let offsets: Vec<_> = index.samples().iter().map(|s| s.offset).collect(); @@ -381,6 +411,7 @@ mod tests { None, ), None, + TEST_FILE_LEN, ); let dts: Vec<_> = index.samples().iter().map(|s| s.dts).collect(); @@ -420,6 +451,7 @@ mod tests { None, ), None, + TEST_FILE_LEN, ); let dts: Vec<_> = index.samples().iter().map(|s| s.dts).collect(); @@ -451,6 +483,7 @@ mod tests { None, ), None, + TEST_FILE_LEN, ); let pts: Vec<_> = index.samples().iter().map(|s| s.pts).collect(); @@ -475,6 +508,7 @@ mod tests { None, ), None, + TEST_FILE_LEN, ); assert!(index.samples().iter().all(|s| s.is_sync)); @@ -497,6 +531,7 @@ mod tests { Some(vec![1, 4]), ), None, + TEST_FILE_LEN, ); let sync: Vec<_> = index.samples().iter().map(|s| s.is_sync).collect(); @@ -523,7 +558,9 @@ mod tests { entries: vec![0x1_0000_0000], }); - let index = SampleIndex::build(&tables, None); + // Addressing past 4 GB is the whole point of co64, so this needs a + // file length that can hold it. + let index = SampleIndex::build(&tables, None, 0x1_0000_0100); assert_eq!(index.samples()[0].offset, 0x1_0000_0000); } @@ -543,6 +580,7 @@ mod tests { Some(vec![1, 4]), // sync at pts 0 and 300 ), None, + TEST_FILE_LEN, ); // Exactly on a sync sample. @@ -593,10 +631,11 @@ mod tests { // Without the edit list every timestamp is offset by the reorder // delay, and rawshift would disagree with every other tool reading the // same file. - let unshifted = SampleIndex::build(&reordered_tables(), None); + let unshifted = SampleIndex::build(&reordered_tables(), None, TEST_FILE_LEN); assert_eq!(unshifted.samples()[0].pts, 100); - let shifted = SampleIndex::build(&reordered_tables(), Some(&edts(Some(100)))); + let shifted = + SampleIndex::build(&reordered_tables(), Some(&edts(Some(100))), TEST_FILE_LEN); assert_eq!(shifted.samples()[0].pts, 0, "presentation must start at 0"); // Decode time goes negative, which is correct: the frame is decoded // before the presentation timeline begins. @@ -606,7 +645,7 @@ mod tests { #[test] fn an_empty_edit_shifts_nothing() { // media_time == -1 is a dwell with no media, not an offset. - let index = SampleIndex::build(&reordered_tables(), Some(&edts(None))); + let index = SampleIndex::build(&reordered_tables(), Some(&edts(None)), TEST_FILE_LEN); assert_eq!(index.samples()[0].pts, 100); } @@ -631,7 +670,7 @@ mod tests { ], }), }; - let index = SampleIndex::build(&reordered_tables(), Some(&cut_list)); + let index = SampleIndex::build(&reordered_tables(), Some(&cut_list), TEST_FILE_LEN); assert_eq!(index.samples()[0].pts, 100, "left untrimmed, not shifted"); } @@ -659,11 +698,113 @@ mod tests { None, ), None, + TEST_FILE_LEN, ); assert_eq!(index.len(), 2, "only the samples in the known chunk"); } + #[test] + fn a_sample_count_beyond_the_file_length_is_clamped() { + // An stsz claiming u32::MAX samples would otherwise allocate tens of + // gigabytes before a byte of media is read. A sample occupies at least + // one byte, so the file length is the honest ceiling. + let tables = stbl( + StszSamples::Identical { + count: u32::MAX, + size: 1, + }, + vec![StscEntry { + first_chunk: 1, + samples_per_chunk: 4, + sample_description_index: 1, + }], + vec![0], + vec![run(4, 100)], + None, + None, + ); + + // A 64-byte file cannot hold four billion samples. + let index = SampleIndex::build(&tables, None, 64); + assert!( + index.len() <= 64, + "clamped to the file length, got {}", + index.len() + ); + } + + #[test] + fn an_explicit_size_table_longer_than_the_file_is_truncated() { + let tables = stbl( + StszSamples::Different { + sizes: vec![1; 10_000], + }, + vec![StscEntry { + first_chunk: 1, + samples_per_chunk: 10_000, + sample_description_index: 1, + }], + vec![0], + vec![run(10_000, 1)], + None, + None, + ); + assert!(SampleIndex::build(&tables, None, 32).len() <= 32); + } + + #[test] + fn a_sample_running_past_the_end_of_the_file_is_dropped() { + // Indexing it would only produce a failing read later, and a size of + // u32::MAX would allocate four gigabytes to do it. + let tables = stbl( + StszSamples::Different { + sizes: vec![10, u32::MAX], + }, + vec![StscEntry { + first_chunk: 1, + samples_per_chunk: 2, + sample_description_index: 1, + }], + vec![0], + vec![run(2, 100)], + None, + None, + ); + + let index = SampleIndex::build(&tables, None, 100); + assert_eq!(index.len(), 1, "only the sample that fits"); + assert_eq!(index.samples()[0].size, 10); + } + + #[test] + fn an_offset_plus_size_that_overflows_is_dropped_rather_than_wrapping() { + // Only a 64-bit co64 offset can get near the top of the range; two + // 32-bit values cannot overflow a u64 between them. + let mut tables = stbl( + StszSamples::Different { + sizes: vec![u32::MAX], + }, + vec![StscEntry { + first_chunk: 1, + samples_per_chunk: 1, + sample_description_index: 1, + }], + vec![0], + vec![run(1, 100)], + None, + None, + ); + tables.stco = None; + tables.co64 = Some(Co64 { + entries: vec![u64::MAX - 1], + }); + + // Wrapping would place the sample at a small offset and read the + // wrong bytes; it must be dropped instead. + assert!(SampleIndex::build(&tables, None, u64::MAX).is_empty()); + } + #[test] fn a_zero_first_chunk_does_not_underflow() { // first_chunk is 1-based; a zero is malformed and must not wrap. @@ -681,6 +822,7 @@ mod tests { None, ), None, + TEST_FILE_LEN, ); assert!(index.is_empty()); } @@ -700,7 +842,7 @@ mod tests { None, ); tables.stco = None; - assert!(SampleIndex::build(&tables, None).is_empty()); + assert!(SampleIndex::build(&tables, None, TEST_FILE_LEN).is_empty()); } #[test] @@ -721,6 +863,7 @@ mod tests { None, ), None, + TEST_FILE_LEN, ); assert_eq!(index.len(), 4); @@ -747,6 +890,7 @@ mod tests { None, ), None, + TEST_FILE_LEN, ); assert_eq!(index.len(), 3); diff --git a/crates/rawshift-video-isobmff/src/movie.rs b/crates/rawshift-video-isobmff/src/movie.rs index 2183c8e..2822f67 100644 --- a/crates/rawshift-video-isobmff/src/movie.rs +++ b/crates/rawshift-video-isobmff/src/movie.rs @@ -105,7 +105,11 @@ impl Movie { for trak in &moov.trak { tracks.push(map_track(trak)); - indices.push(SampleIndex::build(&trak.mdia.minf.stbl, trak.edts.as_ref())); + indices.push(SampleIndex::build( + &trak.mdia.minf.stbl, + trak.edts.as_ref(), + end, + )); } let metadata = build_metadata(&moov, ftyp.as_ref(), container);