feat(hwdec): VideoToolbox backend — HEVC + AV1 still decode on macOS/iOS - #65
Open
justin13888 wants to merge 6 commits into
Open
feat(hwdec): VideoToolbox backend — HEVC + AV1 still decode on macOS/iOS#65justin13888 wants to merge 6 commits into
justin13888 wants to merge 6 commits into
Conversation
Implements the third platform backend behind the existing `HwStillDecoder` contract, so HEIC and AVIF pixel decode work end to end on Apple hardware. `build.rs` already mapped macOS/iOS to `videotoolbox` and `HwBackend` already had the variant; this fills in the module and its dispatch arms. - HEVC: `CMVideoFormatDescriptionCreateFromHEVCParameterSets` from the hvcC parameter sets, with the length-prefixed payload passed through byte for byte — no Annex-B conversion. VideoToolbox parses the SPS, so the advisory (and, from the HEIC adapter, always zero) picture size is never needed. - AV1: `av1C` carried in `SampleDescriptionExtensionAtoms`, dimensions from the sequence header. VideoToolbox *requires* the sequence header in the config atom — a record with empty `configOBUs` fails session creation at every picture size — so when the sample carries it instead, the record is rebuilt with it. - Availability is `VTIsHardwareDecodeSupported` per codec, cached once per process: AV1 is reported only on hardware that has an AV1 decode block (M3 / A17 Pro and later), while HEVC keeps working on older machines. - Sessions are created hardware-pinned on macOS via `RequireHardwareAcceleratedVideoDecoder`, then retried once without it. Apple's hardware HEVC block refuses pictures below roughly 64x64 and HEIF thumbnails are routinely 32x32, so refusing them outright would be worse than letting VideoToolbox decode that one picture itself. Availability is still gated on the hardware probe, so this only widens accepted picture sizes, never the codec list. - The session is cached and reused across pictures sharing a configuration record, which is what a grid HEIC's hundreds of tiles hit. - Decode is synchronous (no async/temporal flags, plus an explicit wait), so the payload is wrapped zero-copy with `kCFAllocatorNull`. - An explicit destination pixel-format list is requested. Left to itself a 10-bit decode natively produces `'p420'`, which appears in no public CoreVideo header and whose plane contents do not match the documented `x420` samples for the same bitstream. Offering all four documented 4:2:0 surfaces keeps the decoder on interpretable layouts while still letting it pick the range-matching one, so video/full range fidelity survives. Bindings are the generated `objc2-*` framework crates rather than a hand-written `sys.rs` — the opposite call from VAAPI, because libva must be dlopen'd to stay headless-safe whereas the Apple frameworks are guaranteed present and link normally. They are taken with `default-features = false` and only the per-header features used, so no Objective-C runtime, Metal, OpenGL or CoreAudio is compiled in. Rationale is recorded in the module docs. Claude-Session: https://claude.ai/code/session_01YTL5nD4tjuppsDGMRFFEoh
Mirrors tests/vaapi_device.rs: compiled only for a VideoToolbox build, and every test skips gracefully when the machine has no hardware decoder for the codec or when the ffmpeg fixture generator is missing. Beyond the probe/HEVC/AV1/garbage set the VAAPI suite has, three tests pin assumptions this backend actually depends on: - `videotoolbox_crops_to_the_conformance_window` — the backend does no cropping arithmetic because `CVPixelBufferGetWidth/Height` report the clean aperture. 60x36 codes as 64x40, so this fails loudly if that ever changes. - `videotoolbox_reuses_one_session_across_many_decodes` — 64 decodes through one decoder must produce identical frames, covering the cached session, the per-decode slot reset and the callback refcon's lifetime. - `videotoolbox_reports_a_colour_range` — records what the emitted surface says for tv- and pc-range sources, since `ColorRange` is read off the surface's four-character code rather than the bitstream. The Main10 test doubles as the P010 bit-alignment check: CoreVideo documents `x420` as 10 bits in the MSBs of 16, and an LSB-aligned misreading would collapse the luma variance by ~4096x. AV1 fixture generation prefers libaom-av1 `-still-picture` and falls back to libsvtav1, which is what common Homebrew ffmpeg builds actually ship. Claude-Session: https://claude.ai/code/session_01YTL5nD4tjuppsDGMRFFEoh
Adds `aarch64-apple-ios` to the compile-boundaries job. The VideoToolbox backend cfg-gates the `kVTVideoDecoderSpecification_*` keys to macOS because they are annotated `ios(17.0)` and the bindings are non-weak `extern` statics — referencing one in a build targeting iOS 14, which docs/SUPPORT.md commits to, would be a dyld launch failure for the whole app. This check is what proves that gating compiles. No hardware decode job is added: hosted runners have no dependable decode block, and every hardware test skips gracefully without one, so such a job would be green without decoding anything. That is covered by the `just test-hw` pre-release gate documented in DEVELOPMENT.md instead. The existing `--features hw --target aarch64-apple-darwin` line now compiles real Apple FFI from the Linux runner. That works because the objc2 framework crates are pure Rust `extern` declarations with no build scripts, no `links` key and no SDK dependency, and `cargo check` does not link. Claude-Session: https://claude.ai/code/session_01YTL5nD4tjuppsDGMRFFEoh
…se gate - `just test-hw` runs the backend device tests plus the end-to-end HEIC/AVIF decode against the machine's real decoder. - DEVELOPMENT.md gains "Pre-release hardware verification": CI cannot test hardware decode, so a maintainer must run `just test-hw` before merging a Release PR for any backend that changed. States plainly that hardware regressions are otherwise invisible on `master`, and warns that a run where everything skipped verifies nothing. - The crate README documents the VideoToolbox scope table, the hardware-first session creation with its narrow small-picture fallback, why an explicit destination pixel format is requested, and why this backend uses generated bindings where VAAPI hand-writes its FFI. - TEST_FIXTURES.md points at the `just` target and notes the silent-skip caveat; CHANGELOG.md gains the `*(hwdec)*` entry. Claude-Session: https://claude.ai/code/session_01YTL5nD4tjuppsDGMRFFEoh
A decoder is opened per codec, not per picture, so it can be handed pictures of different sizes — a HEIF primary image and its thumbnail are exactly that. That drives the session cache's second path, where the configuration record differs and `VTDecompressionSessionCanAcceptFormatDescription` decides whether the session is reused or rebuilt; nothing exercised it before. Alternates 128x96 and 64x64 three times so the cache has to switch in both directions rather than settling on one geometry. Claude-Session: https://claude.ai/code/session_01YTL5nD4tjuppsDGMRFFEoh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the VideoToolbox hardware still-frame decode backend, so HEIC and
AVIF pixel decode work end-to-end on Apple hardware. Before this,
hwonmacOS selected
videotoolboxinbuild.rsand then fell through to theno-backend stub, so
heic_hw_decode_available()andavif_hw_decode_available()both returnedfalse.Closes #28.
What's here
crates/rawshift-hwdec/src/videotoolbox/— the backend behind theexisting
HwStillDecodercontract.HwBackend::VideoToolboxand thebuild.rstarget mapping already existed; this fills in the module and itsdispatch arms.
CMVideoFormatDescriptionCreateFromHEVCParameterSetsfrom thehvcCparameter sets, length-prefixed payload passed through byte for byte(no Annex-B conversion). VideoToolbox parses the SPS, so the advisory
picture size — which the HEIC adapter always passes as
0— is never needed.av1CinSampleDescriptionExtensionAtoms, dimensions from thesequence header.
VTIsHardwareDecodeSupportedper codec, cached once per process.On Apple silicon older than M3 and on Intel,
available_codecs()honestlyomits
Av1while HEVC keeps working.28cad84from the draftMediaCodec branch (feat(hwdec): add Android MediaCodec backend #62), which extracts the shared bitstream parsers into
src/bitstream/. Both branches carry the same patch so whichever landsfirst, the other rebases cleanly. It is amended only with fixes that
refactor needs to be self-consistent on a non-VAAPI build (its unit tests
referenced VAAPI-only builders).
Three things real hardware forced, that are worth review attention
These were all found by running against an M3 Pro; none were predictable from
the headers.
AV1 needs the sequence header in
av1C.configOBUs. Given a record withempty
configOBUs,VTDecompressionSessionCreatefails withkVTVideoDecoderMalfunctionErrat every picture size. AVIF permits thesequence header in either the record or the sample, so when it is in the
sample the record is rebuilt with it.
Apple's hardware HEVC block refuses pictures below roughly 64×64.
Measured: 64×64 and 128×96 succeed; 32×32, 60×36, 61×37, 62×38 all fail
session creation. HEIF thumbnails are routinely 32×32 — the repo's own
fixture is
heif-enc --thumb 32— so sessions are created hardware-pinnedwith
RequireHardwareAcceleratedVideoDecoderand retried once withoutit. Availability is still gated on the hardware probe, so this widens the
accepted picture sizes only, never the codec list. The
heic_thumbnail_hw_decodes_when_listedtest fails without it.A native 10-bit decode emits
'p420', which is in no public CoreVideoheader, and whose plane contents demonstrably differ from the documented
x420samples for the same bitstream — so it cannot be safely interpreted.Rather than reverse-engineer it, the session now requests an explicit list
of all four documented 4:2:0 surfaces. The decoder still picks the one
matching the stream, so video/full range fidelity survives (verified:
tv -> Limited,pc -> Full).Deviation from the issue: no macOS CI smoke test
Issue #28 asks for a "macOS CI smoke test with fixture". This PR
deliberately does not add one, per maintainer direction. Hosted runners have
no dependable hardware decode block, and every hardware test in this repo skips
gracefully without one — so the job would report green without having decoded
anything, at real CI cost.
Instead, hardware decode becomes an explicit pre-release gate: a new
just test-hwtarget, and a "Pre-release hardware verification" section inDEVELOPMENT.mdrequiring a maintainer to run it before merging a Release PRfor any backend that changed. The tradeoff is stated plainly there: hardware
regressions are otherwise invisible on
master, so that gate is the onlybackstop.
CI gains only two
cargo checkinvocations —aarch64-apple-iosis added tocompile-boundaries, which is what proves the macOS-only gating of thekVTVideoDecoderSpecification_*keys compiles. Those symbols are annotatedios(17.0)and the bindings are non-weakexternstatics, so referencing onein an iOS 14 build (which
docs/SUPPORT.mdcommits to) would be a dyld launchfailure for the whole app.
Bindings choice
The
objc2-*framework crates, not a hand-writtensys.rs— the oppositecall from VAAPI. libva must be dlopen'd so a machine without it degrades to
"no decoder" rather than failing to start, which rules out generated bindings;
the Apple frameworks are guaranteed present and link normally, so
CFRetained<T>RAII beats hand-paired
CFRetain/CFRelease. Taken withdefault-features = falseand only the per-header features used: the onlytransitive addition to the tree is
bitflags. Rationale is recorded in thecrate README and module docs, as the issue asked.
Validation
All run on this machine (macOS 26.6, M3 Pro, ffmpeg/heif-enc/avifenc installed).
cargo fmt --all -- --checkcargo clippy --workspace --all-targets -- -D warningscargo clippy -p rawshift-hwdec --features videotoolbox --all-targets -- -D warningscargo clippy --workspace --all-targets --features rawshift-image/full -- -D warningscargo test -p rawshift-hwdec --features videotoolboxcargo test -p rawshift-hwdec(stub)cargo test --workspace --features rawshift-image/fullcargo doc --workspace --no-deps --features rawshift-image/fulljust test-hwHardware-verified behaviour, not just "didn't error":
backend() == Some(VideoToolbox),available_codecs() == [Hevc, Av1].as the P010 MSB-alignment check, since an LSB-aligned misreading would
collapse the variance by ~4096×.
needs no cropping arithmetic of its own.
cached session, per-decode slot reset, and callback refcon lifetime.
2×1 bit-exact from two hardware-decoded tiles, AVIF alpha auxiliary.
Compile boundaries:
hwchecks onaarch64-apple-darwinandaarch64-apple-ios,hw-videotoolboxpinned on macOS, andhw-vaapionmacOS still fails with the expected
compile_error!.Merge readiness
Ready for review. Not merged, per repo policy. Note that
docs/SUPPORT.mdalready listed VideoToolbox as "✅ in" — that was aspirational and is now
accurate; no change was needed there.
https://claude.ai/code/session_01YTL5nD4tjuppsDGMRFFEoh