feat(sensor-oak): stereo+IMU modality, factory calib, on-device H.264 viz, IR projector

[edgarriba]

The full sensor-oak arc, validated live on an Orin + laptop pair:

• IMU in the RGBD pipeline, rotated into the RGB optical frame when the EEPROM carries the extrinsic (imu_aligned is the only record of that — reading it backwards looks exactly like an accelerometer bias).
• Stereo pair modality (oak_open_stereo): hardware-synced left/right GRAY8 on the device capture-stamp timeline, IMU on its own queue, loud failure when CAM_B/CAM_C are missing.
• Factory stereo calibration exposure (stereo_calib) for host rectification — raw pair + calib, the host owns the rectifier.
• Optional on-device H.264 viz stream on the stereo modality (enable_h264): CAM_A + a shared encoder helper, degrading cleanly on boards without a colour camera; only the bitstream crosses the link.
• IR dot projector enabled in the RGBD pipeline.
• Build hardening: depthai as a submodule-pinned build with DEPTHAI_CMAKE_EXTRA passthrough, a $TAG-keyed install stamp so a pin bump actually rebuilds, and kornia named branch = "main" the way kornia-slam does (cargo keys a git source on the exact spec string, so a differently-spelled pin yields two Image/Tensor types in one graph).

Review rounds folded in

An adversarial correctness pass plus a full-branch simplify pass. The IMU rotation direction was verified against depthai's own source (getImuToCameraExtrinsics returns T_camera←imu, applied in that direction, matching depthai's RTABMap/Basalt VIO nodes), as were the stereo extrinsic direction/units and the FFI struct layout.

Fixed as a result:

• The crate did not compile: open_stereo gained a parameter and neither example followed. Nothing caught it because the only job that compiles anything (jetson) is if: false; CI gains a deps job (cargo metadata --locked) that at least catches a stale lock.
• oak_poll_stereo had no null-queue guard, unlike every sibling poll — calling it on an RGBD device dereferenced a null shared_ptr. One handle type serves all three modalities, so the compiler cannot prevent that; it now errors.
• The third H.264 attach site was unguarded, and it is exactly where an RGBD request lands when a board cannot produce depth — an encoder failure took the whole open down. All three sites now share one try_add_h264_encoder; the video-only path, where the stream is the output, fails with an error that says so.
• imu_hz was clamped to the BNO086's 400 Hz on the RGBD path only; out of range on the stereo path, the firmware's sensor-enable throws at pipeline.start() and the caller loses the stereo pair over an IMU rate.
• The four has_* flags were queue != nullptr restated by hand (and the pollers already tested the queue) — deleted; the queue is the capability.
• Dedup: one read_intrinsics(socket), one has_socket(), one attach_imu() (whose call order is load-bearing — the second half early-returns unless the first started the IMU), one clamp_imu_hz, one blank-calibration builder, a static_assert on the C-side struct size to complement the Rust-side layout test, and a cached epoch offset so frame and IMU stamps share one clock read.
• A present-but-zero baseline is rejected before it can reach a rectifier as NaN maps.

Verified on the Orin: cargo test -p sensor-oak --all-targets --release builds every target including both examples; tests green; cargo metadata --locked clean.

Note on the vrt pin

vrt/vrt-xfeat point at a vision-rt commit carrying the same branch = "main" kornia spec, on the line this crate already validates against. vision-rt's main has since diverged on the vrt-xfeat API (EngineProfile.inputs → input, Matcher::submit → submit_match, Descriptors removed), so moving to a main-based rev needs oakd_xfeat_stereo migrated against hardware — separate work.

🤖 Generated with Claude Code

[edgarriba]

Applied all 15 review findings (see latest commit). Hardware-verified on an x86 host with an OAK-D USB: streams + IMU at 98 Hz. One review deviation worth knowing: the proposed accuracy==UNRELIABLE gate on IMU reports had to be dropped — firmware does not populate accuracy on the *_RAW streams, so the gate silenced the entire IMU (measured 0 msgs/5 s); the zero-timestamp gate covers the default-report poison case.

[edgarriba]

Pushed 79ef691 — an adversarial review of this branch turned up 3 merge-blockers and 5 should-fixes, all addressed. The IMU alignment math (the thing most likely to be silently wrong) was verified correct against depthai's own source: getImuToCameraExtrinsics returns T_camera←imu and the shim applies it in that direction, matching depthai's RTABMap/Basalt VIO nodes. Stereo extrinsic direction/units and the FFI layout also checked clean.

Blockers

1. The crate did not compile — open_stereo gained h264 and neither example followed. Nothing caught it because the only job that compiles anything (jetson) is if: false. Both examples fixed, and CI gains a deps job (cargo metadata --locked): it can't compile on a hosted runner (build.rs builds the depthai shim), but it does catch a stale lock.
2. Two kornia graphs in one build — sensor-oak had moved to branch = "main" while vrt still pinned tag = v0.1.15-rc.5, so Image/Tensor existed twice with identical version strings, and Cargo.lock still recorded the old specs (--locked failed outright).
3. imu_hz clamp missing on the stereo path — an out-of-range rate makes the firmware's sensor-enable throw at pipeline start, failing the whole open. Losing the stereo pair over an IMU rate is the opposite of the header's degrade contract.

Worth your call — the vrt pin

Fixing (2) surfaced something structural: this crate pins vrt e1de223, which is not on vision-rt's main (it lives on feat/batch-extract-and-match / upstream/xfeat-batch), and main has since diverged on the vrt-xfeat API (EngineProfile.inputs → input, Matcher::submit → submit_match, Descriptors removed). Bumping to a main rev breaks oakd_xfeat_stereo.

So this PR pins a spec-only commit on top of e1de223 (chore/kornia-spec-on-xfeat) — one kornia spec string, zero API drift, still the rev this crate validates against. kornia/vision-rt#27 carries the main-side repin for whenever that example gets migrated against hardware.

Should-fixes: H.264 attach wrapped so it degrades instead of killing the stereo open; fps < 1 clamped like the RGBD path; the specific calibration-failure reason returned instead of a generic guess; the zero-timestamp IMU gate now counts and reports its drops (a count near half the requested rate means the gate, not the firmware, sets your IMU rate — measured 98 Hz against a 200 Hz request); next_imu reports shim failures once instead of reading as a quiet sensor; a present-but-zero baseline rejected before it reaches a rectifier as NaN maps.

Verified on the Orin: cargo test -p sensor-oak --all-targets --release builds every target including both examples, tests green, cargo metadata --locked clean.