DJI Avata 360: clips stitch (its own field numbering), colour fitted to DJI's Avata 360 LUT, lens-up levelling

[Kemerd]

Follows #1. That PR made the Avata 360's colour mode readable and added a first curve. This one makes Avata 360 clips open, stitch and reframe in osvtool, Premiere and Resolve, and refits the colour to DJI's own Avata 360 LUT.

Why Avata 360 clips had "no calibration"

The Avata 360 writes the same library messages as the Osmo 360 (DewarpParams, PanoDewarpParams, Quaternion, the scalar wrappers). It numbers the fields of ClipMeta, StreamMeta, FrameMeta and FrameMetaOfCamera differently. Read with the Osmo numbering, an Avata clip:

• has no calibration, because PanoDewarpParams is at StreamMeta 5, not 6;
• takes its IMU rate (ClipMeta 8) for the digital focal length, and flat_res for the sensor size;
• loses its attitude, which is at FrameMetaOfCamera 22 and 23, not 9 and 10.

DjmdDecoder now maps each schema's field numbers onto the canonical Osmo 360 ones before its single set of switch statements. So Types.h and the present bits mean the same on every camera. The ClipMeta header names the schema. Later frames carry no header, so MetadataTrack passes sample 0's schema to them. osvtool probe prints the schema it used, and docs/FORMAT.md has a table of where the Avata 360 differs.

This replaces #1's colour-mode post-pass. Every case its test pinned still holds, except one. camera_stream_meta.color_mode is the same library ColorMode as the Osmo's StreamMeta 4, so HLG now reads as HLG instead of Unknown.

Colour: DJI does ship an Avata 360 LUT

DJI Studio 1.0.0.24724 bundles DJI Avata 360 D-Log M to Rec.709 V1.cube (filter LOG_Avata360_DLogM). It is not the Osmo 360 file: the Osmo 360, Pocket 3 and Avata 2 LUTs are one byte-identical file, and the Avata 360's is its own. So the Avata 360 gets the same three fits the Osmo 360 has, made the same way from its own file:

Constant	Fit
kDlogMAvata360	fit_dlogm.py --from-cube: neutral RMS 0.0223 HLG code
kNativeToRec2020_Avata360	fit_primaries.py: full-cube RMS 0.0567 (Pocket 3 matrix 0.1119); every implied primary has positive luminance
kLookDjiRec709Avata360	fit_look.py (new --curve / --matrix): 1.683 dE2000 mean, p95 3.657

makeLookParams and setLook take the camera fit, so --fit avata360 renders Rec.709 through the Avata 360's own look. The Osmo 360 and the legacy fits keep the look they had. Through osvtool (the kernel, 33³ bake), Avata 360 D-Log M against DJI's Avata 360 LUT:

Fit	dE2000 mean	p95	max
avata360, this PR	1.68	3.66	6.34
osmo360 (the default)	2.52	6.50	17.3
avata360 from #1	7.35	13.0	22.7

For scale, osmo360 against the Osmo 360 file measures 1.23 / 2.80 / 6.12.

#1's curve was fitted to DJI Studio's export under a BT.709-OETF model. OpenOSV renders Rec.709 as HLG-in-709 plus a look, so on that scale the curve sat 0.5 to 1.4 stops below the others above grey. @debsahu, thank you for the groundwork. The decoder fix and the plumbing are yours; only the constants moved.

Plug-ins and LUTs

• "Avata 360" is now in the D-Log M Curve menus: the Premiere importer dialog, the Source Settings effect, and Resolve's OpenOSV Source. It is also in the saved user defaults and osvtool's token lists. PrefsDlogmFit::Avata360 = 3 is appended, so saved projects read as before. Osmo 360 stays the default.
• luts/ gains a DJI_Avata360 set: PQ and HLG in all five HDR styles, plus Rec.709 with the DJI look or OpenOSV's. gen_luts.ps1 bakes one set per DJI D-Log M LUT. The Osmo tables are byte-identical, and test_cube regenerates and anchors all 24.

The drone difference: Horizon Leveling on a lens-up rig

The Avata 360 flies with one lens up and one down, so the body's +Y (the lens axis) is vertical. Levelling decomposes yaw, pitch and roll about +Y, which puts it in gimbal lock: in a synthetic lens-up clip, a 5° tilt swung the view by more than 30°. levellingMount measures, over the whole attitude track, how vertical the lens axis is:

• If the lens axis is closer to the horizon on average, the mount is the identity, and levelling is bit-for-bit unchanged. This is tested on random handheld poses and on the sample clip.
• Otherwise it levels about the body's horizontal axis, and the heading holds to within 0.5° through the same tilts.

Tests

On Windows with MSVC, CPU only:

• Core: all 396 test cases pass (8.17M assertions), with [cuda], [gpu], [opencl] and [metal] excluded.
• Plug-ins: Premiere common 57, Source Settings 62, importer 87 (+1 skipped), reframe 215 and OFX 29 test cases pass.
• gen_luts.ps1 -Check passes.

The new tests cover the full Avata schema (calibration, focal length, attitude, schema hint), per-camera looks, the kernel against DJI's whole Avata 360 LUT ([dji-lut], skipped without DJI Studio), and the levelling mount.

Not verified here

There is no Avata 360 footage in this repository. @debsahu, when you have a moment, could you try this branch on your clips?

• osvtool probe <clip> should say Avata 360 schema, list the calibration slots, and show a sensible digital focal length.
• osvtool render <clip> --out f.png should give a clean seam. If the seam is off, the sensor-to-stream scaling or the extrinsic convention may differ from the Osmo's.
• --stab smooth-horizon should give a level horizon with a steady heading.

Which horizontal axis counts as "forward" on the aircraft is the one choice this PR cannot settle without footage. If the default view faces backwards, one sign in levellingMount flips it.

[debsahu]

Thank you for this PR, and for the kind credit.

I tested 343e77e on macOS (arm64, CPU build) with one Avata 360 D-Log M clip. It was recorded after DJI's Panorama Stitching Calibration (fw 01.00.06.00, 3840x3840 per lens, 59.94 fps, 634 frames). I also probed an older clip recorded before that calibration (fw 01.00.05.00). The visual reference is DJI Studio's 6000x3000 export of the same clip. That export had DJI's horizon stabilization on.

Short version: probe reads the schema and the lens records correctly. render and --stab both fail on these clips as they stand. With three local test patches (described below, not a proposed fix) the stitch looks right, and the levelled view faces backwards.

1. probe

• It says Avata 360 schema.
• Slots 3/4 (native_refine_far) hold the lens records, and they match what we read ourselves: slave fx 1046.98, cx/cy 1906.92/1911.23, k1 0.0674; master fx 1038.19, cx/cy 1926.75/1923.78, k1 0.0910.
• Slots 1/2 (native_refine) hold only a few fields (temperature and similar). fx/fy/cx/cy are zero. All other slots are empty.
• Digital focal length is 0.0000. ClipMeta has no field 6 on these clips (present fields are 1-5 and 8-14). The stream and the calibration are both 3840, so scale 1.0 from the calibration is fine.
• Both sets show partial, and selected calibration: none.

The cause: the Avata records have no cam_extri_q (field 28). The 4-float q (field 21) is present but all zeros. The extrinsic is only in the Euler fields 12/13/14: slave yaw -179.489, pitch 90.159, roll 0.334; master yaw 0.326, pitch 90.126, roll 1.342. Field 26 (cam_imu_extri_q) is also present: master is near identity, slave is near 180° about z.

2. render

As is, it stops with Malformed: clip carries no lens calibration (not a dual-fisheye OSV?).

To get further I patched two things locally:

• Build cam_extri_q from the Euler angles with the relation the Osmo 360's own records follow. On an Osmo 360 clip, q = qY(yaw) * qZ(roll) * qX(pitch) reproduces cam_extri_q in all 16 filled slots to within 1e-5°. For the Avata it gives master (w,x,y,z) = (0.7063, 0.7079, 0.0103, 0.0063) and slave (0.0052, 0.0011, -0.7061, 0.7081), close to the Osmo's values.
• Let the selector fall back to 3/4 when 1/2 is incomplete, as FORMAT.md says DJI's tools do.

With those, the stitch holds up:

• osvtool seam overlap NCC is 0.70-0.71 on frames 0, 300 and 600. The alternatives are far worse: scale 0.78125 gives 0.01-0.05, lens2body -0.18 to -0.20, xyzw -0.01 to 0.01. So stream scale 1.0 and wxyz/body2lens look right. Two Osmo 360 clips of mine score 0.88 and 0.71 on the same measure.
• The seam search's median shift is +0.12 to +0.20°. There is no constant offset, so I see no sign of a scaling error.
• The parallax grid gives mean disparity 0.71-0.73° and max 2.2°. Its across-seam part has almost no constant term (-0.1°) but a once-around sinusoid of about 0.95° amplitude, the same on all three frames. Negating or zeroing roll lowers NCC (0.56 and 0.65), so the Osmo relation seems the right one. I can't tell from this scene whether the sinusoid is near-field parallax or a small residual tilt between the lenses.
• At full resolution (6000x3000, levelled) the seam runs along the horizon. With the default blend, near walls and a window frame show doubling of about 12-13 px (about 0.75°). With --no-blend there is a hard step of about the same size. The step also shows on a house farther away, so part of it may be calibration rather than parallax. DJI's export shows no step at those places.
• A grey translucent smear crosses a tree near the horizon in the blended render. It is gone with --no-blend, and DJI's export does not have it. I did not find the cause.

3. --stab smooth-horizon

As is, it fails with AttitudeTrack: no usable samples. These clips have no FrameMetaOfCamera 22/23. camera_frame_meta carries fields 1-19 only, on both firmwares. The fused IMU batch is there (66 samples per frame at 4 kHz), but the sparse track reads only camera.attitude. I patched locally to use the batch's anchor sample when camera.attitude is missing.

With that and the default convention (xyzw-b2w-ny; the auto probe says inconclusive), the output is upside down and tilted. I registered the renders against DJI's frame (luma NCC 0.965). True up in the body frame is -Y to within 1.4°, so the slave lens points up. The default convention puts up 141° away from that. None of the 16 --attitude-convention choices gets closer than 36.7°, so it is not just an order or sign choice.

A lead, from one static clip only: drone_frame_meta (the field the map drops) has a quaternion at 14.4 that is mostly a 37° yaw. Some compositions of it with the IMU quaternion land within 1-2° of the true up.

Heading steadiness: I can't judge it. This clip is almost static; the attitude moves at most 2° over 10.6 s.

4. Forward axis

To test levellingMount apart from the attitude problem, I gave the renderer a fixed attitude measured from DJI's frame. The result is level to 0.13°. Its default view faces 179° away from DJI Studio's default forward. So it faces backwards, and the sign in levellingMount looks like the one to flip. In the body frame, DJI's forward is -Z (within 1.6°) and up is -Y. With +Y down, the mount currently picks +Z.

Without stabilisation, the default reframe (yaw 0, pitch 0) looks straight at the ground, since it faces the master lens and that lens points down.

5. Other

• The pre-calibration clip has the same layout and the same result. DJI's recalibration changed only a few values: slave fx 1046.81 to 1046.98 and cx 1905.44 to 1906.92; master fx 1037.90 to 1038.19 and cx 1927.24 to 1926.75; slave yaw -179.374 to -179.489. k and cam_imu_extri_q did not change.
• Colour, rough only: Rec.709 with --fit avata360 against DJI's export gives median dE2000 2.86 (osmo360: 3.02), and mean L* 50.6 against 50.7. That includes some geometric misalignment, and I don't know which LUT DJI Studio applied on export.
• I can share the three local patches as a diff if they help. They are test hacks, not a fix.

6. Correction: the attitude is in the .OSV too

An earlier version of this section said the per-frame attitude was only in the .LRF. That was wrong. I had compared a single frame packet. Over the whole clip, the OSV carries the same data:

• 3.3.2.1.3: the fused attitude batch, 42,331 samples over 10.6 s (about 4 kHz). The LRF has the same stream.
• 3.4.14.4: one quaternion per frame, 634 of them. It moves 2.04° over the clip, matching the video.
• 3.3.2.2.3: a second batch. In the OSV it appears in one frame only; in the LRF it is in every frame. It agrees with the other two to within 0.05°.

So nothing needs the LRF. "AttitudeTrack: no usable samples" may come from reading fields 22/23 rather than from missing data. Sorry for the noise.

I've left out screenshots because the footage shows my home and my neighbours' houses. I can send frames privately if that would help.