Move reviewers' confirmed occurrences and identifications between Antenna instances

[mihow]

Summary

When a project is re-imported or re-processed, every capture, detection and occurrence gets a new id, and the work reviewers did on the old rows is lost: which detections they confirmed as one individual, and which taxon they identified each occurrence as. This PR adds a way to carry that work from one Antenna database to another. A project's confirmed occurrences and identifications are exported to a bundle that names everything by stable facts (capture path, time and station, bounding box, detector name, taxon GBIF key, reviewer email), and that bundle can be replayed onto a fresh project where the ids are different. The same is done for detection feature vectors, which are expensive to recompute. This is the replay route chosen over migrating the draft tables for the model-outputs schema change (see #1412 and #1208).

List of Changes

#	Change (what it does for an operator)	How (implementation)
1	Export a project's confirmed occurrences and identifications to a portable JSON bundle.	export_validated_occurrences --project N --output file.json; models_future/validated_occurrences.py builds a versioned bundle keyed by natural keys only (DetectionKey, TaxonKey, user email).
2	Replay a bundle onto another project; dry run by default, with a match report.	import_validated_occurrences --project N --input file.json [--execute] [--iou-threshold 0.7] [--create-missing-detections] [--report out.json]. The dry run only reads and prints exact / overlap / missing counts.
3	Find detections again without their ids, exactly or by overlap.	models_future/detection_matching.py: capture by path (fallback station + timestamp), box exact within 0.5 px, else highest IoU above the threshold; each target detection is claimed once. Two queries per batch.
4	Rebuild confirmed occurrences with the same safeguards the review interface has.	Regrouping uses tracks.detach_detection and tracks.add_detections, so chain links, track statistics and session counts stay right. An occurrence with a missing detection is reported as partial and never confirmed.
5	Keep the original reviewer and time on each confirmation, and make sure the review row exists.	verify_grouping() gains an optional timestamp for a confirmation made elsewhere; confirm_grouping_as_of() is the import's one call to it. "Already confirmed" means the cached grouping_verified_at/by match and, where the schema has ValidationReview, a standing grouping review by that person at that time exists; a cache without its review is re-confirmed through verify_grouping so the row gets written.
6	Re-create identifications under their original user and timestamp.	Oldest first so Identification.save() replays the withdraw behaviour; created_at restored afterwards; "agreed with" links resolved by prediction taxon and by bundle ref. Missing users or taxa are reported and skipped, never created.
7	Running the import twice changes nothing.	Already grouped and confirmed records report as unchanged; identifications that exist with the same user, taxon and time are skipped.
8	Export and import detection feature vectors by detection key, from both places they live.	export_embeddings --project N --algorithm KEY --output dir/ writes one vectors.<source>.npy per store (DetectionEmbedding rows and the classifier features on Classification.features_2048, which differ in length), plus index.csv and manifest.json; import_embeddings matches the keys and writes embeddings as DetectionEmbedding rows and classifier features back onto the matching classification, reporting the ones whose detection has no such classification.
9	Tests with synthetic data.	ami/main/test_validated_occurrences.py: round trip, overlap fallback and a stricter threshold, partial not confirmed, hand-drawn box recreated, idempotent re-run, dry run writes nothing, missing user or taxon skipped, cache-only confirmation re-confirmed, vector round trips for both stores. Two tests skip on this base and run on the model-outputs branch (review row written; embedding rows written).

Detailed description

What the bundle is not. It does not carry occurrences, captures or detections themselves, only what people decided about them. The target project must already hold the captures and have been processed by a detector; the bundle finds the boxes again. An unconfirmed occurrence that was identified keeps whatever grouping the target has; its identifications land on the occurrence holding most of its detections and the report says over how many occurrences they were spread.

Where this sits in the outputs stack. The model-outputs work lands as a stack: main → results and reviews → detection embeddings → the tracking backend (#1272) → the tracking UI (#1432). This PR builds on #1432 and moves onto its rebased head when that is pushed. Because a branch may have any subset of those tables and columns, the code here finds out what the schema has instead of assuming it: the vector export reads DetectionEmbedding rows where that table exists and classifier feature vectors where classifications have a features_2048 column, and notes in its manifest any store it could not read; the import writes embedding rows through the model's own insert-mostly store() writer where it has one (a row holding the same vector is left alone, a different one replaced), sets key and project when the model has those fields, leaves job null (a replayed vector has no run of its own on the target), and counts rows it has nowhere to put as skipped_no_field rather than failing. The "already confirmed" check looks for a grouping review only when the review model exists and has the grouping aspect, which arrives with the tracking backend, and otherwise uses the cached fields alone.

Checked against the stack's branches. On this base: 32 tests pass (2 skip, for tables it lacks). The vector modules alone, copied onto the embeddings branch (DetectionEmbedding in its final shape, no features_2048): the embedding round trip passes, writing through store() with key and project set, and the manifest notes the missing feature column. makemigrations --check reports no changes on both.

Measured on a copy of a partner database (same data on both sides, so exact matches are expected): a read-only dry run of a bundle with 183 occurrences, 49 of them confirmed, matched 1,626 of 1,626 detections exactly (all 1,261 in the confirmed occurrences) and reported 182 records as already in place. The one record reported as changed was an identification made after the copy was taken, which is the right answer. The overlap fallback has not yet been exercised against a real detector rerun; the 0.7 default is the plan's starting point, to be tuned from the IoU distribution in the report.

How to test

docker compose run --rm django python manage.py test ami.main.test_validated_occurrences

On a stack with data:

python manage.py export_validated_occurrences --project <source> --output bundle.json
python manage.py import_validated_occurrences --project <target> --input bundle.json --report report.json   # dry run
python manage.py import_validated_occurrences --project <target> --input bundle.json --execute
python manage.py import_validated_occurrences --project <target> --input bundle.json --execute   # second run: all unchanged

Refs #1412, #1208. Based on #1432; moves onto its rebased head once that is pushed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01C7Xf6VPbwWtTumhjjF15g8

[netlify]

✅ Deploy Preview for antenna-preview canceled.

Name	Link
🔨 Latest commit	d374bf9
🔍 Latest deploy log	https://app.netlify.com/projects/antenna-preview/deploys/6ac010d3d12827000814de54

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[mihow]

Claude says: The merge order and plan for tracking, agreed with the owner today, are on #1412: #1412 (comment)

This PR's place: it waits for the review PRs that follow #1469.