Add tracking settings that avoid merging different insects, calibrate them per image model, and tune them against confirmed tracks

[mihow]

Summary

People reviewing moth tracks told us what matters most: a track that lumps two insects together is much worse than a missed link, tracks of moths that sit still for a long time are very valuable, and even a short correct track helps. Tracking today has one knob, the cost threshold, and its default of 0.2 leaves most real tracks in pieces.

This PR gives tracking the settings that those priorities call for, and a way to measure them before anyone changes a default. Operators (staff) can now weigh each part of the matching cost, stop links between detections that a classifier confidently calls different species, make movement tolerance stricter on a crowded sheet, and link still insects first. Every new setting is off by default, and a default run makes exactly the links it made before; a test pins that. The evaluation command can now score a whole grid of settings in one read-only run and write a table that puts wrong merges first.

Stacked on #1439 (itself on #1432 and #1272) and merges after it, because tracking now reads vectors through #1439's embedding reader and records each run in its history. No default changes here.

Tests on this head (240722cd), run locally in the CI compose stack, since GitHub runs the backend tests only on PRs based on main: makemigrations --check reports no changes, and the full backend suite ran 1,095 tests, OK (2 skipped). An earlier merge of #1439 (before 0966b634) hit one import error in test_tracking_cost_terms, for a helper #1439 removed; that import is fixed on this branch. Sweep results on a partner's evaluation project are below, including a general-purpose image model and settings to calibrate the appearance term per model; which defaults to change is a decision for after review.

List of Changes

Change (effect)	How	Notes
1. Staff can weight each term of the matching cost (appearance, overlap, size, distance).	appearance_weight, iou_weight, size_weight, distance_weight on TrackingConfig; weighted_cost() equals total_cost() bit for bit at 1.0.	No time term exists (only consecutive captures are paired), so none was added.
2. Tracking can refuse, or penalise, a link between two detections whose confident top labels name unrelated taxa. A genus and one of its species are not a conflict.	species_gate (off/penalty/forbid), species_gate_min_score, species_gate_penalty; labels read once per session (top_labels, two queries), not per pair.	Label = highest-score terminal classification per detection, excluding rows copied from another classification (applied_to).
3. On a crowded sheet, tracking tolerates less movement.	activity_scaling (log or steps) multiplies the distance term by a function of the larger detection count of the two captures.
4. Insects that sit still can be linked first, so a moving insect cannot take their place; optionally even when a crop has no embedding.	stationary_first pass in choose_links() with its own threshold, max centre shift and min IoU, then the normal pass; still one link per detection per side.
5. The evaluation command scores many settings in one read-only run.	evaluate_tracking --sweep (list or grid), --config, --vectors-file (.npz), --output-dir (Markdown + JSON), --per-track-csv. Pair terms are computed once per session and re-scored per setting.	168 settings over three one-hour sessions (about 12k detections) ran in 47 s locally.
6. The scores put precision first.	Exact recovery and completeness over tracks of 2+ detections (single-detection tracks reported apart, never counted as exact), cross-species merges, session-wide occurrences and distinct determinations before/after, predicted track lengths, and links between confidently different labels as a proxy for wrong links outside confirmed tracks.	Sweep tables cut scores to three places so only a perfect score reads 1.000.
7. All new settings are staff-only and validated.	Pydantic field constraints and validators; not added to the member allow-list in registry.py.
8. The appearance cost can be calibrated to the image model that produced the embeddings, so a model whose similarities all sit near 1 still separates insects.	appearance_similarity_floor / _ceiling map similarity linearly onto 0..1 (appearance_term()); defaults 0 and 1 keep 1 - similarity exactly.	Values are per model; see the calibration table below.
9. Tracking can refuse to link two detections that look clearly different, whatever their boxes say.	appearance_min_similarity, checked in choose_links(); pairs missing an embedding are not gated.
10. Experimental: an insect that moved clear of its old box can link when the two crops look alike.	motion_min_similarity, motion_max_shift: the overlap term becomes min(1 - IoU, shift in box sizes / motion_max_shift) (overlap_term(), shift_in_box_sizes(), new optional PairTerms.shift).	Measured gain is small (below); kept off.
11. A sweep whose output folder does not exist yet no longer fails at the end.	The per-track CSV creates its folder.
12. The merge picker and the capture-by-capture preview score pairs the way a run with the same settings would, so calibration, weights, the move rule and the appearance gate change what they rank and mark as a link.	_pair_scores() uses weighted_cost() under config.link_options(); would_link applies appearance_min_similarity; the preview passes the options to select_links() and applies the crowd multiplier. Response shape unchanged.	The picker still reads the default settings (tracking_config_for()), not the settings of the run that made the track; see "Still open" below.
13. A detection box with no area no longer stops a run.	shift_in_box_sizes() returns infinity for a box with zero or negative area, which leaves the overlap term as it was.	Every pair computes the shift, move rule or not, so this would otherwise have been a new failure on the default path.

First sweep results (measured)

Read-only runs against a local copy of a partner's evaluation project: three one-hour windows of 180 captures (a busy night of small moths, a quiet night with a few large moths, a busy night of large moths), 49 tracks confirmed by a person (40 of two or more detections), 1,212 true links. Embeddings are a classifier backbone's (2048-d). The database session was opened read-only on top of the command's rollback.

Setting	Link precision	Link recall	Exact tracks (2+ detections)	Merges of two confirmed tracks (under-counts joins, see the visual check below)	Occurrences (12,168 detections)	Distinct determinations (631)
Default (0.2, embeddings required)	1.000	0.624	7/40	0	9,774	598
1.0, embeddings not required	1.000	0.960	22/40	0	7,415	565
1.5, embeddings not required, crowd-aware movement (log)	1.000	0.983	28/40	0	3,226	420
1.5, same, crowd-aware movement off	0.994	0.979	25/40	4 (all on the busy night of small moths)	2,089	346

Observations:

• No setting with a threshold of 1.0 or less joined two confirmed tracks, across 168 settings. That is a statement about the confirmed tracks only; the visual check below shows how often the other tracks join two insects. This follows from the cost: 1 - IoU is 1 for boxes that do not overlap, so at 1.0 only overlapping boxes can link.
• Above 1.0, merges appear on the busy night of small moths, and the crowd-aware movement term is what kept them at zero at 1.5.
• The species gate at a score of 0.5 rarely fired where merges happened, because this classifier's per-crop scores are low (mean 0.42). At a score of 0.2–0.3 it fires but also cuts many true links, since labels change from frame to frame on the same moth; a penalty works better than forbidding.
• Linking still insects first changed nothing when embeddings are not required, and linked still moths without an embedding when they are (135 more links on one night, no change on confirmed tracks).

What these numbers cannot show yet (to verify before changing defaults):

• Precision is measured only between confirmed tracks. A link from a confirmed track into a moth nobody confirmed shows up only as lost recall, and a join between two unconfirmed moths does not show up at all. The session-wide count of links between confidently different labels rises from 19 at the default to 160 at 1.5; the visual check below found it misses most real joins, so it is not a substitute for looking.
• 49 tracks is small, the confirmed tracks are probably the clearer ones, and every setting above is chosen and scored on the same data.

Second sweep: a general-purpose image model, calibration, gate and move rule (measured)

Same benchmark. Embeddings from BioCLIP (1024-d, every detection has one, passed with --vectors-file) against the classifier backbone above (2048-d, 74% of detections have one). 720 and 1,440 settings.

How well each model's cosine similarity tells the same insect from a different one nearby (true consecutive pairs against different insects in the adjacent capture within 3 box sizes):

	Classifier backbone	BioCLIP
True pairs, p1 / p50	0.958 / 0.992	0.47 / 0.95
Different insects nearby, p50 / p95	0.957 / 0.979	0.39 / 0.78
AUC	0.980	0.979
True pairs that moved clear of their box (29 of 1,212), p50	0.981	0.74

Both rank pairs equally well, but the backbone squeezes everything into 0.87–1, so its 1 - similarity term moves the cost by about 0.05 and cannot stop a merge.

Overall results, crowd-aware movement on (log), move rule off. Columns: link recall / exact tracks of 40 / merges of two confirmed tracks / links between confidently different labels / occurrences after tracking (from 12,168):

Threshold	Appearance	BioCLIP	Backbone, embeddings required	Backbone, geometry when an embedding is missing
1.5	plain	0.971 / 24 / 0 / 69 / 5,678	0.819 / 23 / 1 / 178 / 5,755	0.983 / 28 / 0 / 160 / 3,226
1.5	plain + gate	0.965 / 24 / 0 / 68 / 5,712	0.814 / 21 / 1 / 130 / 5,961	0.978 / 26 / 0 / 117 / 3,393
1.5	calibrated + gate	0.962 / 24 / 0 / 63 / 5,689	0.808 / 19 / 0 / 52 / 7,367	0.969 / 23 / 0 / 43 / 4,376
2.0	plain	0.987 / 31 / 0 / 172 / 2,510	0.824 / 20 / 5 / 310 / 4,338	0.989 / 31 / 2 / 239 / 1,384
2.0	plain + gate	0.979 / 29 / 0 / 162 / 2,651	0.817 / 22 / 2 / 196 / 4,683	0.982 / 29 / 3 / 152 / 1,560
2.0	calibrated + gate	0.976 / 27 / 0 / 132 / 3,061	0.818 / 22 / 1 / 83 / 5,598	0.980 / 29 / 0 / 62 / 2,149
3.0	plain	0.988 / 31 / 2 / 254 / 939	0.824 / 20 / 7 / 352 / 3,935	0.989 / 31 / 2 / 262 / 840
3.0	plain + gate	0.980 / 30 / 0 / 228 / 1,268	0.817 / 22 / 2 / 210 / 4,294	0.982 / 29 / 4 / 164 / 957
3.0	calibrated + gate	0.979 / 30 / 0 / 214 / 1,353	0.819 / 23 / 1 / 147 / 4,419	0.981 / 29 / 1 / 102 / 1,133

Calibration used floor = median of the nearby different insects, ceiling = 25th percentile of true pairs, gate = 1st percentile of true pairs (BioCLIP 0.39 / 0.89 / 0.47; backbone 0.957 / 0.988 / 0.958). "Plain + gate" uses 1 - similarity with the same gate. Every threshold, percentile and gate value was chosen and scored on the same three one-hour windows, so all of these numbers are in-sample; there is no held-out set yet.

Observations:

• With embeddings required, BioCLIP tracks better: no merges up to threshold 2.0 where the backbone has 1–5, and much higher recall because every detection has an embedding. The backbone catches up only when calibrated and allowed to fall back to geometry.
• The gate at the 1st percentile of true pairs costs little recall and removes the two merges BioCLIP makes at threshold 3.0.
• Allowing one merge buys nothing: the best one-merge settings add at most 0.001 recall.
• Moving insects are not solved by appearance. Only 29 of 1,212 true links are moves. The move rule at threshold 1.0 recovers 3–8 of them with no merges, but adds about one link from the end of a confirmed track into an unconfirmed detection for each correct move; at 1.5 and above it changes almost nothing.
• The "different labels" column uses the backbone's own labels, so it favours the backbone; the reverse check (links BioCLIP rates below 0.47) favours BioCLIP. Neither is ground truth.
• The task on this branch reads embeddings only from classifications; running it with BioCLIP embeddings needs the detection-embedding work first.

Visual check of all predicted tracks (measured, by eye)

The merge counts above only see joins between two confirmed tracks, and the confirmed tracks are mostly clear, well-separated moths, so those counts under-detect joins. To measure precision on everything else, every predicted track of 4 or more detections in a sample was looked at by eye: 365 tracks in the same three one-hour windows and 99 on three full nights, 464 in all, labelled as one insect, two or more insects, or unsure. This is the precision measure to trust over the merge columns above. All three settings use the classifier backbone with no calibration and no gate; unsure tracks count as correct, and the worst case counting them as joins is given in brackets.

Setting	Tracks of 4+ detections that join different insects, hour windows	Full nights (busy / medium / quiet)
Default (0.2, embeddings required)	0 of 120	not checked
1.0, embeddings not required	4 of 122, 3.3% (up to 4.9%)	4 of 40 (up to 9) / 1 of 40 (up to 3) / 0 of 19
1.5, embeddings not required, crowd-aware movement (log)	94 of 123, 76% (90% on the busy night of small moths)	not checked

• The 1.5 setting that reported zero merges between confirmed tracks joins different insects in most long tracks on busy sessions, nearly always across species. The first sweep's suggestion to try 1.5 on busy nights was wrong.
• Every join at 1.0 is the same failure: one insect leaves, a different-looking one lands on the same spot within one capture interval, and the boxes overlap.
• The different-label proxy missed all four joins at 1.0 and caught 18 of 94 at 1.5.
• Not checked by eye: the appearance gate, calibration, BioCLIP embeddings or the move rule. They are the obvious next test for the "one insect leaves, another lands" joins, and the wrong links found in this check are kept as a list to test them against.

A direction to discuss, not a decision: threshold 1.0 without requiring embeddings is the only looser setting checked by eye, and it joins two insects in about 3% of long tracks in the hour windows and about 10% on the busiest full night, while cutting the occurrences to review on the full nights by 29%, 50% and 93%. The default joined nothing in the sample but saves little on busy nights (11% on the busiest). The best result in the second sweep, BioCLIP at 1.5 with crowd-aware movement and the gate at 0.47 ("plain + gate" above: link recall 0.965, 24 of 40 exact tracks, no merges between confirmed tracks), has not been checked by eye. Given what the check found for the ungated 1.5 setting, it should not be adopted until it has. It also cannot run on this branch yet, because the task cannot read BioCLIP embeddings until the detection-embedding work lands.

Detailed Description

• choose_links() is the single matcher: the task's select_links() (used by runs and by the session view's link preview) and the evaluation's precomputed path both call it, and a test checks that both give identical links for non-default settings.
• Regression: test_default_settings_give_the_same_links_as_before_the_new_rules compares default-config links on a synthetic session against a frozen copy of the previous matcher, at four threshold/feature combinations.
• event_transition_pairs() converts each embedding to an array once per detection instead of once per pair, keeping its dtype, so costs are unchanged.

Still open

• The merge picker and the capture preview judge pairs with the default tracking settings, because nothing stores which settings a session was tracked with. Once a staff run uses non-default settings, the picker's "would link" can disagree with what the run did. The history records in Store a feature vector for every detection and show what happened to each occurrence in one timeline #1439 store each run's settings, which is one place the picker could read them from.

How to test

docker compose -f docker-compose.ci.yml run --rm django python manage.py test ami.ml.post_processing
python manage.py evaluate_tracking --project <id> --sweep '{"grid": {"cost_threshold": [0.2, 1.0], "require_features": [true, false]}}' --output-dir out/

Tested locally: ami.ml.post_processing, ami.jobs and the tracking test classes in ami.main.tests pass (422 tests; two NATS result tests failed once on a transient name lookup of the test ML service and passed on rerun); makemigrations --check reports no changes. Frontend untouched. After the calibration, gate and move rule: ami.ml.post_processing passes (136 tests) and makemigrations --check reports no changes.

🤖 Generated with Claude Code

https://claude.ai/code/session_01C7Xf6VPbwWtTumhjjF15g8

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[netlify]

✅ Deploy Preview for antenna-preview canceled.

Name	Link
🔨 Latest commit	61bb112
🔍 Latest deploy log	https://app.netlify.com/projects/antenna-preview/deploys/6abd33bce90d9600097a2800

[mihow]

Claude says: The merge order and plan for tracking, agreed with the owner today, are on #1412: #1412 (comment)

This PR's place: on hold. Its settings work feeds the calibration in #1468, which follows #1469.