Store a feature vector for every detection and show what happened to each occurrence in one timeline

[mihow]

Summary

Three things tracking needs before it leaves the development deployment, built on the tracking server (#1272) and interface (#1432) branches.

First, every detection can now carry a feature vector, including the crops the moth/non-moth filter rejects. Until now the only vectors came from the species classifier, which never sees those crops, so between a fifth and a third of the detections in the benchmark sessions could not be compared by appearance, and every frame reviewers could not attach to a track was one of them. The vectors are stored in their own table and never become a classification, so storing them on an already-annotated project changes no identification.

Second, an occurrence keeps a history of what happened to it: which tracking run grouped it, what class masking or the size filter changed, and when a person marked its track complete and accurate. The occurrence page shows that history in one timeline, beside human identifications and machine predictions, newest first. A review is dated, says who confirmed which detections, and the page can say whether the track changed since.

Third, detections that already exist can be given vectors afterwards. An ordinary ML job run with a feature-only pipeline sends the boxes Antenna already has to the processing service and stores only the vectors that come back, so a project processed months ago can gain appearance vectors without new detections, classifications or changed identifications. Vectors can be of any length, because different feature extractors produce different sizes, and each extractor keeps one length.

Merge order. This PR is based on #1432 and merges after it, and before the tracking settings PR (#1442), which is now based on this branch. It stays one PR: the vectors, the history and the feature-only jobs share one reader and one set of migrations, and splitting them would mean a second round of the same conflicts.

Checked end to end. On a local copy of a development database, three feature-only jobs run through a local processing service (RolnickLab/ami-data-companion#175) stored one 1024-float vector for each of 12,168 existing detections in three one-hour sessions, and the counts of detections, classifications and occurrences, and every determination, were unchanged.

Why it is still a draft. GitHub runs the backend tests only on PRs based on main, so this stacked PR has no CI result. On this head (32da6cfb), the full backend suite ran locally in the CI compose stack: makemigrations --check reports no changes, and 1,065 tests passed (2 skipped). It comes out of draft once it is based on main and its CI is green.

List of Changes

Change (what a reviewer or operator gets)	How
A processing service can send a feature vector with any detection, and Antenna stores it without touching determinations	DetectionResponse.embeddings (optional list of {features, algorithm}; the vector is read from features only, which is what the processing service sends), stored in DetectionEmbedding (migration 0102): one row per detection and algorithm, replaced when the same results are saved again, with the job that stored it (SET_NULL when that job is deleted). An unregistered algorithm key stops the batch the same way it does for classifications
Tracking, merge ranking, the extend-mode preview and the "has a vector" counts use these vectors	One reader (models_future/embeddings.py) reads embeddings and classification vectors in one query, always for one algorithm. Merge ranking scores every candidate with the algorithm that has vectors on most of the occurrence's frames
Vectors of any length are accepted; each feature extractor keeps the length it first sent	The vector column has no fixed size (migration 0102, plus 0104 which drops the size on databases that ran an earlier draft; it stays because a shared demo database ran that draft). Algorithm.embedding_dimensions (ml migration 0029) is set from /info or from the first vector stored; a vector of another length is refused for the whole batch
An ML job can add vectors to detections that already exist, without creating or changing anything else	A pipeline is feature-only when it has an embedding or feature_extraction algorithm and no classifier (a detector may be listed, since the service names the detector of the boxes it echoes back). Its jobs send each capture's existing boxes, skip detections that already have a vector from that extractor, never queue a capture with no boxes, and save only DetectionEmbedding rows matched by capture and box. Classifications in the response are ignored
A feature-only batch that stores no vector fails the job instead of being sent again on every run, and detections left without a vector are counted	After each feature-only save, the detections on the batch's images that still have no vector are counted on the results stage (without_vector), and returned boxes that match no stored detection are counted as unmatched with the first few named in the job log. A batch that stores no vector while such detections remain (no box matches, the boxes come back without vectors, or no boxes come back at all) raises FeatureResultsStoredNothing and fails the job; in the async path the message is acknowledged rather than redelivered. A batch that stores some vectors but misses others does not fail; its missing detections are counted and will be sent again by the next run. A response that reports an error is left to the existing error handling. In the synchronous path the batch still fails the job, but the two counts are not added to its progress
A classifier pipeline that also returns vectors does not reprocess captures it already classified	Feature extractors are left out of the check for already processed captures, since they never write a classification
When a session has vectors from several extractors, tracking and merge ranking compare the one that covers the most detections	Coverage decides the default; an extractor the project runs is preferred only when it covers nearly as many detections. Comparisons always use a single extractor
The tracks CSV counts an embedding as a feature vector	has_feature_vector is true for an embedding or a classification vector
Tracking runs, class masking, the size filter and track reviews leave a dated record on each occurrence they change	OccurrenceHistoryRecord (migration 0103) with a kind (algorithm result or review) and a typed payload. Records move with the keeper when occurrences merge, and a confirmation carries over to the pieces when a regroup splits a track
A review is posted only when the confirmed detections or the confirming person changed, and says who confirmed which detections	The server reports whether the track changed since its last review
One endpoint returns everything that happened to an occurrence	GET /occurrences/{id}/history/: algorithm results, reviews, identifications and one prediction per algorithm, newest first. Visible to whoever can open the occurrence, including occurrences the default filters hide; users are shown by name and picture only
The occurrence page's identification tab becomes one timeline	Cards for identifications, predictions, algorithm results and reviews. Confirming the current identification from a card moves to the next occurrence. Signed-in users without a display name are labelled consistently rather than as anonymous
The size filter saves its last batch even when the final detection is skipped	Found while adding its history record

Open before merge

• Vector size (decided). The column is unsized and each extractor records its own length, as described above. Pairwise comparison in tracking does not need an approximate-nearest-neighbour index; if one is needed later it can be a partial index per extractor.
• Classification vectors. Classification vectors (features_2048) are still written and read beside the new table. Whether new processing services should stop sending them is worth deciding here too.
• Manual track edits (split, move, merge) do not yet write their own history entries; the timeline shows their effect through reviews and the tracking record.

How to test

1. Run the migrations and the suites: python manage.py test ami.ml ami.main ami.exports ami.jobs.
2. Process captures with a processing service that sends embeddings on its detections, then check that DetectionEmbedding has one row per detection and algorithm, including detections with no species classification.
3. Register a processing service that offers a feature-only pipeline (a detector plus an embedding algorithm), run an ML job with it on captures that already have detections, and check that DetectionEmbedding gains one row per real detection while the counts of detections, classifications and occurrences, and every determination, stay the same.
4. Run tracking on a session, mark a track complete, then open the occurrence: the identification tab shows the tracking result and the review in one timeline, and GET /api/v2/occurrences/{id}/history/?project_id=P returns the same entries.
5. In ui/: yarn tsc --noEmit, yarn lint, yarn test.

Part of #1431. Part of #1433. Refs #1417. Based on #1432. The processing-service side of feature-only pipelines is RolnickLab/ami-data-companion#175.

🤖 Generated with Claude Code

https://claude.ai/code/session_01C7Xf6VPbwWtTumhjjF15g8

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[netlify]

✅ Deploy Preview for antenna-preview canceled.

Name	Link
🔨 Latest commit	fd559de
🔍 Latest deploy log	https://app.netlify.com/projects/antenna-preview/deploys/6abf06400b7fb600082aca29

[mihow]

Will this support semantic embeddings? Or is the algorithm field enough? Where should we store a description of what the vector represents?

[mihow]

Can we split this PR and create one focused only on adding the DetectionEmbedding model? The features introduced in #1407 needs to use it as well

[mihow]

Will this approach be appropriate for storing logits in their own table? Otherwise what's the plan for moving logits out of the Classification table?

[mihow]

How can we specify the type of embedding from the algorithm? what if we store a softmax-ed copy or a reduced version of the vector after running PCA? We could use different Algorithm entries since those include post-processing. Or just notes or a enum field?

[mihow]

Claude says: The merge order and plan for tracking, agreed with the owner today, are on #1412: #1412 (comment)

This PR's place: its content was split into #1461 (the occurrence history and algorithm results) and #1462 (feature vectors for every detection).