Give every detection a BioCLIP feature vector, and embed existing detections without re-detecting

[mihow]

Summary

Tracking an insect from one capture to the next works best when every detection carries a feature vector from the same model, whether or not the moth/non-moth filter kept it. This PR lets the processing service give every detection a BioCLIP 2.5 embedding, and adds a feature-only pipeline that embeds detections a platform already has, without running the detector again or adding any classification. That second part matters for backfilling: existing detections, including ones people have already reviewed, keep their boxes and gain a vector.

It builds on the per-classification feature vectors in #77 and the "vector for every detection" work tracked in #170, which are merged into this branch, and it reuses the BioCLIP 2.5 ViT-H/14 backbone loading from #174 without its classification heads. The diff against main therefore includes #77. This is a draft until those land and the platform side can store 1024-float vectors.

List of Changes

Change (effect)	How (implementation)	Notes
1. Any pipeline can attach a BioCLIP embedding to every detection, including those the moth filter rejected, with no extra classification.	New request config embedding_extractor (and service setting AMI_EMBEDDING_EXTRACTOR) naming bioclip_2_5_embeddings; vectors go in detections[].embeddings.	A key the pipeline does not list in /info (unknown, or not the one in the setting) returns 422, so a response never names an unregistered algorithm; an invalid setting stops the service at startup.
2. Existing detections can be embedded without re-detecting or re-classifying.	New pipeline bioclip_2_5_features. When the request carries detections with boxes, the detector is skipped and the same boxes and detector references come back with an embedding each. Without detections, it detects first.	Detections without a box, or with a box of no area, are skipped.
3. A caller can register the new algorithm and pipeline from /info.	The extractor is described from its class (task type embedding, no category map) without loading the backbone; it is listed in every pipeline when the setting is on, and always in the feature-only pipeline.
4. The backbone loads once, not per request.	The open_clip model is cached per process and device.	About 4.7 GB of GPU memory with the detector also loaded.
5. The async worker is unaffected.	The worker and ami worker register skip feature-only pipelines, which only the API serves.	ami worker register never advertises the embedding extractor, even with AMI_EMBEDDING_EXTRACTOR set, because the worker does not run it.
6. Tests never download the backbone.	The encoder is replaced by a small deterministic module; the detector and classifiers are real.	trapdata/api/tests/test_embedding_extractor.py

Detailed Description

The contract the platform relies on

Antenna's side of this (RolnickLab/antenna#1439) reads the response under these rules, so they should not change without a matching change there:

1. Task type. The extractor is described in /info with task_type: "embedding". Antenna treats a pipeline whose algorithms are only embedding extractors and detectors as feature-only, and sends it existing boxes instead of images to detect on.
2. The features key. Each vector is sent as detections[].embeddings[].features, next to the extractor's algorithm reference. Antenna reads only features; no other key is accepted.
3. Echoed boxes. A feature-only response returns exactly the boxes it was sent, with the same coordinates and the same detector algorithm reference. Antenna matches each returned box to a stored detection by image and box (rounded to three decimals); a box that matches nothing is counted and logged, and a batch in which no box matches fails the job.
4. /info lists both algorithms. The feature-only pipeline lists the detector whose boxes it echoes and the embedding extractor. Antenna stores a vector only under an algorithm key the pipeline declared in /info, and refuses a key it did not.
5. One length per extractor. Every vector from one extractor has the same length (1024 for BioCLIP 2.5). Antenna records the length from the first batch and refuses vectors of another length from the same extractor.

Request (feature-only pipeline, embedding existing detections):

{
  "pipeline": "bioclip_2_5_features",
  "source_images": [{"id": "123", "url": "https://example.org/capture.jpg"}],
  "detections": [
    {
      "source_image": {"id": "123", "url": "https://example.org/capture.jpg"},
      "bbox": {"x1": 10, "y1": 20, "x2": 110, "y2": 140},
      "algorithm": {"name": "FasterRCNN for AMI Moth Traps 2023", "key": "fasterrcnn_for_ami_moth_traps_2023"}
    }
  ]
}

Each response detection keeps its box and algorithm, has classifications: [], and one embedding:

"embeddings": [
  {"features": [0.0123, -0.0456, "... 1024 floats, L2-normalised"],
   "algorithm": {"name": "BioCLIP 2.5 ViT-H/14 embeddings", "key": "bioclip_2_5_embeddings"}}
]

For any other pipeline, add "config": {"embedding_extractor": "bioclip_2_5_embeddings"}; this combines with features_for_all_detections, so a detection can carry both the species classifier's vector and the BioCLIP one, each under its own algorithm key.

Measurements

One RTX 3090, 32 synthetic 6080x3420 frames served over local HTTP, 8 images per request, global_moths_2024, 562 detections. Measured once, after a warm-up request.

Run	Wall time	Images/s	Detections/s	Peak GPU memory
Species vectors on every detection	110 s	0.29	5.1	11.3 GB
The same plus BioCLIP on every detection	125 s	0.26	4.5	11.1 GB
Feature-only, given the 562 detections	17 s	1.84	32.3	4.7 GB
Feature-only, detector runs	26 s	1.24	21.8	11.1 GB

The feature-only run returned the same 562 boxes, no classifications, and vectors identical (cosine 1.0) to the ones from the full pipeline. Most of the full pipeline's time is the large-vocabulary species classifier's post-processing, not BioCLIP.

End to end with the platform

Run against a local copy of a development database with the Antenna side of this contract (RolnickLab/antenna#1439): three feature-only jobs over three one-hour sessions stored one 1024-float vector for each of 12,168 existing detections, and the counts of detections, classifications and occurrences, and every determination, were unchanged.

What is not verified yet

• The async worker path, which ignores embedding_extractor.
• Throughput with num_workers > 0; crops are read in the main process here, as in the other API pipelines.

How to run locally

uv sync
AMI_EMBEDDING_EXTRACTOR=bioclip_2_5_embeddings uv run ami api --port 2010
uv run pytest trapdata/api/tests/test_embedding_extractor.py

The first request downloads the backbone (about 3.6 GB) from the Hugging Face Hub.

🤖 Generated with Claude Code

https://claude.ai/code/session_01C7Xf6VPbwWtTumhjjF15g8

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[mihow]

Claude says: I checked this PR against #160, the general insect detector PR, so that the two can land in either order. In short, they are compatible. Two small changes were needed here, and one integration step will be needed whichever PR merges second.

Pushed to this branch

• 21ab54c The worker's pipeline filter (classifier_pipelines) used to keep a slug only if it was in CLASSIFIER_CHOICES. In Let a pipeline pair any detector with any classifier, and add a general insect-detection pipeline #160 that map is a deprecated view that leaves out pipelines built on a different detector, so after a merge the worker and its registration would have quietly dropped the anybug pipeline. The filter now removes only the feature-only pipelines.
• a8047bd This branch's combined registry was called PIPELINE_CHOICES, and Let a pipeline pair any detector with any classifier, and add a general insect-detection pipeline #160 uses the same name for its map of stage definitions. It is now ALL_PIPELINE_CHOICES. The two tests that described a pipeline from a classifier class now look the pipeline up through select_pipelines, so they pass with either registry shape.

The full suite passes: 228 passed, 1 skipped.

What a merge involves

I merged #160 into this branch locally in a throwaway worktree and resolved it. Most of the conflicts come from #160 predating #171 and #172 on main; five of the eight conflicting files also conflict between #160 and main alone. The part that needs thought is that #160 replaces the inline filter-then-classify code in both /process and the worker with one shared stage engine, and this PR's "embed the detections the filter rejected" logic lives in that inline code. It moves into the engine cleanly:

1. The stage runner passes include_features, include_logits and include_embeddings to the terminal classifier, and only include_logits to the gates.
2. After the terminal stage runs, the engine embeds whatever the gates dropped using the terminal classifier. /process calls classifier.embed(). The worker uses a tensor-based twin of _classify_detections_from_tensors, which also picks up Let a pipeline pair any detector with any classifier, and add a general insect-detection pipeline #160's clamp_to_bounds in place of this branch's unclamped _transform_crops.
3. make_pipeline_config_response takes a PipelineDefinition (or a feature extractor for the feature-only pipeline) and appends the embedding extractor after the stage loop.
4. Two of Let a pipeline pair any detector with any classifier, and add a general insect-detection pipeline #160's tests need adjusting. The advertised/subscribed/dispatchable test has to exclude feature-only pipelines, and the score_scale test's mock needs _embedding_only=False.

On the merged tree, with real models, the anybug pipeline lists exactly one detector plus the BioCLIP extractor in /info. Worker registration still leaves the extractor out. On a test frame it returned 43 detections, 8 of which the Lepidoptera gate rejected, and all 43 carried both vectors. The rejected ones carried only their order label. The merged suite passes apart from three tests that fail on #171's model URL rules, which is #160's rebase and not this PR.

Detection and classification schemas do not collide. #160 adds no response fields, only a clamp helper on BoundingBox. The feature-only pipeline already echoes a detection's detector reference unchanged whatever its key; test_embeds_the_given_boxes_without_detecting_or_classifying uses a non-default key.

Suggestions for #160, not changed there

• Rebase onto main first, which covers the model URL rules and select_pipelines.
• When the registry moves out of api.py, as requested in review, give the stage engine per-stage options and a hook for "detections a gate dropped". Then this PR plugs in without adding inline code back. Feature-only pipelines could also get a place in the registry type, so there is a single typed map.
• One thing worth checking on the platform side before running the feature-only pipeline on detections from another detector: its /info advertises the default moth detector, because that is what runs when no detections are sent.