Record which job created each detection and classification, and filter occurrences by job

[mihow]

Summary

When a pipeline run or a post-processing task writes detections and classifications, nothing on those rows says which run made them. That makes it hard to answer simple questions after the fact, such as "what did last night's run actually change?", and it is the missing link that the occurrence history work in #1461 needs.

This PR records the job on every detection and classification a job creates, shows it in the API, and adds a job filter to the occurrence list so a user can see the occurrences a given job created or updated. An occurrence matches every job that created one of its detections or classifications, so a later job that only adds a classification also matches it. The job details page links straight to that filtered list. Rows that already exist are left alone: if a later run returns the same box or label, the row keeps the job that first wrote it. Existing data has no job recorded; only rows created after deployment will.

Requests without ?job= are unchanged: the filter returns the queryset as it is, so the occurrence list and stats endpoints run the same SQL as before.

This is the first PR in the series that builds on job provenance (#1461, then #1469 and #1462), split out so the schema change can land and be reviewed on its own.

Closes #1156

List of Changes

Change (user effect)	How (implementation)
Detections and classifications remember which job created them.	New nullable job foreign key on Detection and Classification (on_delete=SET_NULL, so deleting a job keeps the rows). Migration main/0096_detection_and_classification_job adds the columns only.
The job filter is fast, and deploying it does not lock the two largest tables while an index builds.	Instead of the foreign keys' default index, each table gets a partial covering index: (job, occurrence) on detections and (job, detection) on classifications, WHERE job_id IS NOT NULL. Built CONCURRENTLY in the non-atomic migration main/0097_detection_and_classification_job_indexes. Measurements below.
Pipeline results record their job.	save_results passes job_id through create_detections / get_or_create_detection and create_classifications / create_classification; only newly created rows get it.
Class masking and the size filter record their job on the classifications they add.	make_classifications_filtered_by_taxa_list takes the task's job; SmallSizeFilterTask sets job=self.job.
API consumers can see the job on a detection or classification.	Read-only job id (PrimaryKeyRelatedField with help text) on the detection and classification serializers. It reads the job_id already on the row, so it adds no query.
Users can filter the occurrence list to the occurrences a job created or updated.	?job=<id> via a new OccurrenceJobFilter backend and OccurrenceQuerySet.created_or_updated_by_job(), built from two EXISTS subqueries (a detection by the job, or a classification by the job). Human identifications are not matched. A non-integer id returns 400. Because the occurrence filter backends are shared, the occurrence stats endpoints accept the same parameter.
The job dropdown lists the project's recent processing jobs without paging.	New /api/v2/jobs/choices/: pipeline and post-processing jobs only, most recently created first, one response capped at 100, the same approach as the capture set choices. Failed jobs are included because they may have written results before failing. The capped pagination class is now shared as ChoicesPagination.
Users can pick a job in the occurrence filters, and a job linked from elsewhere stays visible and clearable.	New JobFilter under "More filters". When the selected job is not among the listed choices (for example, an older job linked from its page), it is loaded on its own and added to the options. job is added to the filters carried over into the occurrence list.
The "Default filters" toggle is the first item in the filter list, since it changes what every other filter returns.	Moved to the top of the first filter section on the occurrences page, and on the species page for consistency.
Screen readers announce each filter's name, not just its placeholder or selected value, and the "Default filters" switch has a name.	FilterControl gives each heading an id and labels the control and its clear button as a group (role="group", aria-labelledby, WCAG technique ARIA17), which covers every filter. The switch gets aria-labelledby.
From a pipeline or post-processing job's page, users can open the occurrences it created or updated.	"View occurrences" link on the job details page. The frontend job-type list also gains post_processing, which previously had no label.

Detailed Description

Why EXISTS and not a join

A join through detections or detections__classifications returns one row per matching result, so an occurrence with three detections from the same job would be listed three times and the paginator count would be inflated. The .distinct() fix for that sorts whole occurrence rows and is slow on large projects. Two EXISTS subqueries return each occurrence once, matching how the existing ?algorithm= filter is written.

Schema and index choice (measured)

All figures below are single runs on one machine against a local copy of production data (about 640,000 detections and 835,000 classifications), with a warm cache. They are measurements, not a benchmark, and production tables are larger.

Adding the columns. Measured inside a transaction that was rolled back:

Step	Detections	Classifications
ALTER TABLE … ADD COLUMN job_id … REFERENCES jobs_job	4 ms	0.3 ms
Plain (job_id) index on the new, all-null column	163 ms, 4.4 MB	185 ms, 5.7 MB
Partial (job_id, occurrence_id / detection_id) WHERE job_id IS NOT NULL	59 ms, 8 kB	61 ms, 8 kB

Adding a nullable foreign key column does not scan the table, so the lock it takes is brief. A plain index on the new column stores an entry for every existing row, all of them null; the partial index skips them, so it builds in a third of the time and starts empty.

Querying. To measure the filter once jobs are recorded, both tables were copied into temporary tables with every row assigned to a job, then vacuumed and analysed so index-only scans were possible. The query is the occurrence count, and the first page ordered by determination score, with only the job filter applied (not the full occurrence list query). Four cases: a job that wrote every occurrence of the largest project (179,466 detections and 197,222 classifications); a job that wrote one night in another project (1,492 detections, 888 classifications); a job that wrote only classifications, as post-processing does (876); and a job with nothing in the queried project.

Case (count / first page, ms)	Plain index, EXISTS	Partial covering index, EXISTS (this PR)	Partial covering index, IN (… UNION ALL …)
Whole project	565 / 61	451 / 34	645 / 656
One night	175 / 2.3	115 / 2.0	6.3 / 6.4
Classifications only	175 / 1.7	110 / 1.6	4.0 / 3.9
Nothing in this project	188 / 92	137 / 85	3.1 / 4.9

• The partial covering index was faster than the plain index in every case. Every job lookup was an index-only scan with no heap fetches. The cost is size once most rows have a job: 19 MB and 20 MB, against 4.6 MB and 5.8 MB for the plain index, which Postgres deduplicates.
• The two query shapes trade off. EXISTS … OR EXISTS … walks every occurrence in the project and checks it against the job's rows, so its cost follows the size of the project: 110 to 190 ms here even for a small job. IN (… UNION ALL …) starts from the job's rows, so its cost follows the size of the job: a few milliseconds for a night, but 650 ms for the first page of a job that wrote a whole project, because every matching occurrence has to be collected before the page can be sorted.
• This PR keeps EXISTS, whose worst case here is about 190 ms, against about 650 ms for UNION ALL. Both shapes use the same indexes, so the schema does not depend on this choice. The query lives in one method, created_or_updated_by_job(), and can change shape, or pick a shape based on the job's size, later without a migration.
• Postgres's own foreign key check when a job row is deleted (WHERE job_id = $1) also uses these indexes, so deleting a job does not scan either table.

An earlier measurement on the real tables, with the plain index and the full occurrence list query for an anonymous request with the project's default score threshold, is kept for reference: a page of 20 took 233 ms without ?job=, 104 ms for a job that wrote one night and 240 ms for a job that wrote nothing; the count took 367 ms, 921 ms and 325 ms. Most of the filtered count was spent outside the job filter, in the list's existing LEFT JOIN to detections.

Migration and deployment note

0096 adds the two columns. It needs a brief exclusive lock on each table, but it waits for running queries on those tables to finish, and new queries queue behind it while it waits, so avoid deploying while a long export is running. 0097 builds both indexes CONCURRENTLY, which blocks neither reads nor writes; it clears the statement timeout for the build and restores it afterwards, as in 0093. A local database that applied an earlier version of this branch has the old plain indexes (main_detection_job_id_d3c071b8 and main_classification_job_id_bae87240); they can be dropped by hand.

Tests

• Saving pipeline results twice with different jobs: new detections and classifications record the first job, and the second run does not overwrite it.
• Class masking records its job on the new masked classification and leaves the source classification's job unchanged.
• The size filter records its job on its "Not identifiable" classifications.
• The job choices endpoint: only pipeline and post-processing jobs, most recent first, failed jobs included, a project is required, a draft project's jobs are listed for its members and hidden from everyone else, the response is capped at 100 whatever limit is requested, and the query count does not grow with the number of jobs listed.
• The job filter, on a fixture of five occurrences: returns exactly the three occurrences the job wrote a detection or a classification for, once each, including one occurrence with three matching detections; another job matches only its own; ?job=abc returns 400; and the filter adds no queries of its own (one query for the rows, one for the count).

makemigrations --check reports no changes, the affected test modules pass locally (105 tests), and the full backend suite passes in CI at 797a6495.

UI verification

The frontend passes tsc, ESLint and Prettier, and was checked in a browser at 797a6495 against a local stack, on a test project where a pipeline job and a size-filter job had recorded their rows:

• The job dropdown lists the project's pipeline and post-processing jobs, newest first, as "Name (#id)"; a populate-captures job in the same project is left out.
• Choosing the pipeline job narrows the list from 27 occurrences to 1, with job=<id> in the request and the page URL; clearing the filter returns all 27. The size-filter job matches all 27, since it classified every one.
• "View occurrences" appears on pipeline and post-processing job pages, not on populate-captures or storage-sync jobs, and lands on the filtered list with "More filters" open and the job selected.
• On a project with 126 processing jobs, opening the list with an older job that is not among the 100 choices loads that job on its own; it shows in the dropdown and can be cleared.
• The detection and classification endpoints return job, or null for rows without one, and ?job=abc returns 400.
• No new console errors. The job details dialog logs Radix's missing DialogTitle warning, which this PR does not touch.

One cosmetic issue for later: a long job name is truncated in the dropdown once selected, because the clear button takes part of the width.

Notes for the PRs that build on this

• Record any algorithm's results on occurrences in a standard way, and show them in each occurrence's history #1461 adds the same two job columns in its own migration. When it rebases onto this PR it should drop those two AddField operations and renumber its migrations to follow 0097.
• Jobs that write neither detections nor classifications (tracking in Run automated tracking from the admin as a post-processing method, with editable settings #1469, for example) match nothing in ?job= yet. When Record any algorithm's results on occurrences in a standard way, and show them in each occurrence's history #1461 adds algorithm results with a job, created_or_updated_by_job() should gain a third branch for them.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HEbXiX2mZk7efVmdgSNcN7

[netlify]

✅ Deploy Preview for antenna-preview ready!

Name	Link
🔨 Latest commit	995eae8
🔍 Latest deploy log	https://app.netlify.com/projects/antenna-preview/deploys/6ac437e9b6698e00080b521b
😎 Deploy Preview	https://deploy-preview-1471--antenna-preview.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.
Lighthouse	1 paths audited Performance: 57 (🔴 down 8 from production) Accessibility: 81 (🔴 down 8 from production) Best Practices: 92 (🔴 down 8 from production) SEO: 92 (no change from production) PWA: 80 (no change from production) View the detailed breakdown and full score reports
🤖 Make changes	Run an agent on this branch

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[netlify]

✅ Deploy Preview for antenna-ssec ready!

Name	Link
🔨 Latest commit	995eae8
🔍 Latest deploy log	https://app.netlify.com/projects/antenna-ssec/deploys/6ac437e99a75f60008d26ff3
😎 Deploy Preview	https://deploy-preview-1471--antenna-ssec.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.
🤖 Make changes	Run an agent on this branch

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: 74f51e3f-cebe-4bcc-9ffa-0ed49f1562bd

📥 Commits

Reviewing files that changed from the base of the PR and between cf6daa8 and 797a649.

📒 Files selected for processing (26)

• ami/jobs/serializers.py
• ami/jobs/tests/test_jobs.py
• ami/jobs/views.py
• ami/main/api/serializers.py
• ami/main/api/views.py
• ami/main/migrations/0096_detection_and_classification_job.py
• ami/main/migrations/0097_detection_and_classification_job_indexes.py
• ami/main/models.py
• ami/main/tests.py
• ami/ml/models/pipeline.py
• ami/ml/post_processing/class_masking.py
• ami/ml/post_processing/small_size_filter.py
• ami/ml/post_processing/tests/test_class_masking.py
• ami/ml/tests.py
• ui/src/components/filtering/filter-control.tsx
• ui/src/components/filtering/filters/job-filter.tsx
• ui/src/data-services/constants.ts
• ui/src/data-services/hooks/jobs/useJobChoice.ts
• ui/src/data-services/models/job.ts
• ui/src/pages/job-details/job-details.tsx
• ui/src/pages/occurrences/occurrence-filters.ts
• ui/src/pages/occurrences/occurrences.tsx
• ui/src/pages/species/species.tsx
• ui/src/utils/getAppRoute.ts
• ui/src/utils/language.ts
• ui/src/utils/useFilters.ts

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

This change records the job that creates detections and classifications, exposes job choices and occurrence filtering by job, and adds UI controls and links for viewing occurrences associated with processing jobs.

Changes

Job Provenance and Occurrence Filtering

Layer / File(s)	Summary
Record job provenance ami/main/models.py, ami/main/migrations/0096_detection_and_classification_job.py, ami/main/migrations/0097_detection_and_classification_job_indexes.py, ami/ml/models/pipeline.py, ami/ml/post_processing/*, ami/ml/tests.py, ami/ml/post_processing/tests/*	Detections and classifications now have nullable job relationships. ML and post-processing paths assign the job to newly created records. Tests cover job assignment and confirm reused records keep their existing job.
Filter occurrences by job ami/main/api/serializers.py, ami/main/api/views.py, ami/main/models.py, ami/main/tests.py	Detection and classification responses include the job ID. The occurrence API accepts a positive job ID and returns occurrences with detections or classifications created by that job. Tests cover matching, validation, duplicate occurrences, and query counts.
Provide project job choices ami/jobs/serializers.py, ami/jobs/views.py, ami/jobs/tests/test_jobs.py, ami/main/api/views.py	The jobs API returns newest-first ML and post-processing job choices for an accessible project. The endpoint requires a project and caps the response at 100 results. The choices paginator is shared with the capture-set choices endpoint.
Expose job filtering in the UI ui/src/components/filtering/*, ui/src/data-services/*, ui/src/pages/job-details/job-details.tsx, ui/src/pages/occurrences/*, ui/src/pages/species/species.tsx, ui/src/utils/*	The occurrence page adds a job filter that loads job choices. Job details link to occurrences filtered by the current job. The UI recognizes post-processing jobs and moves default filter controls ahead of other filters on occurrence and species pages.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant User
  participant JobFilter
  participant JobViewSet
  participant Occurrences
  participant OccurrenceJobFilter
  participant OccurrenceQuerySet
  User->>JobFilter: Open job filter
  JobFilter->>JobViewSet: GET jobs/choices for active project
  JobViewSet-->>JobFilter: Return job choices
  User->>Occurrences: Select job
  Occurrences->>OccurrenceJobFilter: Request occurrences with job ID
  OccurrenceJobFilter->>OccurrenceQuerySet: created_or_updated_by_job(job_id)
  OccurrenceQuerySet-->>Occurrences: Return matching occurrences

Loading

Merge Risk: ⚪ Minimal · up to 797a6

The job-filtering path is ready to merge after normal checks. The frontend has not yet been exercised in a browser.

Security Architecture Review

Security architecture risk: 🔵 Low · up to 797a6

The change is additive and preserves project visibility checks. No access-control regression was established. Remaining uncertainty concerns provenance consistency across job execution paths and recovery from interrupted jobs or deployment.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

• inferred — The new read surface concerns job metadata and result-to-job correlation within requester-visible projects. The occurrence filter cannot add another project's rows because it narrows an already visibility-filtered, project-scoped queryset. No additional credential or service authority was established by the inspected changes.

Trust Boundaries and Controls

• observed — Attacker-supplied project and job query parameters cross separate controls: choices checks project visibility before returning metadata, while occurrence filtering adds predicates to the existing authorized queryset. Result serializers make provenance read-only. These controls address direct private-project enumeration and direct provenance replacement.

Resilience and Maintainability Implications

• observed — Existing job creation checks permissions on the job's project, but related source IDs use unrestricted model querysets. Inspected collection and post-processing scope resolution does not itself enforce equality with the job's project. Those selection paths predate this PR; complete downstream enforcement and the exposure of newly persisted cross-project associations remain unverified.

Hardening Proposals

• proposed — Before using provenance for authorization or security auditing, establish and enforce job-to-result project consistency across source selection and result ingestion. Treat the current relationship as attribution, not an independent permission grant or proof of successful job completion.
• proposed — Document deployment recovery for an interrupted non-atomic index migration, including inspection of completed or invalid indexes before retry and restoration of connection timeout settings. This is an operational proposal, not evidence that production recovery is missing.

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Out of Scope Changes check	⚠️ Warning	The occurrence job filter, choices endpoint, and related job UI use the provenance required by #1156 and are connected to that objective. The species page DefaultFiltersControl relocation is not: it…	Revert the DefaultFiltersControl relocation on the species page, or identify a separate in-scope requirement that calls for that change.
Docstring Coverage	⚠️ Warning	Docstring coverage is 44.44% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 45 functions across 26 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	#1156 requires job relationships on Detection and Classification, saving the job during result processing, and exposing it in the API. The PR adds nullable job foreign keys, assigns them to newly crea…
Title check	✅ Passed	The title clearly summarizes the main changes: recording which job created detections and classifications, and filtering occurrences by job.
Description check	✅ Passed	The description is detailed and covers the summary, changes, related issue, implementation, tests, screenshots, and deployment notes. It does not include the template’s checklist, but the description …

Full details: Out of Scope Changes check

Explanation

The occurrence job filter, choices endpoint, and related job UI use the provenance required by #1156 and are connected to that objective. The species page DefaultFiltersControl relocation is not: it changes filter ordering on a page that does not use the new job filter, and the stated reason is consistency. This change has no demonstrated connection to #1156.

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[Copilot]

Copilot review overview

🟡 Changes recommended

Job choices retain unnecessary per-row database work, and the job selector needs an accessible label.

Review effort: Balanced
Findings: 2

Open (2)

• Avoid per-job permission checks and lazy loads in choices serialization · New
• Associate the Job label with the filter trigger · New

What changed in this PR

Adds job provenance to detections and classifications, supporting occurrence filtering and future history features.

Changes:

• Records creating jobs without overwriting existing provenance.
• Adds job-filtered occurrence queries and capped job choices.
• Connects job details and occurrence filters in the UI.

File	Description
ui/​src/​utils/​useFilters.ts	Registers job filter metadata.
ui/​src/​utils/​language.ts	Adds job-filter and navigation text.
ui/​src/​utils/​getAppRoute.ts	Supports job route filters.
ui/​src/​pages/​species/​species.tsx	Moves default filters first.
ui/​src/​pages/​occurrences/​occurrences.tsx	Adds job filter controls.
ui/​src/​pages/​occurrences/​occurrence-filters.ts	Preserves job filters during navigation.
ui/​src/​pages/​job-details/​job-details.tsx	Links to matching occurrences.
ui/​src/​data-services/​models/​job.ts	Labels post-processing jobs.
ui/​src/​data-services/​hooks/​jobs/​useJobChoice.ts	Loads selected jobs outside recent choices.
ui/​src/​data-services/​constants.ts	Defines job choices endpoint.
ui/​src/​components/​filtering/​filters/​job-filter.tsx	Implements job selector.
ui/​src/​components/​filtering/​filter-control.tsx	Registers job selector component.
ami/​ml/​tests.py	Tests pipeline and size-filter provenance.
ami/​ml/​post_processing/​tests/​test_class_masking.py	Tests class-masking provenance.
ami/​ml/​post_processing/​small_size_filter.py	Records classification jobs.
ami/​ml/​post_processing/​class_masking.py	Passes jobs to new classifications.
ami/​ml/​models/​pipeline.py	Records jobs while saving results.
ami/​main/​tests.py	Tests occurrence job filtering.
ami/​main/​models.py	Adds job relationships and filtering.
ami/​main/​migrations/​0097_detection_and_classification_job_indexes.py	Builds partial indexes concurrently.
ami/​main/​migrations/​0096_detection_and_classification_job.py	Adds nullable job columns.
ami/​main/​api/​views.py	Adds job filtering and shared choices pagination.
ami/​main/​api/​serializers.py	Exposes read-only job IDs.
ami/​jobs/​views.py	Adds project-scoped job choices.
ami/​jobs/​tests/​test_jobs.py	Tests choices visibility, ordering, and limits.
ami/​jobs/​serializers.py	Adds job choices serializer.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.