Repository navigation
Score a classifier against a fixed set of occurrences, and show where it does well - #1491
Closed
mohamedelabbas1996 wants to merge 6 commits into
Closed
mohamedelabbas1996 wants to merge 6 commits into
mohamedelabbas1996 wants to merge 6 commits into
Conversation
A project needs the same occurrences again later: comparing two classifiers only means something if both saw the same rows, and a review pass wants the list it started from. A filter answers differently as data arrives, so the membership is stored rather than described. An OccurrenceSet holds its occurrences and the projects it belongs to. A set with no project is global and is offered everywhere, following how TaxaList already treats a list with no project. Membership is decided when the set is created and nothing adds to or removes from one afterwards, because anything recorded against a set was measured on exactly those occurrences. Creating, renaming and deleting are gated on new project permissions, held by the roles that already curate a project's data. A global set has no single project to check against, so it cannot be edited through the API at all. The occurrence list takes an occurrence_set filter, and the sets are offered as choices the same way capture sets are. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CnKz4AS4iFrYj1GrgkZQbq
Adds the occurrence set to the occurrence filter panel, picked from the set choices endpoint the same way a capture set is. A set is only useful if you can look at what is in it, and this is where someone reviewing one starts. The field is carried over from other views like the existing filters, so arriving with a set already chosen shows it in the panel where it can be cleared. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CnKz4AS4iFrYj1GrgkZQbq
Someone filtering the occurrence list to the rows they care about had no way to keep that selection. Selecting occurrences now offers saving them as a set, next to the identification actions already there. The action does not change any occurrence, so it is not behind update rights on them; the endpoint gates it on the project's own permission instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CnKz4AS4iFrYj1GrgkZQbq
Registering a filter in the shared list is not enough for it to appear: the occurrences page renders one FilterControl per field it offers, and the set was missing from that list, so the filter existed everywhere except on screen. It sits under More filters beside the capture set, and that section now opens on arrival when a set is already applied, as it does for the other filters there. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CnKz4AS4iFrYj1GrgkZQbq
The selection bar holds identification actions and is hidden from anyone without update rights on the occurrences, so saving a set — which changes none of them — was unavailable to a reader who could still create one. It also sat there as an unlabelled icon among three others. It now sits beside Export as a labelled button, and appears only while something is selected, since that is the only time it does anything. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CnKz4AS4iFrYj1GrgkZQbq
A project runs more than one classifier over time and has no way to say whether a newer one is better. Accuracy quoted from different data is not a comparison, so scoring here is always against an occurrence set: the same occurrences, every time, for every model. A run compares an algorithm's stored predictions with what people identified. No images are opened and no model runs, so it is a pass over rows and returns at once. Only occurrences a person identified are counted: determination falls back to the model's own prediction when nobody has identified one, and scoring those would have the model agreeing with itself. The species a model can be asked about are resolved to taxa rather than compared as label text, because a category map label is the name the model was trained under and need not be the name the taxon is stored under here. The result also carries what it was measured over, since an accuracy over six of a model's sixty species reads the same as one over all of them. The scores surface in three places: an algorithm's own list of evaluations, a species page showing how each scored model did on it, and the best model for a taxa list, ranked on the per-species average rather than the plain share because trap data is long-tailed. Stacked on the occurrence sets branch, which the scoring runs against. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CnKz4AS4iFrYj1GrgkZQbq
👷 Deploy request for antenna-ssec pending review.Visit the deploys page to approve it
|
✅ Deploy Preview for antenna-preview ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
Author
|
Reopening from a branch in this repository so it can be based on the occurrence sets branch directly. Replaced by the PR linked below. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A project runs more than one classifier over time and has no way to say whether a newer one
is better. Accuracy quoted from different data is not a comparison, so scoring here is
always against an occurrence set: the same occurrences, every time, for every model.
A run compares an algorithm's stored predictions with what people identified. No images are
opened and no model runs, so it is a pass over rows and returns at once.
List of Changes
evaluate_algorithmjob type;ami/ml/evaluation.pycompares stored classifications with identificationsIdentification, not ondetermination— that falls back to the model's own prediction, which would have a model agreeing with itselfspecies_in_setandspecies_predictablein the result and the job log: 1.00 over six of sixty species reads the same as 1.00 over all of themAlgorithmEvaluation(one per algorithm and set) andTaxonEvaluation(one per species); a re-run replaces rather than appendsevaluationsonAlgorithmSerializeralgorithm_performanceonTaxonSerializerbest_modelonTaxaListSerializer, annotated in the viewset so a page costs one queryrun_evaluate_algorithm_job, with an object-level backfill for existing projectsRelated Issues
Built on #1490 (occurrence sets), which this scores against. Split out of #1407, where this
began.
Detailed Description
model that only handles the common species would otherwise win every list.
Every read is scoped to the project, or to what the user can see when the route carries no
project id — the algorithm detail route does not.
ami/ml/migrations/0029collides by number with Store feature vectors from any model for every detection, indexed for the ways they are read #1462. Whichever lands second getsrenumbered; nothing else about them overlaps.
this PR.
Testing
ami.ml.test_evaluation— 30 tests: scoring, the self-scoring and label-matching guards,coverage reporting, cross-project exposure, the per-species and best-model reads, and a
pinned query count for a page of taxa lists.
🤖 Generated with Claude Code
https://claude.ai/code/session_01CnKz4AS4iFrYj1GrgkZQbq