Skip to content

Add browser SAM2 point-click segmentation (Segment tool) - #545

Draft
BryonLewis wants to merge 17 commits into
mainfrom
hf-sam-integration
Draft

Add browser SAM2 point-click segmentation (Segment tool)#545
BryonLewis wants to merge 17 commits into
mainfrom
hf-sam-integration

Conversation

@BryonLewis

@BryonLewis BryonLewis commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds a Segment tool (wand icon, s hotkey) that outlines an object from a few clicks using SAM2.1 running in the browser via transformers.js / ONNX Runtime Web. Ported from the DIVE implementation; no server, worker or queue setup needed.

  • Left click: include point (green). Shift+click or middle click: exclude point (red).
  • Each click previews the polygon on the selected detection; Enter, right click or Confirm keeps it and switches to normal polygon editing.
  • Esc (or Reset) clears the points and restores the detection; Esc again leaves editing.
  • Works for image sequences and video, per frame, and with continuous Detection / auto-advance Track creation.
  • Model (SAM2.1 Tiny/Small) and device (Auto/GPU/CPU) settings in the new-annotation settings menu.
  • Model files come from the Hugging Face Hub on first use and are kept in the browser Cache API; VUE_APP_SAM_MODEL_HOST can point at a mirror. ONNX Runtime's .wasm ships with the client build.

New docs page: docs/Interactive-Segmentation.md (linked from the shortcuts, editing bar, UI settings and SAM2 mask tracking pages).

Differences from DIVE worth reviewing

  • Mask post-processing is plain TypeScript (samPostprocess.ts) instead of DIVE's /models/sam_postprocess.onnx; dive-dsa serves static files under /static/dive/, and this avoids hosting an extra model.
  • transformers.js is imported only by transformersLoader.ts, typed with local structural types (samTypes.ts): its declarations don't parse with this repo's TypeScript 4.3.
  • The largest mask component is stored under the default polygon key '', so a result edits/deletes/exports like a drawn polygon; extra components are SegmentationPolygon-N. PolygonBase now handles keyed polygons and keeps the box around all of them.
  • Toolbar buttons map to fixed UIEditingTypes indexes (Segment = 5) so saved configurations don't shift.

Pre-existing bugs fixed (separate commits)

  • 7fa20d7 Esc never left edit mode: visualMaskManager.editingMaskId is a Ref, so !== null was always true (since Visual Masking Configuration #420).
  • e0cc351 Clicking over a visual mask while drawing deselected the track.
  • Escape now also removes a selected track left with no features; deleting a track's only keyframe left it out of sync with the store's interval tree (worked around on preview restore; Track.deleteFeature itself is unchanged).

Testing

  • New Jest specs: post-processing, mask geometry, polygon helpers, frame capture, SamOnnx (fake transformers module), the recipe, track snapshot/restore, PolygonBase, and a mode-manager integration spec. 135 tests pass; lint clean; every commit from the runtime onward passes lint + tests on its own; vite build succeeds.
  • Manually verified in the browser against a local stack on a video dataset (WebGPU): include/shift/middle-click points, Esc reset then exit, Enter and right-click confirm into polygon editing, abandoning a new detection, continuous detection mode, settings panel.

Note on branch contents

This branch was based on multi-launch-configs, which isn't merged into main, so its 7 commits (cf063a0…54ba352) also appear here. Merging that PR first (or retargeting this one) will leave only the 9 segmentation commits.

🤖 Generated with Claude Code

@BryonLewis
BryonLewis marked this pull request as draft October 16, 2026 17:54
BryonLewis and others added 10 commits October 20, 2026 07:53
@BryonLewis @claude
Fix Esc not leaving edit mode
a1dd0ac 
editingMaskId is a Ref, so comparing it to null was always true and Esc
always took the "stop editing visual mask" branch, never aborting track
editing. Compare its value instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@BryonLewis @claude
Keep visual mask clicks from deselecting while drawing
5329cec 
A left click over a visual mask cleared the track selection even while
the edit layer was creating a shape, so placing polygon vertices on top
of a visual mask dropped out of editing. Ignore those clicks during
creation, as the track click handler already does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@BryonLewis @claude
Add browser SAM2 runtime for point-prompt segmentation
9f9901e 
Run SAM2.1 (Tiny or Small, q4 ONNX exports from the Hugging Face Hub)
in the browser through transformers.js. The model downloads once and is
kept in the browser Cache API; each frame is encoded once and later
clicks only run the prompt decoder. WebGPU is used on hardware GPUs with
a WebAssembly fallback, and VUE_APP_SAM_MODEL_HOST can point at a model
mirror.

Ported from DIVE with these differences:
- mask post-processing (best candidate, unpad, resize, threshold) is
  plain TypeScript instead of a served ONNX graph, so nothing has to be
  hosted under /models
- transformers.js is only imported by transformersLoader.ts with local
  structural types, since its declarations need a newer TypeScript
- mask outlines are returned largest component first, dropping specks
- frameCapture reads native pixels from the image/video GeoJS quad

Exclude onnxruntime-web from Vite dependency pre-bundling so its .wasm
resolves in the dev server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@BryonLewis @claude
Add point-click segmentation recipe
d63ebb2 
SegmentationPointClick collects include/exclude points per frame, runs
the platform predict function and emits each prediction, so the mode
manager can preview it on the detection. Confirm emits every frame with
a polygon; reset asks listeners to restore the detection. Stale or
rejected predictions are dropped without leaving the tool.

segmentationTrack holds the track side: write a mask as the frame's
polygons and box (largest component under the default polygon key,
others as SegmentationPolygon-N), and snapshot/restore a keyframe. A
restored brand-new detection keeps begin/end in step with the store's
interval tree so it can still be removed.

Recipes may now declare their own toolbar hotkey.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@BryonLewis @claude
Support keyed polygons in PolygonBase
5824235 
Delete and vertex removal now work on any polygon key, editing one
polygon keeps the box around the frame's other polygons, and activate()
accepts the key to edit. Needed for multi-component segmentation masks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@BryonLewis @claude
Place include/exclude segmentation points in the annotator
2217c2c 
In Point mode the edit layer marks shift+click and a quick middle click
(a middle drag still pans) as background points, and a right click on
the map confirms. The confirm waits until GeoJS has handled the click,
since some platforms fire contextmenu on mousedown and leaving point
placement early let other layers treat the click as a selection.

An optional provideSegmentationPrompt() lets LayerManager draw the
prompt points (green include, red exclude) for the frame and camera
being segmented, and forward right-click confirms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@BryonLewis @claude
Run point-click segmentation sessions in the mode manager
163da03 
Each prediction is written to the selected detection as a preview,
snapshotting the keyframe first. Reset restores it while staying in the
tool; leaving the detection or edit mode discards an unconfirmed mask;
switching to another tool keeps it. Confirming switches to polygon
editing of the result, or keeps segmenting the next detection/frame in
continuous creation. Prediction errors are shown in a prompt.

Escape now also removes a selected track left with no features (e.g. a
new detection whose only preview was discarded).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@BryonLewis @claude
Add the Segment (SAM2) tool to the editor
3f383fe 
Wire the point-click recipe into the Viewer through useSamSegmentation,
which reads the selected camera's frame (loading the expected image URL
if the map still shows the previous frame) and runs SamOnnx on it.

- wand button with the `s` hotkey; Enter or right click confirms
- Reset/Confirm controls and model download/encode status in the
  editing bar, with a "Segmenting" header and instructions
- Esc clears segmentation points before it leaves editing
- Model (SAM2.1 Tiny/Small) and device (Auto/GPU/CPU) settings in the
  new-annotation settings menu
- toolbar buttons map to fixed UIEditingTypes indexes so adding Segment
  (index 5) does not shift saved configurations; the ToolBar settings
  editor lists mask and segment

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@BryonLewis @claude
Document interactive SAM2 segmentation
53e1186 
Add a user guide for the Segment tool (usage, what is stored, settings,
model caching, offline hosting, troubleshooting) and reference it from
the shortcuts, editing bar, UI settings and SAM2 mask tracking pages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@BryonLewis
job race condition fix
14a9933

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@BryonLewis