Script for matching User Edits

docs/scripting/Endpoints.md

@@ -35,6 +35,11 @@ Operating on the Annotations for a specific DIVE Dataset Id.  These incloude get
 - **Usage:** This endpoint is used to get detailed information about specific tracks within a dataset, including their attributes and associated detections.  There are options to retrieve the annotations at specific revisions
 - **Query parameters:** `revision` (optional), `set` (optional string). Omit `set` for the default annotation collection.
 
+#### `dive_annotation/track/authors`
+- **Method:** GET
+- **Usage:** Lists each track visible at a revision (latest by default) with the revision that saved it and that revision's author. Each row has `id`, `begin`, `end`, `featureCount`, `rev_created`, `author_id`, `author_name`, `created` and `description`. A `description` of `save` is a manual edit in the annotator; `Import ...` is an uploaded file or pipeline output. A save rewrites the whole track, so the author is the last user to change the track, not necessarily the creator of every detection. Paged with `limit`/`offset` and returns the total in the `Girder-Total-Count` header. For ops dumps that need full history, feature frames, or image-name mapping, see the Mongo sample script in `samples/scripts/userEdits/`.
+- **Query parameters:** `revision` (optional), `set` (optional string), `limit`, `offset`, `sort`.
+
 #### `dive_annotation/group`
 - **Method:** GET
 - **Usage:** Same as `dive_annotation/track`, but returns group annotations. Supports `revision` and `set` query parameters.

samples/scripts/userEdits/README.md

@@ -0,0 +1,87 @@
+# DIVE Track Authors
+
+`trackAuthors.py` lists the current version of each track in a dataset (`trackItem`) along with the user who last saved it and its `rev_created` (from `revisionLogItem`). Pass `--all-versions` to include every replaced or deleted version as well. For image-sequence datasets it also maps frame numbers to image names, using the same ordering as the server.
+
+Each track version is joined to the revision log through `rev_created`. A save writes out the whole track again, so authorship is per track version, not per detection.
+
+The revision `description` shows where a version came from:
+
+| description | Source |
+|-------------|--------|
+| `save` | Manual edit in the DIVE annotator |
+| `Import <type> from <file>` | Uploaded annotation file or pipeline output |
+| `initialize clone` | Dataset clone (earlier history lives on the source dataset) |
+
+## Prefer the HTTP API when you can
+
+For normal scripting against a running DIVE server, use
+[`GET dive_annotation/track/authors`](../../../docs/scripting/Endpoints.md)
+with a Girder token. That endpoint respects dataset ACLs, returns the author of
+each visible track at a revision, and does not need Mongo access.
+
+Use this script only for ops/admin work that needs Mongo-only options the API
+does not expose (`--all-versions`, `--features`, image-name mapping, full
+revision log dump). It talks to Mongo directly and **bypasses Girder access
+control**, so anyone with the URI can read every dataset.
+
+## Requirements
+
+- [uv](https://docs.astral.sh/uv/) (installs `click` and `pymongo` automatically)
+- Network access to the Girder MongoDB
+
+The default `docker-compose.yml` does not publish the Mongo port, so `localhost:27017` is refused. Either pass `--container dive-mongo-1` (the script looks up the container IP with `docker inspect`), pass `--mongo-uri` with that IP, or add `ports: ["27017:27017"]` to the `mongo` service in an override file.
+
+## Usage
+
+From the repository root:
+
+```bash
+# Current version of each track, with its author, rev_created and begin/end image names
+uv run samples/scripts/userEdits/trackAuthors.py <folderId> --container dive-mongo-1
+
+# Full history (every saved version), each feature's frame and image name, and the revision log
+uv run samples/scripts/userEdits/trackAuthors.py <folderId> --all-versions --features --revisions
+
+# A different Mongo host, a named annotation set, output to a file
+uv run samples/scripts/userEdits/trackAuthors.py <folderId> \
+  --mongo-uri mongodb://172.18.0.6:27017/girder --set mySet -o tracks.json
+```
+
+HTTP equivalent (token-authenticated, ACL-aware):
+
+```bash
+curl -H "Girder-Token: $TOKEN" \
+  "$GIRDER/api/v1/dive_annotation/track/authors?folderId=<folderId>"
+```
+
+## Output
+
+```json
+{
+  "datasetId": "...",
+  "name": "...",
+  "type": "image-sequence",
+  "imageCount": 326,
+  "tracks": [
+    {
+      "id": 0,
+      "begin": 238,
+      "end": 238,
+      "beginImage": "20171027.214850.033.029373.png",
+      "endImage": "20171027.214850.033.029373.png",
+      "featureCount": 1,
+      "types": ["Motion"],
+      "rev_created": 10,
+      "rev_deleted": null,
+      "author_name": "jdoe",
+      "author_id": "...",
+      "created": "...",
+      "description": "save"
+    }
+  ]
+}
+```
+
+Track fields use the same names as `GET dive_annotation/track/authors` (`id`, `begin`, `end`, `featureCount`, `rev_created`, `author_id`, `author_name`, `created`, `description`), so output built on this script keeps working if you switch to the API. `beginImage`, `endImage`, `types`, `rev_deleted` and `features` are script-only extras.
+
+`rev_deleted` is always `null` by default; with `--all-versions` it is the revision that replaced or deleted that version. Image names are `null` for video datasets. Pass `--revisions` to add a `revisions` array with the dataset's full revision log (author, time, description, additions, deletions).

samples/scripts/userEdits/trackAuthors.py

@@ -0,0 +1,266 @@
+# /// script
+# requires-python = ">=3.8"
+# dependencies = [
+#     "click",
+#     "pymongo",
+# ]
+# ///
+"""
+List the trackItem versions of a DIVE dataset with who saved each one.
+
+Each saved version of a track is a document in the trackItem collection whose
+rev_created points at a revisionLogItem entry (author, time, description).
+For image-sequence datasets the frame numbers are mapped to image names using
+the same ordering the server uses (server/dive_server/crud.py valid_images).
+
+Run with uv:
+  uv run samples/scripts/userEdits/trackAuthors.py <folderId>
+  uv run samples/scripts/userEdits/trackAuthors.py <folderId> --container dive-mongo-1
+  uv run samples/scripts/userEdits/trackAuthors.py <folderId> --all-versions --features --revisions
+  uv run samples/scripts/userEdits/trackAuthors.py <folderId> --mongo-uri mongodb://localhost:27017/girder -o out.json
+"""
+
+from __future__ import annotations
+
+import functools
+import itertools
+import json
+import re
+import subprocess
+from typing import Any, Dict, List, Optional, Union
+
+import click
+from bson import ObjectId
+from bson.errors import InvalidId
+from pymongo import MongoClient
+from pymongo.errors import ServerSelectionTimeoutError
+
+# dive_utils/constants.py (safeImageRegex is case-insensitive)
+WEB_VALID_IMAGE_FORMATS = ("png", "jpg", "jpeg")
+SAFE_IMAGE_REGEX = r"(\." + r"|\.".join(WEB_VALID_IMAGE_FORMATS) + r")$"
+IMAGE_SEQUENCE_TYPE = "image-sequence"
+FOREIGN_MEDIA_ID_MARKER = "foreign_media_id"
+NUMBERS_REGEX = re.compile(r"(\d+)")
+FEATURE_COUNT = {"$size": {"$ifNull": ["$features", []]}}
+
+
+def _maybe_int(value: str) -> Union[str, int]:
+    try:
+        return int(value)
+    except ValueError:
+        return value
+
+
+def _str_chunks(value: str) -> List[Union[int, str]]:
+    return [_maybe_int(v) for v in NUMBERS_REGEX.split(value) if v != ""]
+
+
+def str_numeric_compare(input1: str, input2: str) -> int:
+    """Mirror of dive_utils.strNumericCompare so frame order matches the server."""
+    if input1 == input2:
+        return 0
+    for a, b in itertools.zip_longest(_str_chunks(input1), _str_chunks(input2), fillvalue=None):
+        if a == b:
+            continue
+        if a is None:
+            return -1
+        if b is None:
+            return 1
+        if type(a) == int and type(b) == int:
+            return a - b
+        if type(a) == int:
+            return -1
+        if type(b) == int:
+            return 1
+        return 1 if str(a) > str(b) else -1
+    return 0
+
+
+def get_clone_root(db, folder: Dict[str, Any]) -> Dict[str, Any]:
+    """Follow foreign_media_id so clones resolve to the folder holding the media."""
+    next_id = folder.get(FOREIGN_MEDIA_ID_MARKER)
+    while next_id:
+        source = db.folder.find_one({"_id": ObjectId(str(next_id))})
+        if source is None:
+            raise click.ClickException(f"Clone source folder {next_id} no longer exists")
+        folder = source
+        next_id = folder.get(FOREIGN_MEDIA_ID_MARKER)
+    return folder
+
+
+def image_names(db, folder: Dict[str, Any]) -> List[str]:
+    media_folder = get_clone_root(db, folder)
+    items = db.item.find(
+        {
+            "folderId": media_folder["_id"],
+            "lowerName": {"$regex": SAFE_IMAGE_REGEX, "$options": "i"},
+        },
+        {"name": 1},
+    )
+    names = [item["name"] for item in items]
+    return sorted(names, key=functools.cmp_to_key(str_numeric_compare))
+
+
+def container_ip(container: str) -> str:
+    try:
+        result = subprocess.run(
+            [
+                "docker",
+                "inspect",
+                "-f",
+                "{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}",
+                container,
+            ],
+            capture_output=True,
+            text=True,
+            check=True,
+        )
+    except (OSError, subprocess.CalledProcessError) as err:
+        stderr = getattr(err, "stderr", "") or str(err)
+        raise click.ClickException(f"Could not inspect container {container}: {stderr.strip()}")
+    ips = result.stdout.split()
+    if not ips:
+        raise click.ClickException(f"Container {container} has no network IP (is it running?)")
+    return ips[0]
+
+
+def frame_name(images: Optional[List[str]], frame: int) -> Optional[str]:
+    if images is None or frame < 0 or frame >= len(images):
+        return None
+    return images[frame]
+
+
+@click.command(name="trackAuthors", help="List trackItem versions and their authors for a dataset")
+@click.argument("folder_id")
+@click.option(
+    "--mongo-uri",
+    default="mongodb://localhost:27017/girder",
+    show_default=True,
+    help="Girder MongoDB URI (database name taken from the path)",
+)
+@click.option(
+    "--container",
+    default=None,
+    help="Docker container running Mongo (e.g. dive-mongo-1); its IP overrides --mongo-uri",
+)
+@click.option(
+    "--all-versions",
+    is_flag=True,
+    help="Include replaced and deleted versions (default: only the current version of each track)",
+)
+@click.option(
+    "--revisions", "include_revisions", is_flag=True, help="Include the dataset's full revision log"
+)
+@click.option("--features", is_flag=True, help="Include every feature frame (and image name)")
+@click.option("--set", "annotation_set", default=None, help="Annotation set name (default set if omitted)")
+@click.option("-o", "--output", type=click.Path(dir_okay=False), help="Write JSON here instead of stdout")
+def main(folder_id, mongo_uri, container, all_versions, include_revisions, features, annotation_set, output):
+    if container:
+        mongo_uri = f"mongodb://{container_ip(container)}:27017/girder"
+    client = MongoClient(mongo_uri, serverSelectionTimeoutMS=5000)
+    try:
+        client.admin.command("ping")
+    except ServerSelectionTimeoutError:
+        raise click.ClickException(
+            f"Could not reach MongoDB at {mongo_uri}. The default docker-compose does not publish "
+            "port 27017; try --container dive-mongo-1 or pass --mongo-uri with the container IP."
+        )
+    db = client.get_default_database()
+    try:
+        dataset_id = ObjectId(folder_id)
+    except InvalidId:
+        raise click.ClickException(f"Invalid folder id: {folder_id}")
+
+    folder = db.folder.find_one({"_id": dataset_id})
+    if folder is None:
+        raise click.ClickException(f"Folder {folder_id} not found")
+    dataset_type = folder.get("meta", {}).get("type")
+    images = image_names(db, folder) if dataset_type == IMAGE_SEQUENCE_TYPE else None
+
+    # Revision numbers are unique per dataset across all sets; the default set is stored as None
+    track_query: Dict[str, Any] = {"dataset": dataset_id, "set": annotation_set or None}
+    if not all_versions:
+        track_query["rev_deleted"] = {"$exists": False}
+
+    revisions = {r["revision"]: r for r in db.revisionLogItem.find({"dataset": dataset_id})}
+    if annotation_set:
+        revisions = {k: r for k, r in revisions.items() if r.get("set") == annotation_set}
+    else:
+        revisions = {k: r for k, r in revisions.items() if not r.get("set")}
+
+    # Count features in Mongo so the heavy features array is only loaded with --features
+    track_fields: Dict[str, Any] = {
+        "_id": 0,
+        "id": 1,
+        "begin": 1,
+        "end": 1,
+        "confidencePairs": 1,
+        "rev_created": 1,
+        "rev_deleted": 1,
+        "featureCount": FEATURE_COUNT,
+    }
+    if features:
+        track_fields["features"] = 1
+
+    tracks = []
+    for track in db.trackItem.find(track_query, track_fields).sort([("id", 1), ("rev_created", 1)]):
+        rev = revisions.get(track.get("rev_created"), {})
+        entry = {
+            "id": track["id"],
+            "begin": track.get("begin"),
+            "end": track.get("end"),
+            "beginImage": frame_name(images, track.get("begin", -1)),
+            "endImage": frame_name(images, track.get("end", -1)),
+            "featureCount": track.get("featureCount", 0),
+            "types": [pair[0] for pair in track.get("confidencePairs", [])],
+            "rev_created": track.get("rev_created"),
+            "rev_deleted": track.get("rev_deleted"),
+            "author_name": rev.get("author_name"),
+            "author_id": str(rev["author_id"]) if rev.get("author_id") else None,
+            "created": rev["created"].isoformat() if rev.get("created") else None,
+            # "save" is a manual edit in the UI, "Import ..." is a file or pipeline import
+            "description": rev.get("description"),
+        }
+        if features:
+            entry["features"] = [
+                {
+                    "frame": f["frame"],
+                    "image": frame_name(images, f["frame"]),
+                    "keyframe": f.get("keyframe"),
+                    "interpolate": f.get("interpolate"),
+                }
+                for f in track.get("features", [])
+            ]
+        tracks.append(entry)
+
+    result = {
+        "datasetId": folder_id,
+        "name": folder.get("name"),
+        "type": dataset_type,
+        "imageCount": len(images) if images is not None else None,
+        "tracks": tracks,
+    }
+    if include_revisions:
+        result["revisions"] = [
+            {
+                "revision": r["revision"],
+                "author_name": r.get("author_name"),
+                "created": r["created"].isoformat() if r.get("created") else None,
+                "description": r.get("description"),
+                "additions": r.get("additions"),
+                "deletions": r.get("deletions"),
+            }
+            for r in sorted(revisions.values(), key=lambda r: r["revision"])
+        ]
+
+    text = json.dumps(result, indent=2)
+    if output:
+        with open(output, "w") as f:
+            f.write(text)
+        click.echo(f"Wrote {len(tracks)} track versions to {output}", err=True)
+    else:
+        click.echo(text)
+
+
+if __name__ == "__main__":
+    main()