feat: list a project's datapoints with filters and pagination

.github/workflows/ci.yml

@@ -0,0 +1,72 @@
+name: CI
+
+on:
+  pull_request:
+  push:
+    branches: [main]
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+env:
+  CARGO_TERM_COLOR: always
+  # Debug info only slows linking and bloats the cache in CI.
+  CARGO_PROFILE_DEV_DEBUG: 0
+
+jobs:
+  lint:
+    name: fmt and clippy
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: dtolnay/rust-toolchain@stable
+        with:
+          components: rustfmt, clippy
+      - uses: Swatinem/rust-cache@v2
+      - run: cargo fmt --all -- --check
+      - run: cargo clippy --all-targets --locked -- -D warnings
+
+  test:
+    name: tests
+    runs-on: ubuntu-latest
+    services:
+      # Matches docker-compose.yml; later releases need an auth token.
+      localstack:
+        image: localstack/localstack:4.14
+        ports:
+          - 4566:4566
+        env:
+          SERVICES: s3
+        options: >-
+          --health-cmd "curl -sf http://localhost:4566/_localstack/health"
+          --health-interval 2s
+          --health-timeout 2s
+          --health-retries 30
+    env:
+      AWS_ENDPOINT_URL: http://localhost:4566
+      AWS_ACCESS_KEY_ID: test
+      AWS_SECRET_ACCESS_KEY: test
+      AWS_REGION: us-east-1
+    steps:
+      - uses: actions/checkout@v4
+      - uses: dtolnay/rust-toolchain@stable
+      - uses: Swatinem/rust-cache@v2
+      # The embedding models are large and change rarely.
+      - uses: actions/cache@v4
+        with:
+          path: .fastembed_cache
+          key: fastembed-${{ hashFiles('src/embed.rs') }}
+          restore-keys: fastembed-
+      - name: Unit tests
+        run: cargo test --locked
+      # s3vectors skips itself without TEST_VECTOR_BUCKET.
+      - name: Integration and e2e tests
+        run: cargo test --locked -- --ignored
+      - name: Upload e2e server logs
+        if: failure()
+        uses: actions/upload-artifact@v4
+        with:
+          name: e2e-logs
+          path: target/e2e-logs/
+          if-no-files-found: ignore

.github/workflows/docker.yml

@@ -0,0 +1,32 @@
+name: Docker
+
+# A release build plus the baked-in model takes many minutes, so PRs only
+# build the image when something that affects it changes.
+on:
+  pull_request:
+    paths:
+      - Dockerfile
+      - .dockerignore
+      - Cargo.toml
+      - Cargo.lock
+      - .github/workflows/docker.yml
+  push:
+    branches: [main]
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  docker:
+    name: docker build
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: docker/setup-buildx-action@v3
+      - uses: docker/build-push-action@v6
+        with:
+          context: .
+          push: false
+          cache-from: type=gha
+          cache-to: type=gha,mode=max

README.md

@@ -9,12 +9,17 @@ can paste straight into a prompt.
   AI API.
 - S3 as the only storage backend, with an optional Amazon S3 Vectors
   index for larger data.
-- Idempotent writes.
+- Idempotent writes, and updates and deletes by your own `sourceId`.
+- Webhooks that turn events from other services into datapoints, with no
+  code in between.
 - Everything scoped to a `projectId`.
 
 ## Status
 
-Early but working: everything below is implemented and tested. See
+Early but working: everything below is implemented and tested. That is
+ingest (single and batch), recall, delete, listing stored datapoints, and
+inbound webhooks. There is no auth yet, so run it on a private network.
+[Not yet](#not-yet) lists what is missing. See
 [ARCHITECTURE.md](ARCHITECTURE.md) for how it works and
 [CONTRIBUTING.md](CONTRIBUTING.md) to run it locally.
 
@@ -34,7 +39,16 @@ cp .env.example .env                     # point the AWS SDK at LocalStack
 cargo run                                # first run downloads the embedding model
 ```
 
-The service listens on `http://localhost:8080`.
+The service listens on `http://localhost:8080`. To have something to
+recall, load a set of sample datapoints and list them:
+
+```sh
+cargo run --example seed                 # or: cargo run --example seed -- http://host:8080
+curl 'localhost:8080/api/v1/datapoints?projectId=0192f0c0-5eed-7000-8000-000000000001'
+```
+
+The seed uses `sourceId`s, so running it again updates the same
+datapoints instead of adding copies.
 
 ### Docker
 
@@ -239,6 +253,35 @@ with the `id`, the `sourceId` when given, and `deleted`. `deleted` is
 Deleting a datapoint that was stored with an `idempotencyKey` and then
 retrying that same request stores it again.
 
+### `GET /api/v1/datapoints`
+
+```
+GET /api/v1/datapoints?projectId=0192...uuid&audience=sales&label=call&limit=20
+```
+
+Lists a project's stored datapoints, read straight from S3, to check what
+is in there. It also works while the index is still loading.
+
+| Parameter        | Notes                                                              |
+|------------------|--------------------------------------------------------------------|
+| `projectId`      | required                                                           |
+| `limit`          | 1 to 200, default 50                                               |
+| `cursor`         | `nextCursor` from the previous page                                |
+| `sourceId`       | only the datapoint with this `sourceId`                            |
+| `audience`       | only this audience; `project` means datapoints with none           |
+| `label`          | only labels containing this text, ignoring case                    |
+| `occurredFrom`   | inclusive start on `timestamp`                                     |
+| `occurredTo`     | inclusive end; start after end is `400 recall-range-invalid`       |
+| `includeContent` | `true` to include `content`; off by default since it costs an extra read each |
+
+Returns `{"datapoints": [...], "nextCursor": "..."}`. Each datapoint has
+the same fields as in ingest, without `content` unless asked for, plus
+`contentTokens`, `contentSha256`, `embeddingModel` and `embeddingDim`.
+`nextCursor` is `null` on the last page. Pages are in `id` order, not by
+time. One request reads at most 1000 datapoints, so with narrow filters a
+page can have fewer than `limit` results, or none, and still a
+`nextCursor`; keep going until it is `null`.
+
 ### Webhooks
 
 A webhook turns events that another service POSTs (Jira, GitHub, an alert
@@ -362,14 +405,16 @@ document that does not fit the remaining budget is skipped.
 ### `GET /healthz`
 
 Always `200`. The body is `{"status": "ok"}` once the search index has
-been rebuilt from S3 and `{"status": "loading"}` before that. Ingest and
-recall return `503` with code `loading` until then.
+been rebuilt from S3 and `{"status": "loading"}` before that. Ingest,
+recall, delete and webhook events return `503` with code `loading` until
+then. Listing datapoints and managing webhooks work meanwhile.
 
 ## Not yet
 
 - Auth
-- Filtering on `audience`
-- Metadata filters on recall
+- Filtering on `audience` in recall
+- Metadata filters on recall and listing
+- Signature checks on webhook events
 
 ## License