Skip to content

feat: list a project's datapoints with filters and pagination - #2

Merged
Artmann merged 4 commits into
mainfrom
feat/list-datapoints
Oct 18, 2026
Merged

Artmann merged 4 commits into
mainfrom
feat/list-datapoints

Conversation

@Artmann

@Artmann Artmann commented Oct 18, 2026

Copy link
Copy Markdown
Owner

Adds GET /api/v1/datapoints to make it easier to see what is stored for a project when debugging.

What it does

  • Lists datapoints read straight from S3, so it also works while the index is still loading.
  • Pagination with limit (1 to 200, default 50) and an opaque cursor; the response carries nextCursor, null on the last page.
  • Filters: sourceId (direct lookup, no scan), audience (project means none), label (case-insensitive substring), occurredFrom / occurredTo.
  • includeContent=true adds content; it is off by default since it costs an extra read per datapoint.
  • Each item has the stored fields plus contentTokens, contentSha256, embeddingModel and embeddingDim, without the vector.

Trade-offs

  • Pages are in id order, so S3 can list them incrementally with StartAfter, rather than sorted by time.
  • One request reads at most 1000 datapoints. With narrow filters a page can be short or empty but still have a nextCursor.

Docs

README: documents the endpoint, adds the seed script to the quick start, fixes which endpoints return 503 while loading, and refreshes the status and "Not yet" sections.

Testing

  • cargo test --test datapoints -- --ignored against LocalStack: paging, each filter, listing while loading, validation.
  • Existing delete and recall suites, unit tests, clippy and fmt pass.

🤖 Generated with Claude Code

@Artmann
Artmann merged commit 2a51245 into main Oct 18, 2026
3 checks passed
@Artmann
Artmann deleted the feat/list-datapoints branch October 18, 2026 11:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant