[matthewberry]

Stacked on #10; merge that first. If #10's branch is deleted on merge, GitHub retargets this PR to main; otherwise change the base to main before merging.

Brings /api/v1/data in line with moleculemaker/CLEANDB-api#20, so the two APIs treat large requests the same way.

Problem

#8 capped every request at 5,000 records, in both formats, to stop single requests from exhausting memory. That made the two APIs inconsistent, and for CSV it removed something useful:

• CSV: a client could no longer get the whole table (35,753 records) in one request. fix: cap explicit page sizes and stop printing query results #8 added Link and X-Total-Count headers so a capped file isn't silently mistaken for a complete one, but a CSV client still has to page.
• JSON: an explicit limit above 5,000 was quietly reduced to 5,000. CLEANDB-api rejects it with a 422 instead.

The memory problem #8 fixed came from the rich.print of every result set, which is gone. Streaming the CSV keeps a full export small.

Changes

• CSV without limit: returns every matching record, up to MAX_DATA_LIMIT (100,000). If the query matches more, it returns a 400 asking the caller to narrow the filters or page with limit/offset; it never returns a partial file. The export requests cap + 1 rows, and write_csv returns the 400 if the extra row arrives, so the cap is checked against the rows actually written, in one query.
• Streaming: rows are read through a server-side cursor (Database.iterate, prefetch 1000) into a SpooledTemporaryFile (in memory up to 8 MB, then on disk). The DB connection is released before the response starts, so a slow client never holds it. The file is closed by a BackgroundTask, which also runs when the client disconnects. The row generators are closed with aclosing, so stopping early (the 400, or an error) releases the cursor's connection immediately.
• CSV with limit: returned as asked, up to 100,000 (fix: reject out-of-range limit and offset; previous past the end goes to the last records #10's bound on limit is now 100,000 instead of the bigint maximum).
• CSV skips the COUNT query, and no longer has fix: cap explicit page sizes and stop printing query results #8's Link/X-Total-Count headers: without a limit there is nothing to page, and with one the caller is paging already.
• CSV columns: the header is columns if given, otherwise the first row's keys, as before. (Unlike CLEANDB-api, this table has more columns than the OEDColumn enum lists.) An empty result with columns has a header-only file; with neither, the file is empty.
• JSON limit capped at the page size: an explicit JSON limit above AUTO_PAGINATION_THRESHOLD (5,000) gets a 422 shaped like the le= error. JSON is paginated, so nothing becomes unreachable. A JSON request without limit is auto-paginated as before, with links on every page.
• get_filtered_data's query building moved into build_data_query, shared with the new iter_filtered_data.
• HTTPException now passes through the endpoint's catch-all instead of being rewrapped as a 500.
• The API description and the /data docstring explain how each format behaves.

API clients that send a JSON limit above 5,000 will get a 422. Within moleculemaker, nothing calls /data that way (the OpenEnzymeDB frontend only links to the API docs, and oed-toolkit doesn't call it).

Testing

• tests/test_data_csv.py, ported from CLEANDB-api's test_search_csv.py: 12 tests with the DB calls mocked, covering the header (from rows, from columns, and header-only when nothing matches), returning more than a JSON page without a limit, the cap (and that the row iterator is closed when it's passed), the explicit-limit path, CSV limits above the JSON cap, a limit above 100,000 (422), a DB error partway through (500), output large enough to spill to disk, and the cursor and row generator being closed early.
• tests/test_auto_pagination.py: fix: cap explicit page sizes and stop printing query results #8's "explicit limit is capped" and "capped CSV has headers" tests are replaced by one asserting a JSON limit of 10 is accepted and 11 gets a 422 with the threshold at 10.
• MockDatabase gains iterate, recording queries like fetch, so the existing CSV integration test runs through the new path.
• Full suite: 44 passed. Against fix: reject out-of-range limit and offset; previous past the end goes to the last records #10's code, the new tests give 4 failures and 9 errors.
• Ran the app against a throwaway Postgres 16 with oed.oed_data created from db_migration/oed_03_data.sql:
  • 35,753 rows (production's count): a full CSV is 35,753 rows, 5.1 MB, in 0.33 s; uvicorn's RSS peaked at about 60 MiB (38 MiB idle), well under fix: give the oed-api container a 1Gi memory limit #7's 1 GiB limit
  • exactly 100,000 matches returns 100,000 rows (7.6 MB, 0.57 s); 100,001 returns the 400; with offset=1 the remaining 100,000 are returned
  • format=csv&limit=100001, JSON limit=5001 and limit=-1 get 422s naming limit; JSON limit=5000 returns a page
  • filtered (ec=…), explicitly paged, and columns= CSV requests return the expected rows and header
  • 15 over-cap exports in parallel all returned 400; before and after, all 10 pool connections were idle, and a JSON request afterwards returned 200
• Not tested on production data or on the cluster.

[matthewberry]

Looks good. Reproduced: full suite 44 passed, and against #10's app/ the new tests give 4 failures and 9 errors, matching the PR body.

Checked:

• Temp file closed on disconnect: holds for the installed versions. uvicorn 0.34.2 reports ASGI spec_version 2.3, which sends Starlette 0.46.2's StreamingResponse down its task-group path, and that path runs background() after a cancel. On spec 2.4 a disconnect would raise ClientDisconnect first and skip the background task. If a future uvicorn upgrade switches to 2.4, the file would then be closed by garbage collection instead.
• Connection release: the nested aclosing (in iter_filtered_data and in write_csv) releases the cursor's connection on the over-cap 400 and on errors. This is from reading the code; the 15-parallel-request run in the PR body is the evidence it works.

Minor: dropping #8's X-Total-Count/Link headers means a CSV client paging with an explicit limit has no total, so it can only stop when it gets a short page. That's reasonable, since without limit it gets everything, but it's worth a line in the docs.

Couldn't reproduce, and it predates this PR: the data query has no ORDER BY, so limit/offset pages aren't guaranteed to be consistent between requests. The new 400 message tells callers to page that way. With a 35k-row table it's unlikely to show up.

[matthewberry]

Thanks for checking the ASGI path. Fixed the docs point in 8605b9e (after merging #10's update in 5e576a9): the API description and the /data docstring now say a CSV response carries no total, so a client paging CSV with limit should stop at the first response with fewer than limit records. Full suite still 44 passed.

On the background task: agreed that on ASGI spec 2.4 a disconnect would skip BackgroundTask and leave the spool file to garbage collection. I haven't changed it, since the installed uvicorn reports 2.3 and CLEANDB-api#20 has the same exposure. If either API moves to a uvicorn that reports 2.4, both should close the file in the response path too.

On ORDER BY: same answer as on #10. It predates these PRs, the table has no key column to sort by, and I'd track it as a separate issue.