Skip to content

CSV search export silently truncates at AUTO_PAGINATION_THRESHOLD #18

Closed

Description

@matthewberry

When /api/v1/search is called with format=csv and no limit, limit defaults to AUTO_PAGINATION_THRESHOLD (5000, app/core/config.py). The CSV response carries no total count and no next-page link, so a query matching more than 5000 records downloads a file that looks complete but isn't (e.g. ?ec_number=1.1.1.1&format=csv).

Related: get_total_count runs on every CSV request (app/routers/search.py) and its result is discarded, costing an extra COUNT over the joined views.

Options:

  • Stream the full result set for CSV.
  • Send the total (e.g. X-Total-Count) and pagination Link headers, which would also put the count to use.
  • Document the limit and require explicit paging.

Raised in review of #17.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions