Skip to content

feat: Track upstream provider rate-limit headers - #7

Merged
tito merged 8 commits into
mainfrom
feature/track-upstream-provi-a31
Sep 14, 2026
Merged

tito merged 8 commits into
mainfrom
feature/track-upstream-provi-a31

Conversation

@tito

@tito tito commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

What changed

  • Providers accept a rate_limit_headers map from metric key to upstream header name. Six keys: limit_requests, remaining_requests, limit_tokens, remaining_tokens, reset_requests, reset_tokens. Empty means the provider is not tracked.
  • One capture site in internal/server/openai.go, after the upstream response head and before the unary/stream branch. Regression tests cover both paths.
  • The latest reading per provider is kept in memory. Finished minute buckets are written to a new provider_rate_limit_sample table, one row per provider per minute with the last and the lowest remaining value. The open bucket is flushed at shutdown.
  • Proxied responses carry x-llmproxy-ratelimit-{requests,tokens}-{limit,remaining} headers.
  • Prometheus gauges llmproxy_provider_rate_limit_remaining and llmproxy_provider_rate_limit_limit, labelled by provider and unit.
  • GET /stats/rate-limits returns the snapshot per provider with a stale flag. GET /stats/rate-limits/series returns the minute buckets for one provider over a time range.
  • Admin API accepts and validates rate_limit_headers on provider create and patch.
  • UI: a rate-limit block on the provider form with a "Use standard headers" preset, and a Providers sub-tab under Usage with one card per tracked provider showing remaining over limit and the age of the reading. Existing dashboards are unchanged.
  • Docs: providers guide, API reference, architecture rule.

Why

Some upstreams, tensorx today, report their quota on every response. The proxy read none of those headers and did not forward them, so neither callers nor operators could tell how close the provider was to refusing requests. This change observes and exposes the values. It does not throttle.

Testing

  • go test -race -count=1 ./... passes.
  • golangci-lint run reports no issues.
  • UI rebuilt; the committed internal/server/uidist/ matches.
  • mkdocs build --strict passes.

Notes

  • Nothing on this path reads a request or response body. The new column names pass the persistence schema test.
  • Two replicas that roll over the same minute hit the UNIQUE constraint. One row wins and the conflict is ignored.
  • PruneRateLimitSamples exists in the store but nothing calls it yet. One active provider adds about 1440 rows a day. A retention job is a follow-up.
  • Rate limits are tracked per provider, not per model. If tensorx scopes limits per model, the sample key needs an upstream_name column.

New provider_rate_limit_sample table stores per-minute snapshots of
upstream rate-limit headers. The provider table gains a
rate_limit_headers column that maps metric keys to header names.
Parse the provider's rate_limit_headers column into the route at catalog
load time. FormatRateLimitHeaders serialises back for the admin API.
Single capture site in openai.go reads upstream rate-limit headers,
updates the in-memory per-provider tracker, sets x-llmproxy-ratelimit-*
response headers and exports Prometheus gauges. Regression tests cover
both unary and streaming paths.
The snapshot endpoint returns the in-memory reading per provider with a
stale flag. The series endpoint returns stored minute-bucket samples
for one provider over a time range.
Providers return rate_limit_headers in the view, accept them as a
key-to-header map on create and patch, and validate against the six
accepted metric keys.
Add rate_limit_headers configuration block to the provider edit form
with a "Use standard headers" preset button. Add a Providers sub-tab
under Usage that shows one card per configured provider with remaining
over limit meters and staleness indicator.
Add a "Track upstream rate limits" section to the providers guide, the
two new stats endpoints to the API reference, and the design rule to
the architecture document.
@tito tito changed the title Track upstream provider rate-limit headers feat: Track upstream provider rate-limit headers Sep 14, 2026
@tito
tito merged commit 5943b81 into main Sep 14, 2026
5 checks passed
@tito
tito deleted the feature/track-upstream-provi-a31 branch September 14, 2026 19:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant