Repository navigation
feat: Track upstream provider rate-limit headers - #7
Merged
Merged
Conversation
New provider_rate_limit_sample table stores per-minute snapshots of upstream rate-limit headers. The provider table gains a rate_limit_headers column that maps metric keys to header names.
Parse the provider's rate_limit_headers column into the route at catalog load time. FormatRateLimitHeaders serialises back for the admin API.
Single capture site in openai.go reads upstream rate-limit headers, updates the in-memory per-provider tracker, sets x-llmproxy-ratelimit-* response headers and exports Prometheus gauges. Regression tests cover both unary and streaming paths.
The snapshot endpoint returns the in-memory reading per provider with a stale flag. The series endpoint returns stored minute-bucket samples for one provider over a time range.
Providers return rate_limit_headers in the view, accept them as a key-to-header map on create and patch, and validate against the six accepted metric keys.
Add rate_limit_headers configuration block to the provider edit form with a "Use standard headers" preset button. Add a Providers sub-tab under Usage that shows one card per configured provider with remaining over limit meters and staleness indicator.
Add a "Track upstream rate limits" section to the providers guide, the two new stats endpoints to the API reference, and the design rule to the architecture document.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
rate_limit_headersmap from metric key to upstream header name. Six keys:limit_requests,remaining_requests,limit_tokens,remaining_tokens,reset_requests,reset_tokens. Empty means the provider is not tracked.internal/server/openai.go, after the upstream response head and before the unary/stream branch. Regression tests cover both paths.provider_rate_limit_sampletable, one row per provider per minute with the last and the lowest remaining value. The open bucket is flushed at shutdown.x-llmproxy-ratelimit-{requests,tokens}-{limit,remaining}headers.llmproxy_provider_rate_limit_remainingandllmproxy_provider_rate_limit_limit, labelled by provider and unit.GET /stats/rate-limitsreturns the snapshot per provider with a stale flag.GET /stats/rate-limits/seriesreturns the minute buckets for one provider over a time range.rate_limit_headerson provider create and patch.Why
Some upstreams, tensorx today, report their quota on every response. The proxy read none of those headers and did not forward them, so neither callers nor operators could tell how close the provider was to refusing requests. This change observes and exposes the values. It does not throttle.
Testing
go test -race -count=1 ./...passes.golangci-lint runreports no issues.internal/server/uidist/matches.mkdocs build --strictpasses.Notes
PruneRateLimitSamplesexists in the store but nothing calls it yet. One active provider adds about 1440 rows a day. A retention job is a follow-up.upstream_namecolumn.