Skip to content

fix(openai): retry a transient 429 instead of ending the run - #358

Open
JumpLink wants to merge 1 commit into
danny-avila:mainfrom
faktenforum:fix/retry-transient-429
Open

fix(openai): retry a transient 429 instead of ending the run#358
JumpLink wants to merge 1 commit into
danny-avila:mainfrom
faktenforum:fix/retry-transient-429

Conversation

@JumpLink

Copy link
Copy Markdown
Contributor

Fixes #357.

The problem

Any 429 whose text reads like a spent allowance is never retried. @langchain/core's classifyRateLimitError matches /insufficient[_ -]?quota/i, returns { action: 'stop' }, and defaultFailedAttemptHandler throws before p-retry gets a second attempt. For OpenAI that is right - insufficient_quota means out of credit.

Scaleway answers all three of its rate limits with that wording and nothing else to go on:

HTTP/2 429
content-type: application/json
{"status":429,"error":"INSUFFICIENT QUOTA",
 "message":"You exceeded your current limit of concurrent requests."}

Tokens-per-minute and requests-per-minute differ only in message, which the OpenAI SDK drops - APIError keeps body.error alone - so the classifier only ever sees 429 "INSUFFICIENT QUOTA" and reads a limit that clears in milliseconds as a billing wall. No x-ratelimit-* and no retry-after on the 429 either, although successful responses carry the former.

The fix

An onFailedAttempt handler on the OpenAI-compatible constructors that retries a 429 unless something structured says the allowance is gone:

  • error.code === 'insufficient_quota' - OpenAI's billing wall - stop
  • wording naming billing, credit balance, out of credits, payment - stop
  • a single request larger than the per-minute allowance - stop, and hand it to the existing context-overflow recovery, which shrinks the prompt instead
  • LangChain's never-retry statuses and cancelled requests - stop
  • everything else - return, so p-retry's randomized exponential backoff runs

The discriminator between the two quota meanings is the underscore: insufficient_quota is OpenAI's error code, INSUFFICIENT QUOTA with a space is Scaleway's rate-limit label. A caller-supplied onFailedAttempt still wins, so nothing that already sets one changes behaviour.

Applied to ChatOpenAI (hence Moonshot and OpenRouter), AzureChatOpenAI, ChatDeepSeek and ChatXAI.

Measured

Against api.scaleway.ai on 2026-07-30, driven through ChatOpenAI from this package with the added probe script (npm run probe:ratelimit):

Load stock policy with the handler
150 concurrent 2-token requests 123/150 in 2.5s 150/150 in 3.3s
4 x 30k-token requests (drains a 100k/min bucket) 2/4 in 4.0s 4/4 in 58.3s

The 58s is the token bucket refilling, and that is the trade this makes: a slow turn instead of a dead one. A rejected request is not billed, so the retries cost waiting and nothing else.

Tests

22 unit tests built on real APIError instances generated by the OpenAI SDK rather than hand-written objects, so the assertions run against the shape production sees - including a wiring test that the handler reaches model.caller.onFailedAttempt for all six classes, and that an explicit handler is left alone. src/utils + src/llm/openai: 300 tests green.

The probe script follows context-overflow-probe.ts: it exists so the retry policy stays grounded in what providers send instead of in phrases someone expected them to send.

LangChain refuses to retry any 429 whose text reads like a spent allowance:
`classifyRateLimitError` matches /insufficient[_ -]?quota/i, returns `stop`, and
its default failed-attempt handler throws before p-retry gets a second attempt.
For OpenAI that is right - `insufficient_quota` means the account is out of
credit.

Scaleway answers all three of its rate limits with that wording and nothing else
to go on:

    HTTP/2 429
    {"status":429,"error":"INSUFFICIENT QUOTA",
     "message":"You exceeded your current limit of concurrent requests."}

The tokens-per-minute and requests-per-minute limits differ only in `message`,
which is exactly what the OpenAI SDK drops - `APIError` keeps `body.error`
alone - so the classifier only ever sees `429 "INSUFFICIENT QUOTA"` and reads a
transient limit as a billing wall. There are no rate-limit or retry-after
headers on the 429 either, although successful responses carry the former.

Measured against api.scaleway.ai on 2026-07-30 through ChatOpenAI itself, with
the new probe script:

    150 concurrent tiny requests, stock policy:   123/150 in 2.5s
    150 concurrent tiny requests, this handler:   150/150 in 3.3s
    4 x 30k-token requests, stock policy:           2/4  in 4.0s
    4 x 30k-token requests, this handler:           4/4  in 58.3s

The 58s is the token bucket refilling, and it is the point: a slow turn beats a
dead one. A rejected request is not billed, so the retries cost nothing beyond
the wait.

Terminal cases stay terminal - a structured `insufficient_quota`, wording that
names billing or credit, the never-retry statuses, cancelled requests, and a
single request too large for the per-minute allowance, which is handed to
overflow recovery instead. A caller-supplied `onFailedAttempt` still wins.

Applied to ChatOpenAI (so also Moonshot and OpenRouter), AzureChatOpenAI,
ChatDeepSeek and ChatXAI.

Signed-off-by: JumpLink <pascal@artandcode.studio>
@JumpLink
JumpLink force-pushed the fix/retry-transient-429 branch from 217af02 to 6ffe7ff Compare August 14, 2026 09:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A provider 429 that says "INSUFFICIENT QUOTA" ends the run, even when the limit is per-minute

1 participant