Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/api-overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,7 @@ Guarantees, whatever the limit:
| `LLMProviderRegistry` | Registry of provider factories (`create`, `register`, `has`). |
| `OpenAIProvider`, `AnthropicProvider`, `OllamaProvider`, `OpenRouterProvider` | Provider classes. |
| `createMockProvider()`, `MockLLMProvider` | Deterministic mock provider for tests and demos. |
| `withRetry(provider, opts?)`, `withFallback(providers, opts?)` | Retry transient provider failures with backoff; fall back to the next provider. See [Configuration](configuration.md#provider-retries-and-fallback). |

## Testing

Expand Down
42 changes: 42 additions & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,48 @@ Real providers are resolved by `resolveProvider('<provider>/<model>')`
The `mock` provider needs no credentials and returns canned responses; it is
what the examples and the Quick Start use by default.

## Provider retries and fallback

`withRetry(provider, options)` and `withFallback(providers, options)` wrap any
`LLMProvider` and return another one, so they compose and can be passed
anywhere a provider is accepted:

```ts
import { createAgent, resolveProvider, withFallback, withRetry } from '@loushy/build-ai-agent';

const provider = withFallback(
[
withRetry(resolveProvider('openai/gpt-4o-mini'), {
maxRetries: 3,
backoff: { initialMs: 500, maxMs: 10_000 },
onRetry: ({ attempt, delayMs }) => console.warn(`retry ${attempt} in ${delayMs}ms`),
}),
withRetry(resolveProvider('anthropic/claude-3-5-haiku-latest')),
],
{ onFallback: ({ from, to }) => console.warn(`falling back from ${from} to ${to}`) }
);

const agent = createAgent({ prompt: 'You are helpful.', provider });
```

- `withRetry` retries `generate()` and `stream()` (default `maxRetries: 2`)
on rate limits, timeouts, network errors and 5xx responses, using the same
classification as `compactProviderError()`. Auth failures, invalid requests
and context-length errors are not retried; neither is a cancellation. A
provider's `retryAfterMs` hint (a `Retry-After` header) replaces the backoff
delay. Pass `retryOn(error, attempt)` to change the rule, `timeoutMs` for a
per-attempt time limit, and `signal` to stop retrying.
- A `stream()` call is retried only when it rejects. An error inside a stream
that was already returned is not retried.
- `withFallback` tries each provider in order and rethrows the last error
when all fail. By default it falls back on any error except a cancellation
(`fallbackOn` changes that). Each fallback runs on its own `defaultModel`.
`name` and `defaultModel` report the provider that served the latest call.
- `resilientProvider(provider, { maxRetries, timeout })` applies the
`LLMProviderConfig` fields of the same names.
- The built-in providers also run the `ai` SDK's own internal retries (2 by
default) inside each attempt.

## `createAgent()` options

| Option | Description |
Expand Down
43 changes: 43 additions & 0 deletions llms-full.txt
Original file line number Diff line number Diff line change
Expand Up @@ -1387,6 +1387,7 @@ Guarantees, whatever the limit:
| `LLMProviderRegistry` | Registry of provider factories (`create`, `register`, `has`). |
| `OpenAIProvider`, `AnthropicProvider`, `OllamaProvider`, `OpenRouterProvider` | Provider classes. |
| `createMockProvider()`, `MockLLMProvider` | Deterministic mock provider for tests and demos. |
| `withRetry(provider, opts?)`, `withFallback(providers, opts?)` | Retry transient provider failures with backoff; fall back to the next provider. See [Configuration](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/configuration.md#provider-retries-and-fallback). |

## Testing

Expand Down Expand Up @@ -2359,6 +2360,48 @@ Real providers are resolved by `resolveProvider('<provider>/<model>')`
The `mock` provider needs no credentials and returns canned responses; it is
what the examples and the Quick Start use by default.

## Provider retries and fallback

`withRetry(provider, options)` and `withFallback(providers, options)` wrap any
`LLMProvider` and return another one, so they compose and can be passed
anywhere a provider is accepted:

```ts
import { createAgent, resolveProvider, withFallback, withRetry } from '@loushy/build-ai-agent';

const provider = withFallback(
[
withRetry(resolveProvider('openai/gpt-4o-mini'), {
maxRetries: 3,
backoff: { initialMs: 500, maxMs: 10_000 },
onRetry: ({ attempt, delayMs }) => console.warn(`retry ${attempt} in ${delayMs}ms`),
}),
withRetry(resolveProvider('anthropic/claude-3-5-haiku-latest')),
],
{ onFallback: ({ from, to }) => console.warn(`falling back from ${from} to ${to}`) }
);

const agent = createAgent({ prompt: 'You are helpful.', provider });
```

- `withRetry` retries `generate()` and `stream()` (default `maxRetries: 2`)
on rate limits, timeouts, network errors and 5xx responses, using the same
classification as `compactProviderError()`. Auth failures, invalid requests
and context-length errors are not retried; neither is a cancellation. A
provider's `retryAfterMs` hint (a `Retry-After` header) replaces the backoff
delay. Pass `retryOn(error, attempt)` to change the rule, `timeoutMs` for a
per-attempt time limit, and `signal` to stop retrying.
- A `stream()` call is retried only when it rejects. An error inside a stream
that was already returned is not retried.
- `withFallback` tries each provider in order and rethrows the last error
when all fail. By default it falls back on any error except a cancellation
(`fallbackOn` changes that). Each fallback runs on its own `defaultModel`.
`name` and `defaultModel` report the provider that served the latest call.
- `resilientProvider(provider, { maxRetries, timeout })` applies the
`LLMProviderConfig` fields of the same names.
- The built-in providers also run the `ai` SDK's own internal retries (2 by
default) inside each attempt.

## `createAgent()` options

| Option | Description |
Expand Down
19 changes: 19 additions & 0 deletions src/providers/abortableDelay.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
/**
* Waits `ms`, rejecting with the signal's reason as soon as it is aborted
* (LOU-V1), or at once when it already is. Shared by the mock provider's
* simulated latency and `withRetry()`'s backoff (LOU-V7.1).
*/
export function abortableDelay(ms: number, signal?: AbortSignal): Promise<void> {
return new Promise((resolve, reject) => {
signal?.throwIfAborted(); // rejects this promise
const onAbort = () => {
clearTimeout(timer);
reject(signal?.reason);
};
const timer = setTimeout(() => {
signal?.removeEventListener('abort', onAbort);
resolve();
}, ms);
signal?.addEventListener('abort', onAbort, { once: true });
});
}
1 change: 1 addition & 0 deletions src/providers/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ export * from './OllamaProvider';
export * from './OpenRouterProvider';
export * from './AnthropicProvider';
export * from './resolveProvider';
export * from './resilience';
export { MissingPeerDependencyError } from './optionalPeer';

// Auto-register built-in providers. The factories only construct provider
Expand Down
19 changes: 1 addition & 18 deletions src/providers/mock.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ import {
StreamChunk,
ToolCall,
} from './llm';
import { abortableDelay } from './abortableDelay';

/**
* Mock response configuration
Expand All @@ -24,24 +25,6 @@ export interface MockProviderConfig extends LLMProviderConfig {
errorMessage?: string;
}

/**
* Waits `ms`, rejecting with the signal's reason as soon as it is aborted
* (LOU-V1), so a simulated slow call can be cancelled mid-delay.
*/
function abortableDelay(ms: number, signal?: AbortSignal): Promise<void> {
return new Promise((resolve, reject) => {
const onAbort = () => {
clearTimeout(timer);
reject(signal?.reason);
};
const timer = setTimeout(() => {
signal?.removeEventListener('abort', onAbort);
resolve();
}, ms);
signal?.addEventListener('abort', onAbort, { once: true });
});
}

/**
* Mock LLM Provider
*/
Expand Down
Loading
Loading