diff --git a/CHANGELOG.md b/CHANGELOG.md index 1c42fffa1..69f0e6e7b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] - 2026-09-28 ### Added +- Structured output (LOU-V4): `createAgent({ output: zodSchema })` (and `ExecuteOptions.output`) makes the final reply a JSON object validated by the schema. `send()`, `run.result` and `agent.approvals.resolve()` resolve with it as `result.object`, typed `z.output` from `send()` and `stream()` of `createAgent()` agents (`SimpleAgent`, `AgentRun` and `ExecutionResult` gain a type parameter that defaults to `unknown`); `result.text` keeps the raw JSON. The system prompt gets an `## Output format` section with the schema as JSON Schema, and each model call carries the new `GenerateOptions.responseFormat: { type: 'json', schema? }` hint, which the `ai`-SDK providers map to `experimental_output` JSON mode. The reply is parsed (a code fence is tolerated) and validated; when invalid, one repair step sends the model the issues (it counts against `maxSteps`), and a reply still invalid ends the run with the new `finishReason: 'output-invalid'`, no `object` and `outputError: { message, issues }`. `run.done` gains an optional `object` field (additive: the event schema version stays 1). See docs/structured-output.md. - Background sub-agents (LOU-Y4): the `task` tool takes `background: true`, which starts the sub-agent and returns `{ taskId, status: 'running', agent }` at once. Agents with `subagents` also get `agent_status({ taskId? })` (`queued`/`running`/`done`/`failed`/`cancelled`/`awaiting-approval`, plus `elapsedMs`), `agent_await({ taskId | taskIds, timeoutMs? })` (the synchronous `task` result, or `status: 'timeout'`) and `agent_cancel({ taskId })`. `withSubagentOptions(subagents, { maxConcurrent })` bounds how many run at once per lead run (default 3; the rest queue). Background sub-agents inherit the lead run's runtime and `maxSubagentDepth`, and aborting the lead cancels them. Not yet (LOU-Y4.2): cancelling or awaiting them when the lead run finishes, and resuming one that paused for approval (it reports `awaiting-approval` with its `approvalId`). A user tool named `agent_status`, `agent_await` or `agent_cancel` now conflicts with `subagents`, like `task`. See docs/sub-agents.md. - React hook (LOU-D15): new `@loushy/build-ai-agent/react` subpath with `useLoushyAgent(source, options?)`. `source` is `{ agent, sessionId? }` (in process, via `agent.stream()` or `agent.session({ id }).stream()`) or `{ url, headers?, fetch? }` (POSTs `{ input }` and reads the SSE or NDJSON event stream). It returns `messages` (text and tool calls with status, args and result), `status` (`idle`/`streaming`/`awaiting-approval`/`error`), `pendingApproval`, `error`, `usage`, `lastEvent`, `send()`, `stop()` (aborts; unmount aborts too), `approve(note?)` and `reject(note?)` (via `agent.approvals.resolve()` in process, or `POST ${approvalsUrl}/${id}` remotely). The framework-neutral `reduceAgentEvents()` reducer and `parseEventStream()` parser are exported from the same subpath. `react` (`^18 || ^19`) is an optional peer dependency. See docs/react.md. - Summarizing compaction (LOU-W3): `summarizeStrategy({ model, protectedTokens?, prompt?, maxSummaryTokens? })` replaces the turns before the protected tail with one `[Conversation summary]` user message written by `model` (an `LLMProvider` or a `"provider/model"` spec), keeping system and pinned messages and each assistant turn together with its tool results; if the summary call fails it falls back to pruning and reports `error`. `twoPhaseStrategy()` (the recommended strategy) prunes first and summarizes only when still above the threshold. `pinMessage()` / `isPinned()` and the new optional `Message.metadata` (never sent to providers) mark messages no built-in strategy compacts. `CompactionStrategy.compact()` may now be async, so `compactMessages()` returns a Promise; `CompactionResult` / `CompactionInfo` gain `summary` and `error`, and `CompactionInput` gains `thresholdTokens` and `signal`. A strategy that throws no longer fails the run: the hook reports the `error` to `onCompaction`. See docs/compaction.md. diff --git a/README.md b/README.md index e9a153562..db7a8080f 100644 --- a/README.md +++ b/README.md @@ -72,6 +72,7 @@ agent can be an `agent.yaml` spec served with `npx loushy dev agent.yaml` - **Tools**: `defineTool()` with a zod `input`; arguments are typed, validated, and run in parallel. [Tools](docs/tools.md) - **Approvals**: `needsApproval` pauses a run; `agent.approvals.resolve()` continues it, or `approve` decides in code. [Approvals](docs/approvals.md) - **Sessions**: `agent.session()` keeps a multi-turn conversation in memory, files or SQLite. [Sessions](docs/sessions.md) +- **Structured output**: `output: zodSchema` makes the final reply a typed, validated `result.object`, with one repair step. [Structured output](docs/structured-output.md) - **Streaming**: `agent.stream()` and `session.stream()` yield typed, versioned JSON events ready for SSE. [Streaming](docs/streaming.md) - **UI bindings**: `useLoushyAgent()` from `@loushy/build-ai-agent/react` turns the event stream into chat state, with approvals. [React](docs/react.md) - **Durable execution**: `sessionId` + `checkpointStore` resume a crashed or paused run without redoing finished tools. [Durable execution](docs/durable-execution.md) @@ -177,6 +178,7 @@ await AgentExecutor.execute({ | [Tools](docs/tools.md) | `defineTool()`, validation and errors, built-in tools, `ToolRegistry` | | [Approvals](docs/approvals.md) | `needsApproval`, `agent.approvals`, the `approve` callback, `resumeAfterApproval()`, stores | | [Sessions](docs/sessions.md) | Multi-turn conversations, `session.stream()`, session stores, `SqliteStore` | +| [Structured output](docs/structured-output.md) | `output: zodSchema`: typed `result.object`, the repair step, `'output-invalid'` | | [Streaming](docs/streaming.md) | `agent.stream()`: the typed event schema, terminal and SSE examples | | [React](docs/react.md) | `useLoushyAgent()`: chat state from the event stream, in process or over HTTP; `reduceAgentEvents()`, `parseEventStream()` | | [Durable execution](docs/durable-execution.md) | Checkpoints, crash resume, approvals mid-batch, at-least-once tools | diff --git a/docs/api-overview.md b/docs/api-overview.md index 8e2fc9e81..63b5d55f6 100644 --- a/docs/api-overview.md +++ b/docs/api-overview.md @@ -83,6 +83,18 @@ in). Pauses are kept in a per-agent `InMemoryApprovalStore` unless you pass decides each call in code without pausing (`stream()` still ends at the pause). See [Approvals](./approvals.md). +### Structured output + +Pass `output: zodSchema` to `createAgent()` (or `AgentExecutor.execute()` / +`stream()`): the final reply must be a JSON object matching the schema, and +`send()` / `run.result` resolve with it validated as `result.object`, typed +`z.output` (`result.text` keeps the raw JSON). Tools still run +first. Each model call carries a `responseFormat: { type: 'json', schema }` +hint, which the `ai`-SDK providers map to JSON mode. An invalid reply gets one +repair step listing the issues (it counts against `maxSteps`); if that is +invalid too, the run ends with `finishReason: 'output-invalid'` and +`outputError: { message, issues }`. See [Structured output](./structured-output.md). + ### Skills Pass `skills: [defineSkill({ name, description, content }), ...(await loadSkills(dir))]` to @@ -153,7 +165,9 @@ How it behaves: `result.finishReason` says why a run ended: the model's own reason for its last turn (`'stop'`, `'length'`, `'tool_calls'`, `'content_filter'`, `'error'`), `'awaiting-approval'` (paused on a tool call that needs a human), `'aborted'` -(cancelled with `signal`), or `'max-steps'`. `'max-steps'` means the `maxSteps` +(cancelled with `signal`), `'max-steps'`, or `'output-invalid'` (the reply +did not match the `output` schema even after the repair step, see +[Structured output](./structured-output.md)). `'max-steps'` means the `maxSteps` budget (default 10) ran out while the model still wanted to continue, so the reply may be empty or partial; a run that finishes naturally within the budget keeps its `'stop'`. Steps carried over by `initialSteps` or an approval resume diff --git a/docs/streaming.md b/docs/streaming.md index 493230cb5..ce1c34eb2 100644 --- a/docs/streaming.md +++ b/docs/streaming.md @@ -111,7 +111,7 @@ The event types and their extra fields: | `error` | `error: { name: string, message: string }` | An error. If it ends the run, `run.done` with `finishReason: 'error'` follows. A provider error retried under `surfaceRetryableProviderErrors` is followed by further steps instead. | | `provider.retry` | `attempt: number`, `maxRetries: number`, `delayMs: number`, `error: { message: string, category?: string }`, `provider: string` | A model call failed and is retried after `delayMs` (`createAgent({ retry })` or any `withRetry()` provider). `attempt` is the attempt that failed (1 = first); `category` is `'rate-limit'`, `'timeout'`, ... and absent when unknown (a 5xx, for example). | | `provider.fallback` | `from: string`, `to: string`, `error: { message: string }` | A model call still failed after its retries and the next provider takes over (`createAgent({ fallbackModels })` or any `withFallback()` provider). `from` and `to` are provider names. | -| `run.done` | `finishReason: string`, `text: string`, `usage?: { promptTokens, completionTokens, totalTokens }` | Last event of every run, exactly once, including aborted, failed and awaiting-approval runs. `finishReason` and `text` match `run.result` (`'max-steps'` when the `maxSteps` budget ran out while the model still wanted to continue); a failed run has `finishReason: 'error'`, `text: ''` and no `usage`. | +| `run.done` | `finishReason: string`, `text: string`, `usage?: { promptTokens, completionTokens, totalTokens }`, `object?: unknown` | Last event of every run, exactly once, including aborted, failed and awaiting-approval runs. `finishReason` and `text` match `run.result` (`'max-steps'` when the `maxSteps` budget ran out while the model still wanted to continue); a failed run has `finishReason: 'error'`, `text: ''` and no `usage`. `object` is `run.result`'s validated `object` for an agent with an `output` schema (see [Structured output](./structured-output.md)), absent otherwise. | Optional fields are left out when they have no value. They are never `undefined`, so `JSON.parse(JSON.stringify(event))` returns an equal object. diff --git a/docs/structured-output.md b/docs/structured-output.md new file mode 100644 index 000000000..3e3f7d3b2 --- /dev/null +++ b/docs/structured-output.md @@ -0,0 +1,82 @@ +# Structured output + +Give an agent a zod schema as `output` and its final reply becomes a typed, +validated object: `result.object`. The agent can still call tools first; only +the final answer has to match the schema. + +```ts +import { z } from 'zod'; +import { createAgent } from '@loushy/build-ai-agent'; + +const agent = createAgent({ + model: 'openai/gpt-4o-mini', + instructions: 'You report the weather.', + output: z.object({ city: z.string(), tempC: z.number(), summary: z.string() }), +}); + +const result = await agent.send('Weather in Paris?'); +if (result.object) { + console.log(result.object.tempC); // typed: number +} else { + console.error(result.finishReason, result.outputError?.issues); // 'output-invalid', [{ path, message }] +} +``` + +`result.object` is typed as `z.output`, so zod defaults and +transforms apply. `result.text` keeps the raw JSON text the model wrote. + +## How it works + +1. The system prompt gets an `## Output format` section asking for the final + answer as only a JSON object matching the schema, rendered as JSON Schema. +2. Every model call carries `responseFormat: { type: 'json', schema }` on + `GenerateOptions`. It is a hint: the built-in `ai`-SDK providers turn on + the model's JSON mode (with the schema, for models that support structured + outputs); a custom provider may use it or ignore it. +3. When the model replies without tool calls, the reply is parsed as JSON (a + ```` ```json ```` code fence around it is tolerated) and validated with + the schema. +4. If it is invalid, the model gets one repair step: a user message + starting with `[output-invalid]` that lists the issues, for example + `1 issue (tempC: Expected number, received string)`. The repair step + counts against `maxSteps`, and there is none when the budget is spent. +5. Still invalid, the run resolves (it does not reject) with + `finishReason: 'output-invalid'`, no `object`, and `outputError`: + `{ message, issues: [{ path, message }] }`. + +## Streaming + +`agent.stream()` works the same way: `run.result` resolves with `object`, +and the final `run.done` event carries `object` (JSON-encoded) when the +reply was valid. The repair step shows up as one more +`step.start` / `step.done` pair. See [Streaming](./streaming.md). + +## Without createAgent() + +`AgentExecutor.execute()` and `AgentExecutor.stream()` take the same +`output` option. There `result.object` is `unknown`; parse it again with your +schema, or use `createAgent()` for the inferred type. + +```ts +import { z } from 'zod'; +import { AgentBuilder, AgentExecutor, createMockProvider } from '@loushy/build-ai-agent'; + +const Ticket = z.object({ title: z.string(), priority: z.enum(['low', 'high']) }); +const agent = AgentBuilder.create().setName('triage').setPrompt('You triage bug reports.').build(); +const result = await AgentExecutor.execute({ agent, input: 'The app crashes on login', provider: createMockProvider(), output: Ticket }); +const ticket = result.object === undefined ? undefined : Ticket.parse(result.object); +``` + +## Testing + +With `mockModel`, script the JSON text the model would write: + +```ts +import { z } from 'zod'; +import { createAgent } from '@loushy/build-ai-agent'; +import { mockModel } from '@loushy/build-ai-agent/testing'; + +const agent = createAgent({ provider: mockModel(['{"city":"Paris","tempC":21}']), output: z.object({ city: z.string(), tempC: z.number() }) }); +const { object } = await agent.send('Weather in Paris?'); +console.log(object?.city); // 'Paris' +``` diff --git a/llms-full.txt b/llms-full.txt index f844a731b..f46ef5f8d 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -71,6 +71,7 @@ agent can be an `agent.yaml` spec served with `npx loushy dev agent.yaml` - **Tools**: `defineTool()` with a zod `input`; arguments are typed, validated, and run in parallel. [Tools](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/tools.md) - **Approvals**: `needsApproval` pauses a run; `agent.approvals.resolve()` continues it, or `approve` decides in code. [Approvals](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/approvals.md) - **Sessions**: `agent.session()` keeps a multi-turn conversation in memory, files or SQLite. [Sessions](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/sessions.md) +- **Structured output**: `output: zodSchema` makes the final reply a typed, validated `result.object`, with one repair step. [Structured output](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/structured-output.md) - **Streaming**: `agent.stream()` and `session.stream()` yield typed, versioned JSON events ready for SSE. [Streaming](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/streaming.md) - **UI bindings**: `useLoushyAgent()` from `@loushy/build-ai-agent/react` turns the event stream into chat state, with approvals. [React](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/react.md) - **Durable execution**: `sessionId` + `checkpointStore` resume a crashed or paused run without redoing finished tools. [Durable execution](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/durable-execution.md) @@ -176,6 +177,7 @@ await AgentExecutor.execute({ | [Tools](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/tools.md) | `defineTool()`, validation and errors, built-in tools, `ToolRegistry` | | [Approvals](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/approvals.md) | `needsApproval`, `agent.approvals`, the `approve` callback, `resumeAfterApproval()`, stores | | [Sessions](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/sessions.md) | Multi-turn conversations, `session.stream()`, session stores, `SqliteStore` | +| [Structured output](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/structured-output.md) | `output: zodSchema`: typed `result.object`, the repair step, `'output-invalid'` | | [Streaming](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/streaming.md) | `agent.stream()`: the typed event schema, terminal and SSE examples | | [React](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/react.md) | `useLoushyAgent()`: chat state from the event stream, in process or over HTTP; `reduceAgentEvents()`, `parseEventStream()` | | [Durable execution](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/durable-execution.md) | Checkpoints, crash resume, approvals mid-batch, at-least-once tools | @@ -768,6 +770,18 @@ in). Pauses are kept in a per-agent `InMemoryApprovalStore` unless you pass decides each call in code without pausing (`stream()` still ends at the pause). See [Approvals](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/approvals.md). +### Structured output + +Pass `output: zodSchema` to `createAgent()` (or `AgentExecutor.execute()` / +`stream()`): the final reply must be a JSON object matching the schema, and +`send()` / `run.result` resolve with it validated as `result.object`, typed +`z.output` (`result.text` keeps the raw JSON). Tools still run +first. Each model call carries a `responseFormat: { type: 'json', schema }` +hint, which the `ai`-SDK providers map to JSON mode. An invalid reply gets one +repair step listing the issues (it counts against `maxSteps`); if that is +invalid too, the run ends with `finishReason: 'output-invalid'` and +`outputError: { message, issues }`. See [Structured output](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/structured-output.md). + ### Skills Pass `skills: [defineSkill({ name, description, content }), ...(await loadSkills(dir))]` to @@ -838,7 +852,9 @@ How it behaves: `result.finishReason` says why a run ended: the model's own reason for its last turn (`'stop'`, `'length'`, `'tool_calls'`, `'content_filter'`, `'error'`), `'awaiting-approval'` (paused on a tool call that needs a human), `'aborted'` -(cancelled with `signal`), or `'max-steps'`. `'max-steps'` means the `maxSteps` +(cancelled with `signal`), `'max-steps'`, or `'output-invalid'` (the reply +did not match the `output` schema even after the repair step, see +[Structured output](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/structured-output.md)). `'max-steps'` means the `maxSteps` budget (default 10) ran out while the model still wanted to continue, so the reply may be empty or partial; a run that finishes naturally within the budget keeps its `'stop'`. Steps carried over by `initialSteps` or an approval resume @@ -4191,7 +4207,7 @@ The event types and their extra fields: | `error` | `error: { name: string, message: string }` | An error. If it ends the run, `run.done` with `finishReason: 'error'` follows. A provider error retried under `surfaceRetryableProviderErrors` is followed by further steps instead. | | `provider.retry` | `attempt: number`, `maxRetries: number`, `delayMs: number`, `error: { message: string, category?: string }`, `provider: string` | A model call failed and is retried after `delayMs` (`createAgent({ retry })` or any `withRetry()` provider). `attempt` is the attempt that failed (1 = first); `category` is `'rate-limit'`, `'timeout'`, ... and absent when unknown (a 5xx, for example). | | `provider.fallback` | `from: string`, `to: string`, `error: { message: string }` | A model call still failed after its retries and the next provider takes over (`createAgent({ fallbackModels })` or any `withFallback()` provider). `from` and `to` are provider names. | -| `run.done` | `finishReason: string`, `text: string`, `usage?: { promptTokens, completionTokens, totalTokens }` | Last event of every run, exactly once, including aborted, failed and awaiting-approval runs. `finishReason` and `text` match `run.result` (`'max-steps'` when the `maxSteps` budget ran out while the model still wanted to continue); a failed run has `finishReason: 'error'`, `text: ''` and no `usage`. | +| `run.done` | `finishReason: string`, `text: string`, `usage?: { promptTokens, completionTokens, totalTokens }`, `object?: unknown` | Last event of every run, exactly once, including aborted, failed and awaiting-approval runs. `finishReason` and `text` match `run.result` (`'max-steps'` when the `maxSteps` budget ran out while the model still wanted to continue); a failed run has `finishReason: 'error'`, `text: ''` and no `usage`. `object` is `run.result`'s validated `object` for an agent with an `output` schema (see [Structured output](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/structured-output.md)), absent otherwise. | Optional fields are left out when they have no value. They are never `undefined`, so `JSON.parse(JSON.stringify(event))` returns an equal object. @@ -4427,6 +4443,91 @@ for await (const event of run) { const paused = await run.result; // paused.finishReason === 'awaiting-approval' ``` +# Structured output + +Source: docs/structured-output.md + +Give an agent a zod schema as `output` and its final reply becomes a typed, +validated object: `result.object`. The agent can still call tools first; only +the final answer has to match the schema. + +```ts +import { z } from 'zod'; +import { createAgent } from '@loushy/build-ai-agent'; + +const agent = createAgent({ + model: 'openai/gpt-4o-mini', + instructions: 'You report the weather.', + output: z.object({ city: z.string(), tempC: z.number(), summary: z.string() }), +}); + +const result = await agent.send('Weather in Paris?'); +if (result.object) { + console.log(result.object.tempC); // typed: number +} else { + console.error(result.finishReason, result.outputError?.issues); // 'output-invalid', [{ path, message }] +} +``` + +`result.object` is typed as `z.output`, so zod defaults and +transforms apply. `result.text` keeps the raw JSON text the model wrote. + +## How it works + +1. The system prompt gets an `## Output format` section asking for the final + answer as only a JSON object matching the schema, rendered as JSON Schema. +2. Every model call carries `responseFormat: { type: 'json', schema }` on + `GenerateOptions`. It is a hint: the built-in `ai`-SDK providers turn on + the model's JSON mode (with the schema, for models that support structured + outputs); a custom provider may use it or ignore it. +3. When the model replies without tool calls, the reply is parsed as JSON (a + ```` ```json ```` code fence around it is tolerated) and validated with + the schema. +4. If it is invalid, the model gets one repair step: a user message + starting with `[output-invalid]` that lists the issues, for example + `1 issue (tempC: Expected number, received string)`. The repair step + counts against `maxSteps`, and there is none when the budget is spent. +5. Still invalid, the run resolves (it does not reject) with + `finishReason: 'output-invalid'`, no `object`, and `outputError`: + `{ message, issues: [{ path, message }] }`. + +## Streaming + +`agent.stream()` works the same way: `run.result` resolves with `object`, +and the final `run.done` event carries `object` (JSON-encoded) when the +reply was valid. The repair step shows up as one more +`step.start` / `step.done` pair. See [Streaming](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/streaming.md). + +## Without createAgent() + +`AgentExecutor.execute()` and `AgentExecutor.stream()` take the same +`output` option. There `result.object` is `unknown`; parse it again with your +schema, or use `createAgent()` for the inferred type. + +```ts +import { z } from 'zod'; +import { AgentBuilder, AgentExecutor, createMockProvider } from '@loushy/build-ai-agent'; + +const Ticket = z.object({ title: z.string(), priority: z.enum(['low', 'high']) }); +const agent = AgentBuilder.create().setName('triage').setPrompt('You triage bug reports.').build(); +const result = await AgentExecutor.execute({ agent, input: 'The app crashes on login', provider: createMockProvider(), output: Ticket }); +const ticket = result.object === undefined ? undefined : Ticket.parse(result.object); +``` + +## Testing + +With `mockModel`, script the JSON text the model would write: + +```ts +import { z } from 'zod'; +import { createAgent } from '@loushy/build-ai-agent'; +import { mockModel } from '@loushy/build-ai-agent/testing'; + +const agent = createAgent({ provider: mockModel(['{"city":"Paris","tempC":21}']), output: z.object({ city: z.string(), tempC: z.number() }) }); +const { object } = await agent.send('Weather in Paris?'); +console.log(object?.city); // 'Paris' +``` + # Sub-agents Source: docs/sub-agents.md diff --git a/llms.txt b/llms.txt index 8ddb61e09..5d7fad1f3 100644 --- a/llms.txt +++ b/llms.txt @@ -23,6 +23,7 @@ - [Sessions](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/sessions.md): `agent.send(text)` is single-turn: every call starts from an empty history. - [Skills](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/skills.md): A skill is a named chunk of instructions (a markdown document) that the model loads only when it needs it. - [Streaming](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/streaming.md): `agent.stream()` runs an agent exactly like `agent.send()` and reports the run as a stream of typed events: model text as it arrives, every tool call, step boundaries, approval requests, and a final `run.done`. +- [Structured output](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/structured-output.md): Give an agent a zod schema as `output` and its final reply becomes a typed, validated object: `result.object`. - [Sub-agents](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/sub-agents.md): A sub-agent is a separate agent - its own instructions, model, tools and skills - that a lead agent hands a self-contained task to. - [Testing agents](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/testing.md): `@loushy/build-ai-agent/testing` ships `mockModel`: a scripted, deterministic `LLMProvider` for unit tests. - [Tools](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/tools.md): A tool is a typed function the model can call. diff --git a/src/createAgent.ts b/src/createAgent.ts index c4da18509..22eec2df1 100644 --- a/src/createAgent.ts +++ b/src/createAgent.ts @@ -37,6 +37,7 @@ import type { ApprovalStore } from './execution/ApprovalGate'; import { InMemoryApprovalStore } from './execution/InMemoryApprovalStore'; import { resumeAfterApproval } from './execution/resume'; import { createAgentApprovals, type AgentApprovals, type ApproveToolCall } from './createAgentApprovals'; +import type { z } from 'zod'; /** * Options for createAgent() that do not depend on how the instructions and @@ -44,7 +45,7 @@ import { createAgentApprovals, type AgentApprovals, type ApproveToolCall } from * AgentConfig.tools) will refer to them by - createAgent() registers each * one into a fresh ToolRegistry under that key. */ -export interface CreateAgentBase { +export interface CreateAgentBase { /** * Optional tools: an array of `defineTool()` results (named by the tool), * or a record of descriptors keyed by the name the agent should call them by. @@ -168,6 +169,22 @@ export interface CreateAgentBase { * ``` */ fallbackModels?: readonly string[]; + /** + * A zod schema for the agent's final reply (LOU-V4): the model is asked + * to answer with a JSON object matching it, and `send()` / `stream()` + * resolve with it parsed and validated as `result.object`, typed + * `z.output` (`result.text` keeps the raw JSON). An invalid + * reply gets one repair step; still invalid, the run ends with + * `finishReason: 'output-invalid'` and `outputError`. See + * docs/structured-output.md and `ExecuteOptions.output`. + * + * @example + * ```ts + * const agent = createAgent({ model: 'openai/gpt-4o-mini', output: z.object({ city: z.string(), tempC: z.number() }) }); + * const { object } = await agent.send('Weather in Paris?'); + * ``` + */ + output?: TOutput; } /** @@ -229,7 +246,9 @@ export type CreateAgentModelSource = * @example * createAgent({ model: 'openai/gpt-4o-mini', instructions: 'You are a helpful assistant.' }); */ -export type CreateAgentConfig = CreateAgentBase & CreateAgentInstructions & CreateAgentModelSource; +export type CreateAgentConfig = CreateAgentBase & + CreateAgentInstructions & + CreateAgentModelSource; /** Per-call options for `SimpleAgent.send()`. */ export interface SendOptions { @@ -246,9 +265,10 @@ export interface SendOptions { signal?: AbortSignal; } -export interface SimpleAgent { +/** `TObject`: the type of `result.object` - `z.output` of the `output` schema. */ +export interface SimpleAgent { /** Send a single user message and get back the full execution result text. */ - send: (message: string, options?: SendOptions) => Promise; + send: (message: string, options?: SendOptions) => Promise>; /** * Send a single user message and stream the run as typed events (LOU-V2): * `text.delta` chunks as the model writes, `tool.start`/`tool.done`, @@ -264,7 +284,7 @@ export interface SimpleAgent { * } * ``` */ - stream: (message: string, options?: SendOptions) => AgentRun; + stream: (message: string, options?: SendOptions) => AgentRun; /** * Start a multi-turn conversation (LOU-W4): every `send()` sees the earlier * exchanges. In memory by default; pass `{ id, store }` (e.g. a @@ -307,7 +327,9 @@ export interface SimpleAgent { * const agent = createAgent({ model: 'openai/gpt-4o-mini', instructions: 'You are a helpful assistant.' }); * const { text } = await agent.send('Hello!'); */ -export function createAgent(config: CreateAgentConfig = {}): SimpleAgent { +export function createAgent( + config: CreateAgentConfig = {} +): SimpleAgent> { assertToolConcurrency(config.toolConcurrency, 'createAgent'); assertMaxSubagentDepth(config.maxSubagentDepth, 'createAgent'); assertSubagents(config.subagents, 'createAgent'); @@ -339,12 +361,14 @@ export function createAgent(config: CreateAgentConfig = {}): SimpleAgent { resume: (approvalStore, decision, signal) => resumeAfterApproval(decision, approvalStore, toolRegistry ?? new ToolRegistry(), provider, { ...runOptions, + output: config.output, approvalStore, signal, }), }); const executeOptions = (input: string | Message[], signal?: AbortSignal): ExecuteOptions => ({ ...spec, + output: config.output, approvalStore: approvals.store, input, signal, @@ -352,12 +376,14 @@ export function createAgent(config: CreateAgentConfig = {}): SimpleAgent { const run = (input: string | Message[], signal?: AbortSignal): Promise => AgentExecutor.execute(executeOptions(input, signal)); - const simpleAgent: SimpleAgent = { - async send(message: string, options: SendOptions = {}): Promise { - return approvals.settle(await run(message, options.signal), options.signal); + // `object` was validated with `config.output`, so it has its output type. + type Typed = z.output; + const simpleAgent: SimpleAgent = { + async send(message: string, options: SendOptions = {}): Promise> { + return approvals.settle(await run(message, options.signal), options.signal) as Promise>; }, - stream(message: string, options: SendOptions = {}): AgentRun { - return AgentExecutor.stream(executeOptions(message, options.signal)); + stream(message: string, options: SendOptions = {}): AgentRun { + return AgentExecutor.stream(executeOptions(message, options.signal)) as AgentRun; }, session: (options?: SessionOptions) => approvals.session(run, (input, signal) => AgentExecutor.stream(executeOptions(input, signal)), options), diff --git a/src/execution/AgentExecutor.ts b/src/execution/AgentExecutor.ts index af8d20f3d..c496756a4 100644 --- a/src/execution/AgentExecutor.ts +++ b/src/execution/AgentExecutor.ts @@ -7,6 +7,7 @@ import type { Skill } from '../skills/defineSkill'; import { withSkills } from '../skills/withSkills'; import type { Subagents } from '../subagents/types'; import { assertMaxSubagentDepth, withSubagents } from '../subagents/withSubagents'; +import type { z } from 'zod'; import { newId } from '../utils/id'; import { LLMProvider, Message, ToolCall, GenerateOptions, GenerateResult, ToolDefinition } from '../providers'; import { AgentConfig } from '../types'; @@ -26,6 +27,7 @@ import { import { HookRegistry, type SubagentInfo } from './hooks'; import { baseAgentOf, + extendAgent, settleSuspensions, suspensionRecord, type ToolCallScope, @@ -64,6 +66,7 @@ import { toExecutionResult, } from './agentRunState'; import { AgentRun, RUN_EVENTS, StreamingExecuteOptions, runEventsOf, startAgentRun } from './agentRun'; +import { OutputError, outputInstruction, outputRepairMessage, validateOutput } from './structuredOutput'; export { PropagatingToolError } from './propagatingToolError'; @@ -93,12 +96,16 @@ export type ExecutionEventType = * - `'max-steps'`: the `maxSteps` budget ran out while the model still * wanted to continue (LOU-U19). A run that finishes naturally within the * budget keeps the model's own reason (usually `'stop'`). + * - `'output-invalid'`: the run has an `output` schema and the final reply + * was still not valid JSON matching it after one repair step (LOU-V4); + * see `ExecutionResult.outputError`. */ export type ExecutionFinishReason = | GenerateResult['finishReason'] | 'awaiting-approval' | 'aborted' | 'max-steps' + | 'output-invalid' | (string & {}); /** @@ -452,12 +459,26 @@ export interface ExecuteOptions { * ``` */ toolConcurrency?: ToolConcurrency; + /** + * LOU-V4: a zod schema the final reply must match, as JSON (tools can + * still be called first). It is validated into `result.object`; an invalid + * reply gets one repair step (counted against `maxSteps`), then the run + * ends with `finishReason: 'output-invalid'` and `outputError`. See + * docs/structured-output.md. + * + * @example + * ```ts + * const { object } = await AgentExecutor.execute({ agent, input: 'Weather in Paris?', provider, output: z.object({ tempC: z.number() }) }); + * ``` + */ + output?: z.ZodTypeAny; } /** - * Execution result + * Execution result. `TObject` is the type of `object` - `z.output` of the + * `output` schema for `createAgent({ output })` agents. */ -export interface ExecutionResult { +export interface ExecutionResult { text: string; messages: Message[]; toolCalls: ToolCall[]; @@ -471,6 +492,10 @@ export interface ExecutionResult { finishReason: ExecutionFinishReason; steps: number; approvalId?: string; + /** LOU-V4: the final reply parsed and validated with `output`; absent unless it was valid. */ + object?: TObject; + /** LOU-V4: why the final reply did not match `output` (`finishReason: 'output-invalid'`). */ + outputError?: OutputError; } /** @@ -522,7 +547,12 @@ export class AgentExecutor { options.subagents, options.maxSubagentDepth ); - return { ...options, ...extended }; + // LOU-V4: the output instruction goes last in the system prompt. + const { agent } = extended; + if (!options.output) return { ...options, ...extended }; + const instruction = outputInstruction(options.output); + const prompt = agent.prompt ? `${agent.prompt}\n\n${instruction}` : instruction; + return { ...options, ...extended, agent: extendAgent(agent, { prompt }) }; } /** @@ -611,6 +641,7 @@ export class AgentExecutor { agentSpanId: string ): Promise { const { maxSteps = 10, signal } = options; + let repaired = false; // Execution loop with tool calling. LOU-V1: the signal is checked // before every model call (here) and every tool call (runToolCalls()). @@ -624,7 +655,12 @@ export class AgentExecutor { this.runStep(options, state, tools, agentSpanId) ); if (outcome === 'stop') { - return this.finishRun(options, state); + const output = await this.checkOutput(options, state, !repaired && state.steps < maxSteps); + if (output === 'repair') { + repaired = true; + continue; + } + return this.finishRun(options, state, output); } if (outcome !== 'continue') { return outcome; @@ -641,6 +677,27 @@ export class AgentExecutor { return this.finishRun(options, state); } + /** + * LOU-V4: with `output`, parses and validates the final reply. When it is + * invalid and `canRepair`, queues the issues for one more step ('repair'); + * otherwise ends the run as 'output-invalid'. + */ + private static async checkOutput( + options: ExecuteOptions, + state: AgentRunState, + canRepair: boolean + ): Promise<'repair' | { object: unknown } | { outputError: OutputError } | undefined> { + if (!options.output) return undefined; + const checked = await validateOutput(options.output, state.finalText); + if ('object' in checked) return checked; + if (canRepair) { + state.messages.push(outputRepairMessage(checked.outputError)); + return 'repair'; + } + state.finishReason = 'output-invalid'; + return checked; + } + /** * Runs one step (runStep(), or the resumed pending tool calls of the * current step - LOU-U7/U9), bracketed by the stream's step events, with @@ -1119,7 +1176,8 @@ export class AgentExecutor { */ private static async finishRun( options: ExecuteOptions, - state: AgentRunState + state: AgentRunState, + output?: Pick ): Promise { const { onEvent } = options; @@ -1154,7 +1212,7 @@ export class AgentExecutor { // resuming this run (or forgetting it). await saveStepCheckpoint(options, state, 'finished'); - return toExecutionResult(state, state.finalText, state.finishReason); + return { ...toExecutionResult(state, state.finalText, state.finishReason), ...output }; } /** diff --git a/src/execution/agentEvents.ts b/src/execution/agentEvents.ts index 372fbfd5f..596dc4377 100644 --- a/src/execution/agentEvents.ts +++ b/src/execution/agentEvents.ts @@ -192,6 +192,8 @@ export interface RunDoneEvent extends AgentEventBase<'run.done'> { text: string; /** Total tokens of the run; absent when the run failed. */ usage?: AgentEventUsage; + /** LOU-V4: `ExecutionResult.object` (the validated `output`), JSON-encoded; absent when there is none. */ + object?: unknown; } /** diff --git a/src/execution/agentRun.ts b/src/execution/agentRun.ts index 746936f75..d5becc347 100644 --- a/src/execution/agentRun.ts +++ b/src/execution/agentRun.ts @@ -50,7 +50,7 @@ import type { Usage } from '../models/usage'; * const result = await run.result; // the same ExecutionResult send() returns * ``` */ -export interface AgentRun extends AsyncIterable { +export interface AgentRun extends AsyncIterable { /** The `runId` every event of this run carries. */ readonly runId: string; /** @@ -60,7 +60,7 @@ export interface AgentRun extends AsyncIterable { * outcomes as `send()`/`execute()`. Not awaiting it never causes an * unhandled rejection. */ - readonly result: Promise; + readonly result: Promise>; } /** @@ -217,6 +217,7 @@ class AgentRunImpl implements AgentRun { finishReason: result.finishReason, text: result.text, usage: toEventUsage(result.usage), + ...(result.object !== undefined && { object: toJsonValue(result.object) }), }); return result; } diff --git a/src/execution/generateStep.ts b/src/execution/generateStep.ts index ebff20002..521d297be 100644 --- a/src/execution/generateStep.ts +++ b/src/execution/generateStep.ts @@ -22,6 +22,7 @@ import { } from './errors'; import type { ExecuteOptions } from './AgentExecutor'; import { runEventsOf } from './agentRun'; +import { outputResponseFormat } from './structuredOutput'; /** * Build tools from agent and registry @@ -97,6 +98,8 @@ export async function prepareGenerateRequest( temperature, maxTokens, tools: tools.length > 0 ? tools : undefined, + // LOU-V4: a JSON-mode hint for runs with an `output` schema. + ...(options.output ? { responseFormat: outputResponseFormat(options.output) } : {}), // LOU-V1: lets the provider cancel the in-flight request. ...(signal ? { signal } : {}), }; diff --git a/src/execution/index.ts b/src/execution/index.ts index dd3cdeb23..5d8201b02 100644 --- a/src/execution/index.ts +++ b/src/execution/index.ts @@ -20,6 +20,7 @@ export * from './guardrails'; export * from './hooks'; export { ToolArgumentsValidationError } from './toolArgsValidation'; export type { ToolArgumentIssue } from './toolArgsValidation'; +export type { OutputError } from './structuredOutput'; export type { ToolConcurrency } from './toolBatch'; export { AGENT_EVENT_SCHEMA_VERSION, diff --git a/src/execution/structuredOutput.test-d.ts b/src/execution/structuredOutput.test-d.ts new file mode 100644 index 000000000..e1ece1cb8 --- /dev/null +++ b/src/execution/structuredOutput.test-d.ts @@ -0,0 +1,25 @@ +import { describe, it, expectTypeOf } from 'vitest'; +import { z } from 'zod'; +import { createAgent, type SimpleAgent } from '../createAgent'; +import type { ExecutionResult } from './AgentExecutor'; +import { mockModel } from '../testing'; + +describe('createAgent({ output }) types (LOU-V4)', () => { + it('types result.object as z.output of the schema', () => { + const output = z.object({ city: z.string(), tempC: z.number().default(0), at: z.string().transform((s) => new Date(s)) }); + const agent = createAgent({ provider: mockModel([]), output }); + expectTypeOf(agent).toEqualTypeOf>(); + + type Sent = Awaited>; + expectTypeOf().toEqualTypeOf<{ city: string; tempC: number; at: Date } | undefined>(); + type Streamed = Awaited['result']>; + expectTypeOf().toEqualTypeOf | undefined>(); + expectTypeOf().toMatchTypeOf(); + }); + + it('types result.object as unknown without output', () => { + const agent = createAgent({ provider: mockModel([]) }); + expectTypeOf(agent).toEqualTypeOf(); + expectTypeOf>['object']>().toEqualTypeOf(); + }); +}); diff --git a/src/execution/structuredOutput.test.ts b/src/execution/structuredOutput.test.ts new file mode 100644 index 000000000..c5ef3c776 --- /dev/null +++ b/src/execution/structuredOutput.test.ts @@ -0,0 +1,148 @@ +/** + * LOU-V4: `createAgent({ output })` / `ExecuteOptions.output` - the final + * reply is parsed and validated into `result.object`, with one repair step. + */ + +import { describe, it, expect } from 'vitest'; +import { z } from 'zod'; +import { createAgent } from '../createAgent'; +import { defineTool } from '../tools/defineTool'; +import { mockModel } from '../testing'; +import type { AgentEvent } from './agentEvents'; + +const weather = z.object({ city: z.string(), tempC: z.number() }); + +describe('structured output (LOU-V4)', () => { + it('validates a JSON reply into result.object and keeps the raw text', async () => { + const model = mockModel(['{"city":"Paris","tempC":21}']); + const agent = createAgent({ provider: model, instructions: 'You report weather.', output: weather }); + + const result = await agent.send('Weather in Paris?'); + + expect(result.object).toEqual({ city: 'Paris', tempC: 21 }); + expect(result.text).toBe('{"city":"Paris","tempC":21}'); + expect(result.finishReason).toBe('stop'); + expect(result.outputError).toBeUndefined(); + const request = model.calls[0]; + expect(request.responseFormat).toMatchObject({ type: 'json', schema: { type: 'object', required: ['city', 'tempC'] } }); + const system = request.messages[0]; + expect(system.role).toBe('system'); + expect(system.content).toMatch(/^You report weather\.\n\n## Output format\n/); + expect(system.content).toContain('"tempC":{"type":"number"}'); + }); + + it('applies the schema (defaults, transforms) to the parsed object', async () => { + const schema = z.object({ tags: z.array(z.string()).default([]), name: z.string().transform((s) => s.toUpperCase()) }); + const agent = createAgent({ provider: mockModel(['{"name":"ada"}']), output: schema }); + + expect((await agent.send('go')).object).toEqual({ tags: [], name: 'ADA' }); + }); + + it('accepts a reply wrapped in a code fence', async () => { + const agent = createAgent({ provider: mockModel(['```json\n{"city":"Oslo","tempC":-3}\n```']), output: weather }); + + const result = await agent.send('Weather in Oslo?'); + + expect(result.object).toEqual({ city: 'Oslo', tempC: -3 }); + expect(result.steps).toBe(1); + }); + + it('repairs an invalid reply with one more step that lists the issues', async () => { + const model = mockModel(['{"city":"Paris","tempC":"warm"}', '{"city":"Paris","tempC":21}']); + const agent = createAgent({ provider: model, output: weather }); + + const result = await agent.send('Weather in Paris?'); + + expect(result.object).toEqual({ city: 'Paris', tempC: 21 }); + expect(result.finishReason).toBe('stop'); + expect(result.steps).toBe(2); + const repair = model.calls[1].messages.at(-1); + expect(repair?.role).toBe('user'); + expect(repair?.content).toBe( + '[output-invalid] The reply does not match the output schema: 1 issue (tempC: Expected number, received string). ' + + 'Reply again with only the corrected JSON object.' + ); + expect(model.calls[1].messages.at(-2)).toEqual({ role: 'assistant', content: '{"city":"Paris","tempC":"warm"}' }); + }); + + it("ends with 'output-invalid' and outputError when the repair is invalid too", async () => { + const model = mockModel(['It is sunny.', '{"city":"Paris"}']); + const agent = createAgent({ provider: model, output: weather }); + + const result = await agent.send('Weather in Paris?'); + + expect(model.calls).toHaveLength(2); + expect(model.calls[1].messages.at(-1)?.content).toMatch(/^\[output-invalid\] The reply is not a JSON object: 1 issue \(\(root\): Not valid JSON: /); + expect(result.finishReason).toBe('output-invalid'); + expect(result.object).toBeUndefined(); + expect(result.text).toBe('{"city":"Paris"}'); + expect(result.outputError).toEqual({ + message: 'The reply does not match the output schema: 1 issue (tempC: Required)', + issues: [{ path: 'tempC', message: 'Required' }], + }); + }); + + it('counts the repair against maxSteps: no budget left means no repair', async () => { + const model = mockModel(['nope']); + const agent = createAgent({ provider: model, output: weather, maxSteps: 1 }); + + const result = await agent.send('go'); + + expect(model.calls).toHaveLength(1); + expect(result.finishReason).toBe('output-invalid'); + expect(result.outputError?.issues[0].path).toBe('(root)'); + }); + + it('runs tool calls before the final JSON answer', async () => { + const lookup = defineTool({ + name: 'lookup', + description: 'Current temperature', + input: z.object({ city: z.string() }), + execute: async () => 21, + }); + const model = mockModel([{ toolCalls: [{ name: 'lookup', args: { city: 'Paris' } }] }, '{"city":"Paris","tempC":21}']); + const agent = createAgent({ provider: model, tools: [lookup], output: weather }); + + const result = await agent.send('Weather in Paris?'); + + expect(result.object).toEqual({ city: 'Paris', tempC: 21 }); + expect(result.toolCalls).toHaveLength(1); + expect(model.calls[0].tools).toHaveLength(1); + expect(model.calls[0].responseFormat?.type).toBe('json'); + }); + + it('stream(): run.result and run.done carry the object', async () => { + const agent = createAgent({ provider: mockModel(['{"city":"Rome"', '{"city":"Rome","tempC":30}']), output: weather }); + + const run = agent.stream('Weather in Rome?'); + const events: AgentEvent[] = []; + for await (const event of run) events.push(event); + + const result = await run.result; + expect(result.object).toEqual({ city: 'Rome', tempC: 30 }); + expect(events.filter((e) => e.type === 'step.start')).toHaveLength(2); + expect(events.at(-1)).toMatchObject({ type: 'run.done', finishReason: 'stop', object: { city: 'Rome', tempC: 30 } }); + }); + + it('stream(): run.done has no object when the output is invalid', async () => { + const agent = createAgent({ provider: mockModel(['x', 'y']), output: weather }); + + const run = agent.stream('go'); + const events: AgentEvent[] = []; + for await (const event of run) events.push(event); + + const done = events.at(-1); + expect(done).toMatchObject({ type: 'run.done', finishReason: 'output-invalid' }); + expect(done).not.toHaveProperty('object'); + expect((await run.result).outputError?.issues).toHaveLength(1); + }); + + it('leaves agents without output unchanged', async () => { + const model = mockModel(['not json']); + const result = await createAgent({ provider: model, instructions: 'Hi.' }).send('go'); + + expect(result).not.toHaveProperty('object'); + expect(model.calls[0]).not.toHaveProperty('responseFormat'); + expect(model.calls[0].messages[0].content).toBe('Hi.'); + }); +}); diff --git a/src/execution/structuredOutput.ts b/src/execution/structuredOutput.ts new file mode 100644 index 000000000..816b7e028 --- /dev/null +++ b/src/execution/structuredOutput.ts @@ -0,0 +1,78 @@ +/** + * Structured output (LOU-V4): what a run with `ExecuteOptions.output` adds + * to the request (an instruction in the system prompt and a + * `responseFormat` hint), and the parse/validate/repair of its final reply. + */ + +import { zodSchema } from 'ai'; +import type { z } from 'zod'; +import type { GenerateOptions, Message } from '../providers'; +import { formatIssues, parseWithIssues, type ToolArgumentIssue } from './toolArgsValidation'; + +/** Why a run's final reply is not a valid `output` object (`finishReason: 'output-invalid'`). */ +export interface OutputError { + /** Model-readable summary, e.g. `The reply does not match the output schema: 1 issue (city: Required)`. */ + message: string; + /** Each problem with its path (`(root)` for the whole reply). */ + issues: ToolArgumentIssue[]; +} + +const jsonSchemas = new WeakMap>(); + +/** The schema as JSON Schema (via the `ai` SDK's zod converter), computed once per schema. */ +function jsonSchemaOf(schema: z.ZodTypeAny): Record { + let json = jsonSchemas.get(schema); + if (!json) { + json = zodSchema(schema).jsonSchema as Record; + jsonSchemas.set(schema, json); + } + return json; +} + +/** The system-prompt block that asks for the final answer as JSON matching `schema`. */ +export function outputInstruction(schema: z.ZodTypeAny): string { + return [ + '## Output format', + '', + 'You may call tools first. Your final answer must be only a JSON object (no other text, no code fences) that matches this JSON Schema:', + '', + JSON.stringify(jsonSchemaOf(schema)), + ].join('\n'); +} + +/** The `responseFormat` hint sent with every model call of the run. */ +export function outputResponseFormat(schema: z.ZodTypeAny): GenerateOptions['responseFormat'] { + return { type: 'json', schema: jsonSchemaOf(schema) }; +} + +/** `text` without surrounding whitespace and a ```/```json code fence. */ +function unfence(text: string): string { + const trimmed = text.trim(); + return /^```(?:json)?\s*([\s\S]*?)\s*```$/i.exec(trimmed)?.[1] ?? trimmed; +} + +/** Parses the final reply as JSON and validates it with `schema`. */ +export async function validateOutput( + schema: z.ZodTypeAny, + text: string +): Promise<{ object: unknown } | { outputError: OutputError }> { + let value: unknown; + try { + value = JSON.parse(unfence(text)); + } catch (error) { + const issues = [{ path: '(root)', message: `Not valid JSON: ${(error as Error).message}` }]; + return { outputError: { message: `The reply is not a JSON object: ${formatIssues(issues)}`, issues } }; + } + const result = await parseWithIssues(schema, value); + if (result.success) return { object: result.data }; + const message = `The reply does not match the output schema: ${formatIssues(result.issues)}`; + return { outputError: { message, issues: result.issues } }; +} + +/** The user message of the one repair step: the issues, and the ask to answer again. */ +export function outputRepairMessage(error: OutputError): Message { + return { + role: 'user', + content: `[output-invalid] ${error.message}. Reply again with only the corrected JSON object.`, + }; +} diff --git a/src/execution/toolArgsValidation.ts b/src/execution/toolArgsValidation.ts index 26c4ae842..8a3aa8966 100644 --- a/src/execution/toolArgsValidation.ts +++ b/src/execution/toolArgsValidation.ts @@ -15,6 +15,12 @@ export interface ToolArgumentIssue { message: string; } +/** `2 issues (to: Required; cc: Expected array, received string)` - the model-readable summary. */ +export function formatIssues(issues: ToolArgumentIssue[]): string { + const detail = issues.map(i => `${i.path}: ${i.message}`).join('; '); + return `${issues.length} ${issues.length === 1 ? 'issue' : 'issues'} (${detail})`; +} + /** * The model's tool-call arguments did not match the tool's parameter schema, * so `execute` was NOT called. Returned to the model as the tool result (the @@ -33,13 +39,7 @@ export class ToolArgumentsValidationError extends ToolExecutionError { toolName: string, public readonly issues: ToolArgumentIssue[] ) { - const detail = issues.map(i => `${i.path}: ${i.message}`).join('; '); - super( - `Invalid arguments for tool '${toolName}': ${issues.length} ${ - issues.length === 1 ? 'issue' : 'issues' - } (${detail})`, - toolName - ); + super(`Invalid arguments for tool '${toolName}': ${formatIssues(issues)}`, toolName); this.name = 'ToolArgumentsValidationError'; } @@ -95,16 +95,25 @@ export async function validateToolArguments( return args; } - const result = schema.safeParseAsync - ? await schema.safeParseAsync(args) - : schema.safeParse(args); + const result = await parseWithIssues(schema, args); if (result.success) { return result.data; } + throw new ToolArgumentsValidationError(toolName, result.issues); +} +/** Parses `value` with a zod-style schema: the parsed value, or every issue with its path. */ +export async function parseWithIssues( + schema: ParseableSchema, + value: unknown +): Promise<{ success: true; data: unknown } | { success: false; issues: ToolArgumentIssue[] }> { + const result = schema.safeParseAsync ? await schema.safeParseAsync(value) : schema.safeParse(value); + if (result.success) { + return { success: true, data: result.data }; + } const issues: ToolArgumentIssue[] = (result.error?.issues ?? []).map(issue => ({ path: issue.path.length > 0 ? issue.path.join('.') : '(root)', message: issue.message, })); - throw new ToolArgumentsValidationError(toolName, issues); + return { success: false, issues }; } diff --git a/src/providers/aiSdkProvider.test.ts b/src/providers/aiSdkProvider.test.ts index 7184e7c78..7cf65e12b 100644 --- a/src/providers/aiSdkProvider.test.ts +++ b/src/providers/aiSdkProvider.test.ts @@ -117,6 +117,24 @@ describe('AiSdkProvider', () => { expect(streamTextMock.mock.calls[0][0].abortSignal).toBe(signal); }); + it('LOU-V4: maps responseFormat to a JSON-mode experimental_output that leaves prompt and text alone', async () => { + generateTextMock.mockResolvedValue(textResult('stop')); + const provider = new OpenAIProvider({ name: 'openai', apiKey: 'k' }); + const schema = { type: 'object', properties: { a: { type: 'number' } } }; + + await provider.generate({ messages: [] }); + await provider.generate({ messages: [], responseFormat: { type: 'json', schema } }); + + expect(generateTextMock.mock.calls[0][0].experimental_output).toBeUndefined(); + const output = generateTextMock.mock.calls[1][0].experimental_output; + expect(output.type).toBe('object'); + expect(output.responseFormat({ model: { supportsStructuredOutputs: true } })).toEqual({ type: 'json', schema }); + expect(output.responseFormat({ model: { supportsStructuredOutputs: false } })).toEqual({ type: 'json', schema: undefined }); + expect(output.injectIntoSystemPrompt({ system: undefined, model: {} })).toBeUndefined(); + expect(output.parsePartial({ text: '{"a"' })).toEqual({ partial: '{"a"' }); + expect(output.parseOutput({ text: 'raw' }, {})).toBe('raw'); + }); + it.each([ ['stop', 'stop'], ['length', 'length'], diff --git a/src/providers/aiSdkProvider.ts b/src/providers/aiSdkProvider.ts index ba0342011..0163b33be 100644 --- a/src/providers/aiSdkProvider.ts +++ b/src/providers/aiSdkProvider.ts @@ -25,6 +25,7 @@ import { TextPart, ToolCallPart, ToolResultPart, + Output, } from 'ai'; import { LLMProvider, @@ -264,6 +265,26 @@ function toStreamResult(result: AiSdkStreamResult): StreamResult { }; } +/** + * LOU-V4: `responseFormat` as an 'ai' SDK output spec - JSON mode, with the + * schema only for models that support structured outputs (as + * `Output.object()` does). Unlike `Output.object()` it leaves the prompt and + * the reply text alone: AgentExecutor instructs the model and validates. + */ +function toOutput(format: GenerateOptions['responseFormat']): Output.Output | undefined { + if (format?.type !== 'json') return undefined; + return { + type: 'object', + responseFormat: ({ model }) => ({ + type: 'json', + schema: typeof model === 'object' && model.supportsStructuredOutputs ? format.schema : undefined, + }), + injectIntoSystemPrompt: ({ system }) => system, + parsePartial: ({ text }) => ({ partial: text }), + parseOutput: ({ text }) => text, + }; +} + /** * Base class for providers built on the 'ai' SDK. Subclasses supply the * model factory (createModel), the fallback model id, and the capability / @@ -310,6 +331,7 @@ export abstract class AiSdkProvider impleme // providers with 0 and retries in its withRetry() wrapper instead. maxRetries: this.config.maxRetries ?? 2, abortSignal: options.signal, + experimental_output: toOutput(options.responseFormat), }; } diff --git a/src/providers/llm.ts b/src/providers/llm.ts index 4aee2384b..da5b086a2 100644 --- a/src/providers/llm.ts +++ b/src/providers/llm.ts @@ -88,6 +88,13 @@ export interface GenerateOptions { tools?: ToolDefinition[]; toolChoice?: 'auto' | 'required' | 'none' | { type: 'function'; function: { name: string } }; seed?: number; + /** + * Asks for a JSON reply (LOU-V4). AgentExecutor sets it on every call of a + * run with an `output` schema. A hint: a provider with a JSON mode uses it + * (the `ai`-SDK providers pass it on as `experimental_output`), others + * ignore it. The executor parses and validates the reply either way. + */ + responseFormat?: { type: 'json'; schema?: Record }; /** * Cancels the request. Providers must reject promptly (with the signal's * `reason`, normally an `AbortError`) once it is aborted. AgentExecutor