[EPIC-V][LOU-V12] agent.send(), session.send(), evals and the React hook accept content parts

CHANGELOG.md

@@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [Unreleased] - 2026-09-28
 
 ### Added
+- Multimodal input through the public API (LOU-V12): `agent.send()` / `agent.stream()`, `session.send()` / `stream()`, `t.send()` in evals and `useLoushyAgent().send()` take the new exported `AgentInput` (`string | ContentPart[] | Message[]`): a string or parts become one user message, a `Message[]` is passed through (in a session it is appended to the transcript). Checkpoints, `agent.resume()` and queued input hold the parts like any message. The React user bubble shows the text with an `[image]` / `[file]` marker per non-text part. `SqliteStore` sessions and checkpoints save `Uint8Array` parts as `{ "$bytes": "<base64>" }`, as `FileSessionStore` does. Finishes LOU-V11 (its note that `agent.send()` / `session.send()` take no parts is out of date). See docs/providers.md#multimodal-input.
 - Built-in `ask_question` tool (LOU-X9): `createAgent({ askQuestion: true })` (off by default) or `askQuestionTool()` gives the agent `ask_question({ question, options?, allowFreeText? })`. A call pauses the run through the approval mechanism, so it waits durably like any approval (sessions, `memoryStore()` / `SqliteStore`, a restart between question and answer). The pending record gains optional `kind: 'question'` and `question: { text, options?, allowFreeText? }` (new types `ApprovalKind`, `ApprovalQuestion`; `describeApproval()` and `ASK_QUESTION_TOOL_NAME` are exported), and so do the `approval.requested` event and the React hook's `pendingApproval`. The new `agent.approvals.answer({ id, answer })` (the same as `resolve({ id, approved: true, note: answer })`) continues the run with `{ answer, option? }` as the tool result (`option`: index of the matching option); `approved: false` gives the model a `kind: 'rejected'` error saying the user declined to answer. `useLoushyAgent()` gets `answer(text)`. The `approve` callback may now return a string (`ApproveToolCall` returns `boolean | string`), which approves with that note, so code can answer questions. A tool run after an approval sees the decision's note as `ctx.approval.note` (new optional `ToolExecutionContext.approval`). See docs/approvals.md "Asking the user a question".
 - Agent Forge time travel API (LOU-D45): Agent Forge's `FileCheckpointStore` keeps the bounded per-session checkpoint history (`history()`, `historyLimit` option, `delete({ keepHistory })`) with the same semantics as `LocalStorageCheckpointStore`, checked by the SDK's history contract suite. The runtime control server gains `GET /runs/:id/history` (each step's status, save time, finish reason, tool calls, tokens and cost), `POST /runs/:id/fork` (`{ fromStep, patch?: { toolResult?, appendInput?, businessState? } }`, forks with `AgentExecutor.fork()` and starts the fork as a run of its own) and `GET /runs/compare?a=&b=` (`compareTrajectories()`). Fixed on the way: Agent Forge's chat transcript compiles against multimodal `Message.content` (LOU-V11) again, keeping each message's text. See docs/agent-forge.md#time-travel.
 - Fork and replay (LOU-D44): `AgentExecutor.fork({ sessionId, fromStep, newSessionId?, checkpointStore, patch? })` takes the newest entry of `fromStep` in the session's checkpoint history, applies `patch` (`messages(messages)`, `businessState`, `toolResult: { toolCallId, result }` replaced in place so the transcript stays provider-valid, `appendInput` queued as a user message) and saves it as the `'in-progress'` checkpoint of `newSessionId` (default `<sessionId>.fork-<n>`), leaving the original session untouched; it returns `{ sessionId, step, checkpoint }`, and `execute({ sessionId, checkpointStore, input: [] })` continues the fork. A missing step throws the new `LOUSHY_CHECKPOINT_NOT_FOUND` code. `createAgent()` agents gain `agent.fork(sessionId, { fromStep, newSessionId?, patch? })` over `store.checkpoints` (resume with `agent.resume()`). `compareTrajectories(a, b)` compares two checkpoints or transcripts step by step: each run's steps (text, tool calls and results), the first diverging step and the `loushy eval --drift` diff. New types `ForkOptions`, `ForkPatch`, `ForkResult`, `TrajectoryComparison`, `TrajectoryStep` and `DriftEntry`. See docs/durable-execution.md#fork-and-replay.

docs/providers.md

@@ -85,12 +85,14 @@ fallback model.
 `Message.content` is a string or a list of parts: `{ type: 'text', text }`,
 `{ type: 'image', image, mimeType? }` (an `http(s)` URL, a `data:` URL or the
 bytes as a `Uint8Array`) and `{ type: 'file', data, mimeType, filename? }`.
-Pass such messages as the `input` of `AgentExecutor.execute()` or
-`stream()` (`agent.send()` and `session.send()` take a string for now):
+`agent.send()`, `agent.stream()`, `session.send()` / `stream()`, `t.send()` in
+evals and the React hook's `send()` all take an `AgentInput`: a string, a list
+of parts (sent as one user message) or a `Message[]` (passed through as it is).
+`AgentExecutor.execute()` takes the same messages as its `input`:
 
 ```ts
 import { readFileSync } from 'node:fs';
-import { AgentBuilder, AgentExecutor, resolveProvider, textOf, type Message } from '@loushy/build-ai-agent';
+import { AgentBuilder, AgentExecutor, createAgent, resolveProvider, textOf, type Message } from '@loushy/build-ai-agent';
 
 const agent = AgentBuilder.create().setName('vision').setPrompt('Describe images briefly.').build();
 const input: Message[] = [
@@ -106,6 +108,13 @@ const input: Message[] = [
 
 const result = await AgentExecutor.execute({ agent, input, provider: resolveProvider('openai/gpt-4o-mini') });
 console.log(textOf(input[0]), '->', result.text); // textOf(): the text parts of a message
+
+// The same through the public API: parts become one user message.
+const answer = await createAgent({ prompt: 'Describe images briefly.', model: 'openai/gpt-4o-mini' }).send([
+  { type: 'text', text: 'What is in this picture?' },
+  { type: 'image', image: 'https://example.com/cat.png' },
+]);
+console.log(answer.text);
 ```
 
 - **Images** go to the model on `user` messages with every built-in provider
@@ -123,7 +132,8 @@ console.log(textOf(input[0]), '->', result.text); // textOf(): the text parts of
   (each image or file part counts as a flat 1,000 tokens), compaction,
   `recordReplay()` cassettes (which store the text and a digest of each
   image or file, or its URL) and `mockModel()`. `FileSessionStore` saves bytes
-  as base64 (`{ "$bytes": "..." }`) and loads them back as `Uint8Array`s.
+  as base64 (`{ "$bytes": "..." }`) and loads them back as `Uint8Array`s;
+  `SqliteStore` (sessions and checkpoints) does the same.
 
 ## Provider classes
 

docs/react.md

@@ -71,7 +71,7 @@ on the server.
 | `error` | `{ name, message }` of the last `error` event or a failed request, else `null`. |
 | `usage` | Token usage of the last finished run (from `run.done`), else `null`. |
 | `lastEvent` | The last event received, for anything the projection does not cover. |
-| `send(input)` | Starts a turn. If a turn is still running, it is aborted first. |
+| `send(input)` | Starts a turn. If a turn is still running, it is aborted first. `input` is a string, content parts or a `Message[]` (an `AgentInput`); the user bubble shows the text with an `[image]` / `[file]` marker per other part (remote mode POSTs it as `{ "input": ... }`). |
 | `stop()` | Aborts the turn in flight through its `AbortSignal`; `status` goes back to `'idle'`. |
 | `approve(note?)`, `reject(note?)` | Decide `pendingApproval` (see below). |
 | `answer(text)` | Answers a question (`pendingApproval.kind === 'question'`); the same as `approve(text)`. `reject()` declines it. |

docs/sessions.md

@@ -33,6 +33,8 @@ console.log(session.id, session.messages.length);
 Concurrent `send()` calls on one session are queued and run one after another in
 call order, so the transcript never interleaves.
 
+`send()` and `stream()` take a string, content parts (`[{ type: 'text', ... }, { type: 'image', ... }]`, one user message) or a `Message[]`; the stores keep the parts, see [Multimodal input](./providers.md#multimodal-input).
+
 A `send()` that throws (a provider error, or a tool that throws a
 `PropagatingToolError`) or is aborted with `signal` leaves the transcript
 exactly as it was before that call. An aborted `send()` resolves with

llms-full.txt

@@ -4479,12 +4479,14 @@ fallback model.
 `Message.content` is a string or a list of parts: `{ type: 'text', text }`,
 `{ type: 'image', image, mimeType? }` (an `http(s)` URL, a `data:` URL or the
 bytes as a `Uint8Array`) and `{ type: 'file', data, mimeType, filename? }`.
-Pass such messages as the `input` of `AgentExecutor.execute()` or
-`stream()` (`agent.send()` and `session.send()` take a string for now):
+`agent.send()`, `agent.stream()`, `session.send()` / `stream()`, `t.send()` in
+evals and the React hook's `send()` all take an `AgentInput`: a string, a list
+of parts (sent as one user message) or a `Message[]` (passed through as it is).
+`AgentExecutor.execute()` takes the same messages as its `input`:
 
 ```ts
 import { readFileSync } from 'node:fs';
-import { AgentBuilder, AgentExecutor, resolveProvider, textOf, type Message } from '@loushy/build-ai-agent';
+import { AgentBuilder, AgentExecutor, createAgent, resolveProvider, textOf, type Message } from '@loushy/build-ai-agent';
 
 const agent = AgentBuilder.create().setName('vision').setPrompt('Describe images briefly.').build();
 const input: Message[] = [
@@ -4500,6 +4502,13 @@ const input: Message[] = [
 
 const result = await AgentExecutor.execute({ agent, input, provider: resolveProvider('openai/gpt-4o-mini') });
 console.log(textOf(input[0]), '->', result.text); // textOf(): the text parts of a message
+
+// The same through the public API: parts become one user message.
+const answer = await createAgent({ prompt: 'Describe images briefly.', model: 'openai/gpt-4o-mini' }).send([
+  { type: 'text', text: 'What is in this picture?' },
+  { type: 'image', image: 'https://example.com/cat.png' },
+]);
+console.log(answer.text);
 ```
 
 - **Images** go to the model on `user` messages with every built-in provider
@@ -4517,7 +4526,8 @@ console.log(textOf(input[0]), '->', result.text); // textOf(): the text parts of
   (each image or file part counts as a flat 1,000 tokens), compaction,
   `recordReplay()` cassettes (which store the text and a digest of each
   image or file, or its URL) and `mockModel()`. `FileSessionStore` saves bytes
-  as base64 (`{ "$bytes": "..." }`) and loads them back as `Uint8Array`s.
+  as base64 (`{ "$bytes": "..." }`) and loads them back as `Uint8Array`s;
+  `SqliteStore` (sessions and checkpoints) does the same.
 
 ## Provider classes
 
@@ -4616,7 +4626,7 @@ on the server.
 | `error` | `{ name, message }` of the last `error` event or a failed request, else `null`. |
 | `usage` | Token usage of the last finished run (from `run.done`), else `null`. |
 | `lastEvent` | The last event received, for anything the projection does not cover. |
-| `send(input)` | Starts a turn. If a turn is still running, it is aborted first. |
+| `send(input)` | Starts a turn. If a turn is still running, it is aborted first. `input` is a string, content parts or a `Message[]` (an `AgentInput`); the user bubble shows the text with an `[image]` / `[file]` marker per other part (remote mode POSTs it as `{ "input": ... }`). |
 | `stop()` | Aborts the turn in flight through its `AbortSignal`; `status` goes back to `'idle'`. |
 | `approve(note?)`, `reject(note?)` | Decide `pendingApproval` (see below). |
 | `answer(text)` | Answers a question (`pendingApproval.kind === 'question'`); the same as `approve(text)`. `reject()` declines it. |
@@ -4776,6 +4786,8 @@ console.log(session.id, session.messages.length);
 Concurrent `send()` calls on one session are queued and run one after another in
 call order, so the transcript never interleaves.
 
+`send()` and `stream()` take a string, content parts (`[{ type: 'text', ... }, { type: 'image', ... }]`, one user message) or a `Message[]`; the stores keep the parts, see [Multimodal input](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/providers.md#multimodal-input).
+
 A `send()` that throws (a provider error, or a tool that throws a
 `PropagatingToolError`) or is aborted with `signal` leaves the transcript
 exactly as it was before that call. An aborted `send()` resolves with

src/createAgent.ts

@@ -47,6 +47,7 @@ import type { z } from 'zod';
 import type { McpServerSpec } from './spec/schema';
 import { agentMcp, streamAfter } from './tools/mcp/agentMcp';
 import { HookRegistry, type AgentHook } from './execution/hooks';
+import { toMessages, type AgentInput } from './providers/content';
 import { compactionHookFor, type AgentCompaction } from './context/agentCompaction';
 
 /**
@@ -355,8 +356,12 @@ export interface SendOptions {
 
 /** `TObject`: the type of `result.object` - `z.output` of the `output` schema. */
 export interface SimpleAgent<TObject = unknown> {
-  /** Send a single user message and get back the full execution result text. */
-  send: (message: string, options?: SendOptions) => Promise<ExecutionResult<TObject>>;
+  /**
+   * Send a single user message and get back the full execution result text.
+   * `message` is a string, content parts (one user message with an image or
+   * file, LOU-V12) or a `Message[]` passed through as it is.
+   */
+  send: (message: AgentInput, options?: SendOptions) => Promise<ExecutionResult<TObject>>;
   /**
    * Send a single user message and stream the run as typed events (LOU-V2):
    * `text.delta` chunks as the model writes, `tool.start`/`tool.done`,
@@ -372,7 +377,7 @@ export interface SimpleAgent<TObject = unknown> {
    * }
    * ```
    */
-  stream: (message: string, options?: SendOptions) => AgentRun<TObject>;
+  stream: (message: AgentInput, options?: SendOptions) => AgentRun<TObject>;
   /**
    * Start a multi-turn conversation (LOU-W4): every `send()` sees the earlier
    * exchanges. Kept in the agent's `store` (in memory without one); pass
@@ -546,12 +551,12 @@ export function createAgent<TOutput extends z.ZodTypeAny = z.ZodUnknown>(
   // `object` was validated with `config.output`, so it has its output type.
   type Typed = z.output<TOutput>;
   const simpleAgent: SimpleAgent<Typed> = {
-    async send(message: string, options: SendOptions = {}): Promise<ExecutionResult<Typed>> {
-      const result = await run(message, options.signal, durable(options.sessionId));
+    async send(message: AgentInput, options: SendOptions = {}): Promise<ExecutionResult<Typed>> {
+      const result = await run(toMessages(message), options.signal, durable(options.sessionId));
       return approvals.settle(result, options.signal) as Promise<ExecutionResult<Typed>>;
     },
-    stream(message: string, options: SendOptions = {}): AgentRun<Typed> {
-      return startStream(executeOptions(message, options.signal, durable(options.sessionId))) as AgentRun<Typed>;
+    stream(message: AgentInput, options: SendOptions = {}): AgentRun<Typed> {
+      return startStream(executeOptions(toMessages(message), options.signal, durable(options.sessionId))) as AgentRun<Typed>;
     },
     session,
     async resume(sessionId: string, { signal } = {}): Promise<ExecutionResult<Typed> | null> {

src/evals/trajectory.test.ts

@@ -58,6 +58,26 @@ function failedMessages(result: EvalResult): string[] {
   return result.assertions.filter((a) => !a.passed).map((a) => a.message ?? '');
 }
 
+describe('t.send() with an AgentInput (LOU-V12)', () => {
+  it('sends content parts to the agent as one user message', async () => {
+    const model = mockModel(['It is a cat.']);
+    const parts = [
+      { type: 'text' as const, text: 'What is this?' },
+      { type: 'image' as const, image: 'https://example.com/cat.png' },
+    ];
+    const result = await run(
+      async (t) => {
+        await t.send(parts);
+        t.check('names the animal', t.reply, includes('cat'));
+      },
+      { agent: createAgent({ provider: model }) }
+    );
+
+    expect(failedMessages(result)).toEqual([]);
+    expect(model.calls[0].messages.filter((m) => m.role === 'user')).toEqual([{ role: 'user', content: parts }]);
+  });
+});
+
 describe('trajectory assertions: passing', () => {
   it('passes every assertion for a matching run and reports steps, tool calls and usage', async () => {
     const result = await run(async (t) => {

src/evals/trajectory.ts

@@ -8,6 +8,7 @@
 import type { SimpleAgent } from '../createAgent';
 import type { ExecutionResult } from '../execution/AgentExecutor';
 import type { LLMProvider } from '../providers/llm';
+import type { AgentInput } from '../providers/content';
 import { llmJudge } from './llmJudge';
 import type { Check } from './checks';
 import type { AssertionKind, AssertionResult, EvalResult, EvalToolCall } from './evalResult';
@@ -50,7 +51,7 @@ export interface CalledToolOptions {
  */
 export interface EvalTestContext {
   /** Runs the agent on `message`. May be called more than once; assertions look at all runs. */
-  send(message: string): Promise<ExecutionResult>;
+  send(message: AgentInput): Promise<ExecutionResult>;
   /** Text of the latest reply (empty before the first `send()`). */
   readonly reply: string;
   /** The latest run, or `undefined` before the first `send()`. */
@@ -130,7 +131,7 @@ class TrajectoryContext implements EvalTestContext {
     return this.recorded;
   }
 
-  async send(message: string): Promise<ExecutionResult> {
+  async send(message: AgentInput): Promise<ExecutionResult> {
     this.agent ??= typeof this.agentSource === 'function' ? await this.agentSource() : this.agentSource;
     const result = await this.agent.send(message);
     this.results.push(result);