LinuxDevil · Oct 1, 2026 · Oct 1, 2026 · Oct 1, 2026
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [Unreleased] - 2026-09-28
 
 ### Added
+- `loushy eval --record / --replay / --drift` (LOU-D46): `--record` runs every eval case against its agent's real provider through `recordReplay()` and writes one cassette per case next to the eval file (`__cassettes__/<eval-name>/<case>.json`, the normal cassette format); `--replay` runs every case from its cassette with no network, and a missing cassette fails the case with the `--record` command to run. Plain `loushy eval` with `CI` set replays the cases that have a cassette; without `CI` it is unchanged. `--drift` re-records into a temp directory and compares each case with its committed cassette (ordered tool names, JSON-normalized tool arguments, step count, finish reason; tokens with `--drift-usage`): the summary gets a `Drift:` table and each drifted case a soft `drift` assertion (a JUnit `<system-out>` note), or a gate failure with `--strict`. Existing eval files need no change: while a case runs, `AgentExecutor.execute()` wraps its provider for that case. `EvalResult` gains `cassettes`. Fixed on the way: trajectory and classic eval results now carry their `file` (it was read before the test ran and was always empty). See docs/evals.md#record-replay-and-drift.
 - OpenTelemetry GenAI metrics and cost (LOU-D48): `createOtelTraceExporter()` (`@loushy/build-ai-agent/otel`) now also records the GenAI client metrics `gen_ai.client.token.usage` (histogram, `{token}`, one record per `gen_ai.token.type` `input`/`output`) and `gen_ai.client.operation.duration` (histogram, `s`, for every model call and tool call) through `@opentelemetry/api`'s metrics API: the global `MeterProvider`, or the new `meter` option; `metrics: false` turns them off. Without a registered `MeterProvider` they are discarded by OpenTelemetry's no-op meter. Spans gain `loushy.cost_usd` (estimated USD: per step on `chat` spans, cumulative on `invoke_agent` spans, absent when the price table does not know a model) and `loushy.usage.estimated` on `invoke_agent` spans (it was already on `chat` spans). Any `TraceExporter` sees the cumulative attributes: `withSpan()` sums the cost of finished child spans onto their parent. See docs/observability.md.
 - One `store` option (LOU-D30): `createAgent({ store })` takes an `AgentStore` (`{ sessions?, checkpoints?, approvals? }`, new export) and wires all three from it. `SqliteStore` is one; the new `memoryStore()` returns in-memory sessions, checkpoints and approvals, and docs/sessions.md shows the file-based combination (`FileSessionStore`, `LocalStorageCheckpointStore`, `StorageServiceApprovalStore`). `agent.session({ id })` then keeps its transcript in `store.sessions` and checkpoints every turn in `store.checkpoints`; `store.approvals` is the default `approvalStore`. `SendOptions.sessionId` makes `send()` / `stream()` a durable run checkpointed in `store.checkpoints` (an error without one), so a one-shot durable job needs no `AgentExecutor`, and `agent.approvals.resolve()` keeps checkpointing it. The new `agent.resume(sessionId)` finishes an interrupted `send(..., { sessionId })` run, or else the session's pending turn (as `session.resume()`), and returns `null` when nothing is pending. A session's own `store` / `checkpointStore` and an explicit `approvalStore` still win, part by part. Without `store`, nothing changes.
 - One tool-error shape (LOU-U14): every failed tool call reaches the model as `{ error, toolName, message, kind }` (message capped at 2,000 characters, transcript message flagged `isError`), whichever path failed: `kind` is `'execution'`, `'validation'` (plus `issues`), `'not-found'`, `'rejected'` (plus `note`), `'not-run'`, `'mcp'` or `'sandbox'`. New exports `toolErrorResult()`, `ToolErrorResult`, `ToolErrorKind` and `ToolErrorInput`; `McpToolError` and the sandbox guard's error (named `SandboxRequiredError`, not exported) carry a `toolErrorKind`. `kind` is additive on thrown-error and validation results. The paths that built their own shapes changed, see Changed. See docs/tools.md.

diff --git a/README.md b/README.md
@@ -87,7 +87,7 @@ agent can be an `agent.yaml` spec served with `npx loushy dev agent.yaml`
 - **Hooks, guardrails, sandboxing**: veto tool calls, gate a patch on fail-closed checks, run tools in Docker. [Guardrails](docs/guardrails.md)
 - **Flows and triggers**: fixed multi-step workflows; webhook, Slack and cron adapters. [Flows](docs/flows.md), [Triggers](docs/api-overview.md#triggers)
 - **Tracing**: OpenTelemetry GenAI spans (`invoke_agent`, `chat`, `execute_tool`); content capture is opt-in. [Observability](docs/observability.md)
-- **Testing and evals**: `mockModel`, `recordReplay` cassettes, `defineEval()` trajectory assertions, `loushy eval`. [Testing](docs/testing.md), [Evals](docs/evals.md)
+- **Testing and evals**: `mockModel`, `recordReplay` cassettes, `defineEval()` trajectory assertions, `loushy eval` with `--record` / `--replay` cassettes and `--drift` trajectory diffs. [Testing](docs/testing.md), [Evals](docs/evals.md)
 - **CLI**: `init`, `doctor`, `dev`, `mcp`, `eval`, `build` and `studio`. [CLI](docs/cli.md)
 - **Agent Forge**: `loushy studio` opens a visual canvas, run debugger and chat with approval cards. [Agent Forge](docs/agent-forge.md)
 

diff --git a/docs/cli.md b/docs/cli.md
@@ -27,7 +27,7 @@ loushy build --target=<name> --agent=<path> [--out=<dir>]
 loushy studio [--port N] [--host H] [--prod|--dev]
 loushy mcp <agent.yaml|json> [--http --port N --host H]
 loushy doctor [agent.yaml|json] [--json]
-loushy eval [globs...] [--tag t] [--junit path] [--json path] [--strict] [--judge]
+loushy eval [globs...] [--tag t] [--junit path] [--json path] [--strict] [--judge] [--record | --replay | --drift [--drift-usage]]
 ```
 
 `npm create loushy-agent my-agent` runs `loushy init` with the same arguments.

diff --git a/docs/evals.md b/docs/evals.md
@@ -167,7 +167,7 @@ repository) runs them.
 ## `loushy eval`
 
 ```text
-loushy eval [globs...] [--tag t] [--junit path] [--json path] [--strict] [--judge]
+loushy eval [globs...] [--tag t] [--junit path] [--json path] [--strict] [--judge] [--record | --replay | --drift [--drift-usage]]
 ```
 
 | Option | Meaning |
@@ -179,6 +179,9 @@ loushy eval [globs...] [--tag t] [--junit path] [--json path] [--strict] [--judg
 | `--strict` | soft failures fail the run |
 | `--judge` | run `*.judge.eval.ts` files instead of the normal ones |
 | `--config path` | use your own vitest config instead of the generated one |
+| `--record` | run against the real provider and write one cassette per case ([below](#record-replay-and-drift)) |
+| `--replay` | run every case from its cassette, with no network; a missing cassette fails the case |
+| `--drift` | re-record into a temp directory and report how each case's trajectory changed (`--drift-usage` also compares tokens) |
 
 It spawns the vitest installed in your project (vitest is *your* dependency; if
 it is missing the command prints `npm install --save-dev vitest` and exits 2),
@@ -229,3 +232,56 @@ jobs:
 
 Run `--tag smoke` on every pull request and the full set nightly; run
 `--judge` only where you accept real LLM calls and their cost.
+
+## Record, replay and drift
+
+An eval against a real model is slow, costs money and needs a key; the same
+eval on `mockModel` only tests the script you wrote. `loushy eval` sits in
+between: it records each case once against the real provider through
+[`recordReplay`](testing.md#record-and-replay) and replays the recording in CI.
+Your eval files do not change.
+
+```bash
+npx loushy eval --record        # real provider: writes the cassettes, commit them
+npx loushy eval --replay        # every case from its cassette, no network, no key
+npx loushy eval --drift         # re-record and diff each case's trajectory
+```
+
+- **`--record`** runs every case with its agent's real provider and writes one
+  cassette per case next to the eval file:
+  `__cassettes__/<eval-name>/<case>.json` (names are slugged, for example
+  `refund-flow/polite.json`; an eval without `cases` writes `default.json`).
+  Give each case a unique `label` so the names stay stable. A case whose agent
+  uses more than one provider (a sub-agent on another model) gets
+  `<case>.2.json` and so on. The files use the normal cassette format, with
+  API keys redacted; review and commit them.
+- **`--replay`** never calls the model. A case with no cassette fails with
+  `no cassette for "<eval> [<case>]" at ...` and the `--record` command to run;
+  when the agent's requests changed, the replayed model call fails with a
+  `CassetteMismatchError` naming the first difference. Plain
+  `loushy eval` with `CI` set replays every case that has a cassette and runs the
+  rest live; without `CI` it runs live as before.
+- **`--drift`** re-records every case into a temp directory (the committed
+  cassettes are not touched) and compares each with its committed cassette:
+  the ordered tool names, each call's arguments (JSON-normalized, so key order
+  does not count), the step count and the finish reason. Token usage changes
+  on every real run, so it is compared only with `--drift-usage`. The summary
+  gets a `Drift:` table (eval, case, field, committed, current); each drifted
+  case gets a soft `drift` assertion, so it shows as `PASS (soft fail)` and as a
+  `<system-out>` note in the JUnit report. With `--strict`, drift is a gate
+  failure: the case fails, JUnit has a `<failure>`, and the run exits 1.
+
+```text
+Drift:
+EVAL         CASE    FIELD  COMMITTED                     CURRENT
+refund flow  polite  args   lookup_order {"orderId":"42"}  lookup_order {"orderId":"43"}
+```
+
+Run `--drift` nightly or before a model upgrade, and `--replay` (or plain
+`loushy eval` in CI) on every pull request. Judges are not recorded: grade with
+a real judge only in `*.judge.eval.ts` files.
+
+How it works: while a case runs under one of these modes, every
+`AgentExecutor.execute()` call (which `createAgent()` agents, streaming,
+sub-agents and approval resumes go through) gets its provider wrapped with
+`recordReplay()` for that case's cassette.
diff --git a/docs/testing.md b/docs/testing.md
@@ -188,6 +188,11 @@ only called when recording (so replay never constructs it). It may be
 
 `mode: 'auto'` replays if the cassette file exists and records otherwise.
 
+For `defineEval()` evals you do not need to wrap the provider yourself:
+`loushy eval --record` writes one cassette per eval case, `--replay` runs from
+them, and `--drift` reports how each case's trajectory changed. See
+[Record, replay and drift](evals.md#record-replay-and-drift).
+
 ### What replay checks
 
 By default call N is answered by entry N, but only if the request still matches

diff --git a/llms-full.txt b/llms-full.txt
@@ -86,7 +86,7 @@ agent can be an `agent.yaml` spec served with `npx loushy dev agent.yaml`
 - **Hooks, guardrails, sandboxing**: veto tool calls, gate a patch on fail-closed checks, run tools in Docker. [Guardrails](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/guardrails.md)
 - **Flows and triggers**: fixed multi-step workflows; webhook, Slack and cron adapters. [Flows](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/flows.md), [Triggers](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/api-overview.md#triggers)
 - **Tracing**: OpenTelemetry GenAI spans (`invoke_agent`, `chat`, `execute_tool`); content capture is opt-in. [Observability](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/observability.md)
-- **Testing and evals**: `mockModel`, `recordReplay` cassettes, `defineEval()` trajectory assertions, `loushy eval`. [Testing](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/testing.md), [Evals](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/evals.md)
+- **Testing and evals**: `mockModel`, `recordReplay` cassettes, `defineEval()` trajectory assertions, `loushy eval` with `--record` / `--replay` cassettes and `--drift` trajectory diffs. [Testing](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/testing.md), [Evals](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/evals.md)
 - **CLI**: `init`, `doctor`, `dev`, `mcp`, `eval`, `build` and `studio`. [CLI](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/cli.md)
 - **Agent Forge**: `loushy studio` opens a visual canvas, run debugger and chat with approval cards. [Agent Forge](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/agent-forge.md)
 
@@ -1988,7 +1988,7 @@ loushy build --target=<name> --agent=<path> [--out=<dir>]
 loushy studio [--port N] [--host H] [--prod|--dev]
 loushy mcp <agent.yaml|json> [--http --port N --host H]
 loushy doctor [agent.yaml|json] [--json]
-loushy eval [globs...] [--tag t] [--junit path] [--json path] [--strict] [--judge]
+loushy eval [globs...] [--tag t] [--junit path] [--json path] [--strict] [--judge] [--record | --replay | --drift [--drift-usage]]
 ```
 
 `npm create loushy-agent my-agent` runs `loushy init` with the same arguments.
@@ -3140,7 +3140,7 @@ repository) runs them.
 ## `loushy eval`
 
 ```text
-loushy eval [globs...] [--tag t] [--junit path] [--json path] [--strict] [--judge]
+loushy eval [globs...] [--tag t] [--junit path] [--json path] [--strict] [--judge] [--record | --replay | --drift [--drift-usage]]
 ```
 
 | Option | Meaning |
@@ -3152,6 +3152,9 @@ loushy eval [globs...] [--tag t] [--junit path] [--json path] [--strict] [--judg
 | `--strict` | soft failures fail the run |
 | `--judge` | run `*.judge.eval.ts` files instead of the normal ones |
 | `--config path` | use your own vitest config instead of the generated one |
+| `--record` | run against the real provider and write one cassette per case ([below](#record-replay-and-drift)) |
+| `--replay` | run every case from its cassette, with no network; a missing cassette fails the case |
+| `--drift` | re-record into a temp directory and report how each case's trajectory changed (`--drift-usage` also compares tokens) |
 
 It spawns the vitest installed in your project (vitest is *your* dependency; if
 it is missing the command prints `npm install --save-dev vitest` and exits 2),
@@ -3203,6 +3206,59 @@ jobs:
 Run `--tag smoke` on every pull request and the full set nightly; run
 `--judge` only where you accept real LLM calls and their cost.
 
+## Record, replay and drift
+
+An eval against a real model is slow, costs money and needs a key; the same
+eval on `mockModel` only tests the script you wrote. `loushy eval` sits in
+between: it records each case once against the real provider through
+[`recordReplay`](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/testing.md#record-and-replay) and replays the recording in CI.
+Your eval files do not change.
+
+```bash
+npx loushy eval --record        # real provider: writes the cassettes, commit them
+npx loushy eval --replay        # every case from its cassette, no network, no key
+npx loushy eval --drift         # re-record and diff each case's trajectory
+```
+
+- **`--record`** runs every case with its agent's real provider and writes one
+  cassette per case next to the eval file:
+  `__cassettes__/<eval-name>/<case>.json` (names are slugged, for example
+  `refund-flow/polite.json`; an eval without `cases` writes `default.json`).
+  Give each case a unique `label` so the names stay stable. A case whose agent
+  uses more than one provider (a sub-agent on another model) gets
+  `<case>.2.json` and so on. The files use the normal cassette format, with
+  API keys redacted; review and commit them.
+- **`--replay`** never calls the model. A case with no cassette fails with
+  `no cassette for "<eval> [<case>]" at ...` and the `--record` command to run;
+  when the agent's requests changed, the replayed model call fails with a
+  `CassetteMismatchError` naming the first difference. Plain
+  `loushy eval` with `CI` set replays every case that has a cassette and runs the
+  rest live; without `CI` it runs live as before.
+- **`--drift`** re-records every case into a temp directory (the committed
+  cassettes are not touched) and compares each with its committed cassette:
+  the ordered tool names, each call's arguments (JSON-normalized, so key order
+  does not count), the step count and the finish reason. Token usage changes
+  on every real run, so it is compared only with `--drift-usage`. The summary
+  gets a `Drift:` table (eval, case, field, committed, current); each drifted
+  case gets a soft `drift` assertion, so it shows as `PASS (soft fail)` and as a
+  `<system-out>` note in the JUnit report. With `--strict`, drift is a gate
+  failure: the case fails, JUnit has a `<failure>`, and the run exits 1.
+
+```text
+Drift:
+EVAL         CASE    FIELD  COMMITTED                     CURRENT
+refund flow  polite  args   lookup_order {"orderId":"42"}  lookup_order {"orderId":"43"}
+```
+
+Run `--drift` nightly or before a model upgrade, and `--replay` (or plain
+`loushy eval` in CI) on every pull request. Judges are not recorded: grade with
+a real judge only in `*.judge.eval.ts` files.
+
+How it works: while a case runs under one of these modes, every
+`AgentExecutor.execute()` call (which `createAgent()` agents, streaming,
+sub-agents and approval resumes go through) gets its provider wrapped with
+`recordReplay()` for that case's cassette.
+
 # Flows
 
 Source: docs/flows.md
@@ -5226,6 +5282,11 @@ only called when recording (so replay never constructs it). It may be
 
 `mode: 'auto'` replays if the cassette file exists and records otherwise.
 
+For `defineEval()` evals you do not need to wrap the provider yourself:
+`loushy eval --record` writes one cassette per eval case, `--replay` runs from
+them, and `--drift` reports how each case's trajectory changed. See
+[Record, replay and drift](https://github.com/LinuxDevil/agent-sdk/blob/main/docs/evals.md#record-replay-and-drift).
+
 ### What replay checks
 
 By default call N is answered by entry N, but only if the request still matches

diff --git a/src/cli/__fixtures__/eval/replay.eval.ts b/src/cli/__fixtures__/eval/replay.eval.ts
@@ -0,0 +1,39 @@
+/**
+ * Fixture for the `--record` / `--replay` / `--drift` tests in
+ * src/cli/eval.test.ts (LOU-D46). The "real" provider is a mockModel set up
+ * from env vars: FIXTURE_PROVIDER=offline makes every model call fail (so
+ * replay provably never calls it) and FIXTURE_ORDER_ID changes the tool
+ * arguments (drift). Its cassettes are written to __cassettes__/ next to this
+ * file and deleted by the test.
+ */
+import { z } from 'zod';
+import { createAgent } from '../../../createAgent';
+import { defineEval } from '../../../evals';
+import { defineTool } from '../../../tools/defineTool';
+import { mockModel } from '../../../testing';
+
+const lookupOrder = defineTool({
+  name: 'lookup_order',
+  description: 'Look up an order',
+  input: z.object({ orderId: z.string() }),
+  execute: ({ orderId }) => ({ orderId, status: 'shipped' }),
+});
+
+const realProvider = () =>
+  process.env.FIXTURE_PROVIDER === 'offline'
+    ? mockModel([])
+    : mockModel([
+        { toolCalls: [{ name: 'lookup_order', args: { orderId: process.env.FIXTURE_ORDER_ID ?? '42' } }] },
+        { text: 'Refunds are possible within 30 days.' },
+      ]);
+
+defineEval({
+  name: 'recorded refund',
+  agent: () => createAgent({ prompt: 'You handle refunds.', tools: [lookupOrder], provider: realProvider() }),
+  cases: [{ input: 'Refund order 42', label: 'plain' }],
+  async test(t, c) {
+    await t.send(c.input);
+    t.completed();
+    t.calledTool('lookup_order');
+  },
+});