[LinuxDevil]

What

• loushy eval --record runs each eval case against its agent's real provider through recordReplay() (record mode). It writes one cassette per case next to the eval file, at __cassettes__/<eval-name>/<case>.json. Names are deterministic slugs, the format is the existing cassette format, and a case with a second provider gets <case>.2.json.
• loushy eval --replay runs every case from its cassette with no network. When a cassette is missing, the case fails with loushy eval --replay: no cassette for "<eval> [<case>]" at <path>. Record it with: npx loushy eval --record <file>.
• A plain loushy eval with CI set replays each case that has a cassette and runs the rest live. Without CI it behaves exactly as before.
• loushy eval --drift re-records into a temp dir (committed cassettes are left alone) and compares each case's trajectory with its committed cassette. It compares ordered tool names, JSON-normalized args, step count and finish reason. Tokens are compared only with --drift-usage.
  • The summary gets a Drift: table with the columns EVAL / CASE / FIELD / COMMITTED / CURRENT.
  • Each drifted case gets a soft drift assertion, which shows up in JUnit as <system-out>.
  • With --drift --strict, drift is a gate failure: a JUnit <failure> and exit code 1.
• Pure comparison helper in src/evals/drift.ts (trajectoryOf, diffTrajectories). The renderDriftTable function is in src/cli/evalReport.ts.
• How it hooks in (src/evals/cassettes.ts): defineEval runs each case inside withEvalCassettes(). While a case runs, an AsyncLocalStorage-scoped wrapper around AgentExecutor.execute() swaps options.provider for that case's recordReplay wrapper. createAgent() agents, AgentExecutor.stream(), sub-agents and approval resumes all go through AgentExecutor.execute(). As a result, existing eval files need no change and src/createAgent.ts / src/execution/* are not modified. Without the env var that loushy eval sets, nothing is installed, so a plain vitest run and the smoke evals are unchanged.
• EvalResult gains cassettes?: string[].
• Fix found on the way: EvalResult.file was read from expect.getState().testPath at collection time, so it was always empty. It is now read when the test runs. Cassette paths depend on it.

Why

This is differentiator 2 in .agent-loop/AUDIT.md §7: record/replay as the default test and eval substrate. eve's evals have mockModel but no record/replay, and open-harness has neither. Evals can now be recorded once against a real model, replayed for free in CI, and checked for trajectory drift before a model or prompt change.

Acceptance criteria

• --record writes one cassette per eval case at __cassettes__/<eval-name>/<case>.json next to the *.eval.ts, with deterministic names and the existing cassette format
• --replay (explicit, or the default under CI when a cassette exists) runs from cassettes with no network; a missing cassette is a clear failure naming the --record command; plain loushy eval keeps today's behaviour
• --drift re-records into a temp location and compares tool names/order, normalized args, step count, finish reason and tokens (on request). It reports a per-case diff table in the summary and <system-out>/failure entries in JUnit; --drift --strict makes drift a gate failure
• The comparison lives in the pure, unit-tested src/evals/drift.ts
• defineEval hook added without changing existing eval files; src/evals/__smoke__.eval.ts passes
• Tests:
  • CLI record → replay round trip, using mockModel as the "real" provider. Replay runs with a provider that fails on any call, which shows replay never calls the provider.
  • CI auto-replay versus a plain local run.
  • Missing-cassette error.
  • Drift helper: no drift, tool order, args, steps/finish reason, usage ignored unless requested.
  • JUnit drift <system-out> and --strict <failure>.
  • In-process withEvalCassettes tests.
• Docs: docs/evals.md "Record, replay and drift"; docs/testing.md cross-link; docs/cli.md usage; README testing/evals line mentions --drift; llms files regenerated; CHANGELOG (Added)
• Scope: only src/cli/eval*.ts, src/evals/*, a new CLI fixture and docs (no src/testing/* change needed). About 233 added non-comment, non-test code lines.

How verified

• npx tsc --noEmit: no errors
• npm run lint: 0 errors, 405 warnings. None of the warnings are in changed files.
• npm run test:types: Type Errors no errors
• npx vitest run (after the builds): Test Files 146 passed (146); Tests 2106 passed | 3 skipped (2109)
• npm run build && npm run build --workspace=packages/create-loushy-agent: both exit 0
• npm run docs:verify-snippets -- --skip-build: all 107 snippet(s) type-check against src/; 6 also run cleanly
• npm run docs:llms:check: exit 0
• npm run test:coverage: exit 0, 146/146 files, All files 90.05% statements
• npm run fallow: the only finding is a pre-existing duplicate clone group in src/providers/AnthropicProvider.ts:35-49 / src/providers/OpenAIProvider.ts:29-43. Those files are identical to main and this PR does not touch them. No dead code or complexity findings come from this change.

🤖 Generated with Claude Code

https://claude.ai/code/session_016bY29ibQgv3MDdjAbUT4YR

---

Generated by Claude Code