Skip to content

LOU-D5: scripted mockModel for deterministic agent tests - #40

Merged
LinuxDevil merged 2 commits into
mainfrom
lou-d5-scripted-mock-model
Oct 1, 2026
Merged

LinuxDevil merged 2 commits into
mainfrom
lou-d5-scripted-mock-model

Conversation

@LinuxDevil

Copy link
Copy Markdown
Owner

What

Adds mockModel(script, options?) to @loushy/build-ai-agent/testing: a typed, scripted LLMProvider test double.

const model = mockModel([
  { toolCalls: [{ name: 'get_weather', args: { city: 'Paris' } }] },
  { text: 'It is 21°C in Paris.' },
]);
const agent = createAgent({ prompt: '...', provider: model, tools });
await agent.send('Weather in Paris?');
expect(model.calls).toHaveLength(2);
model.assertExhausted();
  • Turns: string, { text }, { toolCalls, text? }, { error }, or (request) => turn (sync/async); optional usage, finishReason, delayMs.
  • Deterministic tool-call ids (call_1, ...), calls (deep-frozen snapshots), lastCall, reset(), assertExhausted().
  • Exhaustion error names the unexpected call number, shows the last message and suggests adding a turn or { onExhausted: 'repeat-last' }.
  • stream() emits scripted text as chunks, then tool-call and finish chunks.
  • /testing now exports a new src/testing/index.ts barrel (existing repo mocks + mockModel); package.json exports["./testing"] and the tsup entry were repointed.
  • Docs: new docs/testing.md, linked from README and docs/api-overview.md.

Why

Many tests hand-roll "scripted provider" objects; competitors ship mock models for deterministic evals. This removes that boilerplate.

How verified

  • npx tsc --noEmit: clean
  • npm run lint: 0 errors (no new warnings from my files)
  • npm run build + create-loushy-agent build: success; dist/testing/index.{js,mjs,d.ts} emitted
  • npm run test:coverage: Test Files 89 passed (89), Tests 955 passed | 3 skipped
  • npm run fallow: exit 0 (0 above threshold, maintainability 92.4)
  • npm run docs:verify-snippets (quick-start + docs/testing.md): all snippets type-check and run (vitest-based snippets are marked no-verify since vitest is not in the temp project; the same flow is covered by src/testing/mockModel.test.ts)

Notes for reviewer

  • Dogfood: DelegationTool.test.ts (4 tests + the A<->B loop test) and sandbox-wiring.test.ts converted. Net diff of those two files: 26 insertions, 104 deletions (78 lines saved). Because mockModel records a snapshot at call time, assertions no longer include the assistant message the executor pushes onto the shared array after the call (the old tests had comments about this aliasing).
  • Request snapshots copy plain objects/arrays and keep functions/class instances (e.g. zod schemas in tool definitions) by reference, since those cannot be structurally cloned.

🤖 Generated with Claude Code

LinuxDevil and others added 2 commits October 1, 2026 09:06
Adds `mockModel(script, options?)` to the `/testing` subpath: a typed
LLMProvider test double that replays scripted turns (string, {text},
{toolCalls}, {error}, or a function of the request) with optional usage,
finishReason and delayMs. Tool-call ids are generated deterministically
(call_1, call_2, ...). Every generate()/stream() request is recorded as a
deep-frozen snapshot on `calls` (plus `lastCall`, `reset()`,
`assertExhausted()`). Running out of turns throws an error naming the
unexpected call number and the last message, and suggesting a new turn or
`onExhausted: 'repeat-last'`. stream() emits the scripted text as chunks.

`/testing` now points at a new src/testing/index.ts barrel that re-exports
the existing repository mocks plus the new API (package.json exports and
tsup entry repointed; dist/testing/index.{js,mjs,d.ts}).

Dogfooded in DelegationTool.test.ts and sandbox-wiring.test.ts, replacing
hand-rolled scripted providers (net -78 lines). New docs/testing.md,
linked from README and docs/api-overview.md.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@LinuxDevil
LinuxDevil merged commit c970438 into main Oct 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant