Repository navigation
LOU-G: Evals harness - #8
Merged
Merged
Conversation
Wraps vitest test() around the real static AgentExecutor.execute() call
(AgentExecutor has no instance methods in this codebase). EvalConfig's
execution fields (agent, input, provider, toolRegistry, maxSteps,
temperature, maxTokens) are typed directly off ExecuteOptions rather than
a nested {prompt, toolRegistry} shape, since that's what execute() actually
accepts. Manually verified score:()=>0/threshold:1 fails and score:()=>1
passes via a throwaway eval file targeted with an ad-hoc vitest config
(deleted before commit; LOU-G2 wires *.eval.ts into the default config).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds an explicit test.include (vitest had no prior include, relying on its default test/spec glob) covering both the default test/spec pattern and '**/*.eval.ts', plus a '**/*.judge.eval.ts' exclude so LOU-G6's llmJudge evals never run on the default `vitest run` path once they exist. Adds a real, always-passing defineEval() smoke eval using MockLLMProvider, confirmed picked up by `npx vitest run` and covered by `npm run test:coverage`. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds exactMatch(matcher) in src/evals/scorers.ts, matching a string, RegExp, or predicate against the real ExecutionResult.text field. Never throws for a non-match, including when a predicate itself throws. Unit tested with one fixed fake ExecutionResult per matcher type, covering match and non-match. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds toolCallOrder(expected) in src/evals/scorers.ts. AgentExecutor's real ExecutionResult already exposes a top-level toolCalls: ToolCall[] field (populated across every loop step in runAgentLoop(), in request order), so this scorer reads it directly rather than collecting calls itself via the LOU-E onToolCall/onToolResult tracing hooks. Compares length, per-position tool name, and (when specified) args via JSON.stringify equality against the parsed function.arguments. Unit tests cover exact match, shuffled order, a missing call, an args mismatch, and omitted args. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds budget(limits) and describeBudgetFailure(result, limits) in src/evals/scorers.ts, checking the real result.usage.totalTokens and result.steps fields against optional maxTokens/maxSteps limits. budget() returns 0 if either exceeded, 1 otherwise; describeBudgetFailure() checks both and reports a clause per failure. Unit tests cover a result at 50% of budget (pass), 150% (fail), and assert the failure message contains both the actual and budgeted numbers. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds src/evals/llmJudge.ts using the REAL LLMProvider interface
(src/providers/llm.ts): it has generate()/stream() taking a GenerateOptions
with a `messages` array, no bare-string `.complete()`, so llmJudge() builds
a grading prompt and calls provider.generate({ model, messages: [{role:
'user', content: gradePrompt}] }). parseJudgeScore() clamps a well-formed
numeric response to [0, 1] and explicitly returns { score: 0, reason } for
a malformed/non-numeric response, verified by a dedicated unit test rather
than relying on NaN's accidental fail-closed comparison behavior.
Adds a standalone vitest.judge.config.ts (NOT merged with the base config)
whose include matches ONLY '**/*.judge.eval.ts'; vitest.config.ts's own
'**/*.eval.ts' include (LOU-G2) already excludes that same suffix. Added
"test:evals:judge" npm script; left .github/workflows/ci.yml untouched.
Manually verified (not committed): bumping the smoke judge eval's threshold
above the mock judge's fixed 0.9 score flips it from pass to fail, proving
defineEval()+llmJudge() wiring tracks the fixed score exactly. Confirmed
`npx vitest run` (default config) does not collect *.judge.eval.ts, and
neither vitest.config.ts's include nor ci.yml reference judge evals.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…nning outside its judge-eval config QA found that llmJudge()'s exclusion from default CI relied purely on the '*.judge.eval.ts' file-naming convention and glob patterns — a misnamed or mis-globbed judge-eval file (wrong case, different suffix) could silently make a real, budgeted LLM call as part of the default `vitest run`/CI path, with no structural check to catch it. llmJudge() now throws unless LOUSHY_ALLOW_LLM_JUDGE=1 is set, which only vitest.judge.config.ts sets. This is independent of how the test file is named, so the isolation boundary holds even if a file is accidentally picked up by the wrong vitest config. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…uine AgentExecutor output LOU-G4's AC requires toolCallOrder() to be checked against real AgentExecutor.execute() output, not only hand-fabricated fakeResult()/ fakeToolCall() fixtures that could silently drift from how the real executor actually populates toolCalls (field names, arguments-as-JSON- string, ordering). Add a test that registers two real tools on a real ToolRegistry, drives a scripted LLMProvider (same pattern as AgentExecutor.test.ts) through a real AgentExecutor.execute() call, and feeds the genuine result.toolCalls into toolCallOrder() - asserting 1 for the correct sequence and 0 for a reversed/incomplete one. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements all 6 tickets from LOU-G (Phase 2-3), adding a behavior-regression eval harness on top of the existing vitest workflow — no new test runner introduced:
defineEval()wraps vitest'stest()— runsAgentExecutor.execute()exactly once, scores the result, assertsscore >= thresholdviaexpect, so a failing eval shows up as ordinary red CI output.*.eval.tswired intovitest.config.ts's include pattern — picked up by a barevitest run, no separate command.exactMatch()(string/regex/predicate againstresult.text, never throws on a non-match),toolCallOrder()(assertsresult.toolCallssequence/args),budget()+describeBudgetFailure()(token/step budget checks with a human-readable overage message).llmJudge()for rubric-graded free-form output, calling the realLLMProvider.generate()interface — kept strictly out of the default eval run via a separatevitest.judge.config.tsandnpm run test:evals:judgescript.Codebase adaptations
AgentExecutor.execute()is static;EvalConfigmaps directly to the realExecuteOptionsshape.LLMProvideronly hasgenerate()/stream()(no.complete()), sollmJudge()builds a messages array.result.toolCallsis a real top-level fieldAgentExecutoralready populates across every step —toolCallOrder()reads it directly rather than needing the LOU-E tracing hooks.Validation results, and a defense-in-depth hardening
QA verified every ticket end-to-end, including constructing a real
AgentExecutor.execute()run with a scripted provider requesting two tool calls and feeding the genuine result intotoolCallOrder()(not just hand-fabricated fixtures), and confirmingexactMatch()'s predicate branch is genuinely try/caught rather than letting a throw propagate.QA's one residual finding:
llmJudge()'s exclusion from default CI relied purely on the*.judge.eval.tsnaming convention and glob patterns — a misnamed or differently-cased file wouldn't be caught by the exclude and could silently make a real, budgeted LLM call in default CI. Not blocking, but since the epic explicitly treats this as a cost/safety boundary, added a structural guard:llmJudge()now throws unlessLOUSHY_ALLOW_LLM_JUDGE=1is set, which onlyvitest.judge.config.tssets — independent of how the test file happens to be named.npx vitest run: 612 passing, 2 skipped (Docker, pre-existing from LOU-F). 3 known pre-existing provider-suite failures (missing optional peer deps) — unrelated to this epic.npx tsc --noEmit: clean.npm run test:coverage:src/evals/defineEval.tsat 100% coverage.🤖 Generated with Claude Code