Repository navigation
feat(studio): LLM-judge evaluators - #763
Merged
Merged
Conversation
The observe daemon holds no provider keys, so a matching judge rule now queues an in-memory job (bounded at 500, pending + leased). Agent Forge's server pulls them (POST /api/v1/evaluators/judge-jobs/claim, leased, expired leases requeue at the front), runs each judge as one direct provider call with the playground's provider resolution (no spans, so no observe feedback loop), and posts the result back (POST /api/v1/evaluators/judge-jobs/:job_id/result -> online score). Studio also handles test-on-trace for judge rules and forwards code rules. The evaluator trace builder moves to src/observe/evaluatorTrace.ts and is exported from the traces subpath so Studio builds the identical object. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each judge call now gets an AbortSignal and a cap (judgeTimeoutMs, default 20 s, cut to what is left of the job's lease minus a 1 s post margin); a claimed job whose local lease deadline has passed is skipped and left to requeue. The claim sends lease_ms explicitly. A non-JSON 2xx claim answer goes through the once-per-outage warning. The test-on-trace route rejects unsafe ids and caps its dry run the same way. baseDir is dropped from the loop options; LOUSHO_STUDIO_JUDGE=0 turns the loop off and the e2e server sets it. The daemon accepts a null result comment. Docs note the timeout and that a renamed rule scores under its claim-time name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
LLM-judge online evaluators: the daemon queues judge jobs (bounded 500, leased) and the Agent Forge server pulls them, grades with its own provider keys and posts source:'online' scores back; the daemon never holds keys. Adds Studio test-on-trace for judge rules.
Ticket: O25 (Forge Observe vs LangSmith evaluation follow-up)
Changes
Verification: focused vitest,
tsc --noEmit, eslint on changed files (details in the review package); full suite and fallow run in CI.🤖 Generated with Claude Code