Repository navigation
feat(studio): online evaluators - #764
Merged
Merged
Conversation
#/evals/evaluators lists the observe daemon's online evaluator rules
(kind, filter summary, sample rate, an enabled switch that PUTs
{enabled} optimistically and rolls back with an error toast, last run
and runs in the last 24h from one source=online score read grouped by
rule name). "New evaluator" opens a form for code rules (a monospace
textarea prefilled with a sample valid against the documented trace
object) and judge rules (provider/model + rubric); without a Studio
provider key the judge option is disabled with a hint linking Settings.
#/evals/evaluators/:id edits one rule with the same form (a PUT holding
only the changed fields; a taken name shows inline), deletes it after a
confirm, and tests it on a trace through the Studio route
POST /observe/api/v1/evaluators/:id/test (value + comment; 404 reads
"Trace not found"; ?trace= prefills). List filters ?kind=&enabled=
and ?offset= live in the URL and ride along to the edit view.
The proxy allowlist gains POST /api/v1/evaluators and PUT/DELETE
/api/v1/evaluators/:id. Every Evaluations section now has a screen, so
the generic /evals/:section routes and the Placeholder are gone (an
unknown section is an unknown route). e2e: evaluator fixtures in
observeStub; revamp.spec checks the evaluator list and edit controls.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GET /settings/providers now reports envKey per provider: true when its env var (OPENAI_API_KEY / ANTHROPIC_API_KEY) is set where Studio runs, never the value. The mapping moves to secretsStore's PROVIDER_KEY_ENV, which buildAgent (and so the judge and playground) already resolves runs with. The evaluator form treats a stored key or an env key as usable, so a user with only OPENAI_API_KEY can create judge rules and the judge provider select offers that provider. Settings and ModelPicker are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Studio gets an Evaluators screen: rule list with enabled toggle and last-run counts, code/judge create+edit forms, and test on trace; every Evaluations section now has a real screen.
Ticket: F21 (Forge Observe vs LangSmith evaluation follow-up)
Changes
Verification: focused vitest,
tsc --noEmit, eslint on changed files (details in the review package); full suite and fallow run in CI.🤖 Generated with Claude Code