Repository navigation
fix(observe): bound eval flush waits so a down daemon never slows a case - #717
Merged
Merged
Conversation
The eval span went to a batched reporter that was never flushed before the vitest worker exited, so score rows pointed at a trace the daemon never saw. Add ObserveReporter.flush(), await it after the eval span settles, and make emit() return its promise so spans land before scores. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Cap the per-case span flush and score POST at 1s, and skip both for the rest of the run once the collector stops answering. Flush in a finally that also covers a synchronous throw from withSpan. Document the behaviour in docs/evals.md. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
lousho eval --observewrote score rows whosetrace_idpointed at a trace that never reached the daemon (seen live: the trace returned 404 and Agent Forge's score → trace link dead-ended), because the eval span sat in a batched reporter that was never flushed before the vitest worker exited. Each case now flushes its span before posting scores, with both waits capped at 1 s and skipped after the daemon proves unresponsive, so a down daemon never slows or fails a case. Adds a publicObserveReporter.flush().Ticket: O1 (Forge Observe vs LangSmith evaluation follow-up)
Changes
Verification: focused vitest,
tsc --noEmit, eslint on changed files (details in the review package); full suite and fallow run in CI.🤖 Generated with Claude Code