Add LLM Obs evaluations to DDClient - #3
Open
Kyle-Verhoog wants to merge 1 commit into
Open
Conversation
Evaluation metrics, datasets, experiments and evaluator publishing are now client methods rather than something you reach into ddtrace for. submit_evaluation infers metric_type from the value (bool before int, since bool is an int subclass) and defaults to the active LLM Obs span, so scoring a workflow is a one-liner. It also accepts a live span or the dict from export_span(), and export_span() raises rather than returning None when nothing is in scope, so that's caught to give a useful error. Datasets/experiments need an app key on top of the api key, added as llmobs_app_key (DD_APP_KEY), alongside llmobs_project_name (DD_LLMOBS_PROJECT_NAME) for the project they organize under. Both are passed through to LLMObs.enable. datadog.llmobs re-exports the evaluator base classes, built-ins, judge helpers and dataset types so evaluator code doesn't import ddtrace. It is intentionally not imported by the package __init__, keeping the ddtrace.llmobs import cost off clients with LLM Obs disabled. Also fixes the return type comment on trace(): ddtrace.Span does not exist at runtime in ddtrace 4, it's ddtrace.trace.Span. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings the evaluation half of LLM Observability onto the client, so scoring what the app produced sits next to tracing it instead of requiring a drop down into ddtrace.
Covers evaluation metrics against spans, datasets, experiments (sync and async, plus a build-and-run convenience), and publishing an evaluator for Datadog to run against live spans. The metric type is inferred from the value and the evaluation attaches to the span currently being traced, so the common case is one line; spans from another process can still be joined explicitly or by tag.
Datasets and experiments hit the Datadog API directly and so need an app key on top of the api key — that plus the project they organize under are new config options, both with env var equivalents, and the error you get without an app key says so. Evaluator and dataset types are re-exported under a new module so evaluator code has a single import surface; it stays out of the package import path so clients with LLM Obs off don't pay for it.
Validated with the new example and a scratch script covering type inference, the three join keys, metadata passthrough, project defaulting and overrides, and every error path. Type checking is one error better than before; formatting is unchanged. Experiments were exercised against a recording double, not real credentials — the dataset/experiment round trip to Datadog is untested.
🤖 Generated with Claude Code