Skip to content

feat(cockpit): add tools and runs views to the Observe API #441

Description

Context

v0.11.0 shipped the Observe API with four views and per-agent rollups in ObservedAgent.
Agent-level and model-level aggregation now work. Two levels are still missing.

There is no tool-level aggregation. execute_tool spans carry gen_ai.tool.name,
invocation count and latency, but nothing in the current contract surfaces them. Operators
can see that an agent is expensive and slow without being able to see which tool is
responsible.

There is no run-level aggregation. Spans correlate into traces, but the contract exposes no
notion of a complete run, so questions like "how many turns did this take" and "which runs
failed after partial tool success" cannot be answered.

Both levels are prerequisites for cost allocation, because tool invocation count is the
allocation key for tool-side meters and run duration is the allocation key for compute.

What already exists

  • view enum with overview, agents, models, coverage
  • ObservedAgent.source_kind with foundry, external, unknown
  • CoverageResult.dimension already includes agent attribution, model attribution,
    token usage and trace correlation
  • ObserveFilters with agent and model dimensions
  • /api/observe/agent-detail for bounded per-agent trends

Scope

Extend view with tools and runs.

tools view aggregates by reported project, agent, tool name and source. Fields are
invocations, failures, p95 latency, and last observed activity. Token fields are
deliberately absent, because token attributes are not required on execute_tool spans and
must not be synthesized.

runs view aggregates a correlated trace into a single row. Fields are run key, agent key,
turn count, tool invocation count, total observed input and output tokens, failure state,
duration, and last observed activity.

Extend ObserveFilters with tool_name and run_key, both nullable, both validated
against the configured ObserveScope exactly like the existing dimensions.

Refine ObservedAgent.source_kind. The current three values are too coarse to answer
"which runtime is this agent on", which every downstream cost question depends on. Proposed
values are foundry_hosted, foundry_prompt, external_registered,
external_unregistered, copilot_studio, unknown. Existing values map forward, and
unknown remains the default when the runtime cannot be determined from telemetry.

Extend CoverageResult.dimension with tool attribution and run correlation, so operators
learn that tool data is missing rather than seeing an empty table.

Agent types in scope

All five runtimes, because this issue reads telemetry only and adds no billing dependency.

Runtime Identity source Notes
Foundry hosted Hosted agent server package Full tool and run data expected
Foundry prompt Unconfirmed Emit behavior needs verification, see open questions
External registered otel_agent_id from version registration Full data when instrumented
External unregistered OTel distro plus OpenInference instrumentation Agent identity may be null
Copilot Studio Environment-level telemetry ExecuteTool spans in the dependencies table

Copilot Studio environment-level telemetry emits InvokeAgent, ExecuteTool and
OutputMessages spans aligned to OTel GenAI conventions, correlated through
operation_Id and operation_ParentId. Agent-level Copilot Studio telemetry is
event-based, has no OTel alignment, and stays out of scope.

Out of scope

  • Any monetary value
  • Token attribution to individual tool invocations
  • Changing customer telemetry configuration or generating absent telemetry
  • Copilot Studio agent-level event telemetry

Acceptance criteria

  • tools and runs are accepted by /api/observe/query and rejected outside scope with 422
  • Tool rows never report token fields
  • An agent with no tool spans yields a coverage result of no_data or not_reported,
    never a zero row
  • source_kind reports the refined runtime for every runtime that can be determined, and
    unknown otherwise, with no inference from unrelated telemetry
  • Filters round-trip through the page URL, consistent with ObserveFilterState
  • External-agent telemetry appears in both views when readable in scope, per FR-033

Open questions

  • Do Foundry prompt agents emit gen_ai spans at runtime? If they do not, source_kind
    cannot distinguish them and the value should be withheld rather than guessed.
  • Which attribute is the most stable run correlation key across the five runtimes?
  • Does Copilot Studio ExecuteTool carry a tool name in a field that normalizes cleanly to
    gen_ai.tool.name?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions