Context
v0.11.0 shipped the Observe API with four views and per-agent rollups in ObservedAgent.
Agent-level and model-level aggregation now work. Two levels are still missing.
There is no tool-level aggregation. execute_tool spans carry gen_ai.tool.name,
invocation count and latency, but nothing in the current contract surfaces them. Operators
can see that an agent is expensive and slow without being able to see which tool is
responsible.
There is no run-level aggregation. Spans correlate into traces, but the contract exposes no
notion of a complete run, so questions like "how many turns did this take" and "which runs
failed after partial tool success" cannot be answered.
Both levels are prerequisites for cost allocation, because tool invocation count is the
allocation key for tool-side meters and run duration is the allocation key for compute.
What already exists
view enum with overview, agents, models, coverage
ObservedAgent.source_kind with foundry, external, unknown
CoverageResult.dimension already includes agent attribution, model attribution,
token usage and trace correlation
ObserveFilters with agent and model dimensions
/api/observe/agent-detail for bounded per-agent trends
Scope
Extend view with tools and runs.
tools view aggregates by reported project, agent, tool name and source. Fields are
invocations, failures, p95 latency, and last observed activity. Token fields are
deliberately absent, because token attributes are not required on execute_tool spans and
must not be synthesized.
runs view aggregates a correlated trace into a single row. Fields are run key, agent key,
turn count, tool invocation count, total observed input and output tokens, failure state,
duration, and last observed activity.
Extend ObserveFilters with tool_name and run_key, both nullable, both validated
against the configured ObserveScope exactly like the existing dimensions.
Refine ObservedAgent.source_kind. The current three values are too coarse to answer
"which runtime is this agent on", which every downstream cost question depends on. Proposed
values are foundry_hosted, foundry_prompt, external_registered,
external_unregistered, copilot_studio, unknown. Existing values map forward, and
unknown remains the default when the runtime cannot be determined from telemetry.
Extend CoverageResult.dimension with tool attribution and run correlation, so operators
learn that tool data is missing rather than seeing an empty table.
Agent types in scope
All five runtimes, because this issue reads telemetry only and adds no billing dependency.
| Runtime |
Identity source |
Notes |
| Foundry hosted |
Hosted agent server package |
Full tool and run data expected |
| Foundry prompt |
Unconfirmed |
Emit behavior needs verification, see open questions |
| External registered |
otel_agent_id from version registration |
Full data when instrumented |
| External unregistered |
OTel distro plus OpenInference instrumentation |
Agent identity may be null |
| Copilot Studio |
Environment-level telemetry |
ExecuteTool spans in the dependencies table |
Copilot Studio environment-level telemetry emits InvokeAgent, ExecuteTool and
OutputMessages spans aligned to OTel GenAI conventions, correlated through
operation_Id and operation_ParentId. Agent-level Copilot Studio telemetry is
event-based, has no OTel alignment, and stays out of scope.
Out of scope
- Any monetary value
- Token attribution to individual tool invocations
- Changing customer telemetry configuration or generating absent telemetry
- Copilot Studio agent-level event telemetry
Acceptance criteria
tools and runs are accepted by /api/observe/query and rejected outside scope with 422
- Tool rows never report token fields
- An agent with no tool spans yields a coverage result of
no_data or not_reported,
never a zero row
source_kind reports the refined runtime for every runtime that can be determined, and
unknown otherwise, with no inference from unrelated telemetry
- Filters round-trip through the page URL, consistent with
ObserveFilterState
- External-agent telemetry appears in both views when readable in scope, per FR-033
Open questions
- Do Foundry prompt agents emit
gen_ai spans at runtime? If they do not, source_kind
cannot distinguish them and the value should be withheld rather than guessed.
- Which attribute is the most stable run correlation key across the five runtimes?
- Does Copilot Studio
ExecuteTool carry a tool name in a field that normalizes cleanly to
gen_ai.tool.name?
Context
v0.11.0 shipped the Observe API with four views and per-agent rollups in
ObservedAgent.Agent-level and model-level aggregation now work. Two levels are still missing.
There is no tool-level aggregation.
execute_toolspans carrygen_ai.tool.name,invocation count and latency, but nothing in the current contract surfaces them. Operators
can see that an agent is expensive and slow without being able to see which tool is
responsible.
There is no run-level aggregation. Spans correlate into traces, but the contract exposes no
notion of a complete run, so questions like "how many turns did this take" and "which runs
failed after partial tool success" cannot be answered.
Both levels are prerequisites for cost allocation, because tool invocation count is the
allocation key for tool-side meters and run duration is the allocation key for compute.
What already exists
viewenum withoverview,agents,models,coverageObservedAgent.source_kindwithfoundry,external,unknownCoverageResult.dimensionalready includes agent attribution, model attribution,token usage and trace correlation
ObserveFilterswith agent and model dimensions/api/observe/agent-detailfor bounded per-agent trendsScope
Extend
viewwithtoolsandruns.toolsview aggregates by reported project, agent, tool name and source. Fields areinvocations, failures, p95 latency, and last observed activity. Token fields are
deliberately absent, because token attributes are not required on
execute_toolspans andmust not be synthesized.
runsview aggregates a correlated trace into a single row. Fields are run key, agent key,turn count, tool invocation count, total observed input and output tokens, failure state,
duration, and last observed activity.
Extend
ObserveFilterswithtool_nameandrun_key, both nullable, both validatedagainst the configured
ObserveScopeexactly like the existing dimensions.Refine
ObservedAgent.source_kind. The current three values are too coarse to answer"which runtime is this agent on", which every downstream cost question depends on. Proposed
values are
foundry_hosted,foundry_prompt,external_registered,external_unregistered,copilot_studio,unknown. Existing values map forward, andunknownremains the default when the runtime cannot be determined from telemetry.Extend
CoverageResult.dimensionwith tool attribution and run correlation, so operatorslearn that tool data is missing rather than seeing an empty table.
Agent types in scope
All five runtimes, because this issue reads telemetry only and adds no billing dependency.
otel_agent_idfrom version registrationExecuteToolspans in the dependencies tableCopilot Studio environment-level telemetry emits
InvokeAgent,ExecuteToolandOutputMessagesspans aligned to OTel GenAI conventions, correlated throughoperation_Idandoperation_ParentId. Agent-level Copilot Studio telemetry isevent-based, has no OTel alignment, and stays out of scope.
Out of scope
Acceptance criteria
toolsandrunsare accepted by/api/observe/queryand rejected outside scope with 422no_dataornot_reported,never a zero row
source_kindreports the refined runtime for every runtime that can be determined, andunknownotherwise, with no inference from unrelated telemetryObserveFilterStateOpen questions
gen_aispans at runtime? If they do not,source_kindcannot distinguish them and the value should be withheld rather than guessed.
ExecuteToolcarry a tool name in a field that normalizes cleanly togen_ai.tool.name?