Skip to content

[bot] OpenAI: Realtime API (client.realtime.connect()) has zero instrumentation #798

Description

@braintrust-bot

Summary

The OpenAI Python SDK's top-level Realtime/AsyncRealtime resource (client.realtime.connect() / client.realtime.client_secrets / client.realtime.calls) — the GA WebSocket API for real-time audio/text/tool-use sessions — has no instrumentation anywhere in this repo. This is a distinct, self-contained SDK surface from client.responses.connect() (already tracked in #188): it lives in its own module with its own connection manager and does not share implementation with the Responses API's websocket transport.

What is missing

py/src/braintrust/integrations/openai/integration.py registers exactly these patchers:

patchers = (
    AgentsSessionsPatcher,
    ChatCompletionsPatcher,
    EmbeddingsPatcher,
    ModerationsPatcher,
    AudioSpeechPatcher,
    AudioTranscriptionsPatcher,
    AudioTranslationsPatcher,
    ImagesPatcher,
    ResponsesPatcher,
)

py/src/braintrust/integrations/openai/patchers.py's _WRAP_TARGETS registry (the authoritative list of every OpenAI SDK method this repo patches) contains no realtime entry:

_WRAP_TARGETS: tuple[tuple[str, type[CompositeFunctionWrapperPatcher]], ...] = (
    ("beta.agents.sessions", _WrapAgentsSessions),
    ("chat.completions", _WrapChatCompletions),
    ("embeddings", _WrapEmbeddings),
    ("moderations", _WrapModerations),
    ("audio.speech", _WrapAudioSpeech),
    ("audio.transcriptions", _WrapAudioTranscriptions),
    ("audio.translations", _WrapAudioTranslations),
    ("images", _WrapImages),
    ("responses", _WrapResponses),
    ("responses.with_raw_response", _WrapResponsesRaw),
    ("beta.chat.completions", _WrapChatCompletions),
)

A case-insensitive search for "realtime" across the entire py/src/braintrust tree returns zero hits related to OpenAI's Realtime API (the only matches are unrelated "real-time" prose in other integrations/docs). At minimum, instrumentation should provide a session-level span for connect() (model config, duration, aggregate usage) and turn/tool-call spans for events sent/received over the WebSocket connection, mirroring the pattern already used for streaming responses.

Braintrust docs status: not_found

The OpenAI integration page documents Chat Completions, Responses, Embeddings, Moderation, Batch, and the Agents API as traced. It does not describe any tracing support for the Realtime API or client.realtime/connect() — the only "realtime" mention on the page is a cookbook link title ("Evaluating audio with the OpenAI Realtime API"), with no SDK-level tracing detail.

Upstream sources

Local repo files inspected

  • py/src/braintrust/integrations/openai/integration.py (full file)
  • py/src/braintrust/integrations/openai/patchers.py (_WRAP_TARGETS registry)
  • Repo-wide case-insensitive grep for realtime across py/src/braintrust

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions