feat(gateway): voice agents - talk to a Bud agent over /v1/realtime and /ws

[dittops]

Summary

Voice agents (Bud spec 025, specs/025-voice-agents/ in bud-runtime). A caller can talk to any Bud agent that has a voice block by connecting to prompt:<agent>:

• on /v1/realtime (the OpenAI GA protocol), or
• on /ws with config.agent.

WaaV runs speech-to-text, then turn detection, then the agent's turn through budgateway /v1/responses (streamed, with the caller's credential and bud_channel: "voice"), then text-to-speech. The agent's prompt, tools and governance stay in budprompt, unchanged.

Bud side: BudEcosystem/bud-runtime#3125. budapp validates the voice block and publishes voice_agent:{prompt_id}:v{n} to Valkey; budprompt stores the block and replays only what the caller actually heard.

What changed

bud-auth

• voice_agent.rs: parses the voice_agent:* projection (VoiceAgentEntry).
• hydrate.rs, runtime.rs, snapshot.rs: a full sweep of voice_agent:* at boot and on reconnect, plus keyspace events applied without a restart. An update this build cannot parse takes the agent off the air instead of serving stale settings.
• Sessions are authorized, and re-validated, against the agent's prompt:<name> alias for both API keys and JWTs, not against its speech deployments, which the caller may not reach directly.

gateway: core/agent/ (new), the turn engine:

• brain.rs: the streamed /v1/responses client, failure classification, and a bounded retry when a request never reached the agent.
• engine.rs: turns never overlap. Barge-in truncates the reply to what was heard, and the next turn carries bud_truncate.
• Fillers: an ordered list per wait, a tool's own phrase in place of a list entry, a lead-in flushed before a tool runs, and the timer running from the turn's start, so it also covers budprompt opening its stream.
• Also in the engine: static greeting; idle hang-up; degradation message; <think> stripping; text transforms; speaking one field of structured output; panic recovery.
• accepts_speech(): the caller's speech is input only where it may cut what is playing. That's interruption.enabled for an answer and greeting.interruptible for the greeting. Otherwise the speech is dropped before it becomes an stt_result, as LiveKit's discard_audio_if_uninterruptible does.

gateway: /ws

• handlers/ws/agent.rs: the agent loop. It runs a turn controller with the agent's start strategy (min_words of 0 or 1 means any speech), ignore phrases, the session limit and idle, and maps engine signals to /ws messages (agent_response_*, assistant_transcript, agent_tool, agent_truncated, agent_error).
• bud_legs.rs: resolves the agent's STT and TTS deployments into legs and applies caller overrides only where the agent's session_overrides allows them. A refused override comes back as config_warning deployment_setting_not_applied.

gateway: /v1/realtime

• openai_realtime/cascade.rs: the GA face of an in-process agent session, covering session events, overrides (only before the session starts), manual turns, typed turns, responses and truncation.
• The session reports the agent's own turn-taking: semantic_vad with its eagerness, server_vad for silence, null for manual, and interrupt_response = interruption.enabled. Eagerness is refused on non-semantic agents.

Fixes found while testing live

• core/text/think.rs: the <think> stripper sliced inside a multi-byte UTF-8 character and panicked, which wedged sessions.
• core/tts/elevenlabs.rs: eleven_v3* rejects previous_text, next_text and request stitching with a 400, so every sentence after the first failed. These fields are now gated by supports_context_continuity.
• core/voice_manager: tracks audio delivered and still unplayed, so a truncation measures what was actually heard.

Testing

cd gateway
cargo fmt --all --check                                                       # clean (bud-auth too)
cargo clippy --all-targets --features dag-routing,turn-ensemble,noise-filter,openapi
cargo test --lib -- core::agent handlers::ws handlers::openai_realtime core::turn core::voice_manager   # 430 passed
cargo test --features openapi --test openapi_drift                           # passes; docs/openapi.yaml regenerated
cd ../bud-auth && cargo test                                                # 202 passed
typos --config _typos.toml <changed files>                                    # clean

• Clippy reports nothing in any changed file. Locally (rustc 1.96) it flags one pre-existing manual Option::zip in core/stt/iflytek/client.rs, which main's CI clippy passes.
• Mutation checks: the tests for the fixes made in the last round (fillers during stream open, speech acceptance by greeting vs answer, min_words 1, the session's turn-detection report, eagerness refusal) were each run against a deliberately broken implementation; all 7 failed as they should.
• CI baseline: on main, the only failing CI job is the secret-gated real-provider e2e.

Live on Bud's pde-ditto, with real Deepgram STT/TTS and an agent on deepseek-v4-pro with an MCP tool, over both endpoints:

• Fillers at 0.5 s / 1.6 s / 2.7 s.
• "Caller can interrupt" off: the answer plays out and the caller is ignored, while the greeting can still be talked over.
• min_words 1: a single word cuts the answer; with 2 it doesn't (control).
• Each voice setting (language, key terms, voice, speed, eagerness threshold, greeting, idle and session limits, fallback reply, manual turns) changed what callers hear or what the vendor received.

Deploy note

A fresh pod downloads the Silero VAD (from GitHub) and SmartTurn (from Hugging Face) models at first use. Where GitHub is unreachable, every voice-agent session fails setup with "Failed to initialize Silero VAD". This predates the PR, but voice agents are the first feature that needs the models on every session. Baking them into the image is a good follow-up.

🤖 Generated with Claude Code