feat: segmented speech-to-text on live calls, and a tool-call tone for voice agents

[dittops]

Summary

This is the gateway side of two voice features, plus one shared core for every speech-to-text upload path (section 3). The Bud side (budapp, budadmin, budplayground, chart) is BudEcosystem/bud-runtime#3147.

1. Segmented speech-to-text: file-only models on live calls

A transcription model that only accepts recorded files can now serve a live call, a voice agent or /v1/realtime. The gateway cuts the caller's audio at each pause (Silero with SmartTurn), uploads each piece, and returns the text in order with turn signals.

The full status, by release, is in docs/segmented-stt/IMPLEMENTATION_STATUS.md. It covers where the code is, every switch, how each part was verified, and what is not built.

• Engine: new crate segmented-stt.
  • Front end, detectors, segmenter, end-of-turn ladder, sequencer.
  • Attempt loop, limiter, upload breaker and quality filter.
  • Long-window budgets with admission.
  • Setup probe and language vote.
• Wire families: OpenAI-compatible, ElevenLabs, Deepgram, AssemblyAI, Azure fast, Google.
  • Two commit transports: OpenAI's realtime transcription socket and Cartesia's finalize. Both are behind switches until a live probe.
• Capability map and resolver: every live session is resolved against the map.
  • The result is streaming, segmented, buffered or refused, with a named code.
  • Each transcription deployment gets a published voice_capability:{id} record and GET /capabilities/stt.
• Fallback to a second vendor: the deployment's fallback_models (FRD-022), the same list uploads already use.
• Data region and retention: stt.data_region: eu and stt.data_retention: none.
  • They reach the streaming socket, the upload client and the segmented engine.
  • When a vendor cannot honour them, the call or upload is refused with stt_data_setting_unavailable before any audio is sent.
• The greeting fix: each utterance carries its own may-be-cut flag. A reply after a protected greeting can be interrupted again.
• Defects found and fixed along the way (table in the status doc):
  • Silero v5 was missing its 64-sample context, so real speech scored at most 0.011.
  • The OpenAI and Groq clients lost their callbacks at the first audio_end.
  • A /ws client could point uploads at a private base_url (SSRF).
  • Setup-probe verdicts and breakers were shared across deployments on the same host.
  • A changed limit never reached a limiter already in use.
  • Uploads were sent before the data-settings check on the passthrough path.
  • ONNX Runtime logged about 500 INFO lines per session.
• SDKs (TypeScript and Python):
  • Segmented speech-to-text options.
  • VoiceAgentConfig (agent=) for voice agents.
  • base_url and the TypeScript provider field are now optional.
  • The config drift guards read the envelope from the spec.

2. A gentle tone while a voice agent works on a tool call

When a voice agent calls a tool, the caller hears the tool's phrase ("One moment."). After that, instead of more phrases or silence, a soft pulse plays until the agent speaks again.

• The tone (core/agent/tone.rs):
  • A 360 ms pulse every 1.6 s: 440 Hz with its octave, eased in and out, about -24 dBFS peak.
  • Generated, not recorded, and written in the session's output format (PCM16 or G.711 μ-law/A-law).
  • A compressed output format cannot carry it, so those sessions keep their phrases.
• When it plays: across the agent's work on a tool call, from the call until the agent speaks again.
  • On pde-ditto, an agent's MCP tool returned in 0.4–1.7 s while its model thought for 2–8 s around each call. A tone tied to the tool's own run time never played.
  • The phrase comes once per span of tool work, even when the tool returned before the phrase was due.
  • Each pulse waits for 400 ms of quiet output, so it is heard apart from the phrase.
• How it is delivered (VoiceManager::play_if_epoch): through the TTS egress.
  • It queues after speech, counts in the playout estimate, and is cut by a barge-in.
  • Only the pulses are queued, never the quiet between them, so an answer is never held behind silence.
• Truncation accounting: the spoken ledger counts the tone as sound, not speech. The heard part of a reply cut by a barge-in, and the session's speaking-rate calibration, are unaffected by it.
• Switch: fillers.tool_call_sound in bud-auth, on by default. An entry written before it existed reads as on.

3. One shared core for every speech-to-text upload path

The gateway already had file-upload code for /v1/audio/transcriptions, the batch API, and the OpenAI, Groq and prerecorded clients. Segmented sessions had re-implemented parts of it. Each concern now has one implementation in the segmented-stt crate, which the gateway already depends on.

Each path keeps its own request fields and rules on top of the shared core. Live segments send minimal fields that can be dropped when a vendor refuses one. The REST route still forwards files and returns the vendor's body verbatim, so its output is unchanged.

Shared	Replaces
wav	four WAV writers (gateway, OpenAI client, Groq client, crate)
vendor	four request-id lookups and two error-body parsers; vendor hosts and keep-no-audio switches spelled out in several places; two Azure OpenAI URL builders; Groq's own duration parser
vendor::openai	the OpenAI and Groq clients' response structs, and the crate's hand-walked JSON
vendor::{deepgram, elevenlabs}	two parsers per vendor (prerecorded client and segmented sessions)
net	three private-address (SSRF) classifiers
breaker	two circuit-breaker state machines; the gateway's CircuitBreaker keeps its API
resample	two streaming resampler wrappers
batch::http_request and the AssemblyAI upload	two copies each, in the batch API and the prerecorded client

Bugs fixed while consolidating, each with a test that fails without the fix:

• An AssemblyAI deployment set to stt.data_region: eu sent segmented audio to the US host. This was a data-residency defect in this PR's own code.
• A self-hosted deployment's REST upload followed redirects, with its credential, to any host.
• The gateway breaker stayed half-open until restart when its probe was answered with the caller's 4xx, or never reported. This affected the nine HTTP upload clients and streaming reconnects.
• Groq decided retries by matching text its own messages never contained, so most 429s and 5xx were not retried; 498 and 504 never were.
• The batch API:
  • sent keyterm to every Deepgram model (only Nova-3 reads it);
  • labelled every OpenAI upload audio.wav;
  • sent a source language to the translations route.
• The OpenAI client and the batch API posted to /v1/v1 when their base ended in /v1.
• Groq could not parse the reset durations it actually sends (2m59.56s).
• A present but empty request-id header stopped the lookup.
• Live calls built Azure OpenAI URLs with the deployment unencoded, and with an older default api-version than REST. Segmented sessions also sent an empty api-key header where REST refuses the deployment.

Review fixes (after a full review of this PR, each with a test):

• A Retry-After too large for a Duration (1e30) panicked the shared parser; it is now read as no wait. Groq's retry waits at most 60 s.
• The tool-call tone was made at 24 kHz on mulaw/alaw output, which plays at 8 kHz, so pulses were three times too long. It is now made at 8 kHz for G.711, once per session. A second tool in the same span no longer speaks a phrase over the tone.
• The gateway resampler's flush disarmed the stale-filter reset when an utterance ended on a chunk boundary.
• A caller's content type that is not a media type (mp3) failed the OpenAI batch upload; it goes as audio/wav again. ElevenLabs batch files are named by type.
• The blocked-hostname and IP-literal SSRF rules are now shared too: a client-named segmented base is refused for metadata.google.internal and decimal private addresses, as gateway URLs already were.
• The breaker reports state changes from under its one lock, so the waav_circuit_breaker_state gauge cannot miss one. The FileBreaker alias is gone.

CI: four constant-size chunks_exact calls became as_chunks (new clippy 1.99 lint). gitleaks allowlists Deepgram model ids (pricing_key deepgram:nova-2-medical) and a sequential-hex test key, matched on the extracted secret only. typos accepts speaches, an open-source Whisper server.

Deploy notes

• Segmented STT ships switched off. WAAV_SEGMENTED_STT defaults to allowlist with an empty list, so nothing is segmented until a deployment is listed or the switch is set to on.
• The tone is on by default for every voice agent.
• Either merge order is safe. Older readers ignore the new projection and config keys.
• Not built (sign-off or live-key items, listed in the status doc):
  • pause and resume after a false interruption;
  • switching the default to on;
  • the seven analytics columns;
  • drift and live-probe CI jobs;
  • live LiveKit and SIP tests;
  • removing the OpenAI and Groq OnSilence strategy.
• The shared core changes existing gateway paths:
  • Breakers: a lost or 4xx-answered half-open probe is replaced (after a lease of the cooldown, at least 15 s) instead of stranding the breaker. The window counts the last N outcomes exactly instead of halving counters.
  • SSRF: the gateway's checks now also refuse IPv4 multicast, the reserved 240/4 range and 0/8. None of these is a vendor address.
  • Groq: retries follow the HTTP status.
• Secrets / IAM: no changes.

Testing

Run on this branch:

cargo fmt --all --check                                                                   # clean
cargo clippy --all-targets --features dag-routing,turn-ensemble,noise-filter,openapi     # see note
cargo test --lib --features dag-routing,turn-ensemble,noise-filter,openapi                # 7388 passed, 4 failed (see note)
cargo test --features dag-routing --test e2e_mock_tests --test load_test_with_mocks --test server_startup \
  --test ws_tests --test keystone_wire --test provider_keystone_completeness \
  --test segmented_stt_ws --test segmented_stt_golden                                     # all 8 binaries pass (63 tests)
cargo test --features openapi --test openapi_drift                                         # 3 passed
cd ../segmented-stt && cargo clippy --all-targets -- -D warnings && cargo test             # clean; 483 passed
cd ../bud-auth && cargo test                                                              # 176 + 29 passed

Notes on the two non-green lines:

• clippy: the one lint is pre-existing. It is manual_option_zip at core/stt/iflytek/client.rs:556, a file this PR does not touch, and it fires only with this host's clippy 0.1.96. The clippy (deny warnings) job on main passes.

  • Running the check above found that the message benchmark no longer compiled after ready.stt was added. That is fixed in 498f789.

• lib tests: the 4 failures are core::onnx tests that need libonnxruntime.so. This host does not have it; CI installs it through its native-deps step.

• Tone tests:

  • Engine scenarios on a paused clock: phrase then tone; a quick tool followed by long thinking; the tone ending when the agent speaks; barge-in; tone off; an output that cannot carry the tone; text-only sessions.
  • Encoder and pulse unit tests.
  • Ledger accounting.
  • VoiceManager delivery and epoch tests.
  • Each tone test was shown to fail against a mutated engine.

Live on pde-ditto (overlay image). This predates section 3: the shared-core consolidation is covered by the suites above but has not been deployed there yet.

• Segmented STT
  • scribe-v2 on a voice agent over /v1/realtime and /ws, about 1 s per sentence, including barge-in.
  • The EU host and mip_opt_out were seen in the vendor URL.
  • Data settings fail closed on calls and uploads.
• Tool-call tone
  • Realtime PCM16: "One moment.", then 0.6 s of quiet, then 4 pulses, then the answer.
  • /ws with 8 kHz μ-law: decoded pulse peak 1820, against 1817 on PCM16.
  • Switch off: the phrases go on.
  • response.cancel during the tone: no audio afterwards.

🤖 Generated with Claude Code