Skip to content

fix(gateway): Gaia's 16 Oct issues, DAG barge-in and streaming TTS, realtime close, build and test fixes - #19

Merged
dittops merged 7 commits into
mainfrom
fix/gaia-integration-2026-10-16
Oct 19, 2026
Merged

dittops merged 7 commits into
mainfrom
fix/gaia-integration-2026-10-16

Conversation

@dittops

@dittops dittops commented Oct 17, 2026

Copy link
Copy Markdown
Member

Summary

Root-cause fixes for every Gateway issue in Gaia's 16 Oct report, plus the follow-ups found while verifying them. Each item was checked against the same inputs as Gaia's original release build (mock vendors, a real waav-infer, an auth responder and a real LiveKit 1.13.10 server).

Reported issues

Issue Root cause Fix
DAG TTS never delivers; failures hidden select! dropped the completion receiver; errors only logged Poll the receiver in place; typed DAG error messages
LiveKit room calls 404 livekit-api encodes the path's slashes; LiveKit 1.13 rejects it Vendored livekit-api with a literal path
${VAR} sent literally Conversation path passed the raw key Named credentials, resolved per LLM leg
Inline DAG can send the operator key anywhere Env expansion for client graphs; SSRF bypasses Credentials bound to hosts; SSRF and DNS guard
Debug logs print API keys Derived Debug and tracing spans record keys Redacting Debug; spans skip secrets
API secret disables the auth hook Secret mismatch returned 401 before the responder ran AUTH_MODE chain: api_secret or responder
Responder 200 {} admits callers Empty body treated as an identity Identity required, else 502
STT 5 s and LLM 60 s limits, no knobs Constants Env and config knobs (the DAG LLM knob is now honoured too)
/metrics is public No policy on the route MetricsPolicy: token, separate listener or 403
Parent-directory .env is read dotenv walks parent directories ./.env or --env-file only
Segmented STT off unless allowlisted Default allowlist ignored explicit requests An explicit request counts as opt-in
Denoiser never runs on /ws /ws never called it; LiveKit frames too short Per-session streaming denoiser, opt-in; warms up without blocking ready

Follow-ups

  • DAG barge-in and LLM→TTS pipelining. Each turn runs as its own task on the streaming executor; barge-in cancels it and clears TTS, and a newer turn supersedes an older one. The TTS node opens one provider per turn and speaks each sentence as the LLM writes it. Live: a barge-in cut the old answer within ~1 s and the new one was answered in full; first audio ~1.6 s earlier.
  • DNS rebinding on webhook / HTTP DAG nodes: they now use the connect-time guarded resolver.
  • Realtime close on a slow client. Teardown dropped the socket with the client's unread pongs buffered, so the kernel sent a TCP reset and the client lost its remaining audio and the 1011 close. Teardown now keeps reading while the backlog and close go out; the grace for the close reply runs in the background, so the admission slot is released at once.
  • Transcripts in logs. Content fields are <redacted> in the stdout log unless WAAV_LOG_CONTENT=1 (OTLP export unchanged).
  • Failed setup closes the session. A refused credential or failed DAG / conversation / LiveKit setup ends the session with a 1011 close instead of continuing as STT-only.
  • DAG definition lint reports unknown keys, including inside arrays (nodes[1].voice_ID).
  • OpenAI TTS URL no longer doubles /v1.
  • aarch64 build. Vendored webrtc-sys builds the NVIDIA codec on x86 only, so a fresh aarch64 build with CUDA installed succeeds without CUDA_HOME/clang workarounds; Cargo.lock regenerated.
  • Tests run on a clean checkout. The realtime and voice-span test files generate their credential and JWT keys at run time instead of reading bud-auth's git-ignored *.pem fixtures. Two breaker tests now open their 30 s breaker after the gateway is up, and the slow-client test sends more than the kernel can buffer.

Behaviour changes to note

  • Transcripts are no longer in the stdout log by default (WAAV_LOG_CONTENT=1 restores them).
  • A session whose LLM credential is refused, or whose DAG / conversation setup fails, now closes with 1011.

Test plan

  • 644 / 644 targeted tests (DAG, security, realtime relay and translate, voice spans, LiveKit, credentials, denoiser), with ORT_DYLIB_PATH set (turn-detector tests need ONNX Runtime)
  • DAG live verification against mock vendors and a real waav-infer (256,800 bytes of Kokoro audio delivered)
  • Barge-in and pipelining live check
  • LiveKit setup failure closes 1011 in 0.06 s (original stayed open 20 s)
  • Fresh aarch64 build with real CUDA installed

🤖 Generated with Claude Code

…ealtime close, build and test fixes

Reported issues
- DAG TTS never delivered: select! dropped the completion receiver; it is
  now polled in place, and DAG failures are typed errors, not just logs.
- LiveKit room calls 404: livekit-api encoded the path's slashes, which
  LiveKit 1.13 rejects; vendored livekit-api sends a literal path.
- ${VAR} sent literally: named credentials, resolved per LLM leg.
- Inline DAG could send the operator key anywhere: credentials are bound to
  hosts; SSRF and DNS guards on every outbound DAG call.
- Debug logs printed API keys: redacting Debug; spans skip secrets.
- The API secret disabled the auth hook: AUTH_MODE chain (api_secret or
  responder).
- A responder answering 200 {} admitted callers: an identity is required,
  otherwise 502.
- STT 5 s and LLM 60 s limits had no knobs: env and config knobs (the DAG
  LLM knob is now honoured too).
- /metrics was public: MetricsPolicy (token, separate listener or 403).
- A parent-directory .env was read: ./.env or --env-file only.
- Segmented STT stayed off unless allowlisted: an explicit request opts in.
- The denoiser never ran on /ws: per-session streaming denoiser, opt-in,
  warming up without blocking `ready`.

Follow-ups
- DAG turns run as their own task on the streaming executor: barge-in
  cancels the turn and clears TTS, a newer turn supersedes an older one, and
  node execution races the cancel token. The TTS node opens one provider per
  turn, speaks each sentence as the LLM writes it and forwards audio chunk
  by chunk (first audio ~1.6 s earlier).
- Webhook and HTTP DAG nodes use the connect-time guarded resolver (no DNS
  rebinding).
- Transcripts are redacted from the stdout log unless WAAV_LOG_CONTENT=1.
- A refused credential or failed DAG / conversation / LiveKit setup ends
  the session with a 1011 close instead of an STT-only session.
- Inline DAG definitions are linted for unknown keys, including in arrays.
- OpenAI TTS URL no longer doubles /v1.
- Realtime teardown keeps reading the client while the backlog and close go
  out, so a slow client gets its remaining audio and the 1011
  client_too_slow close instead of a TCP reset; the admission slot is
  released at once.

Build and tests
- Vendored webrtc-sys builds the NVIDIA codec on x86 only, so a fresh
  aarch64 build with CUDA installed succeeds; Cargo.lock regenerated.
- Realtime and voice-span tests generate their credential and JWT keys at
  run time instead of reading bud-auth's git-ignored *.pem fixtures; the
  breaker tests open their breaker after the gateway is up; the slow-client
  test sends more than the kernel can buffer.

Tests: 644 / 644 targeted (DAG, security, realtime relay and translate,
voice spans, LiveKit, credentials, denoiser), with ORT_DYLIB_PATH set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dittops
dittops force-pushed the fix/gaia-integration-2026-10-16 branch from 88cdc63 to b83fdca Compare October 17, 2026 20:12
…runcated replies

- Segmented STT: the waav-infer file profile now sends the session language
  (`language`, single shape, droppable on refusal) — the Gateway used to send
  none, so Infer guessed the language per segment. Routing map, capability map
  and the wire test updated.
- Native Infer STT/TTS adapters default to Infer's real endpoint,
  ws://127.0.0.1:8080/v1/ws (was ws://127.0.0.1:8123/v1/realtime).
- TTS: a speech reply WaaV Infer flags with `x-waav-truncated: true` (its
  length bound cut the utterance; the audio is real but incomplete) is logged
  and counted in waav_gateway_tts_truncated_total{provider}, instead of being
  passed off as a complete reply. Pairs with Infer's new `truncated` terminal.

Tests: segmented-stt 490/490; gateway lib 7110/7113 — the three failures
(core::metrics::provider test_ttfb_recording / test_concurrent_min_max_updates,
core::observability::speaking test_typical_speaking_session) are timing tests
in untouched code that pass on rerun (one intermittent in 5 stress runs).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jithinAB and others added 5 commits October 19, 2026 07:26
…ws test

All of these also failed on #19's head; main was last green on 5 Oct.

- supply-chain: rustls 0.23.45 -> 0.23.46 (RUSTSEC-2026-0346..0349). The rest
  cannot be upgraded here and are ignored with reasons in deny.toml (mirrored in
  gateway/.cargo/audit.toml), like the existing tonic 0.11 entries: rustls 0.22
  only via tonic 0.11 (Google Speech client); zip extract() advisories while
  WaaV never extracts (zip 0.6 is webrtc-sys-build's build-time unpack, zip 2 is
  ndarray-npy's .npz reader via mel_spec); unsafe-libyaml unmaintained via
  serde_yaml 0.9 (operator-authored YAML). `cargo audit --deny warnings` and
  `cargo deny check advisories bans licenses sources` pass.
- clippy -D warnings (Rust 1.99): chunks_exact -> as_chunks, is_multiple_of,
  and an explained allow where a `return` is only needless without features.
- server boot: config.example.yaml configures API secrets, so #19's metrics
  policy locks /metrics (403) — the secure default. The probe now asserts the
  403, then reboots with WAAV_METRICS_PUBLIC=true and checks the series.
- ws_tests::test_websocket_voice_config: #19 closes a session whose voice setup
  failed (1011) instead of leaving it half-configured; the test now asserts that
  close rather than writing into the closed socket.

Verified: clippy (default and --all-features) clean; the CI integration suites
(e2e_mock, load, server_startup, ws, keystone_wire, keystone_completeness,
segmented_stt_ws, segmented_stt_golden) 63/63.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI's rustfmt and segmented-stt jobs failed on #19's head. Formatting only, no
functional change (vendored webrtc-sys untouched); both crates build with all
targets afterwards.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gateway/vendor/ (webrtc-sys, livekit-api, vendored in #19) is upstream code kept
byte-close to its source; its spellings are not ours to change. With cargo audit
and cargo deny now green, typos on vendor/ was the last supply-chain failure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…10-19

feat(gateway): WaaV Infer integration — session language, endpoint, truncated replies
gitleaks flagged gateway/vendor/livekit-api/src/test_token.txt (rule jwt) once #20 merged into this branch
and the scan covered the whole range to main. It is the upstream crate's own test token (iss "myapikey",
sub "identity", expired March 2025), read only by its access_token.rs tests. Allowlisted by exact path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dittops
dittops merged commit dae2f89 into main Oct 19, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants