Skip to content

Voice Agent SDK - #35

Closed
sam-s10s wants to merge 10 commits into
mainfrom
feature/voice-package
Closed

sam-s10s wants to merge 10 commits into
mainfrom
feature/voice-package

Conversation

@sam-s10s

@sam-s10s sam-s10s commented Sep 3, 2025

Copy link
Copy Markdown
Member

This is a beta version of the Voice Agent client for the Real-Time API.

It provides the following features:

  • VoiceAgentClient sub-class of AsyncClient
  • Accumulates partials and finals from the STT and emits these as segmented groups of words
  • Supports focussing and ignoring speakers
  • Provides TTFB metrics

Comment thread examples/voice/README.md
## Installation

```bash
pip install -r requirements-examples.txt

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we'd need to add speechmatics-rt and speechmatics-voice into requirements-examples.txt otherwise the scripts won't run

also it'd be nice to use venv to ensure this is reproducible

python -m venv .venv
source .venv/bin/activate

Comment thread examples/voice/README.md
Comment on lines +26 to +27
cd examples/voice
pip install -r requirements-examples.txt

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i don't think we need to duplicate this setup step since its already mentioned in Installation section

@@ -0,0 +1,188 @@
import asyncio

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should add more explanation about what this example does and what kind of logs the user should expect. Initially, I didn’t realise this file was focused on S2 until I actually saw S2 logs coming through. When I ran it with example.wav, I didn’t see any logs at first and thought something was wrong, but later understood that this was just how the file works.

"""
# Parse command line arguments for audio file path
# Default to example.wav if no file specified
audio_file_path = Path(sys.argv[1]) if len(sys.argv) > 1 else Path(__file__).parent / "../example2.wav"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could we use argparse or fire to expose this as a input flag?

also I think the focus_speakers param should be toggleable so that as a user I can test with different speaker_ids, not just S2

@@ -0,0 +1,132 @@
import asyncio

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some explanation here on what this example demonstrates would be nice too. It's not clear that we're focusing on S1 until I tested it and looked at the code.

Comment thread examples/voice/README.md
Comment on lines +49 to +50
cd examples/voice
pip install -r requirements-examples.txt

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dont need this either

# Send frames after delay
self._processor_task = asyncio.create_task(process_after_delay(self._processor_wait_time))

def _calculate_ttfb(self, end_time: float) -> None:

@aaronng91 aaronng91 Sep 4, 2025 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm still not 100% convinced we call this TTFB. This is measuring audio lead time — how far ahead the client streamed past the audio that produced the first text.

True TTFB/TTFT should be the wall-clock difference between when streaming starts and when the first text arrives. And you only calculate it once per session

@aaronng91

Copy link
Copy Markdown

We've discussed this before but there’s a fair amount of custom logic (like discarding/ignoring speakers and grouping fragments by speaker). It would be valuable to add unit tests around it at some point, not necessarily in this MR. Otherwise, it’ll be difficult to make changes without unintentionally breaking existing functionality.

@sam-s10s sam-s10s closed this Sep 16, 2025
@sam-s10s

Copy link
Copy Markdown
Member Author

Create a new PR.

@sam-s10s
sam-s10s deleted the feature/voice-package branch September 16, 2025 09:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants