Repository navigation
feat: Xiaozhi custom cloud service + self-hosting guide - #22
Merged
Merged
Conversation
Limit MCP build/deploy triggers to MCP-owned paths so future sibling apps can ship independently without redeploying the gateway. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Start the replacement Xiaozhi cloud as a separate protocol-first service so device compatibility can evolve independently from the MCP gateway. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Keep generated Python cache files out of the Xiaozhi cloud app so commits stay source-only and reproducible. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Move the custom cloud from passive websocket logging to explicit session-phase transitions so protocol behavior can be validated before wiring audio and model execution. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Capture the active task sequence and completion percentages in-repo so execution can continue step by step with explicit acceptance targets. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Keep local virtualenv and packaging byproducts out of the branch so validation setup does not pollute source commits. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Wire the first complete voice-turn path through audio buffering, Groq STT, an OpenAI-compatible LLM, Edge TTS, and Xiaozhi-framed response audio so staging can answer simple speech. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Build and deploy xiaozhi-cloud with app-specific workflow triggers and compose files so it can roll out independently from the MCP gateway. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Protocol v1 sends raw Opus bytes with no header. Add v1 passthrough to parse_audio_frame and wrap the call in try/except so bad frames no longer kill the websocket connection. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Decode accumulated Opus to PCM every 5 frames, calculate RMS energy, and auto-trigger turn processing when silence exceeds SILENCE_TIMEOUT_MS. Handles devices that stream continuously without sending listen.stop. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Previous implementation calculated RMS of the entire accumulated buffer, which meant early speech energy dominated the average forever. Now we only decode and check audio added since the last analysis, so silence at the end is properly detected. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
When decode_opus_to_pcm fails on partial Opus data, the exception handler returned without updating audio_analyzed_offset. This caused the same broken chunk to be retried forever, blocking all silence checks. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
ffmpeg -f opus requires each packet prefixed with a 4-byte big-endian length. Raw concatenated Opus frames always failed to decode, causing the silence detection VAD to never fire and the device to listen forever. - Store audio as individual frames instead of flat bytearray - Add get_opus_from() that returns length-prefixed Opus data - Use decode_length_prefixed_opus_to_pcm for energy checks - Use length-prefixed data for full Opus→WAV decode in process_turn
…/CD staging fix - Reduce MIN_SILENCE_TIMEOUT_MS from 2200 to 1000 - Send tts:start immediately after STT so device shows 'Speaking' during LLM processing - Split XiaozhiRuntime.process_turn into transcribe_audio + generate_response - Replace ffmpeg-based Opus decode with direct libopus ctypes - Add webrtcvad-based speech detection with configurable aggressiveness - Add proper 60ms Opus frame packetization for TTS output - Propagate audio params from client hello through session into server_hello - Fix CI/CD staging deploy: force-overwrite docker-compose.yml
- RogoMcpClient: real implementation using mcp SDK (streamablehttp_client), exposes list_tools_as_openai_schema and call_tool - OpenAiCompatibleLlmClient.chat: returns full assistant message dict so callers can read tool_calls, sets tool_choice=auto when tools present - XiaozhiRuntime.generate_response: tool loop capped at 5 hops with per-session tool cache (5 min TTL, refresh on error); falls back to plain LLM if MCP client unavailable - main: move tts.start to after generate_response so device does not enter Speaking state mid-tool-call Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…rst-audio latency Replace sequential LLM-complete -> TTS-complete -> opus-encode -> send with an interleaved pipeline: tokens are split into sentences as they stream from the LLM, each sentence is TTS'd as soon as it lands, and opus frames are emitted while later sentences are still being synthesized. Target: median T_first_audio_frame - T_stt_sent <= 1000ms. - audio.py: OpusStreamEncoder + stream_mp3_to_opus_frames (ffmpeg subprocess pipeline with producer task and queue consumer; BaseException-safe cancellation; stderr captured). - integrations/openai_compatible_llm.py: chat_stream() SSE async generator yielding content / tool_call_delta / finish_reason events. - integrations/edge_tts_client.py: synthesize_stream() raw-mp3 generator with NoAudioReceived semantics. - services/runtime.py: generate_response_stream() dual-task pipeline (LLM splitter + TTS pipeliner) over a FIFO sentence queue, per-sentence retry, session-aware sample rate / frame duration, tts_start emitted on first content delta, RuntimeError on empty stream, BaseException-safe task error propagation. Removes legacy generate_response/process_turn/TurnResult. - main.py: WebSocket handler consumes the streaming event loop (stt -> tts.start -> (sentence_start -> audio frames)* -> tts.stop). - tests: streaming-path coverage for SSE parser, TTS stream, opus encoder, runtime event ordering, tool-loop, retry, iteration cap, error propagation, and empty-stream rejection. 28 passed, 1 skipped (ffmpeg smoke, host-only). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three coordinated runtime fixes surfaced in pre-merge review: - _tts_one_sentence now raises on terminal failure instead of soft-logging an error event. _tts_task wraps the consumer loop in try/finally so the events_q sentinel always fires, letting the exception propagate through gather() to main.py's outer handler (which emits tts_stop+error to the client). Prevents device hangs on "responding" with no audio path. - tts_start now emits when the first sentence is enqueued (or on the residual-buffer flush), not on the first LLM content delta. An LLM stream that errors before producing a complete sentence no longer leaves the client hanging on tts_start with no follow-up. - Sentence splitter replaced with regex that splits on ?!\n; always but on . only when followed by whitespace/EOS. URLs, decimals, and times (10:30) no longer get mangled mid-utterance. Adds 19 regression tests: 7 for _try_split (URL, decimal, colon, EOS, no-terminator) and 5 for tts_start ordering across multi- sentence, single-fragment, llm-error, and tool-only paths, plus the TTS-raise unit contract. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
branches-ignore: [master, main] meant every push on every branch (dependabot, draft, WIP) raced the single staging container. Last write wins. Restrict to the active staging branch. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
.claude/ holds local Claude Code agent memory and per-machine settings; .opencode/ ships a node_modules tree we don't want tracked. Neither belongs in the repo. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The repo grew a sibling Python service (apps/xiaozhi-cloud) and a streaming LLM/TTS pipeline that the previous docs didn't reflect. Refresh everything to match the state at this commit: - AGENTS.md (root) reframed as a monorepo with two siblings. Adds xiaozhi-cloud to the structure tree, where-to-look table, and hierarchy. Disambiguates the two Xiaozhi paths (bridge/xiaozhi vs apps/xiaozhi-cloud). Refreshed metadata. - apps/xiaozhi-cloud/AGENTS.md (new) — per-app hierarchical doc matching the src/mcp + src/tools pattern. Covers the streaming pipeline contract, sentence splitter rules, MCP tool loop, audio framing, conventions, anti-patterns, and known SOON items. - apps/xiaozhi-cloud/README.md rewritten — replaces the "initial scope" stub with the actual shipped feature set. - docs/XIAOZHI_CLOUD_PLAN.md updated — Phases 1-4 marked done, Phase 5 (streaming pipeline) added retroactively, pre-merge cleanup items listed. - docs/TEMP_XIAOZHI_CLOUD_TASKS.md updated — progress 33% -> ~95%, all major slices checked off, pre-merge cleanup checklist added. - plans/slice-a-streaming-turn.md (new) — archived planning doc for the streaming refactor. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Fix MCP client never instantiating: drop mcp_project_api_key from
enable guard and URL construction; subdomain URL already routes correctly
via nginx rewrite without appending /mcp/{key}
- Fix fast/burst audio on short responses: TTS_PREROLL_FRAMES 5→0 so
every frame is throttled at the device's frame duration
- Fix repeating responses: persist assistant+tool_call messages and tool
results to session.conversation_history so the LLM sees the full tool
exchange on subsequent turns
- Fix orphaned tool messages after history truncation: _safe_truncate
strips leading tool-role entries that have no preceding assistant+tool_calls
- Sanitize MCP tool errors: return constant string instead of raw exception
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Send tts/start immediately after STT result so the device transitions from listening to thinking without waiting for first LLM sentence - Skip the redundant tts_start event from generate_response_stream since it was already sent early - Yield tool_thinking event when LLM invokes tools; device displays "[tool_name]" as sentence_start text so the user can track tool calls Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace edge-tts (100-500ms network) with Piper (50-150ms local CPU):
- PiperTtsClient synthesizes raw PCM via thread executor, no ffmpeg needed
- stream_pcm_to_opus_frames feeds PCM directly into OpusStreamEncoder,
skipping the MP3→ffmpeg→PCM pipeline entirely
- _tts_one_sentence dispatches on output_format ("pcm" vs "mp3")
- Dockerfile downloads vi_VN-vivos-medium model at build time
- TTS_PROVIDER=piper in env enables Piper; falls back to edge-tts
Add farewell/stop detection:
- _is_farewell() matches Vietnamese + English goodbye keywords on transcript
- After a farewell turn completes TTS, session closes instead of returning
to listening; device exits conversation naturally
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
wget exit code 8 = server error; HuggingFace LFS uses redirects that wget doesn't follow by default. curl -fsSL follows redirects (-L) and fails loudly on HTTP errors (-f). Also v1.0.0 tag doesn't exist — use main. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
vi_VN-vivos-medium does not exist in rhasspy/piper-voices. Available Vietnamese voices: vais1000-medium, 25hours_single-low, vivos-x_low. vais1000-medium is the best quality option. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Piper vais1000-medium outputs 22050 Hz but the device negotiates 16000 Hz. stream_pcm_to_opus_frames now accepts an optional input_rate and resamples via audioop.ratecv when it differs from the target sample_rate. PiperTtsClient exposes a sample_rate property so the runtime can pass it. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
synthesize_stream_raw() does not exist in the installed piper-tts version. Use synthesize() which yields AudioChunk objects; audio_int16_bytes gives s16le PCM directly with no manual numpy conversion needed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Small models like Gemma 4 E2B time out when given 24 MCP tool schemas. LLM_DISABLE_TOOLS=true bypasses tool fetching entirely so the turn completes without tool-call overhead. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
apps/xiaozhi-cloud/— a new Python WebSocket service that bridges Xiaozhi ESP32 devices to the Rogo IoT Cloud via MCP tool calling, with streaming LLM/TTS/Opus pipelinexiaozhi-cloud.yml,xiaozhi-cloud-staging.yml)docs/SELFHOSTING.md— developer self-hosting guide with env setup and MCP endpoint instructionsKnown follow-ups (tracked in
apps/xiaozhi-cloud/AGENTS.md)SYSTEM_PROMPTenv var through to runtime (currently declared but ignored)TTS_PROVIDER=piperin Dockerfilefeat/xiaozhi-custom-cloud)docker rm -forphan-container guard to prod workflowTest plan
🤖 Generated with Claude Code