Building on this project? Credit StackChan Agent by Henry Yuan in your README and posts.

StackChan Agent

Give your StackChan a brain: it sees, talks and moves on its own.

Firmware, a JSON CLI and a local voice agent that let an AI (Claude Code, Codex, or any OpenAI-compatible vision model)
look through the robot's camera, talk through its speaker and move its head and Feetech STS bus-servo arms.

English · 简体中文

Demo video and updates: X @iamhenryyuan · Xiaohongshu (RedNote) · YouTube demo

The 3D twin on the live page (./stackchan dash) mirrors the real servos; here the robot has dozed off.

▶ Watch it on YouTube: "Hey Buddy", count my fingers, then dance.

Contents

• How it works
• Hardware
• Quick start
• Use from a coding agent
• Agent
• Expressions
• Dance
• Voice
• Simulation
• Episodes
• Project layout
• Roadmap
• Limitations
• Acknowledgements
• License

How it works

 StackChan (M5Stack CoreS3)              computer: ./stackchan dash             any OpenAI-compatible
 face · camera · mic · speaker   ◀──WS :81──▶  CLI + live page + agent   ◀──HTTP──▶  vision LLM
 head servos · STS arms           (or USB)     whisper.cpp STT (local)              (camera frame + what it heard
                                               Kokoro TTS (local)                    -> one step per line)

• Firmware (firmware/stackchan_agent/): an animated two-eye face (pluggable styles), the camera, mic and speaker, the body's SCS0009 head servos, STS3215 arm servos through a Waveshare Bus Servo Adapter A on a Grove port (found automatically), Wi-Fi + mDNS stackchan.local.
• Protocol (docs/protocol.md): NDJSON requests, replies and a state stream, identical over USB serial and WebSocket :81. Binary WebSocket frames carry a 1-byte type: camera JPEG, mic audio, speaker audio.
• CLI (./stackchan): JSON in, JSON out, including camera and screen snapshots.
• Live page (./stackchan dash): camera, face, a 3D twin, servo states, faults, and the agent's thinking and steps shown as icon + plain words. Any machine on the LAN can open it.
• Agent: each turn sends what it heard plus the camera frame to the LLM. The answer streams back as one primitive step per line (say, face, gesture, head, arm), executed as each line arrives.

Hardware

Part	Notes
M5Stack CoreS3	The StackChan head, USB-C to the computer for flashing (and optionally for control).
StackChan body	The official kit: its two SCS0009 head servos are on their own bus (GPIO7/6), handled by the firmware.
Waveshare Bus Servo Adapter A	UART side wired to a Grove port (C, A or B; Port A here) with a Grove cable: adapter TX/RX/GND to the port. Set the adapter's jumper to A (UART / ESP32).
2× Feetech STS3215 servos	The shoulders, id 1 = left, id 5 = right, 1 Mbps, powered from the adapter's own supply. Never feed the servo supply into the Grove 5 V pin.
LEGO Technic arms	Assorted LEGO Technic parts on a 3D-printed horn adapter (hardware/sts3215-lego-adapter.stl), mounted through the three holes on each side of the body.
Computer	Runs the CLI, live page, agent, whisper.cpp and Kokoro. Python 3.
Vision model	Any OpenAI-compatible chat API that accepts images: a cloud API or your own server.

CoreS3 Grove Port A ──▶ Waveshare Bus Servo Adapter (A) ──▶ STS3215 id 1 (left) ──▶ STS3215 id 5 (right)

Build details (BOM, wiring, mounting, the printed part, calibration): hardware/README.md.

Quick start

1. Back up the stock firmware

Before the first flash, read the whole 16 MB flash:

esptool --port PORT read-flash 0 0x1000000 backup.bin     # restore: esptool --port PORT write-flash 0 backup.bin

2. Build and flash

arduino-cli core install esp32:esp32        # 3.3.x
arduino-cli lib install M5Unified M5GFX ArduinoJson WebSockets
./flash.sh                                  # or ./flash.sh --build-only

3. First run

python3 -m venv cli/.venv && cli/.venv/bin/pip install -r cli/requirements.txt
./stackchan info                 # over USB
./stackchan wifi scan            # the networks the robot sees
./stackchan wifi set "MySSID"    # asks for the password; stored on the robot only
./stackchan scan                 # servo ids on the bus
./stackchan selftest             # each servo: +-10 deg at half torque, checks it got there
./stackchan torque 1 off; ./stackchan torque 5 off   # pose both arms hanging straight down, then:
./stackchan calibrate --arms     # that pose becomes home (0) for the agent, the dance and the 3D twin

New STS3215 servos all ship as id 1. With several on the bus, connect one at a time and give each its own id: ./stackchan set-id 1 2. Over Wi-Fi, add --host stackchan.local (or the IP shown on the face).

4. Start the agent

brew install whisper-cpp ffmpeg              # speech-to-text (local)
# Kokoro TTS model files -> ~/.cache/stackchan-agent/
#   (kokoro-v1.0.onnx, voices-v1.0.bin from the thewh1teagle/kokoro-onnx releases)
./stackchan --host stackchan.local dash \
  --llm-url http://localhost:8000/v1 --llm-model my-vision-model \
  --whisper-model ~/models/ggml-large-v3-turbo.bin --agent --lang en

Open http://localhost:8765/ for the live page, then say "Hey Buddy".

Claude instead of an OpenAI-compatible server: --llm-api anthropic sends the same turn to /v1/messages (STT and TTS stay local):

STACKCHAN_LLM_KEY=sk-ant-... ./stackchan --host stackchan.local dash \
  --llm-api anthropic --llm-url https://api.anthropic.com --llm-model claude-sonnet-5-5 \
  --whisper-model ~/models/ggml-large-v3-turbo.bin --agent --lang zh

A nicer voice: run cli/cosyvoice_server.py (CosyVoice 2) on a GPU box, tunnel it, and add --tts-url http://127.0.0.1:50021/tts. It starts talking on the first chunk (about 1 s on a DGX Spark); Kokoro speaks whenever the server is down.

Tools (with --llm-api anthropic): --tools tools.json lets the model look things up mid-turn. It says a short line first, calls the tool, then answers. The file names a command and the tool definitions; a call runs command + [name] with the input as JSON on stdin and reads stdout:

{"command": ["python3", "my_tools.py"], "prompt": "optional extra system prompt",
 "tools": [{"name": "lookup", "description": "...", "input_schema": {"type": "object", "properties": {"q": {"type": "string"}}},
            "secret": false}]}

"secret": true (a verification code, say) keeps that turn out of the episode log and the conversation history.

Use from a coding agent

Every CLI command prints one JSON object (NDJSON for stream), so Claude Code, Codex or any coding agent can drive the robot directly. Point it at AGENTS.md: commands, safety rules (small steps, read back, check volt and temp), suggested servo ids, and how to talk to a running agent:

./stackchan agent say "look at the cup and nod"   # one agent turn, as if spoken
./stackchan agent status                          # what it heard, its thinking, each step (ok / error / ms)
./stackchan agent on | off
./stackchan agent lang zh | en                    # language it speaks and listens for
./stackchan view simple | full                    # 3D twin: two shoulder servos, or the full r7 arms

To write your own client, use docs/protocol.md.

Agent

• Model output: JSON objects, one per line, each run as soon as it arrives: {"say": "...", "face": "excited"}, {"do": "face", "expression": "shy", "ms": 800}, {"do": "gesture", "name": "wave", "times": 2}, {"do": "head", "yaw": 10, "pitch": 5, "ms": 300}, {"do": "arm", "side": "left", "deg": 30, "ms": 400}, {"do": "wait", "s": 1}.
• It composes its own moves: head and arm steps carry their own duration (ms 150..1500), gestures take times 1..3. "Shake your head twice" becomes timed yaw steps; no fixed move list needed.
• Speech runs while it moves: each say is spoken in the background while the next steps execute.
• Gestures (wave, nod, shake) are keyframes in cli/gestures.py; new gestures need no firmware change.
• Reasoning: --llm-think low|high|off. Default low (short reasoning, a few times faster than high); off is fastest. With --llm-api anthropic, only high turns on extended thinking (newer models that think on their own are told not to).
• Talking motion: while it speaks, the head nods and sways on the beats of the speech, an arm now and then (cli/talk_motion.py). Nods go up from the lowest pitch (pitch_min in the head calibration).
• Touch: move its head by hand and it turns shy (face, hides behind its arms, turns away); lift its head while it waits for the wake word and it wakes up (cli/touch.py).
• Looking on demand: --look (Anthropic API) sends the camera frame only when the model calls its look tool, not every turn: faster, and it stops remarking on what it sees.
• More flags: --max-says N caps spoken sentences per turn; --volume 0..255 sets the speaker; --follow-s S keeps listening S seconds after an answer without the wake word (it listens while it thinks too).
• Flags: --no-motion keeps every servo still (faces and speech only, plus a small head lift on wake); --no-record turns off episode recording; --qr URL shows a link as a QR code in the page corner. STACKCHAN_LLM_URL, STACKCHAN_LLM_MODEL, STACKCHAN_LLM_KEY, STACKCHAN_LLM_API and STACKCHAN_WHISPER_MODEL work instead of the flags.

Expressions

24 expressions that morph into each other: neutral, happy, excited, love, shy, wink, proud, smug, sad, cry, angry, doubt, curious, surprised, scared, dizzy, sleepy, sigh, bored, listening, thinking, daydream, yawn, doze.

• The eyes breathe, blink and glance on their own.
• After 5 s without input it dozes off (and is now and then startled awake for a moment).
• The agent picks the expression for every sentence it says, and changes it during a reply.

./stackchan face --list                # all expressions
./stackchan face demo                  # play them all
./stackchan face happy --say "hi" --ms 3000
./stackchan face --idle on | off       # doze off on its own after 5 s without input

Face styles. The face engine (face.cpp) animates blinking, gaze and expressions; a style only draws one frame into a 320 × 240 sprite. To add one: write void drawMine(LGFX_Sprite &canvas, const face::Look &look) in a new .cpp, add {"mine", drawMine} to STYLES in face_styles.cpp, flash, then select it with ./stackchan raw face.set '{"style":"mine"}'. The default style eyes is two eyes and nothing else. Expressions live in face_lib.cpp.

Dance

A fixed, repeatable 10 s dance with its own music, played by the robot's speaker. No model is involved: the CLI starts it, or say "Hey Buddy, music!" (any command with "music" / "dance" / "音乐" / "跳舞"); the agent itself never does. Poses sit on beats of one clock, beat = (now - start - MUSIC_OFFSET_S) * BPM / 60, where start is the moment the robot starts playing; if the music is missing, the dance runs on the same clock alone.

./stackchan dance --dry-run      # print the timeline; checks limits, arm speed and that it ends at neutral (no robot)
./stackchan dance --amp 0.5      # first run: half amplitude, so half speed
./stackchan dance                # full;  --loop 3 repeats it seamlessly
./stackchan dance stop           # emergency stop: music off, servos ease back to neutral

On the robot. ./stackchan --host stackchan.local dance upload bakes the dance (servo curves, faces, music) into a file on the robot (dance_1, about 320 KB). The robot then dances on its own clock: one dance.play command, or hold the screen for 1 s (a tap stops it), so it also dances on battery with no computer around. The dash plays the stored copy when it is current and streams the dance from the computer otherwise (--stream forces that); upload again after changing the choreography, the calibration or the music.

It goes through a running dash, which pauses the agent and voice while it dances. The choreography, tempo (BPM, MUSIC_OFFSET_S = first beat, BEATS), limits, arm speed and volume are at the top of cli/dance.py. The music is not in the repository: put an audio file at ~/.cache/stackchan-agent/dance.mp3; for another song, set its BPM and first beat, then check with --dry-run.

Voice

Enabled with --whisper-model (a whisper.cpp ggml model). Everything runs locally and uses the robot's own mic and speaker.

• Say "Hey Buddy" to wake it: it answers "Listening", looks up, lifts its head a little and drops its arms to home (hanging down, at attention), then listens for the command. After every reply the arms go back to rest (45° forward).
• After answering it keeps listening for about 8 s, so a follow-up needs no wake word.
• --lang zh|en picks the language it speaks and listens for; switch at runtime with ./stackchan agent lang zh|en.
• Speech-to-text: whisper.cpp (whisper-server, started by the CLI). Text-to-speech: Kokoro (--tts-voice, default af_heart / zf_xiaoxiao by language), falling back to macOS say.
• Typing works too: ./stackchan agent say "wave at me".

Simulation (work in progress)

sim/ holds a small kinematic + static model of the short-arm build (shoulder servos 1 / 5, head yaw / pitch): pose clearances, holding torques, balance, and a replay of the default dance through the STS speed / acceleration profile. Pure Python, no robot needed: python3 sim/run.py pose l=90 r=0, python3 sim/run.py dance.

Work in progress: not validated against the real robot; geometry and masses are approximate. See sim/README.md.

Episodes

Every agent turn is recorded under ~/.stackchan/episodes/<YYYYmmdd-HHMMSS>/: steps.jsonl (heard, thinking, raw model answer, each step with ok and ms, servo and head state, seconds) plus the camera frame the model saw (<n>.jpg, 320 × 240). Format: docs/episodes.md.

Project layout

firmware/stackchan_agent/   CoreS3 firmware (Arduino): face engine + styles + expressions, camera, audio,
                            STS + head servo buses, Wi-Fi/mDNS, NDJSON commands
cli/                        stackchan.py (CLI), dashboard (live page + 3D twin), agent.py, llm.py, voice.py,
                            gestures.py, journal.py (episodes), watch.py
hardware/                   printed STS3215-to-LEGO adapter (STL + script), build notes
sim/                        short-arm kinematic model and dance replay (work in progress)
docs/                       protocol, episode format, images
AGENTS.md                   guide for coding agents
flash.sh, stackchan         build/flash script, CLI entry point

Roadmap

• Voice agent with wake word, follow-up listening and per-sentence expressions
• More arm servos (elbow, wrist) and a gripper
• Local audio fallback when the robot's speaker or Wi-Fi drops out
• A simulator driven by recorded episodes

Limitations

• The mic and speaker are half duplex: the mic is off while it speaks, so you cannot interrupt it (no barge-in).
• Camera, audio and commands go over Wi-Fi; latency depends on the network.
• The arm servos are heavy for the body; keep moves small and watch the load.
• The firmware only knows the soft limits set with limits: arms can still hit the head, the table or each other.

Acknowledgements

M5Unified / M5GFX · three.js · Kokoro-82M · kokoro-onnx · whisper.cpp · ArduinoJson · arduinoWebSockets · the CoreS3 camera pin map from M5CoreS3

License

MIT. Uses M5Unified / M5GFX (MIT), three.js (MIT), Kokoro-82M (Apache-2.0), kokoro-onnx (MIT) and whisper.cpp (MIT) at runtime, the CoreS3 camera pin map from M5CoreS3 (MIT), ArduinoJson (MIT) and arduinoWebSockets (LGPL-2.1).