Watch voice pipeline

A webserver that turns the ESP32 watch into a voice assistant:

watch mic ──WiFi──> this server ──> mlx-whisper ──> LM Studio ──> say/Kokoro ──> WAV ──> watch speaker
                      :8080 / :8081      (STT)        (the model)      (TTS)

One turn, warm, measured on this Mac (M4 Max) with google/gemma-4-e4b:

stage	time
Whisper large-v3-turbo on 1.5 s of speech	~430 ms
LM Studio (gemma-4-e4b)	~800 ms
say to 16 kHz PCM	~1.3 s
total	~2.5 s

Checkout status: restore the missing module first

The current checkout is not self-contained: voiceserver/llm.py imports run_tool and tool_specs from voiceserver/tools.py, but that file is absent from Git. Starting the server will fail at this import once dependencies are available. Setting TOOLS_ENABLED=false does not bypass a top-level import. Recover the matching module from the maintained source before following the runtime steps below; installing another Python package does not supply this project-local file.

The protocol and timing sections below describe the original running Mac setup. They are useful integration references, not proof that this recovered checkout can currently reproduce those results. Source review on 2026-10-13 did not run models or a microphone session.

Setup after source recovery

You need an Apple Silicon Mac for the implemented mlx-whisper backend, Git, uv, a microphone, and LM Studio with a downloaded model exposed through its local server. Use Python 3.12 for the path below (pyproject.toml permits ≥3.11). Install ffmpeg on PATH for browser/compressed audio and the macOS say conversion path. The default TTS backend uses macOS say.

git clone https://git.peterscharmueller.com/tompeak/Demokrit.git
cd Demokrit
uv venv --python 3.12
uv pip install --python .venv/bin/python -r pyproject.toml
cp .env.example .env

After restoring voiceserver/tools.py, edit .env: point LM_STUDIO_URL at your LM Studio server (normally http://127.0.0.1:1234 on the same Mac), set LM_STUDIO_MODEL to the exact model ID it serves, and use TTS_BACKEND=say for the first run. Set VOICE_HOST=127.0.0.1 for a laptop-only test. The default 0.0.0.0 is needed for LAN clients but exposes an unauthenticated model endpoint on reachable interfaces. Values containing spaces must be quoted, because run.sh sources .env as shell code.

bash run.sh

The first successful start downloads Whisper weights and warms the model. Keep the terminal running; in a second terminal check:

curl --fail http://127.0.0.1:8080/healthz

Wait for stt.loaded to become true. Open http://127.0.0.1:8080, allow the browser microphone, record a short question and listen to the reply. Success: A transcript and reply are produced and the log contains a turn with stage timings. A responding health endpoint alone does not verify LM Studio or TTS. Ctrl+C stops the server.

For ModuleNotFoundError: voiceserver.tools, resolve the missing source above. For an unavailable model host, check LM Studio's server and model ID. Audio conversion errors require checking ffmpeg; a missing microphone permission is a browser/OS setup issue. Configure a watch only after this local round trip works.

Watching the log

./run.sh prints to the terminal and appends to logs/server.log, so you can follow it from anywhere:

./run.sh                      # run it in a terminal, logs scroll live
tail -f logs/server.log       # follow from another terminal
tail -f logs/server.log | grep 'turn \['    # just the turns

One line per turn, with the per-stage timings:

10:48:02 INFO voiceserver.pipeline  turn [tplay] 2.5s audio | stt 435ms llm 4501ms tts 1259ms |
         'Wie weit ist der Mond von der Erde entfernt?' -> 'The Moon is, on average, about 384,400 km'

VOICE_LOG_LEVEL=debug ./run.sh adds the HTTP and worker detail. Set VOICE_SAVE_DIR=turns to also dump every turn's heard audio, spoken reply and transcript to disk, which is the fastest way to work out whether a bad answer was a mishearing or the model.

The address the watch connects to

No router port forwarding is involved. Port forwarding maps the internet into a LAN; the watch and this Mac are both already on 192.168.1.x, so they talk to each other directly. (Forwarding 8080 to the internet would publish an unauthenticated endpoint that runs a model on your machine — don't.)

By default, and in the original LAN setup, the server binds 0.0.0.0, so it answers on every address the Mac has. The laptop-only quick start above instead sets VOICE_HOST=127.0.0.1; change that setting deliberately before connecting a watch. The following addresses belong to the original installation:

address	note
http://192.168.1.246:8080	the LAN IP — what the firmware defaults to
http://Toms-MacBook-Pro.local:8080	mDNS name, survives a DHCP change
http://127.0.0.1:8080	this Mac only — not reachable from the watch

That IP comes from DHCP, so it can change. Two ways to make it stable: give the Mac a DHCP reservation on the router, or point the firmware at the .local name — voice_assist.c resolves those through the mDNS component (CONFIG_VOICE_MDNS_TIMEOUT_MS), and the Mac already advertises it.

On the original Mac, the firewall allowed that interpreter. On another Mac, verify the firewall permission for your own Python process after the local test succeeds; the original machine's allowance does not transfer with this repository.

Three ways for the firmware to connect

Pick whichever is least work in the firmware. All three run at once.

1. POST /v1/voice — one utterance, one reply (push-to-talk)

The simplest shape, and the one to use if the watch has a button. Record while pressed, POST the buffer, play what comes back.

POST /v1/voice?rate=16000&reply=wav&device=watch-1
Content-Type: application/octet-stream

<raw signed 16-bit little-endian PCM, mono>

Response is a WAV file: a canonical 44-byte header followed by PCM at reply_rate, so firmware can skip 44 bytes and stream the rest straight to I2S, or read bytes 40..43 as a little-endian uint32 to get the exact data length.

query param	default	meaning
rate	16000	sample rate of the body, when it has no WAV header
channels	1	channel count of the body; >1 is averaged to mono
reply	wav	wav, pcm (no header), json (text only), none
reply_rate	16000	sample rate of the returned audio — set this to your I2S rate
device	client IP	conversation key, so follow-up questions have context
lang	auto	force a language, e.g. de; skips detection

Response headers carry the text alongside the audio, percent-encoded because header values cannot hold raw UTF-8:

X-Transcript: Wie%20weit%20ist%20der%20Mond%20entfernt%3F
X-Reply:      Im%20Mittel%20etwa%20384.400%20Kilometer.
X-Timings-Ms: {"stt":442,"llm":802,"tts":1310,"total":2554}
X-Language:   de

The body may be raw PCM, a WAV file, a multipart upload, or anything ffmpeg can read (Opus, MP3, FLAC). A WAV header's own rate wins over ?rate=. Chunked transfer encoding with no Content-Length is fine — which is what an ESP32 streaming straight off I2S will usually send.

2. tcp://<mac>:8081 — continuous PCM over a bare socket

For firmware where an HTTP client is not worth the flash. Connect, then write s16le PCM continuously. The server runs voice-activity detection and answers whenever it hears a complete utterance:

watch  → connect, write PCM at VOICE_INPUT_RATE, forever
server → on each detected utterance, writes back one canonical WAV
          (44-byte header; bytes 40..43 give the data length)

The connection stays open across turns. shutdown(SHUT_WR) flushes a partial utterance and gets a final reply. The input rate is not negotiated on this socket — it comes from VOICE_INPUT_RATE, so set that to match the firmware.

3. ws://<mac>:8080/v1/stream — continuous, with a text channel

Same VAD behaviour, but each turn also reports its transcript and timings. Binary frames are PCM; text frames are JSON control messages ({"flush":true}, {"reset":true}, {"rate":16000}). Each reply is a JSON frame describing the turn, then one binary WAV frame.

Single stages, for bisecting a problem

curl -X POST --data-binary @question.wav localhost:8080/v1/stt          # audio -> text
curl -X POST -d '{"text":"hello"}' localhost:8080/v1/chat              # text  -> text
curl -X POST -d '{"text":"hello"}' localhost:8080/v1/tts -o out.wav    # text  -> audio
curl -X POST 'localhost:8080/v1/reset?device=watch-1'                  # clear history

The firmware client

The watch side now exists, in the firmware repo at main/audio/voice_assist.c — wake word (ESP-SR WakeNet or microWakeWord) → record → POST /v1/voice. It is built against path 1 above and the contract is verified in both directions:

firmware	server
raw s16le mono 16 kHz, no header	matches VOICE_INPUT_RATE, so no ?rate= needed
Content-Type: application/octet-stream, chunked	read without a Content-Length
?reply=json&device=tplay	returns transcript, reply, timings_ms
checks error / error_stage	absent on success, set on failure
branches on 422 vs 503	422 = heard nothing, 503 = model host down

Its CONFIG_VOICE_SERVER_URL defaults to http://192.168.1.246:8080/v1/voice?reply=json&device=tplay — the Mac's LAN address, because 127.0.0.1 on the watch means the watch.

Note that reply=json returns text only. To have the watch speak the answer, change that to reply=wav (or reply=pcm) and play the body; the server already returns a canonical 44-byte-header WAV at VOICE_REPLY_RATE.

Both sides run the same VAD shape and numbers, so the utterance boundaries the firmware picks are the ones this server's streaming paths would pick.

Testing without the watch

tools/fake_firmware.py speaks all three protocols, so you can exercise the exact path the firmware will use:

tools/fake_firmware.py health
tools/fake_firmware.py post --wav question.wav   # push-to-talk shape
tools/fake_firmware.py post --mic --seconds 5    # record from this Mac
tools/fake_firmware.py tcp  --mic                # raw socket + server VAD
tools/fake_firmware.py ws   --mic                # websocket + server VAD

tools/stub_lmstudio.py --port 1234 is a fake OpenAI-compatible server, for testing the audio path when the real model host is down.

Web search and weather

The watch can answer questions about now — weather, news, results — because the server runs tools for the model.

Why it has to live here. LM Studio's plugins and MCP servers only work inside its own chat window. The OpenAI-compatible API it serves follows the standard contract: it returns tool_calls and expects the client to execute them. Nothing was doing that, so the watch got no search no matter what was installed in LM Studio. voiceserver/tools.py was that client in the running setup; it must be restored to this checkout as described above.

Two tools are offered:

tool	source	note
get_weather	wttr.in	no API key; real figures
web_search	DuckDuckGo (ddgs)	top 4 results, snippets trimmed to 220 chars

Weather is a separate tool on purpose. Search snippets for weather are link descriptions — "Check current conditions in Vienna" — with no numbers in them, and a model handed that writes a template instead of an answer ("The current conditions are [insert actual condition here]"). One keyless request to wttr.in is both correct and four times faster.

Measured through the full voice pipeline:

question	tool	total
"What is the weather in Vienna?"	get_weather	3.0 s
"Who won the most recent Formula 1 race?"	web_search	6.8 s
"What is the capital of France?"	none	1.5 s
"How tall is the Eiffel Tower?"	none	1.3 s

Gating is the point: ordinary questions never pay for a search. That is done with the tool description ("use this ONLY when...") plus a system-prompt line, not with keyword matching.

TOOL_MAX_ROUNDS=1 means one tool call then an answer. On the final round the tools are withheld and the model is told so explicitly — without that line a small model dithers ("I need to check the current race") instead of reading what it already fetched.

Two failure modes this had to be hardened against, both seen in real turns:

• The bare promise. Asked as a command — "mach einen Websearch für X" — a small model answers the instruction rather than the question: it replies "Searching the web for printers." in under a second, calls nothing, and the watch goes quiet. Announcing is fine, but the answer has to come with it, so when a reply contains a promise and nothing survives once the promise is removed, the server runs the search itself on the user's own words and asks again with the results in hand.
• The template. Given tool output with no facts in it, the model fills the gap: "The current conditions are [insert actual condition here]." The system prompt forbids placeholders, and a tool that returns nothing says so, so the reply becomes "I could not find specific information about that."

Tools are skipped for devices with HomeKit buttons configured, because that path constrains the reply to a JSON schema and a model pinned to a schema cannot emit tool calls. Set TOOLS_ENABLED=false to turn the whole thing off.

Configuration

Everything is an environment variable; see .env.example, which run.sh sources. The ones that matter:

variable	default	note
LM_STUDIO_URL	http://127.0.0.1:1234	see the note below
LM_STUDIO_MODEL	(first listed)	pin it — google/gemma-4-e4b
WHISPER_MODEL	mlx-community/whisper-large-v3-turbo	whisper-tiny is 20× smaller and much less accurate
WHISPER_LANGUAGE	auto	pin to en/de to skip detection
VOICE_INPUT_RATE	16000	what the firmware sends, headerless
VOICE_REPLY_RATE	16000	what we send back — match your I2S rate
TTS_BACKEND	say	or kokoro (better voice, ~330 MB, slower)
TTS_VOICE	system	say -v '?' lists 184 installed voices
VOICE_SAVE_DIR	off	set a path to dump every turn as WAV + transcript

About the LM Studio address

A historical setup request specified http://192.168.1.174:1234, but that host did not exist on the original network — it does not answer ARP, let alone HTTP. LM Studio is in fact running on this Mac, which is 192.168.1.246, and it is already bound to all interfaces, so the watch can reach it directly too.

These addresses describe the original installation, not your machine. The default is http://127.0.0.1:1234. To point it back at the original address once that machine is up:

LM_STUDIO_URL=http://192.168.1.174:1234 ./run.sh

Voices

184 voices are installed and all of them work. tools/voices.py lists and auditions them:

tools/voices.py --lang de            # just German
tools/voices.py --lang en --demo     # hear every English voice
tools/voices.py --pick Samantha Daniel Anna --demo

The ones worth using for an assistant:

voice	locale
Samantha	en_US — the macOS default, clearest of the set
Daniel	en_GB
Karen	en_AU
Moira	en_IE
Tessa	en_ZA
Anna	de_DE

Most other languages have exactly one voice (Thomas fr_FR, Alice it_IT, Mónica es_ES, Milena ru_RU, Kyoko ja_JP, and so on) — tools/voices.py --lang <code> will show it.

Alongside those, eight expressive voices (Eddy, Flo, Grandma, Grandpa, Reed, Rocko, Sandy, Shelley) exist in 14 languages each (Reed in 13). Their names carry the language, so they must be quoted exactly:

TTS_VOICE="Eddy (German (Germany))"

Nineteen more (Bells, Boing, Bubbles, Jester, Organ, Trinoids, Zarvox, Whisper, Albert, Fred, …) are sound effects rather than speech. tools/voices.py hides them unless you pass --all; they are a reasonable choice for an alert chime, not for an answer.

Better quality is a free download. Only the compact voices are installed here. System Settings → Accessibility → Spoken Content → System Voice → Manage Voices offers Enhanced and Premium versions of the main voices, which are a clear step up and appear to say automatically once installed. If you want better than that, TTS_BACKEND=kokoro is the neural option.

A voice per language

Kokoro has no German. Its 54 voices cover American and British English, Spanish, French, Hindi, Italian, Japanese, Portuguese and Mandarin — there is no German model at all. Since Whisper detects the question's language and the model answers in kind, one fixed voice means a German answer read with an American accent.

TTS_LANG_VOICES routes per language instead. The language is whatever Whisper detected, so the answer is spoken by a voice that speaks it:

TTS_BACKEND=kokoro
KOKORO_VOICE=am_onyx
TTS_LANG_VOICES="de=say:Rocko (German (Germany))"

German goes to macOS say, everything else stays on Kokoro. Measured median pitch over the same sentence, to keep both registers deep and male:

	voice	median F0
German	Rocko (German (Germany))	72 Hz — deepest of the nine German voices
	Grandpa (German (Germany))	85 Hz
	Reed (German (Germany))	102 Hz
	Anna	165 Hz (female)
English	am_onyx (Kokoro)	89 Hz — deepest Kokoro male
	bm_lewis (Kokoro)	93 Hz
	am_michael (Kokoro)	119 Hz

Format is lang=backend:voice, separated by ;. Add more the same way, e.g. "de=say:Rocko (German (Germany));fr=say:Thomas".

Note that .env is sourced by the shell, so quote any value containing spaces or brackets — run.sh checks the file and says so if you forget, rather than failing with a bare syntax error.

Kokoro, the best voice here

TTS_BACKEND=kokoro runs Kokoro-82M on the GPU. It is both the best-sounding option and the fastest — 0.09 s for a two-second reply once warm, against roughly 0.5 s for say:

TTS_BACKEND=kokoro ./run.sh

Kokoro is an optional second setup. The configured KOKORO_PYTHON below is a personal path and must point to your own prepared interpreter; the default server installation does not create it. Start with say before setting it up.

Kokoro needs PyTorch and spaCy, which lived in their own arm64 venv rather than this server's, so it cannot be an in-process import. The server spawns voiceserver/kokoro_worker.py under that interpreter and keeps it alive, so the model load and the MPS kernel compilation — about 7 s in total — happen once at startup rather than on someone's first question. The worker is a child process and is terminated with the server; if it ever dies mid-turn it is restarted and the turn retried.

setting	default
KOKORO_PYTHON	/Users/tompeakwalcher/Documents/project/.venv-kokoro/bin/python
KOKORO_VOICE	af_heart
KOKORO_LANG	a — must match the voice prefix
KOKORO_DEVICE	mps, falling back to cpu by itself

Voice prefixes pick the language: af_/am_ American English (lang a), bf_/bm_ British (lang b), and there are Spanish, French, Japanese and Mandarin sets. Only af_heart is cached locally — any other voice downloads from HuggingFace the first time it is used.

Measured here, same sentence, warm:

backend	per reply	notes
kokoro	0.09 s	best quality, GPU, 7 s startup
say	0.50 s	no setup at all
openvox	n/a	needs a model installed first

OpenVox, for a neural voice

TTS_BACKEND=openvox routes synthesis to the local OpenVox app, which serves an OpenAI-compatible speech API on http://127.0.0.1:8000/v1. The model and voice are discovered automatically — the first installed model and its first voice — so normally the only setting needed is the backend itself:

TTS_BACKEND=openvox ./run.sh

Pin them with OPENVOX_MODEL / OPENVOX_VOICE if you want a specific voice; GET /v1/models and GET /v1/models/<model>/voices list what is available, and /healthz reports which pair a turn would actually use.

OpenVox needs a TTS model installed before this works. As of now its /v1/models is empty, so a speech call fails with Unsupported model. Add one in the app under Served Models; /healthz says so explicitly until you do, and nothing else in the pipeline is affected.

Whatever sample rate and container OpenVox returns is resampled to VOICE_REPLY_RATE mono PCM, so the watch is unaffected by the choice of voice.

Pick a non-reasoning model

qwen/qwen3.8-27b thinks before it answers, and LM Studio returns that thinking in a separate reasoning_content field. On a short token budget it can spend the entire budget reasoning and return an empty answer. The server detects exactly that case and retries with four times the budget, but the turn then costs 64 seconds — unusable when someone is holding a watch to their mouth. google/gemma-4-e4b answers the same question in 0.8 s.

LM_DISABLE_THINKING=true (the default) asks the chat template to skip thinking, but this qwen build does not reliably honour it. Pinning a non-reasoning model is the fix.

Notes for the firmware side

• Reply format. Ask for exactly what your I2S is configured for via reply_rate, and use reply=pcm if you would rather not skip a header. Mono 16-bit is all the server emits.
• Rates. 16 kHz mono is 32 kB/s, which is comfortable over WiFi and is Whisper's native rate — sending 44.1 kHz only costs bandwidth and a resample.
• Don't send silence. A turn under 0.25 s is rejected, and pure silence returns no speech detected rather than a hallucinated transcript.
• Errors. A failed turn returns JSON with error and error_stage (input, stt, llm, or tts), plus X-Error / X-Error-Stage headers. Status is 503 when the model host is unreachable and 422 when nothing was heard, so the firmware can tell "network problem" from "say that again".
• Conversation state. Pass a stable device id to get follow-up context; the last 6 turns are replayed to the model.

Layout

voiceserver/
├── app.py          HTTP + WebSocket routes
├── tcpserver.py    raw PCM socket listener
├── pipeline.py     one turn: audio -> transcript -> reply -> audio
├── stt.py          mlx-whisper, serialised behind a lock
├── llm.py          LM Studio client, history, markdown/think stripping
├── tts.py          say | kokoro | none
├── vad.py          energy VAD with hysteresis, for the streaming paths
├── audio.py        decode, downmix, resample, WAV framing
├── config.py       every setting, env-overridable
└── static/         browser test page
tools/
├── fake_firmware.py   stands in for the watch
└── stub_lmstudio.py   stands in for LM Studio

HomeWatch: pressing HomeKit buttons by voice

develop/homewatch is a watch that exposes 8 HomeKit buttons. For a device listed in homewatch.toml (copy homewatch.toml.example), the model gets the list of buttons and what each press is bound to in the Home app. Its answer is then grammar-constrained (LM Studio response_format JSON schema) to one of those actions or "none", so it cannot invent a button. Measured with gemma-4-e4b: plain prompting ignored the action convention and claimed to open a garage that does not exist; with the schema, all test phrases mapped correctly and "open the garage" came back as none.

• POST /v1/voice?device=homewatch&reply=json → JSON gains "action": {"tile": 1, "press": "single"} (or null)
• POST /v1/chat {"text": "...", "device": "homewatch"} → same action, handy for testing without audio
• GET /v1/home/tiles?device=homewatch → tile names for the watch's labels

The file is re-read when it changes; no restart needed (code changes do need one). Logic: voiceserver/home.py.