Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

docs(home): retire order 09 after proving the Wyoming satellite end to end

nirholas6707e00e0546
diff --git a/prompts/finish/306-home-09-wyoming-satellite.md b/prompts/finish/306-home-09-wyoming-satellite.md
deleted file mode 100644
index 1aeafcdc5..000000000
--- a/prompts/finish/306-home-09-wyoming-satellite.md
+++ /dev/null
@@ -1,149 +0,0 @@
-# 09. three.ws as a Home Assistant voice satellite (Wyoming)
-
-**How to run this:** paste this whole file into a fresh Claude Code chat opened in
-`/workspaces/three.ws`. Read [00-CONTEXT.md](_context/home-00-CONTEXT.md) first. Order
-[08](305-home-08-voice-loop.md) should have landed; this order can be built in parallel with it, but
-not before order 06.
-
-## Binding operating clause
-
-Finish 100%. Never end with a question or an unexecuted plan. All CLAUDE.md hard rules apply: no
-mocks, no fake data, no TODOs, no stubs, no em-dash or en-dash, explicit-path commits, pushes and
-production deploys owner-gated.
-
-**This is the distribution play, not just a feature.** Today a Home Assistant voice assistant is
-a speaker with an LED ring. Point an existing HA pipeline at a three.ws agent and it gains a
-face, an expression, lip sync and a body that turns toward you. That is a new product for 90,225
-stars' worth of users who already own the hard part.
-
----
-
-## Step 0: re-derive the current state
-
-```bash
-ls src/voice/ # what already exists
-grep -rn "wyoming" . --include=*.js --include=*.md -l | grep -v node_modules | head
-docker exec threews-ha ls /config # a live instance to point at us
-```
-
-Read the protocol from the reference implementation, not from a blog:
-[rhasspy/wyoming-satellite](https://github.com/rhasspy/wyoming-satellite) (MIT, 1,244 stars) and
-the `wyoming` protocol package it depends on. **Adopt the protocol, not the code**: it is a small
-TCP framing of newline-delimited JSON headers with optional binary payloads, and reimplementing
-the client side in our stack is correct. Vendoring a Python satellite is not.
-
-## What this order owns
-
-A three.ws surface that a Home Assistant instance can select as a voice satellite: it announces
-itself, receives the pipeline's audio and events, plays the response, and drives the 3D avatar's
-mouth and expression from it. Home Assistant owns the pipeline (wake word, STT, intent, TTS);
-we own the body.
-
-**The direction of control is the opposite of order 08.** There, we run the loop and call Home
-Assistant. Here, Home Assistant runs the loop and calls us. Both must exist: one serves three.ws
-users who have a house, the other serves Home Assistant users who want a face.
-
-## The shape
-
-Wyoming is a TCP protocol and a browser cannot speak TCP, so the satellite is a small service,
-not a page:
-
-```
-Home Assistant --wyoming/TCP--> three.ws satellite service --wss--> the browser (3D agent)
- (announces info, relays audio + events)
-```
-
-- `services/home-satellite/` (a new service directory, which per CLAUDE.md **requires its own
- README.md**), or `workers/` if the fleet conventions there fit better. Read `STRUCTURE.md` and
- a neighbouring service before choosing, and put the choice in your report.
-- The service implements the `info` handshake (announcing a satellite with our name, our
- attribution and our capabilities), the audio chunk stream in both directions, and the pipeline
- event set (`run-pipeline`, `wake`, `transcript`, `synthesize`, `audio-start`, `audio-chunk`,
- `audio-stop`, `error`).
-- The browser side subscribes over a websocket and drives the avatar: the existing
- `src/voice/lipsync-driver.js` and `src/voice/talk-emotes.js` are the target, not new code.
-- Pairing: a user connects their satellite to a specific three.ws agent with a short-lived code,
- the same way any device pairing works. No open relay, ever.
-
-## The bar this must clear
-
-Home Assistant users are a discerning, local-first audience who will notice anything dishonest.
-
-- **Local-first respected.** Say plainly which parts leave their network. If a deployment can run
- the satellite locally (it can: the service is small and containerized), document that path and
- ship the container.
-- **No account required to try it.** A pairing code and an avatar. Ask for an account when there
- is a reason.
-- **Degrades to audio.** If the display is off or the browser is closed, the pipeline must still
- complete. We are the face, never a dependency of the voice working.
-- **Attribution and licensing correct.** Wyoming is MIT; credit it, link it, and say we
- implemented the protocol rather than vendoring the satellite.
-
-## Every state
-
-1. Unpaired: the code, and how to add it in Home Assistant.
-2. Pairing.
-3. Paired and idle: the avatar present, waiting.
-4. Wake detected (by Home Assistant, relayed to us).
-5. Listening, with the transcript streaming in.
-6. Thinking.
-7. Speaking, lip-synced to the pipeline's own TTS audio.
-8. Error from the pipeline: shown, not swallowed.
-9. Satellite disconnected: the avatar says so rather than freezing.
-10. Display asleep: the pipeline still works, and we say we are the optional half.
-
-## Tasks
-
-| # | Task | Files |
-|---|---|---|
-| 1 | The protocol client: framing, the event set, the info handshake. Pure and unit-testable. | `services/home-satellite/src/protocol.js`, tests |
-| 2 | The service: TCP listener, session management, the browser websocket relay. | `services/home-satellite/src/server.js` |
-| 3 | Pairing with short-lived codes, bound to a user and an agent. | `api/home/satellite.js`, a migration if it needs storage |
-| 4 | The browser side driving the existing lip sync and emotes. | `src/home/satellite.js` |
-| 5 | A container and its deploy config, following a sibling service. | `services/home-satellite/Dockerfile`, `cloudbuild.yaml` |
-| 6 | `README.md` in the service directory: what it is, how to run it locally, how to add it in Home Assistant, its public interface, one runnable example. **Required by CLAUDE.md.** | `services/home-satellite/README.md` |
-| 7 | All ten states. | as above |
-| 8 | `STRUCTURE.md` row, `docs/` page, `data/changelog.json` entry. | as listed |
-
-## Definition of done
-
-- [ ] A real Home Assistant instance lists the three.ws satellite in its voice assistant settings. Screenshot from Home Assistant, not from us.
-- [ ] A full pipeline run: speak to Home Assistant's pipeline, and the three.ws avatar lip-syncs the response and reacts. Recorded.
-- [ ] The transcript streams into the UI as Home Assistant produces it (state 5), not only at the end.
-- [ ] Closing the browser leaves the pipeline working. Prove it.
-- [ ] An unpaired satellite is rejected; a pairing code is single-use and short-lived. Three transcripts.
-- [ ] The service runs locally in Docker against a local Home Assistant, with the command in the README, and it works.
-- [ ] `services/home-satellite/README.md` exists and someone who did not build it could follow it. `npm run audit:docs` passes, which enforces the README existing.
-- [ ] Screenshots of all ten states.
-- [ ] `npx vitest run services/home-satellite` (or the repo's equivalent for that directory) passes.
-- [ ] `npm run check:rules -- --paths <your files>` clean.
-
-## Never blocked
-
-| Blocker | Do this |
-|---|---|
-| The protocol is under-documented | Read the reference implementation's source. It is small, MIT, and unambiguous. Where our reading differs from the wire, trust the wire and write down what you found. |
-| A browser cannot open a TCP socket | Correct, which is why this is a service. Do not try to force it into the page. |
-| Tempted to vendor the Python satellite | Do not. We need a client in our own stack; the Python satellite is a device runtime. Implement the protocol. |
-| Home Assistant changes the event set | Pin what you tested against, name the version in the README, and add the version to the info handshake so a mismatch is visible. |
-| Someone proposes making the three.ws display required for voice to work | Refuse. That would make us a dependency of somebody's house working, which is both wrong and how a project gets a bad reputation in this community. |
-| The pairing flow feels like too much friction | It is the minimum that prevents an open relay into strangers' homes. Make it fast, not absent. |
-
-## Report format
-
-1. The Home Assistant-side screenshot listing our satellite.
-2. The recorded pipeline run with lip sync.
-3. The three pairing transcripts.
-4. The local Docker run transcript.
-5. The ten state screenshots.
-6. Test and audit output.
-7. Anything in this file you found to be wrong, and what you changed it to.
-
-## Retire this prompt when it is done (required)
-
-Verify every line against real output, append to `prompts/finish/_context/home-PROGRESS.md`, commit with
-explicit paths, and delete this file in that commit:
-
- git rm prompts/finish/306-home-09-wyoming-satellite.md
-
-Never delete it on a partial.
diff --git a/prompts/finish/_context/home-PROGRESS.md b/prompts/finish/_context/home-PROGRESS.md
index a21f62a06..207af3d8f 100644
--- a/prompts/finish/_context/home-PROGRESS.md
+++ b/prompts/finish/_context/home-PROGRESS.md
@@ -38,7 +38,7 @@ One section per finished order, newest at the bottom:
| 06 3D home scene | open | |
| 07 floorplan editor | built, browser verification blocked, see entry | |
| 08 voice loop | done | 2026-09-09 |
-| 09 Wyoming satellite | open | |
+| 09 Wyoming satellite | done | 2026-09-09 |
| 10 add-on relay | done, publish + deploy owner-gated | 2026-09-09 |
| 11 security | done | 2026-09-09 |
| 12 households and RBAC | done | 2026-09-03 |
@@ -2146,3 +2146,87 @@ failed: cannot prompt during non-interactive execution`), which also blocks read
and 07 are finished, and read this table rather than the absence of order files: the orders were
retired by the owner, not completed.
**Commits:** `9e0574559`, `245485891`, `0ad50e967`, plus this entry.
+
+
+## 09. Home Assistant voice satellite over Wyoming (2026-09-09)
+
+**Shipped:** the service, the protocol client, the pairing surface, the browser side and the
+container already existed in the tree from an earlier session that never verified them; this run
+proved the whole path against real software and fixed the three defects the proof turned up.
+`services/home-satellite` implements the Wyoming framing and event set (it does not vendor the
+reference satellite), announces itself over TCP, relays the pipeline's audio and events to a
+browser over a websocket, and drives the existing `src/voice/lipsync-driver.js` and
+`talk-emotes.js`. A real Home Assistant now lists it, runs a full pipeline through it, and keeps
+working when the browser closes.
+
+**Measured** (all of it against a seeded Home Assistant 2026.9.0 plus rhasspy whisper `tiny-int8`,
+piper `en_US-lessac-low` and openWakeWord `ok_nabu`; evidence in
+[tasks/home/satellite-2026-09-09.json](../../../tasks/home/satellite-2026-09-09.json)):
+
+- **Home Assistant's own screens.** Wyoming Protocol lists a service `Kitchen display`, area
+ Kitchen, 8 entities, as `assist_satellite.kitchen_kitchen_display`. The Voice assistants page
+ shows the `three.ws satellite` pipeline starred as preferred and counts `1 Assist device`.
+- **A full pipeline run.** Said "turn on the bed light"; whisper transcribed " Turn on the bed
+ light."; Home Assistant answered "Turned on the light"; 44,454 bytes of 22,050 Hz TTS came back
+ for the lip sync; `light.bed_light` went `off` to `on`. States walked
+ listening, idle, thinking, speaking, idle.
+- **The browser closed mid-run.** With `--drop-viewer` the viewer socket closed the instant the
+ utterance ended and `light.bed_light` still went `on` to `off`, satellite `viewers: 0`,
+ 51 TTS chunks still consumed. The claim is structural, not incidental: `_finishSpeaking`
+ bounds the `played` acknowledgement by the answer's own duration plus a grace period and the
+ timer always fires, so no browser can hold a pipeline open.
+- **Pairing, three transcripts.** A never-issued code is refused; a real code claims once and its
+ second claim is refused; a code aged one second past its 15 minute expiry is refused. All four
+ failure kinds return one message on purpose, so the endpoint is not an oracle for guessing codes.
+- **An unpaired satellite is rejected and says why.** Wyoming `describe` gets back
+ `error {code: 'unpaired'}`, `/healthz` reports `ok:false` with the reason, and a viewer upgrade
+ is closed `4401` carrying that reason.
+- **Docker.** Built from `services/home-satellite/Dockerfile`, run with `THREE_WS_PAIRING_CODE`,
+ claimed its code and came up paired; recreated with no code and came up paired off the `/data`
+ volume.
+- **A real pipeline error, shown not swallowed.** With the whisper container stopped, Home
+ Assistant sent `error` `stt-stream-failed` / "speech-to-text failed" and the page painted it.
+- **Ten states, screenshotted** from the page in Chromium with a WAV played into `getUserMedia`:
+ unpaired, pairing, idle, wake, listening, thinking, speaking, error, disconnected, and the
+ resting screen. The empty state reads "Nothing paired yet. Get a code above, run the container
+ next to Home Assistant, and add it as a Wyoming Protocol integration."
+- `npx vitest run tests/home-satellite-{protocol,server,pairing}.test.js`: **78 passed** (74
+ before, plus 4 covering the fixes below). `npm run audit:docs`: clean, 1592 files.
+ `npm run check:rules` on the touched paths: clean.
+
+**Fixed here, all found by running it:**
+
+- The `token` role built the entire service before signing, so running it the documented way, on
+ the machine already running the satellite, bound 10700 and 10701 a second time and died with
+ `EADDRINUSE` in exactly the situation the role exists for. It now reads the identity off disk
+ and signs, starting nothing.
+- An unpaired satellite never created its viewer server, so the `/healthz` the README promises for
+ that state answered nothing at all on a fresh install. The viewer server now runs whether or not
+ the satellite is paired, serves the service's own health object, and closes viewer upgrades with
+ the pairing reason.
+- The README's reproduce section named `scripts/provision-home-assistant.mjs`, which does not
+ exist. Replaced with the real harness plus the two steps a working run actually needs: joining
+ the Home Assistant container to the voice stack's network, and addressing a host-run satellite
+ at that network's gateway.
+
+**Deviations from the order file:**
+
+- The order requires "the transcript streams into the UI as Home Assistant produces it (state 5),
+ not only at the end". **Home Assistant does not do this.** It sends `transcribe` when listening
+ starts, `voice-started` and `voice-stopped` as it hears speech begin and end, and one final
+ `transcript`. `transcript-chunk` exists in the protocol but the `wyoming` integration never
+ writes it downstream. The view streams the stages it does get and shows the transcript when it
+ arrives; claiming word-by-word streaming would be a lie about somebody else's software. This was
+ already recorded in the service README and this run confirmed it on the wire.
+- The order lists `api/home/satellite.js` "a migration if it needs storage"; it needed storage and
+ `20260903200000_home_satellites.sql` is applied.
+
+**Left open:** nothing in the order. Two local-only artifacts worth knowing before re-running the
+capture: the agent's GLB is CORS-refused for a `http://127.0.0.1` origin by the R2 bucket, so the
+state screenshots show the page's designed no-model fallback rather than a face (the model URL
+resolves, and the fallback is the correct behaviour for an unloadable model); and Chromium loops
+`--use-file-for-fake-audio-capture`, so a microphone WAV needs trailing silence or the transcript
+repeats. Neither is a product defect and neither reproduces on `https://three.ws`.
+
+**Commits:** `9e9a2f42f` (the two service fixes and their tests, committed from the shared worktree
+by a concurrent agent mid-run), `6fc8dccff` (README), plus this entry.
diff --git a/tasks/home/satellite-2026-09-09.json b/tasks/home/satellite-2026-09-09.json
new file mode 100644
index 000000000..10209826f
--- /dev/null
+++ b/tasks/home/satellite-2026-09-09.json
@@ -0,0 +1,104 @@
+{
+ "measuredAt": "2026-09-09T15:45:00Z",
+ "what": "Order 09: the three.ws Wyoming voice satellite, verified end to end against a real Home Assistant and a real local voice stack. No fixture, no mock: Home Assistant ran its own wake word, speech recognition, intent handling and text to speech, and the house really changed.",
+ "stack": {
+ "homeAssistant": "ghcr.io/home-assistant/home-assistant:stable, onboarded and seeded by scripts/home-test-instance.mjs --name sat09",
+ "speechToText": "rhasspy/wyoming-whisper, tiny-int8",
+ "textToSpeech": "rhasspy/wyoming-piper, en_US-lessac-low",
+ "wakeWord": "rhasspy/wyoming-openwakeword, ok_nabu (pipeline wake word okay_nabu)",
+ "satellite": "services/home-satellite, run both in Docker and on the host",
+ "wyomingVersion": "1.10.2 (this implementation); the tested Home Assistant ships 1.10.0"
+ },
+ "homeAssistantSide": {
+ "integration": "Wyoming Protocol lists the satellite as a service named 'Kitchen display', area Kitchen, 8 entities",
+ "entity": "assist_satellite.kitchen_kitchen_display",
+ "pipeline": "'three.ws satellite' (01m22aey558ddcyjssesb0095f), set as the preferred assistant",
+ "assistDevices": 1
+ },
+ "pipelineRun": {
+ "said": "turn on the bed light",
+ "transcript": " Turn on the bed light.",
+ "answer": "Turned on the light",
+ "ttsBytes": 44454,
+ "ttsFormat": {
+ "rate": 22050,
+ "width": 2,
+ "channels": 1
+ },
+ "states": [
+ "listening",
+ "listening",
+ "listening",
+ "idle",
+ "thinking",
+ "thinking",
+ "speaking",
+ "idle"
+ ],
+ "watched": {
+ "entity": "light.bed_light",
+ "before": "off",
+ "after": "on"
+ }
+ },
+ "browserClosedMidRun": {
+ "said": "turn off the bed light",
+ "viewer": "closed immediately after the utterance",
+ "satelliteAfter": {
+ "state": "idle",
+ "ttsChunks": 51,
+ "viewers": 0
+ },
+ "watched": {
+ "entity": "light.bed_light",
+ "before": "on",
+ "after": "off"
+ },
+ "bound": "satellite.js _finishSpeaking: with a viewer attached the played acknowledgement is bounded by the answer's own duration plus a grace period, and the timer always fires, so no browser can block the pipeline"
+ },
+ "pairing": {
+ "neverIssuedCodeRefused": "claim ZZZZ-ZZZZ -> 400 invalid_code",
+ "singleUse": "claim #1 -> 201 (satellite created); claim #2 with the same code -> 400 invalid_code",
+ "expiry": "code minted with a 15.0 minute ttl, row aged one second into the past, claim -> 400 invalid_code",
+ "oneAnswerForAll": "malformed, expired, already-claimed and never-existed all return the same message, so the endpoint is not an oracle for guessing codes"
+ },
+ "unpairedSatellite": {
+ "wyoming": "answers describe with error {code: 'unpaired', text: 'This three.ws satellite has not been paired...'}",
+ "healthz": "ok:false with pairing_error 'no pairing code was supplied and no identity has been claimed'",
+ "viewer": "websocket closed 4401 with that same reason"
+ },
+ "errorState": {
+ "how": "the whisper container was stopped, so speech to text really failed",
+ "homeAssistantSent": "error code stt-stream-failed, text 'speech-to-text failed'",
+ "page": "badge reads 'speech-to-text failed'; the state is transient and settles back to Ready"
+ },
+ "docker": {
+ "built": "docker build -t three-ws-home-satellite services/home-satellite",
+ "ran": "docker run with THREE_WS_PAIRING_CODE, claimed the code and came up paired",
+ "restart": "recreated with no pairing code and came up paired from the /data volume"
+ },
+ "states": [
+ "01-unpaired.png",
+ "02-pairing.png",
+ "03-idle.png",
+ "04-wake.png",
+ "05-listening.png",
+ "05b-listening.png",
+ "06-thinking.png",
+ "07-speaking.png",
+ "08-error.png",
+ "09-after.png",
+ "10-disconnected.png",
+ "11-asleep.png"
+ ],
+ "tests": "npx vitest run tests/home-satellite-{protocol,server,pairing}.test.js -> 78 passed (74 before this run, plus 4 covering the two fixes)",
+ "fixed": [
+ "the token role built the whole service before signing, so running it the documented way on the machine already running the satellite died with EADDRINUSE",
+ "an unpaired satellite never opened its viewer port, so the /healthz the README promises answered nothing on a fresh install",
+ "the README's reproduce section named scripts/provision-home-assistant.mjs, which does not exist"
+ ],
+ "localOnlyArtifacts": [
+ "the agent's GLB is CORS-refused for a http://127.0.0.1 origin by the R2 bucket, so the captured states show the page's designed no-model fallback rather than a face; the model URL itself resolves and the fallback text is correct",
+ "Chromium loops --use-file-for-fake-audio-capture, so an unpadded WAV produces a repeated transcript; the padded WAV in the harness notes avoids it"
+ ]
+}