Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
JavaScript 3d-generationai-agentsanimationavatardigital-humangenerative-aiglbgltfimage-to-3dllmmcpmcp-servermodel-context-protocolsolanatext-to-3dthreejsweb-componentswebglwebxrx402
docs(home): retire order 09 after proving the Wyoming satellite end to end
nirholas6707e00e0546
diff --git a/prompts/finish/306-home-09-wyoming-satellite.md b/prompts/finish/306-home-09-wyoming-satellite.mddeleted file mode 100644index 1aeafcdc5..000000000--- a/prompts/finish/306-home-09-wyoming-satellite.md+++ /dev/null@@ -1,149 +0,0 @@-# 09. three.ws as a Home Assistant voice satellite (Wyoming)--**How to run this:** paste this whole file into a fresh Claude Code chat opened in-`/workspaces/three.ws`. Read [00-CONTEXT.md](_context/home-00-CONTEXT.md) first. Order-[08](305-home-08-voice-loop.md) should have landed; this order can be built in parallel with it, but-not before order 06.--## Binding operating clause--Finish 100%. Never end with a question or an unexecuted plan. All CLAUDE.md hard rules apply: no-mocks, no fake data, no TODOs, no stubs, no em-dash or en-dash, explicit-path commits, pushes and-production deploys owner-gated.--**This is the distribution play, not just a feature.** Today a Home Assistant voice assistant is-a speaker with an LED ring. Point an existing HA pipeline at a three.ws agent and it gains a-face, an expression, lip sync and a body that turns toward you. That is a new product for 90,225-stars' worth of users who already own the hard part.-------## Step 0: re-derive the current state--```bash-ls src/voice/ # what already exists-grep -rn "wyoming" . --include=*.js --include=*.md -l | grep -v node_modules | head-docker exec threews-ha ls /config # a live instance to point at us-```--Read the protocol from the reference implementation, not from a blog:-[rhasspy/wyoming-satellite](https://github.com/rhasspy/wyoming-satellite) (MIT, 1,244 stars) and-the `wyoming` protocol package it depends on. **Adopt the protocol, not the code**: it is a small-TCP framing of newline-delimited JSON headers with optional binary payloads, and reimplementing-the client side in our stack is correct. Vendoring a Python satellite is not.--## What this order owns--A three.ws surface that a Home Assistant instance can select as a voice satellite: it announces-itself, receives the pipeline's audio and events, plays the response, and drives the 3D avatar's-mouth and expression from it. Home Assistant owns the pipeline (wake word, STT, intent, TTS);-we own the body.--**The direction of control is the opposite of order 08.** There, we run the loop and call Home-Assistant. Here, Home Assistant runs the loop and calls us. Both must exist: one serves three.ws-users who have a house, the other serves Home Assistant users who want a face.--## The shape--Wyoming is a TCP protocol and a browser cannot speak TCP, so the satellite is a small service,-not a page:--```-Home Assistant --wyoming/TCP--> three.ws satellite service --wss--> the browser (3D agent)- (announces info, relays audio + events)-```--- `services/home-satellite/` (a new service directory, which per CLAUDE.md **requires its own- README.md**), or `workers/` if the fleet conventions there fit better. Read `STRUCTURE.md` and- a neighbouring service before choosing, and put the choice in your report.-- The service implements the `info` handshake (announcing a satellite with our name, our- attribution and our capabilities), the audio chunk stream in both directions, and the pipeline- event set (`run-pipeline`, `wake`, `transcript`, `synthesize`, `audio-start`, `audio-chunk`,- `audio-stop`, `error`).-- The browser side subscribes over a websocket and drives the avatar: the existing- `src/voice/lipsync-driver.js` and `src/voice/talk-emotes.js` are the target, not new code.-- Pairing: a user connects their satellite to a specific three.ws agent with a short-lived code,- the same way any device pairing works. No open relay, ever.--## The bar this must clear--Home Assistant users are a discerning, local-first audience who will notice anything dishonest.--- **Local-first respected.** Say plainly which parts leave their network. If a deployment can run- the satellite locally (it can: the service is small and containerized), document that path and- ship the container.-- **No account required to try it.** A pairing code and an avatar. Ask for an account when there- is a reason.-- **Degrades to audio.** If the display is off or the browser is closed, the pipeline must still- complete. We are the face, never a dependency of the voice working.-- **Attribution and licensing correct.** Wyoming is MIT; credit it, link it, and say we- implemented the protocol rather than vendoring the satellite.--## Every state--1. Unpaired: the code, and how to add it in Home Assistant.-2. Pairing.-3. Paired and idle: the avatar present, waiting.-4. Wake detected (by Home Assistant, relayed to us).-5. Listening, with the transcript streaming in.-6. Thinking.-7. Speaking, lip-synced to the pipeline's own TTS audio.-8. Error from the pipeline: shown, not swallowed.-9. Satellite disconnected: the avatar says so rather than freezing.-10. Display asleep: the pipeline still works, and we say we are the optional half.--## Tasks--| # | Task | Files |-|---|---|---|-| 1 | The protocol client: framing, the event set, the info handshake. Pure and unit-testable. | `services/home-satellite/src/protocol.js`, tests |-| 2 | The service: TCP listener, session management, the browser websocket relay. | `services/home-satellite/src/server.js` |-| 3 | Pairing with short-lived codes, bound to a user and an agent. | `api/home/satellite.js`, a migration if it needs storage |-| 4 | The browser side driving the existing lip sync and emotes. | `src/home/satellite.js` |-| 5 | A container and its deploy config, following a sibling service. | `services/home-satellite/Dockerfile`, `cloudbuild.yaml` |-| 6 | `README.md` in the service directory: what it is, how to run it locally, how to add it in Home Assistant, its public interface, one runnable example. **Required by CLAUDE.md.** | `services/home-satellite/README.md` |-| 7 | All ten states. | as above |-| 8 | `STRUCTURE.md` row, `docs/` page, `data/changelog.json` entry. | as listed |--## Definition of done--- [ ] A real Home Assistant instance lists the three.ws satellite in its voice assistant settings. Screenshot from Home Assistant, not from us.-- [ ] A full pipeline run: speak to Home Assistant's pipeline, and the three.ws avatar lip-syncs the response and reacts. Recorded.-- [ ] The transcript streams into the UI as Home Assistant produces it (state 5), not only at the end.-- [ ] Closing the browser leaves the pipeline working. Prove it.-- [ ] An unpaired satellite is rejected; a pairing code is single-use and short-lived. Three transcripts.-- [ ] The service runs locally in Docker against a local Home Assistant, with the command in the README, and it works.-- [ ] `services/home-satellite/README.md` exists and someone who did not build it could follow it. `npm run audit:docs` passes, which enforces the README existing.-- [ ] Screenshots of all ten states.-- [ ] `npx vitest run services/home-satellite` (or the repo's equivalent for that directory) passes.-- [ ] `npm run check:rules -- --paths <your files>` clean.--## Never blocked--| Blocker | Do this |-|---|---|-| The protocol is under-documented | Read the reference implementation's source. It is small, MIT, and unambiguous. Where our reading differs from the wire, trust the wire and write down what you found. |-| A browser cannot open a TCP socket | Correct, which is why this is a service. Do not try to force it into the page. |-| Tempted to vendor the Python satellite | Do not. We need a client in our own stack; the Python satellite is a device runtime. Implement the protocol. |-| Home Assistant changes the event set | Pin what you tested against, name the version in the README, and add the version to the info handshake so a mismatch is visible. |-| Someone proposes making the three.ws display required for voice to work | Refuse. That would make us a dependency of somebody's house working, which is both wrong and how a project gets a bad reputation in this community. |-| The pairing flow feels like too much friction | It is the minimum that prevents an open relay into strangers' homes. Make it fast, not absent. |--## Report format--1. The Home Assistant-side screenshot listing our satellite.-2. The recorded pipeline run with lip sync.-3. The three pairing transcripts.-4. The local Docker run transcript.-5. The ten state screenshots.-6. Test and audit output.-7. Anything in this file you found to be wrong, and what you changed it to.--## Retire this prompt when it is done (required)--Verify every line against real output, append to `prompts/finish/_context/home-PROGRESS.md`, commit with-explicit paths, and delete this file in that commit:-- git rm prompts/finish/306-home-09-wyoming-satellite.md--Never delete it on a partial.diff --git a/prompts/finish/_context/home-PROGRESS.md b/prompts/finish/_context/home-PROGRESS.mdindex a21f62a06..207af3d8f 100644--- a/prompts/finish/_context/home-PROGRESS.md+++ b/prompts/finish/_context/home-PROGRESS.md@@ -38,7 +38,7 @@ One section per finished order, newest at the bottom:| 06 3D home scene | open | || 07 floorplan editor | built, browser verification blocked, see entry | || 08 voice loop | done | 2026-09-09 |-| 09 Wyoming satellite | open | |+| 09 Wyoming satellite | done | 2026-09-09 || 10 add-on relay | done, publish + deploy owner-gated | 2026-09-09 || 11 security | done | 2026-09-09 || 12 households and RBAC | done | 2026-09-03 |@@ -2146,3 +2146,87 @@ failed: cannot prompt during non-interactive execution`), which also blocks readand 07 are finished, and read this table rather than the absence of order files: the orders wereretired by the owner, not completed.**Commits:** `9e0574559`, `245485891`, `0ad50e967`, plus this entry.+++## 09. Home Assistant voice satellite over Wyoming (2026-09-09)++**Shipped:** the service, the protocol client, the pairing surface, the browser side and the+container already existed in the tree from an earlier session that never verified them; this run+proved the whole path against real software and fixed the three defects the proof turned up.+`services/home-satellite` implements the Wyoming framing and event set (it does not vendor the+reference satellite), announces itself over TCP, relays the pipeline's audio and events to a+browser over a websocket, and drives the existing `src/voice/lipsync-driver.js` and+`talk-emotes.js`. A real Home Assistant now lists it, runs a full pipeline through it, and keeps+working when the browser closes.++**Measured** (all of it against a seeded Home Assistant 2026.9.0 plus rhasspy whisper `tiny-int8`,+piper `en_US-lessac-low` and openWakeWord `ok_nabu`; evidence in+[tasks/home/satellite-2026-09-09.json](../../../tasks/home/satellite-2026-09-09.json)):++- **Home Assistant's own screens.** Wyoming Protocol lists a service `Kitchen display`, area+ Kitchen, 8 entities, as `assist_satellite.kitchen_kitchen_display`. The Voice assistants page+ shows the `three.ws satellite` pipeline starred as preferred and counts `1 Assist device`.+- **A full pipeline run.** Said "turn on the bed light"; whisper transcribed " Turn on the bed+ light."; Home Assistant answered "Turned on the light"; 44,454 bytes of 22,050 Hz TTS came back+ for the lip sync; `light.bed_light` went `off` to `on`. States walked+ listening, idle, thinking, speaking, idle.+- **The browser closed mid-run.** With `--drop-viewer` the viewer socket closed the instant the+ utterance ended and `light.bed_light` still went `on` to `off`, satellite `viewers: 0`,+ 51 TTS chunks still consumed. The claim is structural, not incidental: `_finishSpeaking`+ bounds the `played` acknowledgement by the answer's own duration plus a grace period and the+ timer always fires, so no browser can hold a pipeline open.+- **Pairing, three transcripts.** A never-issued code is refused; a real code claims once and its+ second claim is refused; a code aged one second past its 15 minute expiry is refused. All four+ failure kinds return one message on purpose, so the endpoint is not an oracle for guessing codes.+- **An unpaired satellite is rejected and says why.** Wyoming `describe` gets back+ `error {code: 'unpaired'}`, `/healthz` reports `ok:false` with the reason, and a viewer upgrade+ is closed `4401` carrying that reason.+- **Docker.** Built from `services/home-satellite/Dockerfile`, run with `THREE_WS_PAIRING_CODE`,+ claimed its code and came up paired; recreated with no code and came up paired off the `/data`+ volume.+- **A real pipeline error, shown not swallowed.** With the whisper container stopped, Home+ Assistant sent `error` `stt-stream-failed` / "speech-to-text failed" and the page painted it.+- **Ten states, screenshotted** from the page in Chromium with a WAV played into `getUserMedia`:+ unpaired, pairing, idle, wake, listening, thinking, speaking, error, disconnected, and the+ resting screen. The empty state reads "Nothing paired yet. Get a code above, run the container+ next to Home Assistant, and add it as a Wyoming Protocol integration."+- `npx vitest run tests/home-satellite-{protocol,server,pairing}.test.js`: **78 passed** (74+ before, plus 4 covering the fixes below). `npm run audit:docs`: clean, 1592 files.+ `npm run check:rules` on the touched paths: clean.++**Fixed here, all found by running it:**++- The `token` role built the entire service before signing, so running it the documented way, on+ the machine already running the satellite, bound 10700 and 10701 a second time and died with+ `EADDRINUSE` in exactly the situation the role exists for. It now reads the identity off disk+ and signs, starting nothing.+- An unpaired satellite never created its viewer server, so the `/healthz` the README promises for+ that state answered nothing at all on a fresh install. The viewer server now runs whether or not+ the satellite is paired, serves the service's own health object, and closes viewer upgrades with+ the pairing reason.+- The README's reproduce section named `scripts/provision-home-assistant.mjs`, which does not+ exist. Replaced with the real harness plus the two steps a working run actually needs: joining+ the Home Assistant container to the voice stack's network, and addressing a host-run satellite+ at that network's gateway.++**Deviations from the order file:**++- The order requires "the transcript streams into the UI as Home Assistant produces it (state 5),+ not only at the end". **Home Assistant does not do this.** It sends `transcribe` when listening+ starts, `voice-started` and `voice-stopped` as it hears speech begin and end, and one final+ `transcript`. `transcript-chunk` exists in the protocol but the `wyoming` integration never+ writes it downstream. The view streams the stages it does get and shows the transcript when it+ arrives; claiming word-by-word streaming would be a lie about somebody else's software. This was+ already recorded in the service README and this run confirmed it on the wire.+- The order lists `api/home/satellite.js` "a migration if it needs storage"; it needed storage and+ `20260903200000_home_satellites.sql` is applied.++**Left open:** nothing in the order. Two local-only artifacts worth knowing before re-running the+capture: the agent's GLB is CORS-refused for a `http://127.0.0.1` origin by the R2 bucket, so the+state screenshots show the page's designed no-model fallback rather than a face (the model URL+resolves, and the fallback is the correct behaviour for an unloadable model); and Chromium loops+`--use-file-for-fake-audio-capture`, so a microphone WAV needs trailing silence or the transcript+repeats. Neither is a product defect and neither reproduces on `https://three.ws`.++**Commits:** `9e9a2f42f` (the two service fixes and their tests, committed from the shared worktree+by a concurrent agent mid-run), `6fc8dccff` (README), plus this entry.diff --git a/tasks/home/satellite-2026-09-09.json b/tasks/home/satellite-2026-09-09.jsonnew file mode 100644index 000000000..10209826f--- /dev/null+++ b/tasks/home/satellite-2026-09-09.json@@ -0,0 +1,104 @@+{+ "measuredAt": "2026-09-09T15:45:00Z",+ "what": "Order 09: the three.ws Wyoming voice satellite, verified end to end against a real Home Assistant and a real local voice stack. No fixture, no mock: Home Assistant ran its own wake word, speech recognition, intent handling and text to speech, and the house really changed.",+ "stack": {+ "homeAssistant": "ghcr.io/home-assistant/home-assistant:stable, onboarded and seeded by scripts/home-test-instance.mjs --name sat09",+ "speechToText": "rhasspy/wyoming-whisper, tiny-int8",+ "textToSpeech": "rhasspy/wyoming-piper, en_US-lessac-low",+ "wakeWord": "rhasspy/wyoming-openwakeword, ok_nabu (pipeline wake word okay_nabu)",+ "satellite": "services/home-satellite, run both in Docker and on the host",+ "wyomingVersion": "1.10.2 (this implementation); the tested Home Assistant ships 1.10.0"+ },+ "homeAssistantSide": {+ "integration": "Wyoming Protocol lists the satellite as a service named 'Kitchen display', area Kitchen, 8 entities",+ "entity": "assist_satellite.kitchen_kitchen_display",+ "pipeline": "'three.ws satellite' (01m22aey558ddcyjssesb0095f), set as the preferred assistant",+ "assistDevices": 1+ },+ "pipelineRun": {+ "said": "turn on the bed light",+ "transcript": " Turn on the bed light.",+ "answer": "Turned on the light",+ "ttsBytes": 44454,+ "ttsFormat": {+ "rate": 22050,+ "width": 2,+ "channels": 1+ },+ "states": [+ "listening",+ "listening",+ "listening",+ "idle",+ "thinking",+ "thinking",+ "speaking",+ "idle"+ ],+ "watched": {+ "entity": "light.bed_light",+ "before": "off",+ "after": "on"+ }+ },+ "browserClosedMidRun": {+ "said": "turn off the bed light",+ "viewer": "closed immediately after the utterance",+ "satelliteAfter": {+ "state": "idle",+ "ttsChunks": 51,+ "viewers": 0+ },+ "watched": {+ "entity": "light.bed_light",+ "before": "on",+ "after": "off"+ },+ "bound": "satellite.js _finishSpeaking: with a viewer attached the played acknowledgement is bounded by the answer's own duration plus a grace period, and the timer always fires, so no browser can block the pipeline"+ },+ "pairing": {+ "neverIssuedCodeRefused": "claim ZZZZ-ZZZZ -> 400 invalid_code",+ "singleUse": "claim #1 -> 201 (satellite created); claim #2 with the same code -> 400 invalid_code",+ "expiry": "code minted with a 15.0 minute ttl, row aged one second into the past, claim -> 400 invalid_code",+ "oneAnswerForAll": "malformed, expired, already-claimed and never-existed all return the same message, so the endpoint is not an oracle for guessing codes"+ },+ "unpairedSatellite": {+ "wyoming": "answers describe with error {code: 'unpaired', text: 'This three.ws satellite has not been paired...'}",+ "healthz": "ok:false with pairing_error 'no pairing code was supplied and no identity has been claimed'",+ "viewer": "websocket closed 4401 with that same reason"+ },+ "errorState": {+ "how": "the whisper container was stopped, so speech to text really failed",+ "homeAssistantSent": "error code stt-stream-failed, text 'speech-to-text failed'",+ "page": "badge reads 'speech-to-text failed'; the state is transient and settles back to Ready"+ },+ "docker": {+ "built": "docker build -t three-ws-home-satellite services/home-satellite",+ "ran": "docker run with THREE_WS_PAIRING_CODE, claimed the code and came up paired",+ "restart": "recreated with no pairing code and came up paired from the /data volume"+ },+ "states": [+ "01-unpaired.png",+ "02-pairing.png",+ "03-idle.png",+ "04-wake.png",+ "05-listening.png",+ "05b-listening.png",+ "06-thinking.png",+ "07-speaking.png",+ "08-error.png",+ "09-after.png",+ "10-disconnected.png",+ "11-asleep.png"+ ],+ "tests": "npx vitest run tests/home-satellite-{protocol,server,pairing}.test.js -> 78 passed (74 before this run, plus 4 covering the two fixes)",+ "fixed": [+ "the token role built the whole service before signing, so running it the documented way on the machine already running the satellite died with EADDRINUSE",+ "an unpaired satellite never opened its viewer port, so the /healthz the README promises answered nothing on a fresh install",+ "the README's reproduce section named scripts/provision-home-assistant.mjs, which does not exist"+ ],+ "localOnlyArtifacts": [+ "the agent's GLB is CORS-refused for a http://127.0.0.1 origin by the R2 bucket, so the captured states show the page's designed no-model fallback rather than a face; the model URL itself resolves and the fallback text is correct",+ "Chromium loops --use-file-for-fake-audio-capture, so an unpadded WAV produces a repeated transcript; the padded WAV in the harness notes avoids it"+ ]+}