Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
JavaScript 3d-generationai-agentsanimationavatardigital-humangenerative-aiglbgltfimage-to-3dllmmcpmcp-servermodel-context-protocolsolanatext-to-3dthreejsweb-componentswebglwebxrx402
test(home-voice): prove barge-in stops audible sound and drive a real light by voice
nirholasdf4d2091ca1c
diff --git a/STRUCTURE.md b/STRUCTURE.mdindex 9ca593fbe..26ce7fad7 100644--- a/STRUCTURE.md+++ b/STRUCTURE.md@@ -69,7 +69,7 @@ bottom.| Notification bell & preferences | Event vocabulary [api/_lib/feed.js](api/_lib/feed.js) (`USER_EVENT_TYPES` → `publishUserEvent()` → `user_notifications`), endpoints [api/notifications/](api/notifications), preference matrix [api/_lib/notify-prefs.js](api/_lib/notify-prefs.js), bell client [src/notifications.js](src/notifications.js), avatar announcements [src/notification-herald.js](src/notification-herald.js), settings panel [src/dashboard-next/pages/settings.js](src/dashboard-next/pages/settings.js) | Live | In-app inbox covering remixes ([api/x402/remix-asset.js](api/x402/remix-asset.js)), DMs ([api/friends/messages.js](api/friends/messages.js)), coin bonding-curve graduations (pump cron in [api/cron/[name].js](api/cron/%5Bname%5D.js)), follows, sales, purchases, IRL, market, and account events. Preferences are a category × channel matrix (`in_app` always on; push/email/telegram/avatar are per-category opt-outs) edited from `/dashboard/settings`. **The `avatar` channel delivers in person:** for a fresh, unread notification in an opted-in category the corner companion walks on screen, gestures, and says the line with a link straight to it, summoning itself for that one message and retreating if it was off ([walk-sdk](walk-sdk) `control.announce`). Rules (15-minute freshness, once per browser, two per batch, visible tab only, device-level mute) live in [src/notification-herald.js](src/notification-herald.js). Docs: [docs/notifications.md](docs/notifications.md), [docs/social-layer.md](docs/social-layer.md). Tests: [tests/notification-herald.test.js](tests/notification-herald.test.js), [tests/notify-prefs.test.js](tests/notify-prefs.test.js) || Personal companion (`/companion`) | Page [pages/companion.html](pages/companion.html) `→ /companion` + [src/companion/](src/companion) (page, source cards, voice), API [api/companion/](api/companion) (settings, sources, contacts, events, `ingest`, `stream`, `poll`), engine [api/_lib/companion/](api/_lib/companion) (store, encrypted BYOK configs, triage, lanes for Telegram / iCal / IMAP), cron [api/cron/companion-poll.js](api/cron/companion-poll.js), migration [api/_lib/migrations/20260828140000_companion_harness.sql](api/_lib/migrations/20260828140000_companion_harness.sql) | Live | Turns a person's own inbox, chats, calendar and phone into deliveries a 3D character makes **in person**. BYOK per source (a Telegram bot the user made, a private iCal URL, IMAP with an app password) plus a universal bridge token anything can POST to. Every message is scored 0-100 by the shared deterministic rules ([@three-ws/companion/triage](packages/companion-sdk/src/triage-rules.js), the same module the CLI runs locally) and optionally refined by the user's own Anthropic key; above their threshold and outside quiet hours it is delivered through `companion_delivery` notifications, so the bell, Web Push, and the avatar channel all carry it. A contact's message is spoken by **that contact's avatar and voice** (`control.announce({ avatar })` in [walk-sdk](walk-sdk)). A delivery whose lane can carry an answer (Telegram, through the user's own bot) offers a reply box in place, and the reply is stored with the message. Live SSE at `/api/companion/stream` feeds the desktop app and any third-party body. Docs: [docs/companion.md](docs/companion.md). Tests: [tests/companion-triage.test.js](tests/companion-triage.test.js), [tests/companion-calendar.test.js](tests/companion-calendar.test.js), [packages/companion-sdk/tests/](packages/companion-sdk/tests) || three.ws Drive: the agent in the car (`/drive`) | Page [pages/drive.html](pages/drive.html) `→ /drive` + [src/drive/](src/drive) (page controller, surface presets, native bridge, local commands, motion gate); native CarPlay scene [ios/native/App/App/CarPlaySceneDelegate.swift](ios/native/App/App/CarPlaySceneDelegate.swift) + [DriveLink.swift](ios/native/App/App/DriveLink.swift); Android Auto car app [solana-mobile/android-overlay/app/src/main/java/ws/three/app/drive/](solana-mobile/android-overlay/app/src/main/java/ws/three/app/drive) | Live (web) / built and compiled (Android Auto) / built, entitlement pending (CarPlay) | The existing talk loop (`/api/asr` → `/api/chat` → `/api/tts` → lipsync on a real rigged avatar) rebuilt for a car panel: one glanceable two-line reply instead of a transcript, exactly four controls, night by default, and a keyboard that hides itself while the wheels turn. Speed comes from `navigator.geolocation`'s real ground speed, so moving above walking pace locks text entry back and a sustained stop is what unlocks it. Control phrases ("repeat that", "stop talking", "louder", "night mode") are answered locally with no round trip, matched against the whole utterance so "stop at the next charger" still reaches the agent. Hands free ends a turn from the real captured mic level on the Riva lane. **The car screen is never ours to draw on, on either platform:** Apple's voice-based conversational category (iOS 26.4) grants templates and an audio session and no drawing surface, and Android grants a canvas only to navigation apps and has no conversational category at all, so both car screens render the same four rows and this page runs beside them, joined by the versioned `threeWsDrive` channel (a WebKit message handler on iOS, a JavaScript interface on Android). Android Auto declares `androidx.car.app.category.IOT` because that is accurate: the home tools in `api/chat.js` reach a real house, so "turn the porch light on" from the car is the product. Its web view has no window and therefore no animation frames, so `?surface=androidauto` runs the loop as audio with the stage off, and refuses tap-to-approve for guarded actions because a person who cannot see what they are approving must not be asked. A guarded action while moving is refused on every surface and offered for when the car stops. Docs: [docs/carplay.md](docs/carplay.md), [ios/docs/CARPLAY.md](ios/docs/CARPLAY.md), [solana-mobile/docs/ANDROID-AUTO.md](solana-mobile/docs/ANDROID-AUTO.md). Tests: [tests/drive-surface.test.js](tests/drive-surface.test.js) |-| Hands-free voice for the house (`/voice/home`) | Loop [src/voice/home-voice.js](src/voice/home-voice.js), VAD [src/voice/vad.js](src/voice/vad.js), wake word [src/voice/wake-word.js](src/voice/wake-word.js) + catalog [src/voice/wake-words.js](src/voice/wake-words.js), panel [src/voice/home-voice-ui.js](src/voice/home-voice-ui.js) + [public/home-voice.css](public/home-voice.css); page [pages/voice-home.html](pages/voice-home.html) `→ /voice/home` + [src/voice-home.js](src/voice-home.js); models committed under [public/models/voice/wake-word/](public/models/voice/wake-word) with the silero + onnxruntime runtime staged by [scripts/copy-voice-models.mjs](scripts/copy-voice-models.mjs) | Live | The browser half of three.ws Home: mic → silero VAD → openWakeWord → `/api/asr` → `/api/chat` (home tools) → `/api/tts/speak` → lipsync, with one `getUserMedia` shared by all three consumers. Wake-word matching runs entirely in the tab on committed openWakeWord v0.5.1 weights served from three.ws, so nothing is uploaded until after the wake word; nothing about detection is even downloaded until the user opts in. Barge-in cuts playback ~110 ms after the user starts talking, and a playback-aware guard means the agent saying its own wake word (score 0.99) wakes it zero times. A pause is not an ending: a closed segment stays open 450 ms so "Hey Jarvis… turn the kitchen light off" is one command. Guarded actions are spoken in full, shown on screen by name and entity id, and satisfied only by the token `confirm` redeemed at `POST /api/home/:id/confirm`; a general "yeah" is refused by name and a screenless surface is refused outright. Twelve designed states, mute that ends the track rather than hiding a dot, and every leg timed live. Docs: [docs/home-voice.md](docs/home-voice.md). Proof: [scripts/check-home-voice.mjs](scripts/check-home-voice.mjs) (34 assertions in a real Chromium). Tests: [tests/home-voice.test.js](tests/home-voice.test.js) |+| Hands-free voice for the house (`/voice/home`) | Loop [src/voice/home-voice.js](src/voice/home-voice.js), VAD [src/voice/vad.js](src/voice/vad.js), wake word [src/voice/wake-word.js](src/voice/wake-word.js) + catalog [src/voice/wake-words.js](src/voice/wake-words.js), panel [src/voice/home-voice-ui.js](src/voice/home-voice-ui.js) + [public/home-voice.css](public/home-voice.css); page [pages/voice-home.html](pages/voice-home.html) `→ /voice/home` + [src/voice-home.js](src/voice-home.js); models committed under [public/models/voice/wake-word/](public/models/voice/wake-word) with the silero + onnxruntime runtime staged by [scripts/copy-voice-models.mjs](scripts/copy-voice-models.mjs) | Live | The browser half of three.ws Home: mic → silero VAD → openWakeWord → `/api/asr` → `/api/chat` (home tools) → `/api/tts/speak` → lipsync, with one `getUserMedia` shared by all three consumers. Wake-word matching runs entirely in the tab on committed openWakeWord v0.5.1 weights served from three.ws, so nothing is uploaded until after the wake word; nothing about detection is even downloaded until the user opts in. Barge-in cuts playback ~110 ms after the user starts talking, and a playback-aware guard means the agent saying its own wake word (score 0.99) wakes it zero times. A pause is not an ending: a closed segment stays open 450 ms so "Hey Jarvis… turn the kitchen light off" is one command. Guarded actions are spoken in full, shown on screen by name and entity id, and satisfied only by the token `confirm` redeemed at `POST /api/home/:id/confirm`; a general "yeah" is refused by name and a screenless surface is refused outright. Twelve designed states, mute that ends the track rather than hiding a dot, and every leg timed live. Docs: [docs/home-voice.md](docs/home-voice.md). Proof: [scripts/check-home-voice.mjs](scripts/check-home-voice.mjs) (36 assertions in a real Chromium, plus `--live` against a real Home Assistant that proves the spoken command really turns the light off). Tests: [tests/home-voice.test.js](tests/home-voice.test.js) || Knock: a priced door to a person (`/knock`) | Hub [pages/knock.html](pages/knock.html) `→ /knock` + [src/knock/page.js](src/knock/page.js), public door [pages/knock-door.html](pages/knock-door.html) `→ /knock/<handle>` + [src/knock/door.js](src/knock/door.js), API [api/knock/](api/knock) (door, directory, send, reply, settings, inbox, escrowed, escrow-sync), paid lane [api/x402/knock.js](api/x402/knock.js), escrowed lane [api/_lib/knock/escrow.js](api/_lib/knock/escrow.js) + browser client [src/knock/escrow-program.js](src/knock/escrow-program.js) / [src/knock/escrow-checkout.js](src/knock/escrow-checkout.js), engine [api/_lib/knock/](api/_lib/knock) (pure policy, store, delivery, receipt tokens), migration [api/_lib/migrations/20260828210000_knock_doors.sql](api/_lib/migrations/20260828210000_knock_doors.sql), SDK + CLI [packages/knock-sdk/](packages/knock-sdk) (`@three-ws/knock`), MCP [packages/knock-mcp/](packages/knock-mcp) (`@three-ws/knock-mcp`) | Live | Every account can publish a door at `/knock/<username>` and set what one message from a stranger costs. Paying it buys exactly one message through, and the delivery is **in person**: the knock becomes a `companion_events` row whose importance is derived from the amount paid, so the recipient's corner avatar walks on wherever they are and says who is at the door and what they paid. **The USDC settles straight to the recipient** (`payTo` on the 402 challenge is their own address, and a priced door cannot be opened without one), so there is no platform wallet in the path and no cut. Solana leads; Base is advertised only when the owner sets an address for it. **Every refusal happens before the 402** (shut door, daily cap, block list, length limits) and again before settlement, so a knock that was never going to land is never a knock somebody paid for. Only the subject line is spoken; an untrusted body never uses the recipient's avatar as a speaker. The sender gets a derived HMAC receipt URL and reads the reply through it with no account, which is why answering someone never hands out an address. Free doors take a plain POST on a tight per-IP bucket. A door can also take **escrowed** knocks (row below), which the door page offers beside paying up front and the owner runs from their own wallet in the settings panel; nothing on this server can move an escrowed payment. Spec: [specs/KNOCK_PROTOCOL.md](specs/KNOCK_PROTOCOL.md). Docs: [docs/knock.md](docs/knock.md). Tests: [tests/knock-policy.test.js](tests/knock-policy.test.js), [packages/knock-sdk/test/](packages/knock-sdk/test), [packages/knock-mcp/test/](packages/knock-mcp/test) || Visitor feedback loop | Recorder [@three-ws/witness](packages/witness), capture + panel [src/feedback-companion.js](src/feedback-companion.js) (mounted by [src/walk-companion.js](src/walk-companion.js)), intake [api/feedback/report.js](api/feedback/report.js), queue API [api/feedback/index.js](api/feedback/index.js), scoring [api/_lib/feedback/triage.js](api/_lib/feedback/triage.js), store [api/_lib/feedback/store.js](api/_lib/feedback/store.js), cron [api/cron/feedback-triage.js](api/cron/feedback-triage.js), review page [pages/feedback.html](pages/feedback.html) `→ /feedback` + [src/feedback-page.js](src/feedback-page.js) | Live | The return channel on the corner companion: the visitor tells the avatar what broke and the report carries the route, build SHA, viewport, and the last console errors and failed requests the browser captured, so a vague sentence lands as an actionable row. The companion offers proactively at the one moment worth asking (right after the page actually threw), once per route per session, never in a background tab. **Feedback is data, never an instruction:** the model that reads untrusted visitor text writes six scalars to that report's own row and has no path to the repo, a deploy, or any other write; the admin-only `/feedback` queue groups reports into one row per problem and a person decides. **A report arrives as a runnable failing test:** the recorded session compiles into a Playwright spec asserting the failure is GONE (red until fixed, green after), reachable from the queue's Copy test button, `GET /api/feedback/repro`, `npm run feedback:repro`, and the `get_feedback_repro` MCP tool. Docs: [docs/feedback.md](docs/feedback.md), [docs/witness.md](docs/witness.md). Tests: [tests/feedback-triage.test.js](tests/feedback-triage.test.js) || Session-to-test recorder | [packages/witness/](packages/witness) `→ @three-ws/witness`: recorder [recorder.js](packages/witness/src/recorder.js), selector synthesis [selector.js](packages/witness/src/selector.js), capture-time privacy [redact.js](packages/witness/src/redact.js), spec compiler [compile.js](packages/witness/src/compile.js); compiled repros land in [tests/repros/](tests/repros) with [their own Playwright config](tests/repros/playwright.config.mjs) | Published as `@three-ws/witness` (Apache-2.0, zero deps) | Turns a real user session into a runnable failing test. Records INTENT (about 2 KB) rather than DOM mutations (megabytes), so it can be always-on, which is the point: the trace already exists when somebody decides to complain. Every element gets the strongest handle that is unique in the live document (test id > id > aria-label > role+name > text > name > placeholder > class > structural path), framework-generated ids and utility classes refused, and each trace carries a replay confidence equal to its WEAKEST link. The compiled spec asserts the failure is gone, never that it happened, so it locks in the fix rather than the bug; a session with no machine-visible failure emits `test.fail()` and says which assertion a human must add rather than faking green. Privacy is enforced at capture: typed values are never held (only a length and a shape), password/hidden/email/tel/date inputs and `data-witness="off"` subtrees record nothing, URLs lose their credentials before storage. Docs: [docs/witness.md](docs/witness.md). Tests: [packages/witness/test/](packages/witness/test) (36, `npm run test:witness`) |diff --git a/docs/home-voice.md b/docs/home-voice.mdindex 5cd904ef8..e05b6ebe1 100644--- a/docs/home-voice.md+++ b/docs/home-voice.md@@ -155,6 +155,12 @@ Playback and capture run together, the VAD runs during playback, and speech cutsthe audio stops, whatever synthesis is still in flight is aborted, and the interrupting utterancebecomes the next request without needing a second wake word.+Either way the host is told. `onEvent` fires `playback-stopped` when an utterance is cut, whether it+was already audible or still being synthesized, and `audible` says which. An interruption that lands+inside the synthesis window cancels a sentence nobody heard, and a surface that clears its speaking+affordance on that event would otherwise hold the affordance forever, waiting on audio that was+aborted in flight.+Interruption needs four consecutive frames (128 ms) above a high speech probability, not oneframe at the idle threshold. One frame would fire on the agent's own voice leaking past echocancellation on a laptop speaker at volume, and an agent that interrupts itself is worse than one@@ -223,12 +229,35 @@ npx vite --port 3457 # /api proxies to productionnode scripts/check-home-voice.mjs --port 3457```-Thirty-four assertions across ten scenarios, in a real Chromium with a real microphone stream fed+Thirty-six assertions across ten scenarios, in a real Chromium with a real microphone stream fedfrom speech the platform's own TTS lane synthesized. It writes the measured legs and a frame ofeach of the twelve states to `.cache/home-voice/`. Add `--headed` to watch it.-Set `AUDIT_EMAIL` and `AUDIT_PASSWORD` (the QA account already in `.env`) to run against the-signed-in rate limits rather than the tighter anonymous ones.+`--only <names>` runs a subset, comma separated, from `cold-load, happy, barge, self-trigger,+guarded-yeah, guarded-token, mute, unavailable, permission-denied, gallery, live`. Use it when+re-checking one fix: a full run spends the ASR bucket for this IP, and the block that follows+lengthens each time it is hit, so repeated full runs cost the next hour of them.++`--live` adds an eleventh scenario that drives a **real Home Assistant** and asserts the device+actually changed:++```bash+node scripts/check-home-voice.mjs --port 3457 --live+```++It brings up a real Home Assistant through+[`scripts/home-test-instance.mjs`](../scripts/home-test-instance.mjs), starts a local API server+that can reach it, connects it through the real `POST /api/home`, then says "Hey Jarvis, turn the+kitchen light off" out loud and reads `light.kitchen_lights` back out of Home Assistant's own REST+API. The assertion is on the entity state, not on what the agent said it did. It needs docker, and+it removes the home connection it created on the way out. The local server exists only because a+Home Assistant on loopback is not reachable from Cloud Run; ASR and TTS still go to production,+because those are the two lanes whose credentials live there.++The run signs in with `AUDIT_EMAIL` and `AUDIT_PASSWORD`, which it reads from `.env` itself, so it+gets the signed-in rate limits rather than the tighter anonymous ones. Without them it still works,+it just cannot repeat as often: the anonymous ASR bucket is spent by roughly one full run, and the+next one fails part-way through the happy path with a 429 that reads like a broken speech lane.The unit tests cover the parts that must not be wrong without a browser:diff --git a/scripts/check-home-voice.mjs b/scripts/check-home-voice.mjsindex 8c0250877..12d066ff3 100644--- a/scripts/check-home-voice.mjs+++ b/scripts/check-home-voice.mjs@@ -39,7 +39,8 @@import { chromium } from 'playwright';import { mkdirSync, writeFileSync, existsSync, readFileSync } from 'node:fs';-import { createHash } from 'node:crypto';+import { createHash, randomBytes } from 'node:crypto';+import { spawn } from 'node:child_process';import { resolve, join } from 'node:path';const args = process.argv.slice(2);@@ -51,6 +52,26 @@ const PORT = Number(flag('port', process.env.HOME_VOICE_PORT || 3457));const BASE = `http://localhost:${PORT}`;const OUT = resolve(flag('out', '.cache/home-voice'));const HEADED = args.includes('--headed');+/**+ * Also drive a REAL Home Assistant, end to end, and assert the device changed.+ *+ * Off by default because it needs docker and a local API server: the loop's+ * /api/chat has to run somewhere that can reach a Home Assistant on loopback,+ * and production cannot route into this machine. Everything it adds is real.+ */+const LIVE = args.includes('--live');+const LIVE_API_PORT = Number(flag('live-api-port', 3798));+const LIVE_VITE_PORT = Number(flag('live-vite-port', 3799));+/**+ * Run only the named scenarios, comma separated. A full run spends the ASR+ * bucket for the IP, and the block that follows lengthens each time it is hit,+ * so re-verifying one fix by re-running all ten scenarios costs the next hour of+ * runs. Names are the keys of SCENARIOS below.+ */+const ONLY = (flag('only', '') || '')+ .split(',')+ .map((name) => name.trim())+ .filter(Boolean);const AUDIO = join(OUT, 'audio');mkdirSync(AUDIO, { recursive: true });@@ -63,6 +84,26 @@ mkdirSync(AUDIO, { recursive: true });*/let SESSION_COOKIE = null;+/**+ * The credentials live in .env, and nothing in a bare `node script.mjs` reads+ * that file. Without this the run silently falls back to the anonymous buckets+ * and dies part-way through the happy path on a 429 that reads like a broken+ * speech lane. Same shape as the loader in scripts/agent-wallet-smoke.mjs; a+ * value already in the environment always wins.+ */+(function loadEnvFiles() {+ for (const name of ['.env', '.env.local']) {+ const file = resolve(process.cwd(), name);+ if (!existsSync(file)) continue;+ for (const line of readFileSync(file, 'utf8').split('\n')) {+ const m = line.match(/^([A-Z0-9_]+)=(.*)$/i);+ if (!m) continue;+ const value = m[2].trim().replace(/^"(.*)"$/, '$1').replace(/^'(.*)'$/, '$1');+ if (process.env[m[1]] == null || process.env[m[1]] === '') process.env[m[1]] = value;+ }+ }+})();+async function signIn() {const email = process.env.AUDIT_EMAIL;const password = process.env.AUDIT_PASSWORD;@@ -500,7 +541,13 @@ async function scenarioBargeIn() {await bootPage(page);await optIn(page);// Speak a long enough answer that the user's interruption lands inside it.- await page.evaluate((line) => window.homeVoice.loop._speak(line), BARGE_IN_ANSWER);+ // Started, not awaited. _speak resolves when the utterance finishes playing,+ // and the whole point of this scenario is that it never does: the promise a+ // barge-in cancels is left unsettled on purpose, so returning it here would+ // hang the evaluate for the rest of the run.+ await page.evaluate((line) => {+ window.homeVoice.loop._speak(line);+ }, BARGE_IN_ANSWER);// Wait for sound actually leaving the speaker, not merely for the state// flip: _speak enters `speaking` before the synthesis request returns, so a// failed TTS call would otherwise read as a silent, un-interruptible agent.@@ -790,6 +837,214 @@ async function scenarioStateGallery() {}}+/**+ * The whole point of the feature, proved against hardware-shaped reality: a+ * person says one sentence out loud and a real device in a real Home Assistant+ * changes state.+ *+ * Every leg here is the real one. The speech is synthesized audio played into+ * Chromium's capture device, the wake word is matched by the committed+ * openWakeWord weights in the tab, the transcript comes from the live ASR lane,+ * and the agent turn runs api/chat.js with the real home tools against a real+ * Home Assistant, which really turns the light off. The assertion is not on+ * what the agent SAID it did: it is on the entity state read back out of Home+ * Assistant's own REST API afterwards, because an agent that reports success it+ * did not achieve is the failure this scenario exists to catch.+ *+ * The one piece of scaffolding is topology, not behaviour: the page is served+ * by a Vite whose /api proxy points at a local server instead of production,+ * because a Home Assistant on 127.0.0.1 is not reachable from Cloud Run. The+ * two lanes that need cloud credentials, ASR and TTS, are routed back to+ * production so they stay the real lanes too.+ */+/** The seeded demo light the spoken command names. Its friendly name is "Kitchen Lights". */+const LIVE_LIGHT = 'light.kitchen_lights';++/**+ * Bring up everything the live scenario needs, and hand back a teardown.+ *+ * A real Home Assistant from the lane's shared harness, a real API server that+ * can reach it, a real session for the QA account, and a real home connection+ * created through the real POST /api/home. The connection row is deleted again+ * on the way out: it points at a loopback URL that means nothing anywhere else,+ * and its token is encrypted with a key generated for this run.+ */+async function liveStack() {+ const sh = (cmd, cmdArgs, opts = {}) =>+ new Promise((resolveRun, rejectRun) => {+ const child = spawn(cmd, cmdArgs, { cwd: process.cwd(), ...opts });+ let out = '';+ child.stdout?.on('data', (d) => (out += d));+ child.stderr?.on('data', (d) => (out += d));+ child.on('error', rejectRun);+ child.on('close', (code) => (code === 0 ? resolveRun(out) : rejectRun(new Error(`${cmd} exited ${code}: ${out.slice(-500)}`))));+ });++ console.log('[live] bringing up a real Home Assistant...');+ const raw = await sh(process.execPath, [+ 'scripts/home-test-instance.mjs',+ '--up', '--onboard', '--seed', '--json', '--name', 'voiceloop',+ ]);+ const ha = JSON.parse(raw.slice(raw.indexOf('{')));+ console.log(`[live] Home Assistant ${ha.haVersion} at ${ha.baseUrl}`);++ // The API server needs its own signing and encryption keys. They are local to+ // this run: they sign the session it mints and encrypt the row it writes, both+ // of which this function deletes again.+ const env = {+ ...process.env,+ PORT: String(LIVE_API_PORT),+ HOME_ALLOW_LOCAL_INSTANCE: '1',+ JWT_SECRET: process.env.JWT_SECRET || randomBytes(32).toString('hex'),+ WALLET_ENCRYPTION_KEY: process.env.WALLET_ENCRYPTION_KEY || randomBytes(32).toString('hex'),+ };+ const api = spawn(process.execPath, ['server/index.mjs'], { env, stdio: 'ignore' });+ const apiBase = `http://localhost:${LIVE_API_PORT}`;+ await waitForHttp(`${apiBase}/api/healthz`, 60000);++ // The same real login the site's own form performs, against the local server+ // so the session is one this server will accept.+ const login = await fetch(`${apiBase}/api/auth/login`, {+ method: 'POST',+ headers: { 'content-type': 'application/json' },+ body: JSON.stringify({ email: process.env.AUDIT_EMAIL, password: process.env.AUDIT_PASSWORD }),+ });+ if (!login.ok) throw new Error(`local login failed: ${login.status} ${(await login.text()).slice(0, 200)}`);+ const cookie = cookieHeader(login);+ if (!/sid=/.test(cookie)) throw new Error('the local server returned no session cookie');++ // Cookie sessions are CSRF-protected, so the token has to be carried the same+ // way the browser carries it: cookie plus matching header.+ const csrfRes = await fetch(`${apiBase}/api/csrf-token`, { headers: { cookie } });+ const csrf = (await csrfRes.json().catch(() => ({})))?.data?.token || '';+ const authed = `${cookie}; ${cookieHeader(csrfRes)}`;+ if (!csrf) throw new Error('the local server issued no CSRF token');++ const created = await fetch(`${apiBase}/api/home`, {+ method: 'POST',+ headers: { 'content-type': 'application/json', cookie: authed, 'x-csrf-token': csrf },+ body: JSON.stringify({ baseUrl: ha.baseUrl, token: ha.token }),+ });+ const body = await created.json().catch(() => ({}));+ if (!created.ok) throw new Error(`could not connect the house: ${created.status} ${JSON.stringify(body).slice(0, 300)}`);+ const homeId = body?.home?.id || body?.id;+ if (!homeId) throw new Error(`the connect response carried no home id: ${JSON.stringify(body).slice(0, 300)}`);+ console.log(`[live] connected as home ${homeId}`);++ // A second Vite, whose /api goes to the local server rather than production.+ const vite = spawn('npx', ['vite', '--port', String(LIVE_VITE_PORT)], {+ env: { ...process.env, DEV_API_PROXY: apiBase },+ stdio: 'ignore',+ });+ const viteBase = `http://localhost:${LIVE_VITE_PORT}`;+ await waitForHttp(`${viteBase}/voice/home`, 90000);++ return {+ ha,+ homeId,+ cookie: authed,+ viteBase,+ async teardown() {+ await fetch(`${apiBase}/api/home/${homeId}`, {+ method: 'DELETE',+ headers: { cookie: authed, 'x-csrf-token': csrf },+ }).catch(() => {});+ vite.kill();+ api.kill();+ },+ };+}++/** Every cookie a response set, folded into one request-shaped header. */+function cookieHeader(res) {+ const raw = res.headers.getSetCookie?.() ?? [res.headers.get('set-cookie')].filter(Boolean);+ return raw.map((c) => c.split(';')[0]).join('; ');+}++/** Poll a URL until it answers, so a spawned server is never raced. */+async function waitForHttp(url, timeoutMs) {+ const deadline = Date.now() + timeoutMs;+ while (Date.now() < deadline) {+ const res = await fetch(url).catch(() => null);+ if (res?.ok) return;+ await new Promise((r) => setTimeout(r, 1000));+ }+ throw new Error(`nothing answered at ${url} within ${timeoutMs} ms`);+}++async function scenarioLive({ ha, viteBase, homeId, cookie }) {+ const state = async (entity) => {+ const res = await fetch(`${ha.baseUrl}/api/states/${entity}`, {+ headers: { authorization: `Bearer ${ha.token}` },+ });+ if (!res.ok) throw new Error(`home assistant ${res.status} reading ${entity}`);+ return (await res.json()).state;+ };+ const call = (domain, service, entity) =>+ fetch(`${ha.baseUrl}/api/services/${domain}/${service}`, {+ method: 'POST',+ headers: { authorization: `Bearer ${ha.token}`, 'content-type': 'application/json' },+ body: JSON.stringify({ entity_id: entity }),+ });++ // Put the light in the state the sentence is going to change, so "off" at the+ // end cannot be the state it was already in.+ await call('light', 'turn_on', LIVE_LIGHT);+ const before = await state(LIVE_LIGHT);+ check('the real light starts on, so turning it off is a change', before === 'on', { before });++ const { browser, page } = await launch(CLIPS.command.path);+ try {+ // ASR and TTS are the only lanes whose credentials live in the cloud, so+ // they go to production while everything else talks to the local server.+ for (const lane of ['**/api/asr', '**/api/tts/speak']) {+ await page.route(lane, async (route) => {+ const req = route.request();+ const res = await fetch(`https://three.ws${new URL(req.url()).pathname}`, {+ method: req.method(),+ headers: { ...req.headers(), host: 'three.ws', ...(SESSION_COOKIE ? { cookie: SESSION_COOKIE } : {}) },+ body: req.method() === 'GET' ? undefined : req.postDataBuffer(),+ });+ await route.fulfill({+ status: res.status,+ headers: { 'content-type': res.headers.get('content-type') || 'application/octet-stream' },+ body: Buffer.from(await res.arrayBuffer()),+ });+ });+ }+ // launch() put the production session on the context for the ASR and TTS+ // lanes; the page itself now talks to the local server, which needs the+ // local one. The two routes above keep sending production's.+ await page.setExtraHTTPHeaders({ cookie });+ await page.goto(`${viteBase}/voice/home?home=${homeId}`, { waitUntil: 'domcontentloaded' });+ await page.waitForFunction(() => !!window.homeVoice, null, { timeout: 30000 });+ await page.evaluate(TAP);+ await optIn(page);++ await waitForEvent(page, 'transcript', 90000);+ const transcript = (await events(page)).find((e) => e.type === 'transcript');+ check('the spoken command is transcribed by the real lane', /kitchen/i.test(transcript?.text || ''), {+ text: transcript?.text,+ });++ // Poll Home Assistant itself. The agent's own reply is not evidence.+ let after = before;+ for (let i = 0; i < 60 && after !== 'off'; i++) {+ await page.waitForTimeout(1000);+ after = await state(LIVE_LIGHT);+ }+ check('the real light is off, read back from Home Assistant', after === 'off', { before, after });++ const done = (await events(page)).find((e) => e.type === 'reply' || e.type === 'turn-done');+ const spoken = await page.evaluate(() => window.homeVoice.loop.stateDetail?.text || '');+ check('and the agent says what it did', !!(spoken || done), { spoken: (spoken || '').slice(0, 120) });+ await page.locator('#voice-panel').screenshot({ path: join(OUT, 'live-real-device.png') });+ return { before, after, transcript: transcript?.text, spoken };+ } finally {+ await browser.close();+ }+}+/*** Run one scenario, and run it again if a dev-server reload pulled the page out* from under it.@@ -832,16 +1087,35 @@ async function main() {await warmUp();const measured = {};- await run(scenarioColdLoad);- measured.happy = await run(scenarioHappyPath);- measured.barge = await run(scenarioBargeIn);- measured.selfTrigger = await run(scenarioSelfTrigger);- await run(() => scenarioGuarded({ clip: CLIPS.ambientYeah, expectRedeemed: false, label: 'ambient yes' }));- await run(() => scenarioGuarded({ clip: CLIPS.confirmToken, expectRedeemed: true, label: 'the token' }));- measured.mute = await run(scenarioMute);- await run(scenarioUnavailable);- await run(scenarioPermissionDenied);- await run(scenarioStateGallery);+ const SCENARIOS = {+ 'cold-load': scenarioColdLoad,+ happy: async () => (measured.happy = await scenarioHappyPath()),+ barge: async () => (measured.barge = await scenarioBargeIn()),+ 'self-trigger': async () => (measured.selfTrigger = await scenarioSelfTrigger()),+ 'guarded-yeah': () => scenarioGuarded({ clip: CLIPS.ambientYeah, expectRedeemed: false, label: 'ambient yes' }),+ 'guarded-token': () => scenarioGuarded({ clip: CLIPS.confirmToken, expectRedeemed: true, label: 'the token' }),+ mute: async () => (measured.mute = await scenarioMute()),+ unavailable: scenarioUnavailable,+ 'permission-denied': scenarioPermissionDenied,+ gallery: scenarioStateGallery,+ };++ const unknown = ONLY.filter((name) => name !== 'live' && !SCENARIOS[name]);+ if (unknown.length) throw new Error(`unknown scenario(s): ${unknown.join(', ')}. Known: ${Object.keys(SCENARIOS).join(', ')}, live`);+ const wanted = (name) => ONLY.length === 0 || ONLY.includes(name);++ for (const [name, scenario] of Object.entries(SCENARIOS)) {+ if (wanted(name)) await run(scenario);+ }++ if (LIVE && wanted('live')) {+ const stack = await liveStack();+ try {+ measured.live = await run(() => scenarioLive(stack));+ } finally {+ await stack.teardown();+ }+ }const failed = results.filter((r) => !r.pass);writeFileSync(