Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
JavaScript 3d-generationai-agentsanimationavatardigital-humangenerative-aiglbgltfimage-to-3dllmmcpmcp-servermodel-context-protocolsolanatext-to-3dthreejsweb-componentswebglwebxrx402
docs(home-voice): replace the latency budgets with what was actually measured
nirholasf663e9c9a130
diff --git a/docs/home-voice.md b/docs/home-voice.mdindex e05b6ebe1..eb91bb5d5 100644--- a/docs/home-voice.md+++ b/docs/home-voice.md@@ -1,7 +1,7 @@# The browser voice loop: wake word, barge-in, and a confirmation a "yeah" cannot satisfy**Status:** shipped. Live at [`/voice/home`](https://three.ws/voice/home) and mountable on any-surface that wants it. **Last measured against the tree:** 2026-09-03.+surface that wants it. **Last measured against the tree:** 2026-09-09.Hands-free is the only interface that works when you are carrying groceries into a dark kitchen.This is the part of [three.ws Home](./smart-home.md) that lets you say "turn the kitchen light@@ -137,12 +137,12 @@ is a worse network position than a real user's:| Leg | Budget | Measured | Note ||---|---|---|---|-| Wake word detection | 200 ms | 8 to 16 ms | Inference only. The model needs about 140 ms of audio context past the end of the word before it crosses threshold, measured offline. |-| End of speech | 400 ms | 343 to 369 ms | 11 silero frames of trailing silence, plus the frame the decision is made on. |-| Transcription round trip | 900 ms | 684 to 1810 ms | Misses from here. The path is Codespace to Cloud Run to NVIDIA Riva; a browser near the region is materially closer. |+| Wake word detection | 200 ms | 21 to 49 ms | Inference only. The model needs about 140 ms of audio context past the end of the word before it crosses threshold, measured offline. |+| End of speech | 400 ms | 389 ms settled, 480 ms loaded | 11 silero frames of trailing silence, plus the frame the decision is made on. The window itself is 352 ms; the rest is frame quantisation and per-frame inference on the main thread. On a machine also running the wake-word model and a 3D scene it has been seen to stretch to 2.3 s for a single utterance, which is scheduling, not the window. |+| Transcription round trip | 900 ms | 738 ms best, 5.4 s worst | Misses from here under load. The path is Codespace to Cloud Run to NVIDIA Riva; a browser near the region is materially closer. || Agent turn to first tool call | 1200 ms | measured live, varies by provider | The `home_tool` frame arrives ahead of the model's closing sentence. || Action to device change | 700 ms | needs a connected house | Mostly Home Assistant's own latency. |-| Playback stops after you interrupt | 200 ms | 104 to 125 ms | Four consecutive frames above a high speech probability. |+| Playback stops after you interrupt | 200 ms | 91 to 101 ms | Four consecutive frames above a high speech probability, measured against sound that was genuinely audible. |The transcription leg is the one that misses, and the number above is the real one. It is notwidened here to make the table green.@@ -234,11 +234,11 @@ from speech the platform's own TTS lane synthesized. It writes the measured legseach of the twelve states to `.cache/home-voice/`. Add `--headed` to watch it.`--only <names>` runs a subset, comma separated, from `cold-load, happy, barge, self-trigger,-guarded-yeah, guarded-token, mute, unavailable, permission-denied, gallery, live`. Use it when+guarded-yeah, guarded-token, mute, unavailable, permission-denied, gallery, live, live-confirm`. Use it whenre-checking one fix: a full run spends the ASR bucket for this IP, and the block that followslengthens each time it is hit, so repeated full runs cost the next hour of them.-`--live` adds an eleventh scenario that drives a **real Home Assistant** and asserts the device+`--live` adds two more scenarios that drive a **real Home Assistant** and assert the real deviceactually changed:```bash@@ -254,6 +254,26 @@ it removes the home connection it created on the way out. The local server existHome Assistant on loopback is not reachable from Cloud Run; ASR and TTS still go to production,because those are the two lanes whose credentials live there.+`live-confirm` is the other half, and it is the one that proves the door. It drives+[`api/_lib/home/tools.js`](../api/_lib/home/tools.js), the same module `api/chat.js` runs+server-side between its two model passes, so a real confirmation row is minted; then it redeems+that id through the real `POST /api/home/:id/confirm` and reads `lock.front_door` back out of+Home Assistant. It asserts the door was locked before, that minting alone did not move it, that+redeeming really unlocked it, and that replaying the same id is refused (410 `confirmation_spent`)+with the door still locked. No model is involved on purpose: the model picking a tool is the one+leg that needs a provider key, and none of the safety properties depend on it.++Measured 2026-09-09: 36/36 browser assertions and 6/6 live-confirm assertions against Home+Assistant 2026.9.0.++**The `live` scenario needs a tool-calling provider, and says so when there is none.** Every rung+of [`api/_lib/llm-tool-chain.js`](../api/_lib/llm-tool-chain.js) needs either a provider key or a+usable GCP token, so a checkout with no keys in `.env` and an expired `gcloud` login has zero+rungs and no agent turn can run at all. That used to surface as "the light is still on", which+reads like a broken home lane and is not one. The run now counts the rungs first and skips the+spoken end-to-end leg with the reason and the fix, rather than reporting a red. `live-confirm`+still runs, because it needs no model.+The run signs in with `AUDIT_EMAIL` and `AUDIT_PASSWORD`, which it reads from `.env` itself, so itgets the signed-in rate limits rather than the tighter anonymous ones. Without them it still works,it just cannot repeat as often: the anonymous ASR bucket is spent by roughly one full run, and thediff --git a/scripts/home-test-instance.mjs b/scripts/home-test-instance.mjsindex 947ab3646..f50ad6349 100755--- a/scripts/home-test-instance.mjs+++ b/scripts/home-test-instance.mjs@@ -929,32 +929,92 @@ function describeInstance(name, version) {* An exclusive lock around one named instance, held for the whole up/onboard/* seed sequence. `mkdir` is the atomic primitive here: it either creates the* directory or fails, with no window between the two.+ *+ * The lock records WHO holds it, and that is the difference between a lock and+ * a fifteen-minute outage. Callers of this harness kill it on their own+ * timeouts (tests/_helpers/home-instance.js gives `--stop` 120 seconds), and a+ * killed process never reaches the `finally` below, so the directory outlives+ * it. Abandoning purely on age meant the next call on that lane sat in the wait+ * loop below until the stale lock aged out, then died on ITS caller's timeout+ * and left a fresh lock behind: one kill guaranteed the next several. It was+ * seen as `home-test-instance timed out after 120s` on a `--stop` that takes+ * seven seconds when it runs at all, which reads as a hung container and is a+ * dead process's leftovers.+ *+ * So the holder writes its pid, and a waiter asks the operating system whether+ * that process still exists. A dead holder's lock is reclaimed at once; a live+ * holder's is waited on exactly as before. Signal 0 is the standard "is this+ * pid alive" probe: it performs the permission check and delivers nothing.*/async function withLock(inst, fn) {const lock = `${inst.configDir}.lock`;+ const owner = path.join(lock, 'pid');const deadline = Date.now() + 900_000;for (;;) {try {fs.mkdirSync(lock);+ fs.writeFileSync(owner, `${process.pid}\n`);break;} catch (err) {if (err.code !== 'EEXIST') throw err;- // A killed run leaves its lock behind. Nothing here takes longer than- // a cold Home Assistant boot, so a much older lock is abandoned.+ const holder = lockHolder(owner);+ // No pid file yet means the holder is between its mkdir and its write,+ // which is microseconds; anything else is a lock whose owner is gone.+ if (holder !== null && !isAlive(holder)) {+ log(`[lock] reclaiming the lock on "${inst.name}" from pid ${holder}, which is no longer running`);+ fs.rmSync(lock, { recursive: true, force: true });+ continue;+ }+ // Belt and braces for the pre-pid window and for a lock written by an+ // older copy of this script: nothing here outlasts a cold boot.const age = Date.now() - (fs.statSync(lock).mtimeMs || 0);if (age > 900_000) {fs.rmSync(lock, { recursive: true, force: true });continue;}if (Date.now() > deadline) throw new Error(`another run has held the lock on "${inst.name}" for 15 minutes`);- log(`[lock] waiting for another run to finish with "${inst.name}"`);+ log(`[lock] waiting for pid ${holder ?? 'unknown'} to finish with "${inst.name}"`);await new Promise((r) => setTimeout(r, 2000));}}+ // SIGKILL cannot be caught, which is why the pid check above exists; SIGTERM+ // and SIGINT can be, and releasing on them means the common case (a caller+ // asking politely first, a developer pressing ctrl-c) leaves nothing behind.+ const release = () => fs.rmSync(lock, { recursive: true, force: true });+ const onSignal = (signal) => {+ release();+ process.exit(signal === 'SIGINT' ? 130 : 143);+ };+ process.once('SIGTERM', onSignal);+ process.once('SIGINT', onSignal);try {return await fn();} finally {- fs.rmSync(lock, { recursive: true, force: true });+ process.off('SIGTERM', onSignal);+ process.off('SIGINT', onSignal);+ release();+ }+}++/** The pid inside a lock directory, or null when it has not been written yet. */+function lockHolder(pidFile) {+ try {+ const pid = Number.parseInt(fs.readFileSync(pidFile, 'utf8').trim(), 10);+ return Number.isInteger(pid) && pid > 0 ? pid : null;+ } catch {+ return null;+ }+}++/** Whether a pid is a process this user could signal. Sends nothing. */+function isAlive(pid) {+ try {+ process.kill(pid, 0);+ return true;+ } catch (err) {+ // EPERM means it exists and belongs to somebody else, which still counts+ // as held. Only ESRCH is proof that nothing is there.+ return err.code === 'EPERM';}}diff --git a/scripts/measure-home-scene.mjs b/scripts/measure-home-scene.mjsindex d68517e0b..4582ef508 100644--- a/scripts/measure-home-scene.mjs+++ b/scripts/measure-home-scene.mjs@@ -461,16 +461,32 @@ async function captureLostHouse(page, shot) {await page.locator('#hs-status[data-status="stale"]').waitFor({ timeout: 180_000 });out.stale = await shot(page, '05-stale');out.staleRoomsHeld = await page.evaluate(() => window.__homeScene.model.rooms.length);+ } catch (err) {+ out.lostHouseError = err.message;+ } finally {+ await docker(['start', HOUSE.container]);+ }- await page.locator('#hs-status[data-status="disconnected"]').waitFor({ timeout: 300_000 });+ // Disconnected is NOT a house that stopped answering: while the platform is+ // still retrying that is stale, and the page says so on purpose. It is the+ // state where nobody is retrying, and the case a wall display really meets is+ // its own network going away underneath it, which kills the event stream this+ // page holds. The capability is the browser's, so nothing here is stubbed.+ try {+ await page.context().setOffline(true);+ await page.locator('#hs-status[data-status="disconnected"]').waitFor({ timeout: 180_000 });out.disconnected = await shot(page, '06-disconnected');out.disconnectedRoomsHeld = await page.evaluate(() => window.__homeScene.model.rooms.length);+ out.disconnectedOffersReconnect = await page.locator('#hs-reconnect:not([hidden])').isVisible();} catch (err) {- out.lostHouseError = err.message;+ out.disconnectedError = err.message;} finally {- await docker(['start', HOUSE.container]);+ await page.context().setOffline(false);}- // Back on its own, with no reload: the page recovers or the run says so.+ // Back on its own, with no reload: the page recovers or the run says so. The+ // house has to be answering again for that, so this waits on the container+ // that was just restarted as well as on the network that just came back.+ await waitForHouse();await page.locator('#hs-status[data-status="live"]').waitFor({ timeout: 300_000 })@@ -483,6 +499,18 @@ async function captureLostHouse(page, shot) {return out;}+/** Wait until the house itself answers again after a stop and start. */+async function waitForHouse() {+ for (let i = 0; i < 90; i += 1) {+ const ok = await haFetch('/api/')+ .then((r) => r.ok)+ .catch(() => false);+ if (ok) return true;+ await new Promise((resolve) => setTimeout(resolve, 2000));+ }+ return false;+}+async function docker(args) {const { execFile } = await import('node:child_process');await new Promise((resolve, reject) => execFile('docker', args, (err) => (err ? reject(err) : resolve())));@@ -542,16 +570,32 @@ async function captureLiveChange(page, dir) {* back to confirm which one was actually hit.*/async function clickEntity(page, entityId) {+ const selected = async () =>+ (+ await page+ .locator('.hs-entity-id')+ .textContent({ timeout: 5000 })+ .catch(() => null)+ )?.trim() === entityId;+const at = await page.evaluate((id) => window.__homeScene.project(id), entityId);- if (!at?.visible) return false;const box = await page.locator('#hs-stage canvas').boundingBox();- if (!box) return false;- await page.mouse.click(box.x + at.x, box.y + at.y);- const shown = await page- .locator('.hs-entity-id')- .textContent({ timeout: 5000 })- .catch(() => null);- return shown?.trim() === entityId;+ if (at?.visible && box) {+ await page.mouse.click(box.x + at.x, box.y + at.y);+ if (await selected()) return true;+ }++ // The raycast can legitimately miss: the object is behind the camera, another+ // object is in front of it, or this box is loaded enough that the camera has+ // moved between the projection and the click. None of that is a device a+ // person cannot reach, because the room rail lists every one of them and is+ // the keyboard route to the same selection. Falling back to it keeps the+ // measurement about the product rather than about pointer luck.+ const row = page.locator(`.hs-room-device[data-entity-id="${entityId.replace(/"/g, '\\"')}"]`).first();+ if (!(await row.count())) return false;+ await row.scrollIntoViewIfNeeded().catch(() => {});+ await row.click();+ return selected();}/** Every drawn object of a domain, room by room. */