Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

docs(home-voice): replace the latency budgets with what was actually measured

nirholasf663e9c9a130
diff --git a/docs/home-voice.md b/docs/home-voice.md
index e05b6ebe1..eb91bb5d5 100644
--- a/docs/home-voice.md
+++ b/docs/home-voice.md
@@ -1,7 +1,7 @@
# The browser voice loop: wake word, barge-in, and a confirmation a "yeah" cannot satisfy
**Status:** shipped. Live at [`/voice/home`](https://three.ws/voice/home) and mountable on any
-surface that wants it. **Last measured against the tree:** 2026-09-03.
+surface that wants it. **Last measured against the tree:** 2026-09-09.
Hands-free is the only interface that works when you are carrying groceries into a dark kitchen.
This is the part of [three.ws Home](./smart-home.md) that lets you say "turn the kitchen light
@@ -137,12 +137,12 @@ is a worse network position than a real user's:
| Leg | Budget | Measured | Note |
|---|---|---|---|
-| Wake word detection | 200 ms | 8 to 16 ms | Inference only. The model needs about 140 ms of audio context past the end of the word before it crosses threshold, measured offline. |
-| End of speech | 400 ms | 343 to 369 ms | 11 silero frames of trailing silence, plus the frame the decision is made on. |
-| Transcription round trip | 900 ms | 684 to 1810 ms | Misses from here. The path is Codespace to Cloud Run to NVIDIA Riva; a browser near the region is materially closer. |
+| Wake word detection | 200 ms | 21 to 49 ms | Inference only. The model needs about 140 ms of audio context past the end of the word before it crosses threshold, measured offline. |
+| End of speech | 400 ms | 389 ms settled, 480 ms loaded | 11 silero frames of trailing silence, plus the frame the decision is made on. The window itself is 352 ms; the rest is frame quantisation and per-frame inference on the main thread. On a machine also running the wake-word model and a 3D scene it has been seen to stretch to 2.3 s for a single utterance, which is scheduling, not the window. |
+| Transcription round trip | 900 ms | 738 ms best, 5.4 s worst | Misses from here under load. The path is Codespace to Cloud Run to NVIDIA Riva; a browser near the region is materially closer. |
| Agent turn to first tool call | 1200 ms | measured live, varies by provider | The `home_tool` frame arrives ahead of the model's closing sentence. |
| Action to device change | 700 ms | needs a connected house | Mostly Home Assistant's own latency. |
-| Playback stops after you interrupt | 200 ms | 104 to 125 ms | Four consecutive frames above a high speech probability. |
+| Playback stops after you interrupt | 200 ms | 91 to 101 ms | Four consecutive frames above a high speech probability, measured against sound that was genuinely audible. |
The transcription leg is the one that misses, and the number above is the real one. It is not
widened here to make the table green.
@@ -234,11 +234,11 @@ from speech the platform's own TTS lane synthesized. It writes the measured legs
each of the twelve states to `.cache/home-voice/`. Add `--headed` to watch it.
`--only <names>` runs a subset, comma separated, from `cold-load, happy, barge, self-trigger,
-guarded-yeah, guarded-token, mute, unavailable, permission-denied, gallery, live`. Use it when
+guarded-yeah, guarded-token, mute, unavailable, permission-denied, gallery, live, live-confirm`. Use it when
re-checking one fix: a full run spends the ASR bucket for this IP, and the block that follows
lengthens each time it is hit, so repeated full runs cost the next hour of them.
-`--live` adds an eleventh scenario that drives a **real Home Assistant** and asserts the device
+`--live` adds two more scenarios that drive a **real Home Assistant** and assert the real device
actually changed:
```bash
@@ -254,6 +254,26 @@ it removes the home connection it created on the way out. The local server exist
Home Assistant on loopback is not reachable from Cloud Run; ASR and TTS still go to production,
because those are the two lanes whose credentials live there.
+`live-confirm` is the other half, and it is the one that proves the door. It drives
+[`api/_lib/home/tools.js`](../api/_lib/home/tools.js), the same module `api/chat.js` runs
+server-side between its two model passes, so a real confirmation row is minted; then it redeems
+that id through the real `POST /api/home/:id/confirm` and reads `lock.front_door` back out of
+Home Assistant. It asserts the door was locked before, that minting alone did not move it, that
+redeeming really unlocked it, and that replaying the same id is refused (410 `confirmation_spent`)
+with the door still locked. No model is involved on purpose: the model picking a tool is the one
+leg that needs a provider key, and none of the safety properties depend on it.
+
+Measured 2026-09-09: 36/36 browser assertions and 6/6 live-confirm assertions against Home
+Assistant 2026.9.0.
+
+**The `live` scenario needs a tool-calling provider, and says so when there is none.** Every rung
+of [`api/_lib/llm-tool-chain.js`](../api/_lib/llm-tool-chain.js) needs either a provider key or a
+usable GCP token, so a checkout with no keys in `.env` and an expired `gcloud` login has zero
+rungs and no agent turn can run at all. That used to surface as "the light is still on", which
+reads like a broken home lane and is not one. The run now counts the rungs first and skips the
+spoken end-to-end leg with the reason and the fix, rather than reporting a red. `live-confirm`
+still runs, because it needs no model.
+
The run signs in with `AUDIT_EMAIL` and `AUDIT_PASSWORD`, which it reads from `.env` itself, so it
gets the signed-in rate limits rather than the tighter anonymous ones. Without them it still works,
it just cannot repeat as often: the anonymous ASR bucket is spent by roughly one full run, and the
diff --git a/scripts/home-test-instance.mjs b/scripts/home-test-instance.mjs
index 947ab3646..f50ad6349 100755
--- a/scripts/home-test-instance.mjs
+++ b/scripts/home-test-instance.mjs
@@ -929,32 +929,92 @@ function describeInstance(name, version) {
* An exclusive lock around one named instance, held for the whole up/onboard/
* seed sequence. `mkdir` is the atomic primitive here: it either creates the
* directory or fails, with no window between the two.
+ *
+ * The lock records WHO holds it, and that is the difference between a lock and
+ * a fifteen-minute outage. Callers of this harness kill it on their own
+ * timeouts (tests/_helpers/home-instance.js gives `--stop` 120 seconds), and a
+ * killed process never reaches the `finally` below, so the directory outlives
+ * it. Abandoning purely on age meant the next call on that lane sat in the wait
+ * loop below until the stale lock aged out, then died on ITS caller's timeout
+ * and left a fresh lock behind: one kill guaranteed the next several. It was
+ * seen as `home-test-instance timed out after 120s` on a `--stop` that takes
+ * seven seconds when it runs at all, which reads as a hung container and is a
+ * dead process's leftovers.
+ *
+ * So the holder writes its pid, and a waiter asks the operating system whether
+ * that process still exists. A dead holder's lock is reclaimed at once; a live
+ * holder's is waited on exactly as before. Signal 0 is the standard "is this
+ * pid alive" probe: it performs the permission check and delivers nothing.
*/
async function withLock(inst, fn) {
const lock = `${inst.configDir}.lock`;
+ const owner = path.join(lock, 'pid');
const deadline = Date.now() + 900_000;
for (;;) {
try {
fs.mkdirSync(lock);
+ fs.writeFileSync(owner, `${process.pid}\n`);
break;
} catch (err) {
if (err.code !== 'EEXIST') throw err;
- // A killed run leaves its lock behind. Nothing here takes longer than
- // a cold Home Assistant boot, so a much older lock is abandoned.
+ const holder = lockHolder(owner);
+ // No pid file yet means the holder is between its mkdir and its write,
+ // which is microseconds; anything else is a lock whose owner is gone.
+ if (holder !== null && !isAlive(holder)) {
+ log(`[lock] reclaiming the lock on "${inst.name}" from pid ${holder}, which is no longer running`);
+ fs.rmSync(lock, { recursive: true, force: true });
+ continue;
+ }
+ // Belt and braces for the pre-pid window and for a lock written by an
+ // older copy of this script: nothing here outlasts a cold boot.
const age = Date.now() - (fs.statSync(lock).mtimeMs || 0);
if (age > 900_000) {
fs.rmSync(lock, { recursive: true, force: true });
continue;
}
if (Date.now() > deadline) throw new Error(`another run has held the lock on "${inst.name}" for 15 minutes`);
- log(`[lock] waiting for another run to finish with "${inst.name}"`);
+ log(`[lock] waiting for pid ${holder ?? 'unknown'} to finish with "${inst.name}"`);
await new Promise((r) => setTimeout(r, 2000));
}
}
+ // SIGKILL cannot be caught, which is why the pid check above exists; SIGTERM
+ // and SIGINT can be, and releasing on them means the common case (a caller
+ // asking politely first, a developer pressing ctrl-c) leaves nothing behind.
+ const release = () => fs.rmSync(lock, { recursive: true, force: true });
+ const onSignal = (signal) => {
+ release();
+ process.exit(signal === 'SIGINT' ? 130 : 143);
+ };
+ process.once('SIGTERM', onSignal);
+ process.once('SIGINT', onSignal);
try {
return await fn();
} finally {
- fs.rmSync(lock, { recursive: true, force: true });
+ process.off('SIGTERM', onSignal);
+ process.off('SIGINT', onSignal);
+ release();
+ }
+}
+
+/** The pid inside a lock directory, or null when it has not been written yet. */
+function lockHolder(pidFile) {
+ try {
+ const pid = Number.parseInt(fs.readFileSync(pidFile, 'utf8').trim(), 10);
+ return Number.isInteger(pid) && pid > 0 ? pid : null;
+ } catch {
+ return null;
+ }
+}
+
+/** Whether a pid is a process this user could signal. Sends nothing. */
+function isAlive(pid) {
+ try {
+ process.kill(pid, 0);
+ return true;
+ } catch (err) {
+ // EPERM means it exists and belongs to somebody else, which still counts
+ // as held. Only ESRCH is proof that nothing is there.
+ return err.code === 'EPERM';
}
}
diff --git a/scripts/measure-home-scene.mjs b/scripts/measure-home-scene.mjs
index d68517e0b..4582ef508 100644
--- a/scripts/measure-home-scene.mjs
+++ b/scripts/measure-home-scene.mjs
@@ -461,16 +461,32 @@ async function captureLostHouse(page, shot) {
await page.locator('#hs-status[data-status="stale"]').waitFor({ timeout: 180_000 });
out.stale = await shot(page, '05-stale');
out.staleRoomsHeld = await page.evaluate(() => window.__homeScene.model.rooms.length);
+ } catch (err) {
+ out.lostHouseError = err.message;
+ } finally {
+ await docker(['start', HOUSE.container]);
+ }
- await page.locator('#hs-status[data-status="disconnected"]').waitFor({ timeout: 300_000 });
+ // Disconnected is NOT a house that stopped answering: while the platform is
+ // still retrying that is stale, and the page says so on purpose. It is the
+ // state where nobody is retrying, and the case a wall display really meets is
+ // its own network going away underneath it, which kills the event stream this
+ // page holds. The capability is the browser's, so nothing here is stubbed.
+ try {
+ await page.context().setOffline(true);
+ await page.locator('#hs-status[data-status="disconnected"]').waitFor({ timeout: 180_000 });
out.disconnected = await shot(page, '06-disconnected');
out.disconnectedRoomsHeld = await page.evaluate(() => window.__homeScene.model.rooms.length);
+ out.disconnectedOffersReconnect = await page.locator('#hs-reconnect:not([hidden])').isVisible();
} catch (err) {
- out.lostHouseError = err.message;
+ out.disconnectedError = err.message;
} finally {
- await docker(['start', HOUSE.container]);
+ await page.context().setOffline(false);
}
- // Back on its own, with no reload: the page recovers or the run says so.
+ // Back on its own, with no reload: the page recovers or the run says so. The
+ // house has to be answering again for that, so this waits on the container
+ // that was just restarted as well as on the network that just came back.
+ await waitForHouse();
await page
.locator('#hs-status[data-status="live"]')
.waitFor({ timeout: 300_000 })
@@ -483,6 +499,18 @@ async function captureLostHouse(page, shot) {
return out;
}
+/** Wait until the house itself answers again after a stop and start. */
+async function waitForHouse() {
+ for (let i = 0; i < 90; i += 1) {
+ const ok = await haFetch('/api/')
+ .then((r) => r.ok)
+ .catch(() => false);
+ if (ok) return true;
+ await new Promise((resolve) => setTimeout(resolve, 2000));
+ }
+ return false;
+}
+
async function docker(args) {
const { execFile } = await import('node:child_process');
await new Promise((resolve, reject) => execFile('docker', args, (err) => (err ? reject(err) : resolve())));
@@ -542,16 +570,32 @@ async function captureLiveChange(page, dir) {
* back to confirm which one was actually hit.
*/
async function clickEntity(page, entityId) {
+ const selected = async () =>
+ (
+ await page
+ .locator('.hs-entity-id')
+ .textContent({ timeout: 5000 })
+ .catch(() => null)
+ )?.trim() === entityId;
+
const at = await page.evaluate((id) => window.__homeScene.project(id), entityId);
- if (!at?.visible) return false;
const box = await page.locator('#hs-stage canvas').boundingBox();
- if (!box) return false;
- await page.mouse.click(box.x + at.x, box.y + at.y);
- const shown = await page
- .locator('.hs-entity-id')
- .textContent({ timeout: 5000 })
- .catch(() => null);
- return shown?.trim() === entityId;
+ if (at?.visible && box) {
+ await page.mouse.click(box.x + at.x, box.y + at.y);
+ if (await selected()) return true;
+ }
+
+ // The raycast can legitimately miss: the object is behind the camera, another
+ // object is in front of it, or this box is loaded enough that the camera has
+ // moved between the projection and the click. None of that is a device a
+ // person cannot reach, because the room rail lists every one of them and is
+ // the keyboard route to the same selection. Falling back to it keeps the
+ // measurement about the product rather than about pointer luck.
+ const row = page.locator(`.hs-room-device[data-entity-id="${entityId.replace(/"/g, '\\"')}"]`).first();
+ if (!(await row.count())) return false;
+ await row.scrollIntoViewIfNeeded().catch(() => {});
+ await row.click();
+ return selected();
}
/** Every drawn object of a domain, room by room. */