Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

docs(home): register the satellite and security pages and model the relay threat

nirholasf9a8780d8056
diff --git a/data/pages.json b/data/pages.json
index 387deb436..cae0091b7 100644
--- a/data/pages.json
+++ b/data/pages.json
@@ -1025,14 +1025,14 @@
"changefreq": "weekly",
"added": "2026-09-03"
},
- {
- "path": "/smart-home/satellite",
- "title": "Voice satellite",
- "description": "Give your existing Home Assistant voice assistant a face. Pair a three.ws satellite and the same pipeline you already use gains a 3D agent that lip-syncs its answers and reacts, on any screen in the house. Home Assistant keeps the wake word, the speech recognition and the answer; closing the page never stops the voice working.",
- "priority": 0.7,
- "changefreq": "monthly",
- "added": "2026-09-03"
- },
+ {
+ "path": "/smart-home/satellite",
+ "title": "Voice satellite",
+ "description": "Give your existing Home Assistant voice assistant a face. Pair a three.ws satellite and the same pipeline you already use gains a 3D agent that lip-syncs its answers and reacts, on any screen in the house. Home Assistant keeps the wake word, the speech recognition and the answer; closing the page never stops the voice working.",
+ "priority": 0.7,
+ "changefreq": "monthly",
+ "added": "2026-09-03"
+ },
{
"path": "/inspect",
"title": "Model Inspector",
@@ -2680,12 +2680,20 @@
},
{
"path": "/docs/home-security",
- "title": "Docs \u00b7 Home security",
+ "title": "Docs · Home security",
"description": "The threat model for a connected house, written about a building rather than a database: what each actor can do and what stops them, why the confirmation gate is the load-bearing control and prompt filtering is depth, the eleven checks that run with every build, the SSRF guard every server-side dial goes through and why a hostname check is not one, the prompt-injection proof that ends by asking a real lock whether it opened, and the residuals accepted on purpose.",
"priority": 0.6,
"changefreq": "monthly",
"added": "2026-09-03"
},
+ {
+ "path": "/docs/home-relay-threat-model",
+ "title": "Docs · Home relay threat model",
+ "description": "What an attacker gets at every position around the socket a LAN-only house dials out through, and what stops them: why the relay carries a fixed list of message types instead of proxying HTTP into a home network, what a stolen install token is and is not worth, why the allowlist is enforced again inside the house so a compromised relay widens nothing, and the residuals accepted on purpose.",
+ "priority": 0.6,
+ "changefreq": "monthly",
+ "added": "2026-09-03"
+ },
{
"path": "/docs/home-privacy",
"title": "Docs · Home privacy and retention",
diff --git a/docs/home-relay-threat-model.md b/docs/home-relay-threat-model.md
new file mode 100644
index 000000000..ce7bf3c61
--- /dev/null
+++ b/docs/home-relay-threat-model.md
@@ -0,0 +1,143 @@
+# The home relay: threat model
+
+**What an attacker gets, at every position around the socket a LAN-only house dials out through,
+and what stops them.**
+
+This is the document [`services/home-relay/src/server.js`](../services/home-relay/src/server.js),
+[`protocol.js`](../services/home-relay/src/protocol.js) and
+[`api/home/pair.js`](../api/home/pair.js) tell you to read before changing them. It covers the
+relay specifically. The platform-wide model, the prompt-injection boundary and the abuse surface
+are in [home-security.md](home-security.md).
+
+## What the relay is
+
+Most Home Assistant installs answer only on their own network. three.ws is served over https from
+Cloud Run and cannot route to RFC1918 space, so for those houses there is no URL we could ever
+dial. The house dials us instead: the three.ws integration inside it opens **one outbound
+WebSocket** to [`services/home-relay`](../services/home-relay/README.md) and keeps it. The
+platform then asks the relay to open a session over that socket.
+
+Nothing listens on the user's network. No port is forwarded. No inbound firewall rule exists.
+
+```
+ three.ws api ──WS /v1/bridge──► home-relay ◄──WS /v1/agent── Home Assistant
+ (platform) (this service) (the house)
+```
+
+## The design decision the whole model rests on
+
+**The obvious implementation would be the worst thing this platform could ship.** "Forward any
+HTTP request into the LAN" is a caller-chosen path into somebody's home network, and no amount of
+authentication makes that safe, because the authenticated caller is exactly who you have to worry
+about once anything upstream is compromised.
+
+So the relay is not a proxy. It carries a **fixed, enumerated set** of Home Assistant WebSocket
+message types and refuses everything else. The set is derived from what
+[`packages/home-bridge`](../packages/home-bridge/README.md) actually sends, which is what the
+product's three channels need: the entity subscription, four registry reads, and service calls.
+Nothing else has ever been needed, so nothing else is permitted.
+
+## The five positions, and what each one gets
+
+### 1. A stranger on the internet
+
+Gets nothing. There is no inbound path to the house at all, and the relay's own endpoints require
+either a signed install token (`/v1/agent`) or the platform's service token (`/v1/bridge`).
+
+### 2. Someone who steals an install token
+
+An install token (`hr1.<payload>.<hmac>`, see [`token.js`](../services/home-relay/src/token.js))
+identifies which house is dialling in. It is minted by three.ws at pairing time, stored by the
+integration, and presented on every dial-in. The relay verifies the HMAC with the shared signing
+key and reads the relay id out of the payload, so it stays **stateless about ownership**: it never
+queries a database on the connect path, and it cannot be tricked into binding a socket to a relay
+id its holder was not granted.
+
+A stolen token gets an attacker a socket to the relay. It does not get them a house:
+
+- The relay only ever joins that socket to **the owning user's own sessions**.
+- The physical-action gate still sits above every one of those sessions, so a guarded action still
+ needs a person.
+- The token is **not** a Home Assistant credential. In relay mode three.ws stores no Home
+ Assistant token at all (`home_connections.access_token_enc` is empty), because the integration
+ authenticates to Home Assistant locally with a refresh token it mints for itself and never sends
+ off the machine.
+
+Revocation is push-based and immediate: three.ws calls `POST /v1/revoke` when a home is revoked,
+the relay drops the socket and denies the relay id. The token deliberately carries no expiry the
+user would have to re-pair around, because it is not the last line of defence: the platform
+re-reads the connection row on every bridge connect, so a revoked home never gets a session opened
+even if the relay were to forget the denial.
+
+### 3. A compromised or buggy platform-side caller
+
+This is the position the allowlist in the relay exists for. Even holding the service token, a
+platform caller can send only the enumerated outbound types, and:
+
+- `subscribe_events` is limited to `state_changed`. Anything else "would subscribe to more of the
+ house than the room graph needs", and is refused by name.
+- `call_service` is refused outright for `shell_command`, `python_script`, `hassio`, `supervisor`,
+ `backup`, `update`, `cloud`, `config`, `auth` and `command_line`, plus the named services that
+ survive a permitted domain: `homeassistant.restart`, `homeassistant.stop`,
+ `homeassistant.check_config`, `homeassistant.reload_all`, `homeassistant.reload_core_config`,
+ `persistent_notification.create`.
+- Only `result`, `event` and `pong` come back out of a house.
+
+Rate limits are per install and cover both frame rate and actuation rate (40 outbound frames per
+second, 60 service calls per minute, 8 concurrent sessions, 4 MB per frame): a bug in an
+integration must not be able to flood a house, and a compromised platform caller must not be able
+to hammer a lock.
+
+### 4. A compromised relay
+
+**This is the position that matters most, because the relay is operated by us and is therefore
+exactly the component a user has to trust least.**
+
+The allowlist is enforced **twice, independently**: once in the relay
+([`server.js`](../services/home-relay/src/server.js)) so a bad platform caller cannot reach past
+it, and again in the integration inside the house (`relay_client.py`) so a bad relay cannot
+either. Compromising the relay does not widen what may enter a house by one message type.
+
+Both enforcement points read the same enumeration rather than transcribing it:
+[`protocol.js`](../services/home-relay/src/protocol.js) is the source of truth,
+[`allowlist.json`](../services/home-relay/allowlist.json) is the generated copy the Python side
+reads, and `tests/home-relay-protocol.test.js` fails if the checked-in copy is stale. Two
+enforcement points that drift apart would be worse than one.
+
+The relay also holds nothing worth stealing: no database, no Home Assistant credential, and no
+persistent record of what passed through it.
+
+### 5. Someone inside the house
+
+Out of scope, and honestly so. Anyone with access to the Home Assistant instance can already do
+everything the relay could ever ask for, directly, without us.
+
+## What is deliberately accepted
+
+- **The relay sees traffic in transit.** It terminates both sockets, so it can read the state and
+ service calls that cross it. It cannot read a Home Assistant credential, because none crosses,
+ and it cannot widen the message set, because the far end enforces the list again. Encrypting
+ end to end past the relay would require the integration and the platform to share a key the
+ relay never sees, which is worth doing and is not done today.
+- **The install token has no expiry.** See position 2 for why: revocation is push-based and the
+ platform re-checks the connection row on every session.
+- **A house can lie about its own state.** An instance that reports a door as locked when it is
+ not is a compromised house, and no relay design fixes that.
+
+## The invariant
+
+**Nothing on this path may open a building without a person.** The relay does not implement the
+gate, and it must never be given a way to bypass one: a service call arriving over `/v1/bridge`
+has already been through
+[`api/_lib/home/tools.js`](../api/_lib/home/tools.js) and, if guarded, has already been redeemed
+by a signed-in human at `POST /api/home/:id/confirm`. If a change to this service would let a call
+reach a house without having passed that, the change is wrong.
+
+## Read next
+
+- [`services/home-relay/README.md`](../services/home-relay/README.md): running it, its endpoints,
+ and regenerating the allowlist.
+- [home-security.md](home-security.md): the platform threat model, the injection boundary, the
+ eleven checks.
+- [smart-home.md](smart-home.md): why the reachability problem exists and the three answers to it.
+- [tutorials/connect-your-home.md](tutorials/connect-your-home.md): the user-facing walkthrough.
diff --git a/scripts/home-load.mjs b/scripts/home-load.mjs
index 788132aa7..a8fc817c2 100644
--- a/scripts/home-load.mjs
+++ b/scripts/home-load.mjs
@@ -673,10 +673,16 @@ async function measureCloudRunStream({ url = 'https://three.ws/api/pump/trades-s
* hundred would ever fire, and the ladder would be decoration.
*
* It is measured rather than reasoned about, against the running service, using
- * a streaming endpoint already deployed there. Cloud Run autoscales on
- * concurrency, so if streams consume slots the instance count rises with them
- * and falls again afterwards. The instance count comes from Cloud Monitoring,
- * which is the platform's own accounting rather than ours.
+ * a streaming endpoint already deployed there. Cloud Run publishes the observed
+ * concurrency per instance as `container/max_request_concurrencies`, which is
+ * the platform's own accounting of what it counts as an in-flight request. If
+ * holding N streams moves that number, a stream is a concurrency slot; if it
+ * does not, a stream is free and the cap can be sized on memory alone.
+ *
+ * Reading that metric rather than watching for a scaling event is deliberate:
+ * forcing this service to autoscale would take more than a thousand concurrent
+ * streams, which is a load test on production, and the concurrency gauge answers
+ * the same question with two hundred.
*
* @param {{ url?: string, streams?: number, seconds?: number, project?: string, service?: string }} options
*/
@@ -687,7 +693,10 @@ async function measureStreamConcurrency({
project = 'aerial-vehicle-466722-p5',
service = 'three-ws-api',
} = {}) {
- const before = await readInstanceCount({ project, service, minutesBack: 20 });
+ const before = {
+ concurrency: await readConcurrency({ project, service, minutesBack: 20 }),
+ instances: await readInstanceCount({ project, service, minutesBack: 20 }),
+ };
const controllers = [];
let opened = 0;
@@ -724,29 +733,78 @@ async function measureStreamConcurrency({
}
await sleep(Math.max(0, seconds * 1000 - (Date.now() - started)));
- const during = await readInstanceCount({ project, service, minutesBack: 8 });
+ const during = {
+ concurrency: await readConcurrency({ project, service, minutesBack: 6 }),
+ instances: await readInstanceCount({ project, service, minutesBack: 6 }),
+ };
clearInterval(keepAlive);
for (const controller of controllers) controller.abort();
- const delta = during.peakActive != null && before.peakActive != null ? during.peakActive - before.peakActive : null;
+ const rise = during.concurrency.peak != null && before.concurrency.peak != null
+ ? Number((during.concurrency.peak - before.concurrency.peak).toFixed(1))
+ : null;
+ const instancesDuring = during.instances.peakActive ?? null;
return {
url,
streamsRequested: streams,
streamsOpened: opened,
streamsRefused: failed,
heldSeconds: seconds,
- instancesBefore: before,
- instancesDuring: during,
- peakActiveDelta: delta,
- streamsPerAddedInstance: delta && delta > 0 ? Number((opened / delta).toFixed(1)) : null,
- occupiesConcurrencySlot: delta != null ? delta > 0 : null,
- note: 'Cloud Run autoscales on concurrent requests. An instance count that rises while N streams are held, and only while they are held, is the platform saying each stream is an in-flight request holding a concurrency slot.',
+ concurrencyBefore: before.concurrency,
+ concurrencyDuring: during.concurrency,
+ concurrencyRise: rise,
+ instancesBefore: before.instances,
+ instancesDuring: during.instances,
+ expectedRiseIfStreamsHoldSlots: instancesDuring ? Number((opened / instancesDuring).toFixed(1)) : null,
+ occupiesConcurrencySlot: rise != null ? rise > 2 : null,
+ note: 'Cloud Run counts an in-flight request against containerConcurrency. Holding N streams and watching the platform own concurrency gauge rise by roughly N divided by the instance count is the platform saying a held stream is one of those requests.',
+ };
+}
+
+/** Observed concurrent requests per instance, from Cloud Run's own gauge. */
+async function readConcurrency({ project, service, minutesBack }) {
+ const series = await readTimeSeries({
+ project,
+ service,
+ metric: 'run.googleapis.com/container/max_request_concurrencies',
+ minutesBack,
+ aligner: 'ALIGN_PERCENTILE_99',
+ });
+ if (series.error) return { error: series.error, peak: null };
+ const values = (series.timeSeries[0]?.points || []).map((p) => Number(p.value.doubleValue ?? p.value.int64Value ?? 0));
+ if (!values.length) return { peak: null, points: 0 };
+ return {
+ windowMinutes: minutesBack,
+ points: values.length,
+ peak: Number(Math.max(...values).toFixed(1)),
+ median: Number(values.slice().sort((a, b) => a - b)[Math.floor(values.length / 2)].toFixed(1)),
};
}
/** Cloud Run instance counts, straight from Cloud Monitoring. */
async function readInstanceCount({ project, service, minutesBack }) {
+ const series = await readTimeSeries({
+ project,
+ service,
+ metric: 'run.googleapis.com/container/instance_count',
+ minutesBack,
+ aligner: 'ALIGN_MEAN',
+ });
+ if (series.error) return { error: series.error, peakActive: null };
+ const body = series;
+
+ const byState = {};
+ for (const series of body.timeSeries || []) {
+ const state = series.metric.labels.state;
+ const values = series.points.map((p) => Number(p.value.doubleValue ?? p.value.int64Value ?? 0));
+ byState[state] = { peak: Math.max(...values, 0), mean: Number((values.reduce((a, b) => a + b, 0) / Math.max(1, values.length)).toFixed(2)), points: values.length };
+ }
+ return { windowMinutes: minutesBack, byState, peakActive: byState.active?.peak ?? null };
+}
+
+/** One Cloud Monitoring read, with the access token gcloud already holds. */
+async function readTimeSeries({ project, service, metric, minutesBack, aligner }) {
const token = await new Promise((resolve, reject) => {
const child = spawn('gcloud', ['auth', 'print-access-token'], { stdio: ['ignore', 'pipe', 'ignore'] });
let out = '';
@@ -758,25 +816,18 @@ async function readInstanceCount({ project, service, minutesBack }) {
const end = new Date();
const start = new Date(end.getTime() - minutesBack * 60_000);
const params = new URLSearchParams({
- filter: `metric.type="run.googleapis.com/container/instance_count" AND resource.labels.service_name="${service}"`,
+ filter: `metric.type="${metric}" AND resource.labels.service_name="${service}"`,
'interval.startTime': start.toISOString(),
'interval.endTime': end.toISOString(),
'aggregation.alignmentPeriod': '60s',
- 'aggregation.perSeriesAligner': 'ALIGN_MEAN',
+ 'aggregation.perSeriesAligner': aligner,
});
const response = await fetch(`https://monitoring.googleapis.com/v3/projects/${project}/timeSeries?${params}`, {
headers: { authorization: `Bearer ${token}` },
});
const body = await response.json();
- if (body.error) return { error: body.error.message, peakActive: null };
-
- const byState = {};
- for (const series of body.timeSeries || []) {
- const state = series.metric.labels.state;
- const values = series.points.map((p) => Number(p.value.doubleValue ?? p.value.int64Value ?? 0));
- byState[state] = { peak: Math.max(...values, 0), mean: Number((values.reduce((a, b) => a + b, 0) / Math.max(1, values.length)).toFixed(2)), points: values.length };
- }
- return { windowMinutes: minutesBack, byState, peakActive: byState.active?.peak ?? null };
+ if (body.error) return { error: body.error.message };
+ return { timeSeries: body.timeSeries || [] };
}
/** Each rung of the ladder, demonstrated rather than described. */
diff --git a/services/home-satellite/scripts/pipeline-run.mjs b/services/home-satellite/scripts/pipeline-run.mjs
new file mode 100755
index 000000000..21c8cc070
--- /dev/null
+++ b/services/home-satellite/scripts/pipeline-run.mjs
@@ -0,0 +1,229 @@
+#!/usr/bin/env node
+/**
+ * End-to-end proof: speak a sentence into a real Home Assistant pipeline
+ * through this satellite, and print what came back.
+ *
+ * It plays the part of the browser. It connects to the satellite's viewer
+ * WebSocket exactly as src/home/satellite.js does, streams microphone audio
+ * exactly as the browser's capture would, and receives the same events and the
+ * same text-to-speech audio the avatar lip-syncs to. Nothing about the path is
+ * simulated: Home Assistant runs its own speech recognition, its own intent
+ * handling and its own text to speech, and the house really changes.
+ *
+ * The "microphone audio" is real speech, synthesized by the same piper
+ * container Home Assistant uses for its answers, over the same Wyoming protocol
+ * this service implements. There is no recorded fixture and no canned
+ * transcript: whisper hears audio it has never seen and transcribes it.
+ *
+ * node scripts/pipeline-run.mjs \
+ * --viewer ws://127.0.0.1:10701/viewer --token "$(node src/index.js token)" \
+ * --piper 127.0.0.1:10200 \
+ * --say "turn on the kitchen lights" \
+ * --ha http://127.0.0.1:8123 --ha-token <token> --watch light.kitchen_lights
+ */
+
+import { connect } from 'node:net';
+import { writeFileSync } from 'node:fs';
+import { WebSocket } from 'ws';
+
+import { EventDecoder, encodeEvent, EVENT, readAudioFormat } from '../src/protocol.js';
+
+const args = process.argv.slice(2);
+const arg = (name, fallback) => {
+ const i = args.indexOf(`--${name}`);
+ return i === -1 ? fallback : args[i + 1];
+};
+
+const VIEWER = arg('viewer', 'ws://127.0.0.1:10701/viewer');
+const TOKEN = arg('token', process.env.SATELLITE_VIEWER_TOKEN || '');
+const PIPER = arg('piper', '127.0.0.1:10200');
+const PHRASE = arg('say', 'turn on the kitchen lights');
+const HA = arg('ha', '').replace(/\/+$/, '');
+const HA_TOKEN = arg('ha-token', process.env.HOME_ASSISTANT_TOKEN || '');
+const WATCH = arg('watch', '');
+const OUT = arg('out', '');
+const MODE = arg('mode', 'command');
+
+const log = (...a) => console.error('[run]', ...a);
+
+/** Ask piper for speech, over the same Wyoming protocol this service speaks. */
+function synthesize(hostPort, text) {
+ const [host, port] = hostPort.split(':');
+ return new Promise((resolve, reject) => {
+ const socket = connect(Number(port), host, () => {
+ socket.write(encodeEvent({ type: EVENT.SYNTHESIZE, data: { text } }));
+ });
+ const decoder = new EventDecoder();
+ const chunks = [];
+ let format = null;
+ socket.on('data', (data) => {
+ for (const event of decoder.push(data)) {
+ if (event.type === EVENT.AUDIO_START) format = readAudioFormat(event.data);
+ else if (event.type === EVENT.AUDIO_CHUNK) {
+ if (!format) format = readAudioFormat(event.data);
+ if (event.payload) chunks.push(event.payload);
+ } else if (event.type === EVENT.AUDIO_STOP) {
+ socket.end();
+ resolve({ format, audio: Buffer.concat(chunks) });
+ } else if (event.type === EVENT.ERROR) {
+ socket.end();
+ reject(new Error(`piper: ${event.data?.text}`));
+ }
+ }
+ });
+ socket.on('error', reject);
+ socket.setTimeout(30_000, () => {
+ socket.destroy();
+ reject(new Error('piper did not answer in 30s'));
+ });
+ });
+}
+
+/** Linear resample of interleaved mono s16 to 16 kHz. */
+function to16k(audio, rate) {
+ if (rate === 16000) return audio;
+ const inSamples = audio.length / 2;
+ const ratio = rate / 16000;
+ const outSamples = Math.floor(inSamples / ratio);
+ const out = Buffer.alloc(outSamples * 2);
+ for (let i = 0; i < outSamples; i += 1) {
+ const pos = i * ratio;
+ const idx = Math.floor(pos);
+ const frac = pos - idx;
+ const a = audio.readInt16LE(Math.min(idx, inSamples - 1) * 2);
+ const b = audio.readInt16LE(Math.min(idx + 1, inSamples - 1) * 2);
+ out.writeInt16LE(Math.round(a + (b - a) * frac), i * 2);
+ }
+ return out;
+}
+
+function wav(pcm, rate) {
+ const header = Buffer.alloc(44);
+ header.write('RIFF', 0);
+ header.writeUInt32LE(36 + pcm.length, 4);
+ header.write('WAVE', 8);
+ header.write('fmt ', 12);
+ header.writeUInt32LE(16, 16);
+ header.writeUInt16LE(1, 20);
+ header.writeUInt16LE(1, 22);
+ header.writeUInt32LE(rate, 24);
+ header.writeUInt32LE(rate * 2, 28);
+ header.writeUInt16LE(2, 32);
+ header.writeUInt16LE(16, 34);
+ header.write('data', 36);
+ header.writeUInt32LE(pcm.length, 40);
+ return Buffer.concat([header, pcm]);
+}
+
+async function haState(entity) {
+ if (!HA || !HA_TOKEN || !entity) return null;
+ const res = await fetch(`${HA}/api/states/${entity}`, { headers: { authorization: `Bearer ${HA_TOKEN}` } });
+ if (!res.ok) return null;
+ return (await res.json()).state;
+}
+
+const main = async () => {
+ if (!TOKEN) throw new Error('a viewer token is required: --token, or SATELLITE_VIEWER_TOKEN');
+
+ log(`asking piper for "${PHRASE}"`);
+ const speech = await synthesize(PIPER, PHRASE);
+ const micPcm = to16k(speech.audio, speech.format?.rate || 22050);
+ log(`got ${(micPcm.length / 32000).toFixed(2)}s of speech at 16 kHz`);
+
+ const before = await haState(WATCH);
+ if (WATCH) log(`${WATCH} before: ${before}`);
+
+ const events = [];
+ const ttsChunks = [];
+ let ttsFormat = null;
+ let transcript = null;
+ let answer = null;
+
+ const socket = new WebSocket(`${VIEWER}?token=${encodeURIComponent(TOKEN)}`);
+ socket.binaryType = 'arraybuffer';
+
+ const finished = new Promise((resolve, reject) => {
+ const done = () => resolve();
+ socket.on('open', async () => {
+ log('viewer socket open');
+ socket.send(JSON.stringify({ t: 'mic-start', mode: MODE }));
+ // Stream in real-time-sized chunks. A pipeline with voice activity
+ // detection needs the audio to arrive at roughly the rate a microphone
+ // would produce it; firing the whole utterance in one write makes the
+ // far end decide the speaker stopped before they started.
+ const CHUNK = 1024 * 2; // 1024 samples, 64ms
+ for (let offset = 0; offset < micPcm.length; offset += CHUNK) {
+ socket.send(micPcm.subarray(offset, offset + CHUNK));
+ await new Promise((r) => setTimeout(r, 55));
+ }
+ // A tail of silence: Home Assistant's voice activity detection needs to
+ // hear the end of the sentence, and speech that stops at the last
+ // syllable reads as speech that is still going.
+ const silence = Buffer.alloc(CHUNK);
+ for (let i = 0; i < 12; i += 1) {
+ socket.send(silence);
+ await new Promise((r) => setTimeout(r, 55));
+ }
+ socket.send(JSON.stringify({ t: 'mic-stop' }));
+ log('finished streaming');
+ });
+ socket.on('message', (raw, isBinary) => {
+ if (isBinary) {
+ ttsChunks.push(Buffer.from(raw));
+ return;
+ }
+ const message = JSON.parse(raw.toString('utf8'));
+ events.push(message);
+ if (message.t === 'state') log(`state: ${message.state}${message.detail ? ` (${message.detail})` : ''}`);
+ if (message.t === 'transcript' && message.final) {
+ transcript = message.text;
+ log(`transcript: ${JSON.stringify(message.text)}`);
+ }
+ if (message.t === 'speech') {
+ answer = message.text;
+ log(`answer: ${JSON.stringify(message.text)}`);
+ }
+ if (message.t === 'audio-start') {
+ ttsFormat = { rate: message.rate, width: message.width, channels: message.channels };
+ log(`speaking at ${message.rate} Hz`);
+ }
+ if (message.t === 'audio-stop') {
+ socket.send(JSON.stringify({ t: 'played' }));
+ setTimeout(done, 400);
+ }
+ if (message.t === 'error') log(`pipeline error: ${message.code} ${message.text}`);
+ });
+ socket.on('error', reject);
+ socket.on('close', (code, reason) => {
+ if (code !== 1000 && code !== 1005) reject(new Error(`viewer socket closed ${code} ${reason}`));
+ else done();
+ });
+ setTimeout(() => reject(new Error('no answer within 90s')), 90_000);
+ });
+
+ await finished;
+ socket.close();
+
+ const after = await haState(WATCH);
+ const ttsAudio = Buffer.concat(ttsChunks);
+ if (OUT && ttsAudio.length) {
+ writeFileSync(OUT, wav(ttsAudio, ttsFormat?.rate || 22050));
+ log(`wrote ${OUT} (${(ttsAudio.length / ((ttsFormat?.rate || 22050) * 2)).toFixed(2)}s)`);
+ }
+
+ console.log(JSON.stringify({
+ said: PHRASE,
+ transcript,
+ answer,
+ tts_bytes: ttsAudio.length,
+ tts_format: ttsFormat,
+ states: events.filter((e) => e.t === 'state').map((e) => e.state),
+ watched: WATCH ? { entity: WATCH, before, after } : null,
+ }, null, '\t'));
+ process.exit(0);
+};
+
+main().catch((err) => {
+ console.error(`[run] failed: ${err.message}`);
+ process.exit(1);
+});