Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
JavaScript 3d-generationai-agentsanimationavatardigital-humangenerative-aiglbgltfimage-to-3dllmmcpmcp-servermodel-context-protocolsolanatext-to-3dthreejsweb-componentswebglwebxrx402
test(home): fire all three home alerts, and write the 48-hour launch watch
nirholasd063a6f86b4d
diff --git a/docs/ops/home-operations.md b/docs/ops/home-operations.mdindex 44a7b1525..acea3709a 100644--- a/docs/ops/home-operations.md+++ b/docs/ops/home-operations.md@@ -1079,3 +1079,30 @@ limit 50;That last query is the one an operator owes a user who asks "what did my agent doin my house last Tuesday", and it is why the action log is its own table ratherthan rows in the shared `audit_log`.++## The first 48 hours after launch++Written before launch, not after. The lane's alerts are tuned to catch a+correlated outage, and none of them catch the thing a first launch actually+produces: a slow drift that never trips a threshold. These five checks are the+manual watch that covers the gap, and each one is a command rather than a+feeling.++| Hour | Check | Command | What "fine" looks like |+|---|---|---|---|+| 0 | The `home` block is real, and the first connection and first action landed | `curl -s https://three.ws/api/healthz \| node -e "let s='';process.stdin.on('data',d=>s+=d).on('end',()=>console.log(JSON.stringify(JSON.parse(s).subsystems.home,null,1)))"` | status `ok`, `homes.connected` climbing off zero, `actions.total` non-zero |+| 1 | Handshake success across tenants, and how many confirmations expire unanswered | the correlation query above, plus `confirmations` on the health block | handshake rate at 1, expiry rate under 0.2 |+| 6 | Heap trend on the API service and subscriber counts per instance | `gcloud logging read 'resource.type="cloud_run_revision" resource.labels.service_name="three-ws-api" textPayload:"home-runtime"' --freshness=6h --limit=50` | subscribers tracking open connections, not climbing past them |+| 24 | p95 action latency against the SLO, and the action-log integrity query | `actions.p95LatencyMs` on the health block, then the integrity query in alert 2 | p95 under 1500 ms, integrity violations 0 |+| 48 | The alert set has not fired spuriously, and the error budget spent so far | the ops alert channel, plus the SLO table above | zero pages that were not real, budget spend tracking below linear |++Two notes that matter more than the table.++**The hour-24 p95 is the one to read carefully.** A refusal never reaches the+timing, so a window where most actions were guarded and refused reports a p95+over very few samples. Read `actions.total` next to it: a p95 over fewer than+twenty timed actions is a sample, not a verdict, and the SLO is a 30-day number.++**A quiet alert channel at hour 48 is not proof the alerts work.** That proof is+`tests/cron-home-health-alert.test.js`, which fires all three against the real+cron. If the channel is quiet and that suite is red, believe the suite.diff --git a/tests/cron-home-health-alert.test.js b/tests/cron-home-health-alert.test.jsnew file mode 100644index 000000000..55a1d5424--- /dev/null+++ b/tests/cron-home-health-alert.test.js@@ -0,0 +1,235 @@+// The three home alerts, actually fired (api/cron/home-health-alert.js).+//+// This cron is the only thing that pages a human about the smart-home lane, and+// until this file existed nothing anywhere invoked it. Every property below was+// therefore live in production on the strength of a code read alone, including+// the one that matters most: an unlock that executed with nobody's yes on record+// pages on a single row, with no threshold and no dedup that could swallow it.+//+// The split this file pins is the lane's whole alerting thesis: a per-tenant+// failure NEVER pages (one unplugged router, one expired token), and a+// correlated one always does. Both halves are asserted, because an alert that+// cannot stay quiet is as broken as one that cannot fire.+import { test, expect, vi, beforeEach } from 'vitest';++const sent = [];+const cache = new Map();+let signals = null;+let leakInstances = [];++vi.mock('../api/_lib/alerts.js', () => ({+ sendOpsAlert: async (title, detail, opts) => { sent.push({ title, detail, opts }); },+}));+vi.mock('../api/_lib/cron-auth.js', () => ({ requireCron: () => true }));+vi.mock('../api/_lib/cache.js', () => ({+ cacheGet: async (k) => (cache.has(k) ? cache.get(k) : null),+ cacheSet: async (k, v) => { cache.set(k, v); },+}));+// importOriginal, never a hand-listed export map: the thresholds this cron reads+// (HANDSHAKE_DOWN, MIN_HOMES_FOR_A_VERDICT, WINDOW_MINUTES) must be the real+// shipped numbers, and only the two IO functions are replaced.+vi.mock('../api/_lib/ops/home-health.js', async (importOriginal) => ({+ ...(await importOriginal()),+ readHomeSignals: async () => signals,+ readLeakInstances: async () => leakInstances,+}));++const handler = (await import('../api/cron/home-health-alert.js')).default;+const { HANDSHAKE_DOWN, MIN_HOMES_FOR_A_VERDICT, WINDOW_MINUTES } = await import('../api/_lib/ops/home-health.js');++function makeRes() {+ return {+ statusCode: 0,+ body: null,+ headersSent: false,+ writableEnded: false,+ setHeader() {}, getHeader() {},+ status(code) { this.statusCode = code; return this; },+ end(payload) {+ this.writableEnded = true;+ if (payload && this.body === null) {+ try { this.body = JSON.parse(payload); } catch { this.body = payload; }+ }+ return this;+ },+ json(payload) { this.body = payload; return this; },+ };+}++/** A house that is entirely healthy: every alert must stay silent on this. */+function quiet(overrides = {}) {+ return {+ windowMinutes: WINDOW_MINUTES,+ homes: { live: 40, connected: 39, unreachable: 1, authFailed: 0 },+ handshakes: { attempts: 40, failed: 0, rate: 1 },+ actions: { total: 100, failed: 1, p95LatencyMs: 900 },+ confirmations: { offered: 10, redeemed: 8, expired: 2 },+ integrity: { violations: 0, lastAt: null, grantBacked: 0, grantBackedWithoutGrant: 0, confirmationScrubbed: 0 },+ ...overrides,+ };+}++async function tick() {+ const res = makeRes();+ await handler({ method: 'GET', headers: {}, url: '/api/cron/home-health-alert' }, res);+ return res;+}++beforeEach(() => {+ sent.length = 0;+ cache.clear();+ leakInstances = [];+ signals = quiet();+});++test('a healthy lane fires nothing at all', async () => {+ const res = await tick();+ expect(res.statusCode).toBe(200);+ expect(res.body.ok).toBe(true);+ expect(res.body.fired).toEqual([]);+ expect(sent).toEqual([]);+});++test('alert 2 fires on a SINGLE unconfirmed guarded action, at critical', async () => {+ signals = quiet({ integrity: { violations: 1, lastAt: '2026-09-09T18:04:11.000Z', grantBacked: 0, grantBackedWithoutGrant: 0, confirmationScrubbed: 0 } });++ const res = await tick();++ expect(res.body.fired).toContain('confirmation_integrity');+ expect(sent).toHaveLength(1);+ const [alert] = sent;+ expect(alert.opts.severity).toBe('critical');+ // The word the operator has to see, and the query that finds the rows.+ expect(alert.title).toMatch(/guarded home action\(s\) executed with no confirmation/);+ expect(alert.detail).toContain('2026-09-09T18:04:11.000Z');+ expect(alert.detail).toContain("where guarded = true and confirmed_by is null and outcome = 'ok'");+ expect(alert.detail).toContain('docs/ops/home-operations.md');+});++test('alert 2 has no dedup window a second incident could fall into', async () => {+ signals = quiet({ integrity: { violations: 1, lastAt: '2026-09-09T18:04:11.000Z', grantBacked: 0, grantBackedWithoutGrant: 0, confirmationScrubbed: 0 } });+ await tick();+ const first = sent[0].opts.signature;++ // An hour later, a second door opens. The signature carries the newest+ // violation's timestamp precisely so this cannot be swallowed as a repeat.+ signals = quiet({ integrity: { violations: 2, lastAt: '2026-09-09T19:11:02.000Z', grantBacked: 0, grantBackedWithoutGrant: 0, confirmationScrubbed: 0 } });+ await tick();++ expect(sent).toHaveLength(2);+ expect(sent[1].opts.signature).not.toBe(first);+ expect(sent[1].opts.signature).toContain('2026-09-09T19:11:02.000Z');+});++test('alert 1 stays silent for one person unplugging their router', async () => {+ // Three homes, all three down: a 0% rate, far under the threshold, and still+ // nobody is paged because three houses cannot be a verdict about us.+ const attempts = MIN_HOMES_FOR_A_VERDICT - 1;+ signals = quiet({+ homes: { live: attempts, connected: 0, unreachable: attempts, authFailed: 0 },+ handshakes: { attempts, failed: attempts, rate: 0 },+ });++ const res = await tick();++ expect(res.body.fired).toEqual([]);+ expect(sent).toEqual([]);+});++test('alert 1 fires when enough distinct homes fail at once, then re-escalates hourly rather than every tick', async () => {+ const attempts = MIN_HOMES_FOR_A_VERDICT + 12;+ const failed = Math.ceil(attempts * 0.9);+ const down = () => quiet({+ homes: { live: attempts, connected: attempts - failed, unreachable: failed, authFailed: 0 },+ handshakes: { attempts, failed, rate: 1 - failed / attempts },+ });+ expect(1 - failed / attempts).toBeLessThan(HANDSHAKE_DOWN);++ signals = down();+ const first = await tick();+ expect(first.body.fired).toContain('correlated_unreachability');+ expect(sent).toHaveLength(1);+ expect(sent[0].opts.severity).toBe('critical');+ expect(sent[0].title).toMatch(/Home handshakes failing across tenants/);+ expect(sent[0].detail).toContain('curl -s https://three.ws/api/version');++ // Ticks 2 through 11 are the same outage: page once, not every five minutes.+ for (let i = 0; i < 10; i += 1) await tick();+ expect(sent).toHaveLength(1);++ // The twelfth tick is an hour in and re-escalates.+ await tick();+ expect(sent).toHaveLength(2);+ expect(sent[1].detail).toContain('Still failing after 12 consecutive checks');+});++test('alert 1 sends a recovery notice, at info, once the houses come back', async () => {+ const attempts = MIN_HOMES_FOR_A_VERDICT + 12;+ signals = quiet({+ homes: { live: attempts, connected: 0, unreachable: attempts, authFailed: 0 },+ handshakes: { attempts, failed: attempts, rate: 0 },+ });+ await tick();+ sent.length = 0;++ signals = quiet({ handshakes: { attempts, failed: 0, rate: 1 } });+ const res = await tick();++ expect(res.body.fired).toContain('correlated_unreachability_recovered');+ expect(sent).toHaveLength(1);+ expect(sent[0].opts.severity).toBe('info');+ expect(sent[0].title).toMatch(/Recovered/);+ // And it recovers exactly once: a quiet lane is not a recovery every 5 minutes.+ sent.length = 0;+ await tick();+ expect(sent).toEqual([]);+});++test('alert 3 fires per leaking instance and names the climbing samples', async () => {+ leakInstances = [+ { instanceId: 'api-00420-a', leaking: true, samples: [40, 91, 173] },+ { instanceId: 'api-00420-b', leaking: false, samples: [4, 4, 5] },+ ];++ const res = await tick();++ expect(res.body.fired).toContain('subscriber_leak');+ expect(sent).toHaveLength(1);+ expect(sent[0].opts.severity).toBe('critical');+ expect(sent[0].title).toBe('Home subscriber leak on 1 instance(s)');+ expect(sent[0].detail).toContain('api-00420-a: subscribers climbed 40 then 91 then 173');+ expect(sent[0].detail).not.toContain('api-00420-b');+ expect(sent[0].opts.signature).toBe('home:leak:api-00420-a');+});++test('all three fire together on one tick without any of them swallowing another', async () => {+ const attempts = MIN_HOMES_FOR_A_VERDICT + 5;+ signals = quiet({+ homes: { live: attempts, connected: 0, unreachable: attempts, authFailed: 0 },+ handshakes: { attempts, failed: attempts, rate: 0 },+ integrity: { violations: 4, lastAt: '2026-09-09T18:04:11.000Z', grantBacked: 0, grantBackedWithoutGrant: 0, confirmationScrubbed: 0 },+ });+ leakInstances = [{ instanceId: 'api-00420-a', leaking: true, samples: [40, 91, 173] }];++ const res = await tick();++ expect(res.body.fired).toEqual(['confirmation_integrity', 'correlated_unreachability', 'subscriber_leak']);+ expect(sent).toHaveLength(3);+ expect(new Set(sent.map((s) => s.opts.signature)).size).toBe(3);+ // The integrity page is first out of the door, ahead of the noisier two.+ expect(sent[0].title).toMatch(/no confirmation/);+});++test('a grant the user revoked afterwards reports but never pages', async () => {+ // The live database carries exactly this shape: a lawful grant-backed unlock+ // whose grant row was hard-deleted by a later grant.revoke. It must not page.+ signals = quiet({+ integrity: { violations: 0, lastAt: null, grantBacked: 2, grantBackedWithoutGrant: 1, confirmationScrubbed: 0 },+ });++ const res = await tick();++ expect(res.body.fired).toEqual([]);+ expect(sent).toEqual([]);+ expect(res.body.signals.integrity.grantBackedWithoutGrant).toBe(1);+});