Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

feat(home): send three ops alerts for the house lane and never a fourth

nirholase38f809f97c9
diff --git a/CLAUDE.md b/CLAUDE.md
index 7bc33ba75..da182124f 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -305,7 +305,7 @@ The whole build + submit + purge in one command: `npm run deploy:gcp:full`.
- Frontend: vanilla JS modules + Vite (`npm run dev`, port 3000).
- 3D: Three.js with glTF/GLB.
- Backend touchpoints: serverless-style handlers in `api/`, workers in `workers/`.
-- **Production runs on Google Cloud Run, NOT Vercel** (migrated 2026-07-07 after Vercel disabled the deployment). One container ([server/index.mjs](server/index.mjs)) serves the static frontend, the vercel.json route table, and all `api/**` handlers; the crons (112 as of 2026-09-02, see vercel.json) run on Cloud Scheduler. `vercel.json` is a LIVE config file: `server/index.mjs` reads its `routes` array on boot (split at the `{handle:"filesystem"}` marker into pre- and post-filesystem phases), and `scripts/create-gcp-scheduler.mjs` reads its `crons` array to sync Cloud Scheduler jobs. The server itself never reads `crons`. Never delete `vercel.json` as a leftover. GCP builds/deploys must pin the `three-ws-build@` (build) and `three-ws@` (runtime) service accounts; the project's default compute SA was deleted. Deploys: see the "Deploy runbook" section above.
+- **Production runs on Google Cloud Run, NOT Vercel** (migrated 2026-07-07 after Vercel disabled the deployment). One container ([server/index.mjs](server/index.mjs)) serves the static frontend, the vercel.json route table, and all `api/**` handlers; the crons (113 as of 2026-09-03, see vercel.json) run on Cloud Scheduler. `vercel.json` is a LIVE config file: `server/index.mjs` reads its `routes` array on boot (split at the `{handle:"filesystem"}` marker into pre- and post-filesystem phases), and `scripts/create-gcp-scheduler.mjs` reads its `crons` array to sync Cloud Scheduler jobs. The server itself never reads `crons`. Never delete `vercel.json` as a leftover. GCP builds/deploys must pin the `three-ws-build@` (build) and `three-ws@` (runtime) service accounts; the project's default compute SA was deleted. Deploys: see the "Deploy runbook" section above.
- **Env-var trap:** `vercel env pull` returns EMPTY for secret-type vars. Never trust a Vercel env export as complete. Production env lives on the Cloud Run service (`gcloud run services describe/update three-ws-api --region us-central1`).
- Solana/agent SDKs in `sdk/`, `solana-agent-sdk/`, `agent-payments-sdk/`.
- Real APIs in use: Pump.fun feed, Solana RPC, OpenAI/Anthropic via worker proxies. Never mock these.
diff --git a/api/_lib/ops/home-health.js b/api/_lib/ops/home-health.js
index 733186aed..333b3350a 100644
--- a/api/_lib/ops/home-health.js
+++ b/api/_lib/ops/home-health.js
@@ -25,6 +25,9 @@
// Shaped after api/_lib/ops/index-lag.js: a `read*` that does the IO, a pure
// `*Verdict` that scores it, and a `gather*` that never throws.
+import { randomUUID } from 'node:crypto';
+
+import { cacheGet, cacheSet } from '../cache.js';
import { sql } from '../db.js';
import { stats as runtimeStats } from '../home/runtime.js';
@@ -68,6 +71,22 @@ const LEAK_SAMPLES = 3;
/** The rolling subscriber samples, newest last. Per process, like the pool it measures. */
const leakSamples = [];
+/**
+ * This process's identity for leak reporting.
+ *
+ * Cloud Run gives a service no per-instance environment variable, and it runs
+ * this one at minScale=6 with sessionAffinity off, so a cron tick lands on an
+ * arbitrary instance and can only ever see its own pool. A random id minted at
+ * module load IS the instance identity for this purpose: it lives exactly as
+ * long as the process whose sockets it is counting.
+ */
+const INSTANCE_ID = `${process.env.K_REVISION || 'local'}-${randomUUID().slice(0, 8)}`;
+/** Where every instance parks its rolling leak samples for the alert cron to read. */
+const LEAK_KEY = 'home:leak:instances';
+const LEAK_TTL_S = 60 * 60;
+/** An instance that has not reported inside this window is gone, not leaking. */
+const LEAK_STALE_MS = 15 * 60_000;
+
/**
* Record one subscriber sample and say whether the pool is leaking.
*
@@ -349,6 +368,53 @@ export function homeHealthVerdict(s) {
return { status, detail: parts.join('; ') + '.', ...(hint ? { hint } : {}) };
}
+/**
+ * Park this instance's leak state where the alert cron can read it.
+ *
+ * Fire and forget, and deliberately tolerant of a lost update: several instances
+ * read-modify-write the same map, so a concurrent write occasionally drops one
+ * sample. That delays a slow leak's detection by one tick and never invents one,
+ * which is the correct trade for a gauge nobody is billed on.
+ *
+ * @param {{ leaking: boolean, samples: number[], margins: number[] }} leak
+ */
+export async function publishLeakSample(leak) {
+ try {
+ const map = (await cacheGet(LEAK_KEY)) || {};
+ const now = Date.now();
+ const next = {};
+ for (const [id, entry] of Object.entries(map)) {
+ if (entry && now - Number(entry.at || 0) < LEAK_STALE_MS) next[id] = entry;
+ }
+ next[INSTANCE_ID] = { at: now, leaking: leak.leaking, samples: leak.samples, margins: leak.margins };
+ await cacheSet(LEAK_KEY, next, LEAK_TTL_S);
+ } catch {
+ // A gauge that cannot be parked must not take the health block down.
+ }
+}
+
+/**
+ * Every instance's current leak state, stale entries dropped.
+ * @returns {Promise<Array<{ instanceId: string, leaking: boolean, samples: number[], margins: number[], at: number }>>}
+ */
+export async function readLeakInstances() {
+ try {
+ const map = (await cacheGet(LEAK_KEY)) || {};
+ const now = Date.now();
+ return Object.entries(map)
+ .filter(([, entry]) => entry && now - Number(entry.at || 0) < LEAK_STALE_MS)
+ .map(([instanceId, entry]) => ({
+ instanceId,
+ leaking: Boolean(entry.leaking),
+ samples: Array.isArray(entry.samples) ? entry.samples : [],
+ margins: Array.isArray(entry.margins) ? entry.margins : [],
+ at: Number(entry.at || 0),
+ }));
+ } catch {
+ return [];
+ }
+}
+
/**
* The `home` subsystem block for /api/healthz and /api/status. Never throws.
* @returns {Promise<{ name: string, label: string, status: string, detail: string, hint?: string, metrics?: object }>}
@@ -361,6 +427,10 @@ export async function gatherHomeHealth() {
} catch (err) {
return { ...base, status: 'unknown', detail: err?.message || 'home signals unreadable' };
}
+ // Every instance answers its own /api/healthz, so gathering here is the one
+ // place that reliably samples all of them. Not awaited: the health block must
+ // not wait on a cache write.
+ publishLeakSample(signals.leak);
return { ...base, ...homeHealthVerdict(signals), metrics: signals };
}
diff --git a/api/cron/home-health-alert.js b/api/cron/home-health-alert.js
new file mode 100644
index 000000000..e18e7c442
--- /dev/null
+++ b/api/cron/home-health-alert.js
@@ -0,0 +1,173 @@
+// @ts-check
+// GET /api/cron/home-health-alert: the three alerts the Home lane is allowed to
+// send, and nothing else.
+//
+// The rule this file exists to enforce: A PER-TENANT FAILURE NEVER ALERTS. One
+// person's Home Assistant being unplugged, one expired token, one house behind a
+// dead tunnel: those are states that user sees in their own UI and their own
+// action log. Paging an operator for them trains everyone to ignore the channel,
+// and then the real outage arrives unread.
+//
+// Three alerts, deliberately:
+//
+// 1. Correlated unreachability. Handshake success under 80% across at least
+// ten distinct homes in fifteen minutes. One router is noise; ten at once
+// is us, and it is almost always a deploy, an egress change or DNS.
+// 2. Confirmation integrity. A guarded physical action (an unlock, a garage
+// door, a disarm) recorded as having executed with no confirmation on
+// record. This is supposed to be impossible, so a SINGLE row pages: it has
+// no rate, no threshold and no error budget.
+// 3. Subscriber leak. Registered subscribers exceeding open streams by a
+// margin that grows across three consecutive checks. Nothing about this
+// shows up as an error; instances just quietly fill with sockets into
+// strangers' houses until they die.
+//
+// The alerting channel itself is api/_lib/alerts.js. This adds no second
+// channel, and the generic degradation escalation in api/cron/uptime-check.js
+// still covers the `home` subsystem as one of many.
+
+import { cacheGet, cacheSet } from '../_lib/cache.js';
+import { sendOpsAlert } from '../_lib/alerts.js';
+import { json, method, wrapCron } from '../_lib/http.js';
+import { requireCron } from '../_lib/cron-auth.js';
+import {
+ HANDSHAKE_DOWN,
+ MIN_HOMES_FOR_A_VERDICT,
+ readHomeSignals,
+ readLeakInstances,
+ WINDOW_MINUTES,
+} from '../_lib/ops/home-health.js';
+
+/** Consecutive ticks a correlated outage must persist before it re-pages, so a deploy blip pages once rather than hourly. */
+const RE_ESCALATE_EVERY_TICKS = 12;
+const UNREACHABLE_STREAK_KEY = 'home:alert:unreachable:streak';
+const STREAK_TTL_S = 6 * 60 * 60;
+
+export default wrapCron(async (req, res) => {
+ if (!method(req, res, 'GET')) return;
+ if (!requireCron(req, res)) return;
+
+ const signals = await readHomeSignals();
+ const fired = [];
+
+ fired.push(...(await alertIntegrity(signals)));
+ fired.push(...(await alertCorrelatedUnreachability(signals)));
+ fired.push(...(await alertSubscriberLeak()));
+
+ return json(res, 200, {
+ ok: true,
+ windowMinutes: signals.windowMinutes,
+ fired: fired.map((f) => f.id),
+ signals: {
+ homes: signals.homes,
+ handshakes: signals.handshakes,
+ actions: { total: signals.actions.total, failed: signals.actions.failed, p95LatencyMs: signals.actions.p95LatencyMs },
+ confirmations: signals.confirmations,
+ integrity: signals.integrity,
+ },
+ });
+});
+
+/**
+ * Alert 2. No threshold, no streak, no dedup window that could swallow a second
+ * incident: the signature carries the most recent violation timestamp, so a new
+ * row always pages even if an older one paged an hour ago.
+ */
+async function alertIntegrity(signals) {
+ if (!signals.integrity.violations) return [];
+
+ await sendOpsAlert(
+ `CRITICAL: ${signals.integrity.violations} guarded home action(s) executed with no confirmation`,
+ [
+ `Most recent: ${signals.integrity.lastAt}`,
+ '',
+ 'A physical action that can open a building executed without a human saying yes.',
+ 'This has no error budget. Treat it as Sev 1 and assume the affected homes need their owners told.',
+ '',
+ 'Identify them:',
+ " select id, home_id, user_id, actor, channel, action, entity_ids, risk, created_at",
+ " from home_action_log",
+ " where guarded = true and confirmed_by is null and outcome = 'ok'",
+ " and created_at > now() - interval '24 hours'",
+ ' order by created_at desc;',
+ '',
+ 'Runbook: docs/home-operations.md, "Confirmation integrity violation".',
+ ].join('\n'),
+ { severity: 'critical', signature: `home:integrity:${signals.integrity.lastAt}` },
+ );
+ return [{ id: 'confirmation_integrity' }];
+}
+
+/**
+ * Alert 1. Requires BOTH a low rate and enough distinct homes reporting, because
+ * a low rate over three homes is one person's router and must never page.
+ */
+async function alertCorrelatedUnreachability(signals) {
+ const { attempts, rate, failed } = signals.handshakes;
+ const correlated = rate !== null && rate < HANDSHAKE_DOWN && attempts >= MIN_HOMES_FOR_A_VERDICT;
+
+ const streak = Number((await cacheGet(UNREACHABLE_STREAK_KEY)) || 0);
+
+ if (!correlated) {
+ if (streak > 0) {
+ await cacheSet(UNREACHABLE_STREAK_KEY, 0, STREAK_TTL_S);
+ await sendOpsAlert(
+ 'Recovered: home handshakes are succeeding across tenants again',
+ `${(rate === null ? 0 : rate * 100).toFixed(1)}% success over ${attempts} homes in the last ${signals.windowMinutes} minutes.`,
+ { severity: 'info', signature: 'home:unreachable:recovered' },
+ );
+ return [{ id: 'correlated_unreachability_recovered' }];
+ }
+ return [];
+ }
+
+ const next = streak + 1;
+ await cacheSet(UNREACHABLE_STREAK_KEY, next, STREAK_TTL_S);
+ // Page on the first tick, then hourly while it persists, rather than every
+ // five minutes forever.
+ if (next !== 1 && next % RE_ESCALATE_EVERY_TICKS !== 0) return [];
+
+ await sendOpsAlert(
+ `Home handshakes failing across tenants: ${(rate * 100).toFixed(1)}% success over ${attempts} homes`,
+ [
+ `${failed} of ${attempts} distinct homes failed their last handshake inside ${signals.windowMinutes} minutes.`,
+ `Connected now: ${signals.homes.connected}/${signals.homes.live}. Unreachable: ${signals.homes.unreachable}. Auth failed: ${signals.homes.authFailed}.`,
+ next === 1 ? '' : `Still failing after ${next} consecutive checks.`,
+ '',
+ 'This many houses do not go dark at once on their own. Suspect us first:',
+ ' 1. Did anything deploy in the last 20 minutes? curl -s https://three.ws/api/version',
+ ' 2. Egress: gcloud run services describe three-ws-api --region us-central1 --format="value(spec.template.metadata.annotations)" | tr "," "\\n" | grep vpc',
+ ' 3. Logs: gcloud logging read \'resource.type="cloud_run_revision" resource.labels.service_name="three-ws-api" textPayload:"home-runtime"\' --freshness=30m',
+ '',
+ 'Runbook: docs/home-operations.md, "Correlated unreachability".',
+ ].filter(Boolean).join('\n'),
+ { severity: 'critical', signature: `home:unreachable:${next}` },
+ );
+ return [{ id: 'correlated_unreachability' }];
+}
+
+/**
+ * Alert 3. Reads the per-instance samples every process publishes when its
+ * health block is gathered, because the pool is per-instance and this cron runs
+ * on whichever instance answered the request.
+ */
+async function alertSubscriberLeak() {
+ const leaking = (await readLeakInstances()).filter((i) => i.leaking);
+ if (!leaking.length) return [];
+
+ await sendOpsAlert(
+ `Home subscriber leak on ${leaking.length} instance(s)`,
+ [
+ ...leaking.map(
+ (i) => ` ${i.instanceId}: subscriber surplus over open streams grew ${i.margins.join(' then ')} (subscribers ${i.samples.join(' then ')})`,
+ ),
+ '',
+ 'A subscription is being registered and never released, so the instance is holding sockets into houses nobody is watching.',
+ 'It has no error signature; it ends as an out-of-memory restart.',
+ '',
+ 'Runbook: docs/home-operations.md, "Subscriber leak".',
+ ].join('\n'),
+ { severity: 'critical', signature: `home:leak:${leaking.map((i) => i.instanceId).sort().join(',')}` },
+ );
+ return [{ id: 'subscriber_leak' }];
+}
diff --git a/tests/home-integrity.test.js b/tests/home-integrity.test.js
new file mode 100644
index 000000000..385275e3c
--- /dev/null
+++ b/tests/home-integrity.test.js
@@ -0,0 +1,239 @@
+// The Home lane's scoring rules, exercised directly.
+//
+// The one thing this file is really for: proving that a single dark house does
+// NOT read as an outage, and that ten of them at once does. Everything else in
+// the observability lane is downstream of getting that distinction right, and it
+// is the distinction that is easy to break in a refactor and impossible to
+// notice until somebody is paged at 3am for a stranger's router.
+//
+// `homeHealthVerdict` and `noteSubscriberSample` are pure, so every threshold is
+// asserted here without a database. The end-to-end proof against a real Postgres
+// and a real Home Assistant is in the order 13 report.
+
+import { beforeEach, describe, expect, it } from 'vitest';
+
+import {
+ ACTION_DOWN,
+ EXPIRY_DOWN,
+ HANDSHAKE_DOWN,
+ homeHealthVerdict,
+ LATENCY_DOWN_MS,
+ MIN_HOMES_FOR_A_VERDICT,
+ noteSubscriberSample,
+ resetSubscriberSamples,
+} from '../api/_lib/ops/home-health.js';
+
+/** A platform where nothing is wrong, as `readHomeSignals` shapes it. */
+function healthy(overrides = {}) {
+ return {
+ windowMinutes: 15,
+ homes: { live: 40, connected: 40, unreachable: 0, authFailed: 0 },
+ handshakes: { attempts: 40, ok: 40, failed: 0, rate: 1 },
+ actions: { total: 200, ok: 198, refused: 2, failed: 0, homes: 30, rate: 1, timed: 200, p95LatencyMs: 300 },
+ confirmations: { total: 20, redeemed: 19, expired: 1, expiryRate: 1 / 20 },
+ integrity: { violations: 0, lastAt: null },
+ pool: { open: 12, subscribers: 12, capacity: 200, breakersOpen: 0, byStatus: { connected: 12 }, streams: 12, rung: 'normal' },
+ leak: { leaking: false, samples: [12, 12, 12], margins: [0, 0, 0], growth: 0 },
+ ...overrides,
+ };
+}
+
+describe('confirmation integrity is a zero-budget invariant', () => {
+ it('takes the subsystem down on a single row, however healthy everything else is', () => {
+ const verdict = homeHealthVerdict(healthy({ integrity: { violations: 1, lastAt: '2026-09-03T03:37:14.568Z' } }));
+ expect(verdict.status).toBe('down');
+ expect(verdict.detail).toContain('no confirmation on record');
+ expect(verdict.hint).toContain('Sev 1');
+ });
+
+ it('outranks every other signal, so it is never masked by a busy platform', () => {
+ const verdict = homeHealthVerdict(
+ healthy({
+ integrity: { violations: 3, lastAt: '2026-09-03T03:37:14.568Z' },
+ actions: { total: 1000, ok: 1000, refused: 0, failed: 0, homes: 90, rate: 1, timed: 1000, p95LatencyMs: 120 },
+ }),
+ );
+ expect(verdict.status).toBe('down');
+ expect(verdict.detail).toContain('3 guarded physical actions');
+ });
+});
+
+describe('one dark house is a UI state, not an outage', () => {
+ it('does not score a handshake rate at all below the home floor', () => {
+ // Three homes, one offline: a 33% failure rate that must not page anyone.
+ const verdict = homeHealthVerdict(
+ healthy({
+ homes: { live: 3, connected: 2, unreachable: 1, authFailed: 0 },
+ handshakes: { attempts: 3, ok: 2, failed: 1, rate: 2 / 3 },
+ }),
+ );
+ expect(verdict.status).toBe('ok');
+ expect(verdict.detail).toContain(`under the ${MIN_HOMES_FOR_A_VERDICT}-home floor, reported not scored`);
+ });
+
+ it('still reports the failure, so the operator can see it without being paged for it', () => {
+ const verdict = homeHealthVerdict(
+ healthy({
+ homes: { live: 3, connected: 2, unreachable: 1, authFailed: 0 },
+ handshakes: { attempts: 3, ok: 2, failed: 1, rate: 2 / 3 },
+ }),
+ );
+ expect(verdict.detail).toContain('66.7%');
+ expect(verdict.detail).toContain('2/3 homes connected');
+ });
+
+ it('an expired token on one house of forty does not move the aggregate', () => {
+ const verdict = homeHealthVerdict(
+ healthy({
+ homes: { live: 40, connected: 39, unreachable: 0, authFailed: 1 },
+ handshakes: { attempts: 40, ok: 39, failed: 1, rate: 39 / 40 },
+ }),
+ );
+ expect(verdict.status).toBe('ok');
+ });
+});
+
+describe('a correlated failure is an outage', () => {
+ it('goes down when handshakes fail across enough homes at once', () => {
+ const attempts = MIN_HOMES_FOR_A_VERDICT + 2;
+ const ok = 2;
+ const verdict = homeHealthVerdict(
+ healthy({
+ homes: { live: 40, connected: ok, unreachable: attempts - ok, authFailed: 0 },
+ handshakes: { attempts, ok, failed: attempts - ok, rate: ok / attempts },
+ }),
+ );
+ expect(ok / attempts).toBeLessThan(HANDSHAKE_DOWN);
+ expect(verdict.status).toBe('down');
+ expect(verdict.hint).toContain('almost always us');
+ });
+
+ it('degrades between the two thresholds rather than jumping straight to down', () => {
+ const verdict = homeHealthVerdict(
+ healthy({ handshakes: { attempts: 40, ok: 35, failed: 5, rate: 0.875 } }),
+ );
+ expect(verdict.status).toBe('degraded');
+ });
+});
+
+describe('actions', () => {
+ it('counts a refused action as a success, because the gate working is not a fault', () => {
+ // Every action refused, none failed: the safety gate did its job all day.
+ const verdict = homeHealthVerdict(
+ healthy({ actions: { total: 100, ok: 0, refused: 100, failed: 0, homes: 20, rate: 1, timed: 100, p95LatencyMs: 200 } }),
+ );
+ expect(verdict.status).toBe('ok');
+ });
+
+ it('goes down when actions genuinely fail across homes', () => {
+ const rate = 0.9;
+ expect(rate).toBeLessThan(ACTION_DOWN);
+ const verdict = homeHealthVerdict(
+ healthy({ actions: { total: 100, ok: 90, refused: 0, failed: 10, homes: 25, rate, timed: 100, p95LatencyMs: 300 } }),
+ );
+ expect(verdict.status).toBe('down');
+ expect(verdict.detail).toContain('10 failed');
+ });
+
+ it('says so plainly when nothing on the act path is timed yet', () => {
+ const verdict = homeHealthVerdict(healthy({ actions: { ...healthy().actions, timed: 0, p95LatencyMs: null } }));
+ expect(verdict.status).toBe('ok');
+ expect(verdict.detail).toContain('no action timings recorded');
+ });
+
+ it('goes down on our own leg being slow', () => {
+ const verdict = homeHealthVerdict(
+ healthy({ actions: { ...healthy().actions, p95LatencyMs: LATENCY_DOWN_MS + 1 } }),
+ );
+ expect(verdict.status).toBe('down');
+ expect(verdict.hint).toContain('The house is excluded from this measurement');
+ });
+});
+
+describe('confirmations that expire are a UI failure', () => {
+ it('goes down when most confirmations time out instead of being answered', () => {
+ const expiryRate = EXPIRY_DOWN + 0.1;
+ const verdict = homeHealthVerdict(
+ healthy({ confirmations: { total: 50, redeemed: 25, expired: 25, expiryRate } }),
+ );
+ expect(verdict.status).toBe('down');
+ expect(verdict.hint).toContain('That is a UI failure, not user hesitation');
+ });
+
+ it('ignores an empty window rather than scoring it as perfect or broken', () => {
+ const verdict = homeHealthVerdict(
+ healthy({ confirmations: { total: 0, redeemed: 0, expired: 0, expiryRate: null } }),
+ );
+ expect(verdict.status).toBe('ok');
+ expect(verdict.detail).toContain('no confirmations in window');
+ });
+});
+
+describe('the subscriber leak detector', () => {
+ beforeEach(() => resetSubscriberSamples());
+
+ it('does not accuse anything before it has three samples', () => {
+ expect(noteSubscriberSample({ subscribers: 10, open: 5, streams: 2 }).leaking).toBe(false);
+ expect(noteSubscriberSample({ subscribers: 20, open: 5, streams: 2 }).leaking).toBe(false);
+ });
+
+ it('fires when the surplus over open streams grows across three checks', () => {
+ noteSubscriberSample({ subscribers: 12, open: 10, streams: 10 });
+ noteSubscriberSample({ subscribers: 18, open: 10, streams: 10 });
+ const third = noteSubscriberSample({ subscribers: 26, open: 10, streams: 10 });
+
+ expect(third.leaking).toBe(true);
+ expect(third.margins).toEqual([2, 8, 16]);
+ expect(third.growth).toBe(14);
+ });
+
+ it('clears as soon as the leaked subscriptions are released', () => {
+ noteSubscriberSample({ subscribers: 12, open: 10, streams: 10 });
+ noteSubscriberSample({ subscribers: 18, open: 10, streams: 10 });
+ expect(noteSubscriberSample({ subscribers: 26, open: 10, streams: 10 }).leaking).toBe(true);
+
+ expect(noteSubscriberSample({ subscribers: 10, open: 10, streams: 10 }).leaking).toBe(false);
+ });
+
+ it('does not call honest traffic a leak: streams and subscribers rising together', () => {
+ noteSubscriberSample({ subscribers: 10, open: 10, streams: 10 });
+ noteSubscriberSample({ subscribers: 40, open: 10, streams: 40 });
+ const third = noteSubscriberSample({ subscribers: 90, open: 10, streams: 90 });
+
+ expect(third.leaking).toBe(false);
+ expect(third.margins).toEqual([0, 0, 0]);
+ });
+
+ it('does not call a shrinking surplus a leak', () => {
+ noteSubscriberSample({ subscribers: 30, open: 10, streams: 10 });
+ noteSubscriberSample({ subscribers: 20, open: 10, streams: 10 });
+ expect(noteSubscriberSample({ subscribers: 12, open: 10, streams: 10 }).leaking).toBe(false);
+ });
+
+ it('falls back to the pooled connection count when the runtime reports no stream gauge', () => {
+ noteSubscriberSample({ subscribers: 12, open: 10, streams: null });
+ noteSubscriberSample({ subscribers: 18, open: 10, streams: null });
+ expect(noteSubscriberSample({ subscribers: 26, open: 10, streams: null }).leaking).toBe(true);
+ });
+
+ it('surfaces the leak as degraded, not down: every request is still being served', () => {
+ const verdict = homeHealthVerdict(
+ healthy({ leak: { leaking: true, samples: [12, 18, 26], margins: [2, 8, 16], growth: 14 } }),
+ );
+ expect(verdict.status).toBe('degraded');
+ expect(verdict.hint).toContain('never releasing it');
+ });
+});
+
+describe('an empty platform', () => {
+ it('reports unknown rather than a confident green when no home is connected', () => {
+ const verdict = homeHealthVerdict(
+ healthy({
+ homes: { live: 0, connected: 0, unreachable: 0, authFailed: 0 },
+ handshakes: { attempts: 0, ok: 0, failed: 0, rate: null },
+ }),
+ );
+ expect(verdict.status).toBe('unknown');
+ expect(verdict.detail).toContain('no homes connected yet');
+ });
+});
diff --git a/vercel.json b/vercel.json
index 33eb7b06a..da49b98c5 100644
--- a/vercel.json
+++ b/vercel.json
@@ -1747,6 +1747,10 @@
"src": "/api/agora/act",
"dest": "/api/agora/act.js"
},
+ {
+ "src": "/voice/home/?",
+ "dest": "/voice-home.html"
+ },
{
"src": "/voice/?",
"dest": "/voice.html"
@@ -5641,6 +5645,10 @@
"Location": "/"
}
},
+ {
+ "src": "/home/([0-9a-fA-F-]{36})/?",
+ "dest": "/home-scene.html"
+ },
{
"src": "/home",
"status": 301,
@@ -6367,6 +6375,10 @@
{
"path": "/api/cron/print-orders-sync",
"schedule": "*/15 * * * *"
+ },
+ {
+ "path": "/api/cron/home-health-alert",
+ "schedule": "*/5 * * * *"
}
]
}