Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
JavaScript 3d-generationai-agentsanimationavatardigital-humangenerative-aiglbgltfimage-to-3dllmmcpmcp-servermodel-context-protocolsolanatext-to-3dthreejsweb-componentswebglwebxrx402
fix(home): make the subscriber leak detector able to fire at all
nirholas3357c7daeeee
diff --git a/api/_lib/ops/home-health.js b/api/_lib/ops/home-health.jsindex e006f7e2e..38c56ece4 100644--- a/api/_lib/ops/home-health.js+++ b/api/_lib/ops/home-health.js@@ -82,6 +82,13 @@ const QUERY_TIMEOUT_MS = 4_000;* called a leak. Three, because two is a busy minute and four is a wasted hour.*/const LEAK_SAMPLES = 3;+/**+ * Watchers per open connection above which a rising subscriber count stops+ * looking like a busy household and starts looking like nobody releasing. Four+ * people watching one home live at the same moment is a large family; forty is a+ * bug.+ */+const LEAK_PER_CONNECTION = 4;/** The rolling subscriber samples, newest last. Per process, like the pool it measures. */const leakSamples = [];@@ -104,25 +111,40 @@ const LEAK_STALE_MS = 15 * 60_000;/*** Record one subscriber sample and say whether the pool is leaking.*- * A leak here is a subscriber that was registered and never released: an SSE+ * A leak here is a subscription that was registered and never released: an SSE* stream whose client vanished without the server noticing, repeated until the- * instance dies holding hundreds of sockets into strangers' houses. It has no- * error signature at all, which is why it is measured as a SHAPE over time- * rather than a threshold.+ * instance dies holding sockets into houses nobody is watching. It has no error+ * signature at all, which is why it is measured as a SHAPE over time rather than+ * a threshold.+ *+ * **The obvious signal does not work, and this was measured rather than+ * reasoned.** The plan was to watch the margin between registered subscribers+ * and open streams. Against the real runtime that margin is always zero:+ * `subscribe()` registers the subscriber and admits the stream in the same call,+ * so the two counters move in lockstep by construction and a detector built on+ * their difference can never fire. A run of six deliberately leaked+ * subscriptions produced `margins=[0,0,0]`.+ *+ * What actually leaks is the ABSOLUTE count: subscribers climbing and never+ * coming back down while the number of pooled connections does not grow. Honest+ * traffic fluctuates, because people close tabs. A leak only ever rises.+ *+ * The count alone is not enough either, because many people can legitimately+ * watch one house at once and that also raises subscribers against a flat+ * connection count. So the detector additionally requires more watchers per open+ * connection than a household plausibly has (`LEAK_PER_CONNECTION`), which is+ * where "a family is looking at the same home" stops being the likely+ * explanation and "nobody is releasing" starts.*- * The signal is the MARGIN between registered subscribers and open streams. One- * subscriber per stream is correct at any scale, so a margin that exists and- * keeps growing across three consecutive checks is the fingerprint, and a- * hundred honest streams arriving at once is not. When the runtime reports no- * stream gauge, the pooled connection count stands in: subscribers climbing- * while connections do not is the same shape, one step less precise.+ * The margin is still recorded, because a NON-zero one is a different bug worth+ * seeing: a stream admitted without a subscriber, or the reverse.** @param {{ subscribers: number, open: number, streams?: number|null }} sample* @returns {{ leaking: boolean, samples: number[], margins: number[], growth: number }}*/export function noteSubscriberSample({ subscribers, open, streams = null }) {- const baseline = typeof streams === 'number' ? streams : open;- leakSamples.push({ subscribers, baseline, margin: subscribers - baseline });+ const baseline = typeof streams === 'number' ? streams : subscribers;+ leakSamples.push({ subscribers, open, margin: subscribers - baseline });while (leakSamples.length > LEAK_SAMPLES) leakSamples.shift();const samples = leakSamples.map((s) => s.subscribers);@@ -131,12 +153,15 @@ export function noteSubscriberSample({ subscribers, open, streams = null }) {let climbing = true;for (let i = 1; i < leakSamples.length; i += 1) {- if (leakSamples[i].margin <= leakSamples[i - 1].margin) climbing = false;+ // Strictly rising subscribers, and connections that are not rising with+ // them. A fleet genuinely taking on more houses grows both.+ if (leakSamples[i].subscribers <= leakSamples[i - 1].subscribers) climbing = false;+ if (leakSamples[i].open > leakSamples[i - 1].open) climbing = false;}- const growth = margins[margins.length - 1] - margins[0];- // A negative margin is not a leak in the other direction, it is a stream that- // has not registered its subscriber yet. Only an unexplained surplus counts.- return { leaking: climbing && growth > 0 && margins[margins.length - 1] > 0, samples, margins, growth };+ const latest = leakSamples[leakSamples.length - 1];+ const crowded = latest.subscribers > LEAK_PER_CONNECTION * Math.max(1, latest.open);+ const growth = samples[samples.length - 1] - samples[0];+ return { leaking: climbing && crowded && growth > 0, samples, margins, growth };}/** Forget the rolling samples. Test seam, and the reset after an alert fires. */@@ -416,7 +441,7 @@ export function homeHealthVerdict(s) {const hint = status === 'ok'? undefined: s.leak.leaking- ? `The subscriber surplus over open streams grew across ${s.leak.margins.join(' then ')}. A stream is registering a subscriber and never releasing it. See docs/ops/home-operations.md, "Subscriber leak".`+ ? `Subscribers climbed ${s.leak.samples.join(' then ')} across three checks while open connections did not, at more than ${LEAK_PER_CONNECTION} watchers per connection. A subscription is being registered and never released. See docs/ops/home-operations.md, "Subscriber leak".`: handshakeStatus !== 'ok'? 'Handshakes are failing across tenants, which is almost always us: a deploy, an egress change or a DNS fault. Run the correlation query in docs/ops/home-operations.md before touching anything.': actionStatus !== 'ok'diff --git a/api/cron/home-health-alert.js b/api/cron/home-health-alert.jsindex 6956230e6..b19e79d56 100644--- a/api/cron/home-health-alert.js+++ b/api/cron/home-health-alert.js@@ -17,10 +17,11 @@// door, a disarm) recorded as having executed with no confirmation on// record. This is supposed to be impossible, so a SINGLE row pages: it has// no rate, no threshold and no error budget.-// 3. Subscriber leak. Registered subscribers exceeding open streams by a-// margin that grows across three consecutive checks. Nothing about this-// shows up as an error; instances just quietly fill with sockets into-// strangers' houses until they die.+// 3. Subscriber leak. Registered subscribers climbing across three consecutive+// checks while open connections do not, at more watchers per connection+// than a household plausibly has. Nothing about this shows up as an error;+// instances just quietly fill with sockets into strangers' houses until+// they die.//// The alerting channel itself is api/_lib/alerts.js. This adds no second// channel, and the generic degradation escalation in api/cron/uptime-check.js@@ -159,7 +160,7 @@ async function alertSubscriberLeak() {`Home subscriber leak on ${leaking.length} instance(s)`,[...leaking.map(- (i) => ` ${i.instanceId}: subscriber surplus over open streams grew ${i.margins.join(' then ')} (subscribers ${i.samples.join(' then ')})`,+ (i) => ` ${i.instanceId}: subscribers climbed ${i.samples.join(' then ')} across three checks while open connections did not`,),'','A subscription is being registered and never released, so the instance is holding sockets into houses nobody is watching.',diff --git a/docs/ops/home-operations.md b/docs/ops/home-operations.mdindex f8feda4b4..a7a675ffd 100644--- a/docs/ops/home-operations.md+++ b/docs/ops/home-operations.md@@ -56,7 +56,7 @@ expired of 4 (under the 10-confirmation floor, reported not scored); ...| Confirmation expiry | `home_confirmations` | under 20% | 20 to 40% | over 40% || Breaker-open homes | `runtime.stats()` | under 2% | 2 to 10% | over 10% || p95 action latency (our leg) | `home_action_log.detail->>'latencyMs'` | under 1.5 s | 1.5 to 4 s | over 4 s |-| Subscriber leak | `stats().subscribers` versus open streams | flat | growing | growing |+| Subscriber leak | `stats().subscribers` over three checks | flat or falling | climbing with flat connections | climbing with flat connections || Confirmation integrity | `home_action_log`, excluding standing grants | zero rows | n/a | any row |A **refused** action counts as a success. The safety gate refusing to open a@@ -211,8 +211,22 @@ full root cause to tell somebody their door was opened.## Alert 3: subscriber leak-**Fires when:** on any instance, the surplus of registered subscribers over open-streams grows across three consecutive checks.+**Fires when:** on any instance, registered subscribers climb across three+consecutive checks while the number of open connections does not, at more than+four watchers per open connection.++**The obvious signal does not work, and this was measured rather than reasoned.**+The plan was to watch the margin between registered subscribers and open+streams. Against the real runtime that margin is always zero, because+`subscribe()` registers the subscriber and admits the stream in the same call, so+the two counters move in lockstep by construction. Six deliberately leaked+subscriptions produced `margins=[0,0,0]` and a detector built on the difference+would never have fired once. What actually leaks is the absolute count:+subscribers that climb and never come back down while the pool does not grow.+Honest traffic fluctuates, because people close tabs. A leak only ever rises. The+per-connection ceiling is what keeps a family all watching one house at once from+reading as a leak; the margin is still recorded, because a non-zero one is a+different bug (a stream admitted with no subscriber, or the reverse).**Symptom as reported:** nothing, until instances start restarting. This failurehas no error signature: every request keeps succeeding while the process fills