Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

fix(home): make the subscriber leak detector able to fire at all

nirholas3357c7daeeee
diff --git a/api/_lib/ops/home-health.js b/api/_lib/ops/home-health.js
index e006f7e2e..38c56ece4 100644
--- a/api/_lib/ops/home-health.js
+++ b/api/_lib/ops/home-health.js
@@ -82,6 +82,13 @@ const QUERY_TIMEOUT_MS = 4_000;
* called a leak. Three, because two is a busy minute and four is a wasted hour.
*/
const LEAK_SAMPLES = 3;
+/**
+ * Watchers per open connection above which a rising subscriber count stops
+ * looking like a busy household and starts looking like nobody releasing. Four
+ * people watching one home live at the same moment is a large family; forty is a
+ * bug.
+ */
+const LEAK_PER_CONNECTION = 4;
/** The rolling subscriber samples, newest last. Per process, like the pool it measures. */
const leakSamples = [];
@@ -104,25 +111,40 @@ const LEAK_STALE_MS = 15 * 60_000;
/**
* Record one subscriber sample and say whether the pool is leaking.
*
- * A leak here is a subscriber that was registered and never released: an SSE
+ * A leak here is a subscription that was registered and never released: an SSE
* stream whose client vanished without the server noticing, repeated until the
- * instance dies holding hundreds of sockets into strangers' houses. It has no
- * error signature at all, which is why it is measured as a SHAPE over time
- * rather than a threshold.
+ * instance dies holding sockets into houses nobody is watching. It has no error
+ * signature at all, which is why it is measured as a SHAPE over time rather than
+ * a threshold.
+ *
+ * **The obvious signal does not work, and this was measured rather than
+ * reasoned.** The plan was to watch the margin between registered subscribers
+ * and open streams. Against the real runtime that margin is always zero:
+ * `subscribe()` registers the subscriber and admits the stream in the same call,
+ * so the two counters move in lockstep by construction and a detector built on
+ * their difference can never fire. A run of six deliberately leaked
+ * subscriptions produced `margins=[0,0,0]`.
+ *
+ * What actually leaks is the ABSOLUTE count: subscribers climbing and never
+ * coming back down while the number of pooled connections does not grow. Honest
+ * traffic fluctuates, because people close tabs. A leak only ever rises.
+ *
+ * The count alone is not enough either, because many people can legitimately
+ * watch one house at once and that also raises subscribers against a flat
+ * connection count. So the detector additionally requires more watchers per open
+ * connection than a household plausibly has (`LEAK_PER_CONNECTION`), which is
+ * where "a family is looking at the same home" stops being the likely
+ * explanation and "nobody is releasing" starts.
*
- * The signal is the MARGIN between registered subscribers and open streams. One
- * subscriber per stream is correct at any scale, so a margin that exists and
- * keeps growing across three consecutive checks is the fingerprint, and a
- * hundred honest streams arriving at once is not. When the runtime reports no
- * stream gauge, the pooled connection count stands in: subscribers climbing
- * while connections do not is the same shape, one step less precise.
+ * The margin is still recorded, because a NON-zero one is a different bug worth
+ * seeing: a stream admitted without a subscriber, or the reverse.
*
* @param {{ subscribers: number, open: number, streams?: number|null }} sample
* @returns {{ leaking: boolean, samples: number[], margins: number[], growth: number }}
*/
export function noteSubscriberSample({ subscribers, open, streams = null }) {
- const baseline = typeof streams === 'number' ? streams : open;
- leakSamples.push({ subscribers, baseline, margin: subscribers - baseline });
+ const baseline = typeof streams === 'number' ? streams : subscribers;
+ leakSamples.push({ subscribers, open, margin: subscribers - baseline });
while (leakSamples.length > LEAK_SAMPLES) leakSamples.shift();
const samples = leakSamples.map((s) => s.subscribers);
@@ -131,12 +153,15 @@ export function noteSubscriberSample({ subscribers, open, streams = null }) {
let climbing = true;
for (let i = 1; i < leakSamples.length; i += 1) {
- if (leakSamples[i].margin <= leakSamples[i - 1].margin) climbing = false;
+ // Strictly rising subscribers, and connections that are not rising with
+ // them. A fleet genuinely taking on more houses grows both.
+ if (leakSamples[i].subscribers <= leakSamples[i - 1].subscribers) climbing = false;
+ if (leakSamples[i].open > leakSamples[i - 1].open) climbing = false;
}
- const growth = margins[margins.length - 1] - margins[0];
- // A negative margin is not a leak in the other direction, it is a stream that
- // has not registered its subscriber yet. Only an unexplained surplus counts.
- return { leaking: climbing && growth > 0 && margins[margins.length - 1] > 0, samples, margins, growth };
+ const latest = leakSamples[leakSamples.length - 1];
+ const crowded = latest.subscribers > LEAK_PER_CONNECTION * Math.max(1, latest.open);
+ const growth = samples[samples.length - 1] - samples[0];
+ return { leaking: climbing && crowded && growth > 0, samples, margins, growth };
}
/** Forget the rolling samples. Test seam, and the reset after an alert fires. */
@@ -416,7 +441,7 @@ export function homeHealthVerdict(s) {
const hint = status === 'ok'
? undefined
: s.leak.leaking
- ? `The subscriber surplus over open streams grew across ${s.leak.margins.join(' then ')}. A stream is registering a subscriber and never releasing it. See docs/ops/home-operations.md, "Subscriber leak".`
+ ? `Subscribers climbed ${s.leak.samples.join(' then ')} across three checks while open connections did not, at more than ${LEAK_PER_CONNECTION} watchers per connection. A subscription is being registered and never released. See docs/ops/home-operations.md, "Subscriber leak".`
: handshakeStatus !== 'ok'
? 'Handshakes are failing across tenants, which is almost always us: a deploy, an egress change or a DNS fault. Run the correlation query in docs/ops/home-operations.md before touching anything.'
: actionStatus !== 'ok'
diff --git a/api/cron/home-health-alert.js b/api/cron/home-health-alert.js
index 6956230e6..b19e79d56 100644
--- a/api/cron/home-health-alert.js
+++ b/api/cron/home-health-alert.js
@@ -17,10 +17,11 @@
// door, a disarm) recorded as having executed with no confirmation on
// record. This is supposed to be impossible, so a SINGLE row pages: it has
// no rate, no threshold and no error budget.
-// 3. Subscriber leak. Registered subscribers exceeding open streams by a
-// margin that grows across three consecutive checks. Nothing about this
-// shows up as an error; instances just quietly fill with sockets into
-// strangers' houses until they die.
+// 3. Subscriber leak. Registered subscribers climbing across three consecutive
+// checks while open connections do not, at more watchers per connection
+// than a household plausibly has. Nothing about this shows up as an error;
+// instances just quietly fill with sockets into strangers' houses until
+// they die.
//
// The alerting channel itself is api/_lib/alerts.js. This adds no second
// channel, and the generic degradation escalation in api/cron/uptime-check.js
@@ -159,7 +160,7 @@ async function alertSubscriberLeak() {
`Home subscriber leak on ${leaking.length} instance(s)`,
[
...leaking.map(
- (i) => ` ${i.instanceId}: subscriber surplus over open streams grew ${i.margins.join(' then ')} (subscribers ${i.samples.join(' then ')})`,
+ (i) => ` ${i.instanceId}: subscribers climbed ${i.samples.join(' then ')} across three checks while open connections did not`,
),
'',
'A subscription is being registered and never released, so the instance is holding sockets into houses nobody is watching.',
diff --git a/docs/ops/home-operations.md b/docs/ops/home-operations.md
index f8feda4b4..a7a675ffd 100644
--- a/docs/ops/home-operations.md
+++ b/docs/ops/home-operations.md
@@ -56,7 +56,7 @@ expired of 4 (under the 10-confirmation floor, reported not scored); ...
| Confirmation expiry | `home_confirmations` | under 20% | 20 to 40% | over 40% |
| Breaker-open homes | `runtime.stats()` | under 2% | 2 to 10% | over 10% |
| p95 action latency (our leg) | `home_action_log.detail->>'latencyMs'` | under 1.5 s | 1.5 to 4 s | over 4 s |
-| Subscriber leak | `stats().subscribers` versus open streams | flat | growing | growing |
+| Subscriber leak | `stats().subscribers` over three checks | flat or falling | climbing with flat connections | climbing with flat connections |
| Confirmation integrity | `home_action_log`, excluding standing grants | zero rows | n/a | any row |
A **refused** action counts as a success. The safety gate refusing to open a
@@ -211,8 +211,22 @@ full root cause to tell somebody their door was opened.
## Alert 3: subscriber leak
-**Fires when:** on any instance, the surplus of registered subscribers over open
-streams grows across three consecutive checks.
+**Fires when:** on any instance, registered subscribers climb across three
+consecutive checks while the number of open connections does not, at more than
+four watchers per open connection.
+
+**The obvious signal does not work, and this was measured rather than reasoned.**
+The plan was to watch the margin between registered subscribers and open
+streams. Against the real runtime that margin is always zero, because
+`subscribe()` registers the subscriber and admits the stream in the same call, so
+the two counters move in lockstep by construction. Six deliberately leaked
+subscriptions produced `margins=[0,0,0]` and a detector built on the difference
+would never have fired once. What actually leaks is the absolute count:
+subscribers that climb and never come back down while the pool does not grow.
+Honest traffic fluctuates, because people close tabs. A leak only ever rises. The
+per-connection ceiling is what keeps a family all watching one house at once from
+reading as a leak; the margin is still recorded, because a non-zero one is a
+different bug (a stream admitted with no subscriber, or the reverse).
**Symptom as reported:** nothing, until instances start restarting. This failure
has no error signature: every request keeps succeeding while the process fills