Skip to content

Free, Open Source, Self-Hosted WhatsApp API Gateway

docs(events): state the limits of the burst and pause guarantees

Yudhi Armyndharis74d74e9b31aa
diff --git a/docs/13-horizontal-scaling.md b/docs/13-horizontal-scaling.md
index ef28ee264..0f0749a97 100644
--- a/docs/13-horizontal-scaling.md
+++ b/docs/13-horizontal-scaling.md
@@ -151,21 +151,25 @@
> every form for the rooms to match.
>
> **The fan-out to other replicas is best effort and keeps no replay queue.** A Redis that pauses for less
-> than 10 s (a snapshot fork, a slow fsync) answers afterwards, and every event raised meanwhile reaches
-> the other replicas late. A longer silence, a restart, a failover or an outage drops the connection a node
-> publishes on, and what was written to it may be lost with it, or may still reach the other replicas late
-> and after newer events, which their clients then apply out of order. The events that node raises while it is
-> disconnected, plus its reconnect backoff of up to 5 s once Redis is back, are dropped for the clients
-> connected to the other replicas: they are not delivered late and not twice. Clients connected to the node
-> that raised an event are served directly, and webhooks do not use this path, so neither is affected. Memory
-> is bounded in two ways. A Redis that sends nothing for 10 s has the connection destroyed, and the node holds
-> nothing more for it; what it had already held, at most 10 s of events, is freed once the node has reconnected.
-> A Redis that keeps answering, but slower than events are raised, is never silent for 10 s, so a cap bounds
-> it instead: once the node holds 12,000 unanswered events (10 s at 1,000 events a second, with a fifth to
-> spare) and the backlog has outlived 2 s, meaning the count has not been below 6,000 for that long, it drops
-> further frames for the other replicas until the count is back below 6,000, with a warning when dropping
-> starts and another, with the count, when it ends. A burst raised in a single tick is never cut, however
-> large: a healthy Redis answers it right after, and the node holds it until then. A replica whose own Redis
+> than 10 s (a snapshot fork, a slow fsync) answers afterwards, and every event raised meanwhile reaches the
+> other replicas late, at up to about 1,200 events a second; above that the cap described below applies
+> first and drops the frames beyond it. A longer silence, a restart, a failover or an outage drops the
+> connection a node publishes on, and what was written to it may be lost with it, or may still reach the
+> other replicas late and after newer events, which their clients then apply out of order. The events that
+> node raises while it is disconnected, plus its reconnect backoff of up to 5 s once Redis is back, are
+> dropped for the clients connected to the other replicas: they are not delivered late and not twice.
+> Clients connected to the node that raised an event are served directly, and webhooks do not use this path,
+> so neither is affected. Memory is bounded in two ways. A Redis that sends nothing for 10 s has the
+> connection destroyed, and the node holds nothing more for it; what it had already held, at most 10 s of
+> events, is freed once the node has reconnected. A Redis that keeps answering, but slower than events are
+> raised, is never silent for 10 s, so a cap bounds it instead: once the node holds 12,000 unanswered events
+> (10 s at 1,000 events a second, with a fifth to spare) and the backlog has outlived 2 s, meaning the count
+> has not been below 6,000 for that long, it drops further frames for the other replicas until the count is
+> back below 6,000, with a warning when dropping starts and another, with the count, when it ends. The node
+> can hold the traffic of those 2 s on top of the 12,000. A burst raised in a single tick is not cut as long
+> as raising it past its 6,000th frame takes under 2 s: a healthy Redis answers it right after, and the node
+> holds it until then. A synchronous burst that takes longer than that to raise past 6,000 frames is cut at
+> 12,000 even against a healthy Redis, and the warning names the unanswered count. A replica whose own Redis
> connection is reconnecting also misses what the others publish meanwhile.
>
> **Mid-connection key eviction converges on a timer, not a broadcast, and only on one node.** The
diff --git a/src/modules/events/redis-io.adapter.spec.ts b/src/modules/events/redis-io.adapter.spec.ts
index 1529a0569..026083ce9 100644
--- a/src/modules/events/redis-io.adapter.spec.ts
+++ b/src/modules/events/redis-io.adapter.spec.ts
@@ -270,6 +270,8 @@ describe('RedisIoAdapter', () => {
it('pins the figures docs/13 quotes, and a 10 s silence at 1000 events a second fits under them', () => {
expect(WS_REDIS_MAX_UNANSWERED).toBe(12_000);
expect(WS_REDIS_MAX_UNANSWERED).toBeGreaterThanOrEqual((WS_REDIS_SOCKET_TIMEOUT_MS / 1000) * 1000 * 1.2);
+ // "Up to about 1,200 events a second" in docs/13: the rate at which a pause of one socket deadline fills the cap.
+ expect(WS_REDIS_MAX_UNANSWERED / (WS_REDIS_SOCKET_TIMEOUT_MS / 1000)).toBe(1200);
});
// Slowness is judged by how long the backlog has lasted, not by its depth: a burst raised in one tick is all on the
diff --git a/src/modules/events/redis-io.adapter.ts b/src/modules/events/redis-io.adapter.ts
index 39276e081..fbcdbcc36 100644
--- a/src/modules/events/redis-io.adapter.ts
+++ b/src/modules/events/redis-io.adapter.ts
@@ -34,11 +34,14 @@ export const WS_REDIS_SOCKET_TIMEOUT_MS = 10_000;
* than events are raised never trips the socket deadline, so without a cap the backlog grows with the load for as
* long as the slowness lasts. The rule: frames are dropped for the other replicas while the queue holds this many
* commands and has not been below half of it for longer than a command is given to be answered
- * (WS_REDIS_COMMAND_TIMEOUT_MS), and they go on being dropped until the queue is back below half of it. A burst
- * raised in one tick is never cut, however large, because depth alone says nothing: it is all on the queue before a
- * reply can be read. Dropping starts with a warning and ends with another, with the count. 12,000 holds a full
- * socket deadline of traffic at 1,000 events a second with a fifth to spare, which measured about 25 MB of heap with
- * 1 KiB frames. It counts commands, not bytes: events with inline media are larger.
+ * (WS_REDIS_COMMAND_TIMEOUT_MS), and they go on being dropped until the queue is back below half of it. Depth alone
+ * says nothing, since a burst raised in one tick is all on the queue before a reply can be read, so such a burst is
+ * not cut as long as raising it past half the cap takes less than a command is given. One that takes longer, a
+ * synchronous loop that cannot yield for over 2 s, is cut at the cap whatever the peer is doing, and the warning
+ * names the unanswered count. Dropping starts with a warning and ends with another, with the count. The queue can
+ * pass the cap by the traffic of one command deadline before dropping starts. 12,000 holds a full socket deadline
+ * of traffic at 1,000 events a second with a fifth to spare, which measured about 25 MB of heap with 1 KiB frames.
+ * It counts commands, not bytes: events with inline media are larger.
*/
export const WS_REDIS_MAX_UNANSWERED = 12_000;
@@ -107,13 +110,15 @@ export function rejectRootNamespace(server: Server): void {
*
* Delivery while Redis is away or slow: the fan-out to the other replicas is best effort and keeps no replay
* queue. A Redis that pauses for less than WS_REDIS_SOCKET_TIMEOUT_MS answers afterwards, and every event raised
- * meanwhile reaches them late. Silence for that long destroys the publisher's socket, and so does a restart, a
+ * meanwhile reaches them late, at up to about 1,200 events a second; above that the cap below applies first, and
+ * drops the frames beyond it. Silence for that long destroys the publisher's socket, and so does a restart, a
* failover or an outage: what was written to the dropped connection may be lost with it, or may still reach the
* other replicas late and after newer events. The events this node raises while it is disconnected, plus its
* reconnect backoff of up to 5 s once Redis is back (wsRedisOptions), are dropped for the other replicas, never
* delivered late and never twice. A Redis that keeps answering, but slower than events are raised, never looks
* silent: it is bounded by WS_REDIS_MAX_UNANSWERED, which drops frames for the other replicas once the backlog has
- * outlived a command's deadline and until it is back below half of it (a burst raised in one tick is not cut).
+ * outlived a command's deadline and until it is back below half of it (a burst raised in one tick is not cut
+ * unless raising it past half the cap takes over 2 s).
* Clients on this node are served by the local broadcast and webhooks do not use this path, so neither is
* affected. What the publisher holds is bounded by one socket deadline of traffic for a silent Redis and by that
* cap, plus the traffic of one command deadline, for a slow one.