Free, Open Source, Self-Hosted WhatsApp API Gateway
TypeScript apibotgatewayself-hostedwhatsappwhatsapp-apiwhatsapp-automationwhatsapp-botwhatsapp-gatewaywhatsapp-gateway-apiwhatsapp-webwhatsapp-web-api
docs(events): state the limits of the burst and pause guarantees
Yudhi Armyndharis74d74e9b31aa
diff --git a/docs/13-horizontal-scaling.md b/docs/13-horizontal-scaling.mdindex ef28ee264..0f0749a97 100644--- a/docs/13-horizontal-scaling.md+++ b/docs/13-horizontal-scaling.md@@ -151,21 +151,25 @@> every form for the rooms to match.>> **The fan-out to other replicas is best effort and keeps no replay queue.** A Redis that pauses for less-> than 10 s (a snapshot fork, a slow fsync) answers afterwards, and every event raised meanwhile reaches-> the other replicas late. A longer silence, a restart, a failover or an outage drops the connection a node-> publishes on, and what was written to it may be lost with it, or may still reach the other replicas late-> and after newer events, which their clients then apply out of order. The events that node raises while it is-> disconnected, plus its reconnect backoff of up to 5 s once Redis is back, are dropped for the clients-> connected to the other replicas: they are not delivered late and not twice. Clients connected to the node-> that raised an event are served directly, and webhooks do not use this path, so neither is affected. Memory-> is bounded in two ways. A Redis that sends nothing for 10 s has the connection destroyed, and the node holds-> nothing more for it; what it had already held, at most 10 s of events, is freed once the node has reconnected.-> A Redis that keeps answering, but slower than events are raised, is never silent for 10 s, so a cap bounds-> it instead: once the node holds 12,000 unanswered events (10 s at 1,000 events a second, with a fifth to-> spare) and the backlog has outlived 2 s, meaning the count has not been below 6,000 for that long, it drops-> further frames for the other replicas until the count is back below 6,000, with a warning when dropping-> starts and another, with the count, when it ends. A burst raised in a single tick is never cut, however-> large: a healthy Redis answers it right after, and the node holds it until then. A replica whose own Redis+> than 10 s (a snapshot fork, a slow fsync) answers afterwards, and every event raised meanwhile reaches the+> other replicas late, at up to about 1,200 events a second; above that the cap described below applies+> first and drops the frames beyond it. A longer silence, a restart, a failover or an outage drops the+> connection a node publishes on, and what was written to it may be lost with it, or may still reach the+> other replicas late and after newer events, which their clients then apply out of order. The events that+> node raises while it is disconnected, plus its reconnect backoff of up to 5 s once Redis is back, are+> dropped for the clients connected to the other replicas: they are not delivered late and not twice.+> Clients connected to the node that raised an event are served directly, and webhooks do not use this path,+> so neither is affected. Memory is bounded in two ways. A Redis that sends nothing for 10 s has the+> connection destroyed, and the node holds nothing more for it; what it had already held, at most 10 s of+> events, is freed once the node has reconnected. A Redis that keeps answering, but slower than events are+> raised, is never silent for 10 s, so a cap bounds it instead: once the node holds 12,000 unanswered events+> (10 s at 1,000 events a second, with a fifth to spare) and the backlog has outlived 2 s, meaning the count+> has not been below 6,000 for that long, it drops further frames for the other replicas until the count is+> back below 6,000, with a warning when dropping starts and another, with the count, when it ends. The node+> can hold the traffic of those 2 s on top of the 12,000. A burst raised in a single tick is not cut as long+> as raising it past its 6,000th frame takes under 2 s: a healthy Redis answers it right after, and the node+> holds it until then. A synchronous burst that takes longer than that to raise past 6,000 frames is cut at+> 12,000 even against a healthy Redis, and the warning names the unanswered count. A replica whose own Redis> connection is reconnecting also misses what the others publish meanwhile.>> **Mid-connection key eviction converges on a timer, not a broadcast, and only on one node.** Thediff --git a/src/modules/events/redis-io.adapter.spec.ts b/src/modules/events/redis-io.adapter.spec.tsindex 1529a0569..026083ce9 100644--- a/src/modules/events/redis-io.adapter.spec.ts+++ b/src/modules/events/redis-io.adapter.spec.ts@@ -270,6 +270,8 @@ describe('RedisIoAdapter', () => {it('pins the figures docs/13 quotes, and a 10 s silence at 1000 events a second fits under them', () => {expect(WS_REDIS_MAX_UNANSWERED).toBe(12_000);expect(WS_REDIS_MAX_UNANSWERED).toBeGreaterThanOrEqual((WS_REDIS_SOCKET_TIMEOUT_MS / 1000) * 1000 * 1.2);+ // "Up to about 1,200 events a second" in docs/13: the rate at which a pause of one socket deadline fills the cap.+ expect(WS_REDIS_MAX_UNANSWERED / (WS_REDIS_SOCKET_TIMEOUT_MS / 1000)).toBe(1200);});// Slowness is judged by how long the backlog has lasted, not by its depth: a burst raised in one tick is all on thediff --git a/src/modules/events/redis-io.adapter.ts b/src/modules/events/redis-io.adapter.tsindex 39276e081..fbcdbcc36 100644--- a/src/modules/events/redis-io.adapter.ts+++ b/src/modules/events/redis-io.adapter.ts@@ -34,11 +34,14 @@ export const WS_REDIS_SOCKET_TIMEOUT_MS = 10_000;* than events are raised never trips the socket deadline, so without a cap the backlog grows with the load for as* long as the slowness lasts. The rule: frames are dropped for the other replicas while the queue holds this many* commands and has not been below half of it for longer than a command is given to be answered- * (WS_REDIS_COMMAND_TIMEOUT_MS), and they go on being dropped until the queue is back below half of it. A burst- * raised in one tick is never cut, however large, because depth alone says nothing: it is all on the queue before a- * reply can be read. Dropping starts with a warning and ends with another, with the count. 12,000 holds a full- * socket deadline of traffic at 1,000 events a second with a fifth to spare, which measured about 25 MB of heap with- * 1 KiB frames. It counts commands, not bytes: events with inline media are larger.+ * (WS_REDIS_COMMAND_TIMEOUT_MS), and they go on being dropped until the queue is back below half of it. Depth alone+ * says nothing, since a burst raised in one tick is all on the queue before a reply can be read, so such a burst is+ * not cut as long as raising it past half the cap takes less than a command is given. One that takes longer, a+ * synchronous loop that cannot yield for over 2 s, is cut at the cap whatever the peer is doing, and the warning+ * names the unanswered count. Dropping starts with a warning and ends with another, with the count. The queue can+ * pass the cap by the traffic of one command deadline before dropping starts. 12,000 holds a full socket deadline+ * of traffic at 1,000 events a second with a fifth to spare, which measured about 25 MB of heap with 1 KiB frames.+ * It counts commands, not bytes: events with inline media are larger.*/export const WS_REDIS_MAX_UNANSWERED = 12_000;@@ -107,13 +110,15 @@ export function rejectRootNamespace(server: Server): void {** Delivery while Redis is away or slow: the fan-out to the other replicas is best effort and keeps no replay* queue. A Redis that pauses for less than WS_REDIS_SOCKET_TIMEOUT_MS answers afterwards, and every event raised- * meanwhile reaches them late. Silence for that long destroys the publisher's socket, and so does a restart, a+ * meanwhile reaches them late, at up to about 1,200 events a second; above that the cap below applies first, and+ * drops the frames beyond it. Silence for that long destroys the publisher's socket, and so does a restart, a* failover or an outage: what was written to the dropped connection may be lost with it, or may still reach the* other replicas late and after newer events. The events this node raises while it is disconnected, plus its* reconnect backoff of up to 5 s once Redis is back (wsRedisOptions), are dropped for the other replicas, never* delivered late and never twice. A Redis that keeps answering, but slower than events are raised, never looks* silent: it is bounded by WS_REDIS_MAX_UNANSWERED, which drops frames for the other replicas once the backlog has- * outlived a command's deadline and until it is back below half of it (a burst raised in one tick is not cut).+ * outlived a command's deadline and until it is back below half of it (a burst raised in one tick is not cut+ * unless raising it past half the cap takes over 2 s).* Clients on this node are served by the local broadcast and webhooks do not use this path, so neither is* affected. What the publisher holds is bounded by one socket deadline of traffic for a silent Redis and by that* cap, plus the traffic of one command deadline, for a slow one.