Free, Open Source, Self-Hosted WhatsApp API Gateway
TypeScript apibotgatewayself-hostedwhatsappwhatsapp-apiwhatsapp-automationwhatsapp-botwhatsapp-gatewaywhatsapp-gateway-apiwhatsapp-webwhatsapp-web-api
docs(queue): stop promising per-key isolation in the worker pools
Yudhi Armyndharis65b455d7a7e5
diff --git a/.env.example b/.env.exampleindex 9fb4b9368..e61cdcb2f 100644--- a/.env.example+++ b/.env.example@@ -696,7 +696,14 @@ WEBHOOK_SSRF_PROTECT=true# Concurrency of the BullMQ workers that process queued webhooks and ingress events. Only meaningful# when QUEUE_ENABLED=true. Ordering within one conversation is preserved by a per-conversation lock,-# so raising these parallelizes unrelated conversations rather than reordering them.+# so raising these parallelizes unrelated conversations rather than reordering them. Neither pool is+# isolated per webhook or per conversation:+# - A dead webhook receiver holds a slot for up to WEBHOOK_TIMEOUT per attempt, retries included, so+# keep WEBHOOK_WORKER_CONCURRENCY above its events per second x attempts x WEBHOOK_TIMEOUT (in+# seconds), or every other webhook waits behind it.+# - An ingress job waiting on a busy conversation still holds a slot, so keep+# INGRESS_WORKER_CONCURRENCY above the largest burst on one conversation, and declare a+# conversationId pointer on the route (without one, the whole instance is one conversation).# WEBHOOK_WORKER_CONCURRENCY=10 # webhook delivery workers (default 10)# INGRESS_WORKER_CONCURRENCY=10 # integration ingress workers (default 10)diff --git a/docs/25-integration-fabric.md b/docs/25-integration-fabric.mdindex 9fe781d83..b4f19b95e 100644--- a/docs/25-integration-fabric.md+++ b/docs/25-integration-fabric.md@@ -211,6 +211,12 @@ PostgreSQL state. When the queue is disabled, ingress dispatches inline after peserialize concurrent same-conversation deliveries. Providers already deliver over unordered,at-least-once HTTP, so plugin handlers must be idempotent and treat ingress as a reconciliation trigger.+A job waiting on that lock still holds one of the `INGRESS_WORKER_CONCURRENCY` worker slots (default+10). A burst on one lane larger than that fills every slot with same-lane waiters, and events for other+conversations and instances queue behind the burst until it drains. Size `INGRESS_WORKER_CONCURRENCY`+above the largest burst you expect on a single lane, and declare a `conversationId` pointer on the+route so the lane is one conversation; without it the lane is the whole instance.+Persist-before-acknowledge alone is not delivery: a crash between the persist and the enqueue, or afire-and-forget enqueue on a `response` route whose outcome is never recorded, would strand the rowwith the provider already acknowledged. The **ingress reconciler** closes that window: everydiff --git a/src/modules/queue/processors/ingress.processor.ts b/src/modules/queue/processors/ingress.processor.tsindex 6e407041f..645d4c3e2 100644--- a/src/modules/queue/processors/ingress.processor.ts+++ b/src/modules/queue/processors/ingress.processor.ts@@ -30,10 +30,15 @@ export interface IngressJobData {// The KeyedAsyncLock wrapping dispatch below guarantees no two dispatches for the SAME conversation// run concurrently (mutual exclusion + in-order START for events as they reach the worker), so the-// worker no longer needs concurrency 1 — raising it parallelizes UNRELATED conversations instead of-// head-of-line-blocking every inbound event behind the slowest one. Strict end-to-end order is NOT-// preserved across a BullMQ retry: a retried job re-enters lock.run() after its backoff and chains at-// the conversation's CURRENT tail, so it can overtake a same-conversation successor that dispatched+// worker no longer needs concurrency 1: raising it parallelizes unrelated conversations. The lock is+// taken inside the job, though, so a job waiting on a busy key still holds a worker slot. A burst on+// one key larger than the concurrency fills every slot with waiters, and other keys queue behind it+// until it drains. The key is per instance when the route declares no conversationId pointer, so+// routes should declare one, and the concurrency should exceed the largest expected per-key burst.+//+// Strict end-to-end order is NOT preserved across a BullMQ retry: a retried job re-enters lock.run()+// after its backoff and chains at the conversation's CURRENT tail, so it can overtake a+// same-conversation successor that dispatched// during the backoff window. This is a deliberate tradeoff — BullMQ retries release the worker slot// during backoff (better throughput under transient failure), where retrying inside the lock would// hold it — and is acceptable because ingress order is best-effort regardless: the provider deliversdiff --git a/src/modules/queue/redis-connection.ts b/src/modules/queue/redis-connection.tsindex 6c62975eb..1122208ad 100644--- a/src/modules/queue/redis-connection.ts+++ b/src/modules/queue/redis-connection.ts@@ -37,9 +37,14 @@ const DEFAULT_WEBHOOK_WORKER_CONCURRENCY = 10;/*** Webhook Worker concurrency. BullMQ defaults a Worker to 1, which serializes ALL webhook deliveries* process-wide: one slow or timing-out receiver head-of-line-blocks every other session's webhooks- * until it finishes (up to WEBHOOK_TIMEOUT + retries). Running several in parallel decouples healthy- * receivers from a stuck one. Override via WEBHOOK_WORKER_CONCURRENCY; a non-positive/garbage value- * falls back to the default. (Read at module import like workerConnectionOptions above.)+ * until it finishes (up to WEBHOOK_TIMEOUT + retries). Running several in parallel lets healthy+ * receivers proceed while a few slots wait on a stuck one, but the pool is shared and not isolated+ * per webhook: each attempt against a dead receiver holds a slot for up to WEBHOOK_TIMEOUT, and its+ * retries re-enter the same pool. Once that receiver's events per second x attempts x+ * WEBHOOK_TIMEOUT (in seconds) reach the concurrency, every slot is busy with it and all other+ * webhooks wait. Size the value above that product. Override via WEBHOOK_WORKER_CONCURRENCY; a+ * non-positive/garbage value falls back to the default. (Read at module import like+ * workerConnectionOptions above.)*/export function webhookWorkerConcurrency(): number {const parsed = parseInt(process.env.WEBHOOK_WORKER_CONCURRENCY || '', 10);@@ -52,8 +57,10 @@ const DEFAULT_INGRESS_WORKER_CONCURRENCY = 10;/*** Ingress Worker concurrency. Ordering within a conversation is now guaranteed by the* per-conversation KeyedAsyncLock in the processor, not by a single-worker queue, so raising- * concurrency here parallelizes unrelated conversations instead of head-of-line-blocking every- * inbound event behind the slowest one. Override via INGRESS_WORKER_CONCURRENCY; a+ * concurrency here parallelizes unrelated conversations. A job waiting on a busy key still holds a+ * slot, though: a burst on one key larger than this value blocks every other key until it drains.+ * Size it above the largest expected per-key burst; the key is the whole instance unless the route+ * declares a conversationId pointer. Override via INGRESS_WORKER_CONCURRENCY; a* non-positive/garbage value falls back to the default.*/export function ingressWorkerConcurrency(): number {