Skip to content

Free, Open Source, Self-Hosted WhatsApp API Gateway

docs(queue): stop promising per-key isolation in the worker pools

Yudhi Armyndharis65b455d7a7e5
diff --git a/.env.example b/.env.example
index 9fb4b9368..e61cdcb2f 100644
--- a/.env.example
+++ b/.env.example
@@ -696,7 +696,14 @@ WEBHOOK_SSRF_PROTECT=true
# Concurrency of the BullMQ workers that process queued webhooks and ingress events. Only meaningful
# when QUEUE_ENABLED=true. Ordering within one conversation is preserved by a per-conversation lock,
-# so raising these parallelizes unrelated conversations rather than reordering them.
+# so raising these parallelizes unrelated conversations rather than reordering them. Neither pool is
+# isolated per webhook or per conversation:
+# - A dead webhook receiver holds a slot for up to WEBHOOK_TIMEOUT per attempt, retries included, so
+# keep WEBHOOK_WORKER_CONCURRENCY above its events per second x attempts x WEBHOOK_TIMEOUT (in
+# seconds), or every other webhook waits behind it.
+# - An ingress job waiting on a busy conversation still holds a slot, so keep
+# INGRESS_WORKER_CONCURRENCY above the largest burst on one conversation, and declare a
+# conversationId pointer on the route (without one, the whole instance is one conversation).
# WEBHOOK_WORKER_CONCURRENCY=10 # webhook delivery workers (default 10)
# INGRESS_WORKER_CONCURRENCY=10 # integration ingress workers (default 10)
diff --git a/docs/25-integration-fabric.md b/docs/25-integration-fabric.md
index 9fe781d83..b4f19b95e 100644
--- a/docs/25-integration-fabric.md
+++ b/docs/25-integration-fabric.md
@@ -211,6 +211,12 @@ PostgreSQL state. When the queue is disabled, ingress dispatches inline after pe
serialize concurrent same-conversation deliveries. Providers already deliver over unordered,
at-least-once HTTP, so plugin handlers must be idempotent and treat ingress as a reconciliation trigger.
+A job waiting on that lock still holds one of the `INGRESS_WORKER_CONCURRENCY` worker slots (default
+10). A burst on one lane larger than that fills every slot with same-lane waiters, and events for other
+conversations and instances queue behind the burst until it drains. Size `INGRESS_WORKER_CONCURRENCY`
+above the largest burst you expect on a single lane, and declare a `conversationId` pointer on the
+route so the lane is one conversation; without it the lane is the whole instance.
+
Persist-before-acknowledge alone is not delivery: a crash between the persist and the enqueue, or a
fire-and-forget enqueue on a `response` route whose outcome is never recorded, would strand the row
with the provider already acknowledged. The **ingress reconciler** closes that window: every
diff --git a/src/modules/queue/processors/ingress.processor.ts b/src/modules/queue/processors/ingress.processor.ts
index 6e407041f..645d4c3e2 100644
--- a/src/modules/queue/processors/ingress.processor.ts
+++ b/src/modules/queue/processors/ingress.processor.ts
@@ -30,10 +30,15 @@ export interface IngressJobData {
// The KeyedAsyncLock wrapping dispatch below guarantees no two dispatches for the SAME conversation
// run concurrently (mutual exclusion + in-order START for events as they reach the worker), so the
-// worker no longer needs concurrency 1 — raising it parallelizes UNRELATED conversations instead of
-// head-of-line-blocking every inbound event behind the slowest one. Strict end-to-end order is NOT
-// preserved across a BullMQ retry: a retried job re-enters lock.run() after its backoff and chains at
-// the conversation's CURRENT tail, so it can overtake a same-conversation successor that dispatched
+// worker no longer needs concurrency 1: raising it parallelizes unrelated conversations. The lock is
+// taken inside the job, though, so a job waiting on a busy key still holds a worker slot. A burst on
+// one key larger than the concurrency fills every slot with waiters, and other keys queue behind it
+// until it drains. The key is per instance when the route declares no conversationId pointer, so
+// routes should declare one, and the concurrency should exceed the largest expected per-key burst.
+//
+// Strict end-to-end order is NOT preserved across a BullMQ retry: a retried job re-enters lock.run()
+// after its backoff and chains at the conversation's CURRENT tail, so it can overtake a
+// same-conversation successor that dispatched
// during the backoff window. This is a deliberate tradeoff — BullMQ retries release the worker slot
// during backoff (better throughput under transient failure), where retrying inside the lock would
// hold it — and is acceptable because ingress order is best-effort regardless: the provider delivers
diff --git a/src/modules/queue/redis-connection.ts b/src/modules/queue/redis-connection.ts
index 6c62975eb..1122208ad 100644
--- a/src/modules/queue/redis-connection.ts
+++ b/src/modules/queue/redis-connection.ts
@@ -37,9 +37,14 @@ const DEFAULT_WEBHOOK_WORKER_CONCURRENCY = 10;
/**
* Webhook Worker concurrency. BullMQ defaults a Worker to 1, which serializes ALL webhook deliveries
* process-wide: one slow or timing-out receiver head-of-line-blocks every other session's webhooks
- * until it finishes (up to WEBHOOK_TIMEOUT + retries). Running several in parallel decouples healthy
- * receivers from a stuck one. Override via WEBHOOK_WORKER_CONCURRENCY; a non-positive/garbage value
- * falls back to the default. (Read at module import like workerConnectionOptions above.)
+ * until it finishes (up to WEBHOOK_TIMEOUT + retries). Running several in parallel lets healthy
+ * receivers proceed while a few slots wait on a stuck one, but the pool is shared and not isolated
+ * per webhook: each attempt against a dead receiver holds a slot for up to WEBHOOK_TIMEOUT, and its
+ * retries re-enter the same pool. Once that receiver's events per second x attempts x
+ * WEBHOOK_TIMEOUT (in seconds) reach the concurrency, every slot is busy with it and all other
+ * webhooks wait. Size the value above that product. Override via WEBHOOK_WORKER_CONCURRENCY; a
+ * non-positive/garbage value falls back to the default. (Read at module import like
+ * workerConnectionOptions above.)
*/
export function webhookWorkerConcurrency(): number {
const parsed = parseInt(process.env.WEBHOOK_WORKER_CONCURRENCY || '', 10);
@@ -52,8 +57,10 @@ const DEFAULT_INGRESS_WORKER_CONCURRENCY = 10;
/**
* Ingress Worker concurrency. Ordering within a conversation is now guaranteed by the
* per-conversation KeyedAsyncLock in the processor, not by a single-worker queue, so raising
- * concurrency here parallelizes unrelated conversations instead of head-of-line-blocking every
- * inbound event behind the slowest one. Override via INGRESS_WORKER_CONCURRENCY; a
+ * concurrency here parallelizes unrelated conversations. A job waiting on a busy key still holds a
+ * slot, though: a burst on one key larger than this value blocks every other key until it drains.
+ * Size it above the largest expected per-key burst; the key is the whole instance unless the route
+ * declares a conversationId pointer. Override via INGRESS_WORKER_CONCURRENCY; a
* non-positive/garbage value falls back to the default.
*/
export function ingressWorkerConcurrency(): number {