Skip to content

Free, Open Source, Self-Hosted WhatsApp API Gateway

fix(webhook): share an opt-in stage among a session's webhooks

Yudhi Armyndharis6af59ffa5c8e
diff --git a/.env.example b/.env.example
index 8392755bd..f21bba4ac 100644
--- a/.env.example
+++ b/.env.example
@@ -466,9 +466,9 @@ WEBHOOK_RETRY_DELAY=5000 # Base delay between retries in ms (doubles
# Deliveries to one session's failing receivers park at most
# a quarter of it, so other sessions keep room to wait.
# Opt-in events such as message.receipt are bounded per session
- # instead: a quarter of WEBHOOK_DISPATCH_CONCURRENCY run at
- # once and three quarters of this wait, and further ones are
- # dropped, not replayed, and counted in
+ # instead: a quarter of WEBHOOK_DISPATCH_CONCURRENCY deliveries
+ # run at once and three quarters of this many events wait, and
+ # further ones are dropped, not replayed, and counted in
# openwa_webhook_delivery_failures_total.
# Each inline delivery holds its event's inline media (up to
# 4/3 x WEBHOOK_MEDIA_INLINE_MAX_BYTES as base64), up to about
diff --git a/docs/06-api-specification.md b/docs/06-api-specification.md
index caa9ad444..4f19288d0 100644
--- a/docs/06-api-specification.md
+++ b/docs/06-api-specification.md
@@ -7229,7 +7229,7 @@ These are the events OpenWA actually emits. A webhook is registered with an `eve
| `message.received` | An inbound message arrives | The full message object: `id`, `from`, `to`, `body`, `type`, `timestamp` (epoch **seconds**), `isGroup`, `kind` (user-facing chat discriminator of `chatId` — `individual\|group\|channel\|status\|broadcast\|unknown`), `fromMe`, `chatId`, `media{mimetype,filename?,data?,omitted?,sizeBytes?}` (media messages only), `contact{…}` (plus optional `senderPhone` for `@lid` senders, `order{orderId,token?}` / `product{productId,…}` on `order` / `product` messages, and `poll{name,options,allowMultipleAnswers}` on a poll creation message; on Baileys also `buttons:[{id,text}]` on a WhatsApp Business prompt that offers choices, and `button{id,text?}` when the sender tapped one) |
| `message.sent` | An outbound message is created/sent from this session | Same message object shape as `message.received` |
| `message.ack` | A delivery/read receipt updates an outbound message. On Baileys only 1:1 receipts advance it; opt-in `message.receipt` reports group and status receipts per participant and leaves the message at `sent`, see below | `{ id, messageId, status, ack, chatId? }` — `status` is the canonical state (`pending`/`sent`/`delivered`/`read`/`failed`); `ack` is the deprecated legacy integer derived from it; `chatId` is the conversation the acked message belongs to, present whenever the engine's update names the chat (both engines do on the live path) |
-| `message.receipt` | One recipient receives, reads or plays a group or status message the account sent (**Baileys only**; not included in `*`) | `{ id, messageId, chatId, participant, status, timestamp }` ; one event per recipient and status; `status` is `delivered`/`read`/`played`, `timestamp` is epoch **seconds**. Receipts from the account's own linked devices are not reported. On Baileys 7.0.0-rc14 a played voice note reports `read`. Nothing is stored and `message.ack` is not advanced by it; per session a quarter of `WEBHOOK_DISPATCH_CONCURRENCY` run at once, three quarters of `WEBHOOK_DISPATCH_MAX_QUEUED` wait, and further ones are dropped, not replayed (see Delivery semantics) |
+| `message.receipt` | One recipient receives, reads or plays a group or status message the account sent (**Baileys only**; not included in `*`) | `{ id, messageId, chatId, participant, status, timestamp }` ; one event per recipient and status; `status` is `delivered`/`read`/`played`, `timestamp` is epoch **seconds**. Receipts from the account's own linked devices are not reported. On Baileys 7.0.0-rc14 a played voice note reports `read`. Nothing is stored and `message.ack` is not advanced by it; per session a quarter of `WEBHOOK_DISPATCH_CONCURRENCY` deliveries run at once, three quarters of `WEBHOOK_DISPATCH_MAX_QUEUED` events wait, and further ones are dropped, not replayed (see Delivery semantics) |
| `message.failed` | A receipt resolves to `failed` (dispatched in addition to `message.ack`) | `{ id, messageId, status: "failed", ack: -1, chatId? }`; `chatId` follows the same rules as `message.ack` |
| `message.revoked` | A message is deleted/recalled | `{ id, revokedId?, chatId, from, to, type: "revoked", body: "", timestamp }` — **reconcile on `revokedId`** (the original deleted message's id), falling back to `id`. On whatsapp-web.js `id` is the _revocation notification_ (a distinct message that won't match a stored id) and `revokedId` may be absent when the original isn't cached locally; on Baileys the two coincide |
| `message.reaction` | A reaction is added, changed, or removed | `{ messageId, chatId, reaction, senderId, reactions? }` — `reactions` is the post-apply `{ senderId: emoji }` snapshot, omitted when the gateway holds no stored copy of the message to compute it from (an ephemeral message, or one predating the session going live); treat it as unknown rather than empty and keep the map you already hold. `reaction` is empty when removed |
@@ -7301,7 +7301,7 @@ Webhook delivery is **at-least-once**. A consumer can legitimately receive the s
Outbox creation is best effort, and message persistence and outbox insertion are separate transactions. A write failure or crash between them can still lose the webhook delivery. Failure records with zero attempts are excluded from operator redrive. Capacity and shutdown rejections can still be recovered from an unsettled outbox record; shutdown during retry backoff may follow earlier POSTs. The direct dispatcher leaves its local in-flight work alone during reconciliation. Graceful shutdown drains current queue jobs for at most 10 s; direct sends, an operator redrive replay in flight, and an outbox replay in flight with its plugin hooks, get up to `WEBHOOK_SHUTDOWN_DRAIN_MS` (default 5s). That replay sends at most one more POST, bounded by `WEBHOOK_TIMEOUT`, and no further row is replayed; the next start replays what it left pending.
-**Opt-in events are bounded, not replayed.** `message.receipt` fires once per recipient and status, so one flush of buffered receipts can be thousands of events in a tick. Each session passes a queue of its own before the shared dispatch pool: a quarter of `WEBHOOK_DISPATCH_CONCURRENCY` of its receipts run at once and up to three quarters of `WEBHOOK_DISPATCH_MAX_QUEUED` more wait their turn, which with the defaults delivers a flush of 500 in full and 754 of a flush of 2000. Any further receipt is dropped: it leaves no outbox or failure row and is not replayed, and it is counted in `openwa_webhook_delivery_failures_total` and logged once per burst (`webhook_opt_in_dropping` when the session starts dropping, then `webhook_opt_in_dropped` with the count). Because the queue is per session, one session's flood, or a receiver of its that never answers, does not shed or delay another session's events. Receipts still waiting at shutdown are dropped without being counted.
+**Opt-in events are bounded, not replayed.** `message.receipt` fires once per recipient and status, so one flush of buffered receipts can be thousands of events in a tick. Each session passes a queue of its own before the shared dispatch pool: a quarter of `WEBHOOK_DISPATCH_CONCURRENCY` deliveries of its receipts run at once, shared by the webhooks that name `message.receipt` since one receipt is a delivery to each, and up to three quarters of `WEBHOOK_DISPATCH_MAX_QUEUED` more receipts wait their turn, which with the defaults delivers a flush of 500 in full and at least 750 of a flush of 2000. Any further receipt is dropped: it leaves no outbox or failure row and is not replayed, and it is counted in `openwa_webhook_delivery_failures_total` and logged once per burst (`webhook_opt_in_dropping` when the session starts dropping, then `webhook_opt_in_dropped` with the count). Because the queue is per session, one session's flood, or a receiver of its that never answers, does not shed or delay another session's events. Receipts still waiting at shutdown are dropped without being counted.
Before downgrading to a version that does not recognize the `queued` outbox state, drain or recover those records and take a backup. Its cleanup may otherwise treat an unsettled queued record as settled. Reverting the terminal-identity migration removes its index but does not restore historical duplicates.
diff --git a/docs/10-devops-infrastructure.md b/docs/10-devops-infrastructure.md
index 396b4a243..17724099f 100644
--- a/docs/10-devops-infrastructure.md
+++ b/docs/10-devops-infrastructure.md
@@ -504,8 +504,9 @@ WEBHOOK_DISPATCH_MAX_QUEUED=1000
# The first 2xx from the webhook lifts it. With the queue disabled, a session parks at most a
# quarter of WEBHOOK_DISPATCH_MAX_QUEUED behind that limit and sheds the rest, so other sessions
# keep room. Opt-in events such as message.receipt are bounded per session instead: a quarter of
-# WEBHOOK_DISPATCH_CONCURRENCY run at once and three quarters of WEBHOOK_DISPATCH_MAX_QUEUED wait,
-# and further ones are dropped, not replayed, and counted in openwa_webhook_delivery_failures_total.
+# WEBHOOK_DISPATCH_CONCURRENCY deliveries run at once and three quarters of
+# WEBHOOK_DISPATCH_MAX_QUEUED events wait, and further ones are dropped, not replayed, and counted
+# in openwa_webhook_delivery_failures_total.
# Each running, parked or backoff-waiting inline delivery holds its event's inline media
# (up to 4/3 x WEBHOOK_MEDIA_INLINE_MAX_BYTES as base64), about 1.3 GiB per process at the defaults
# when every slot is taken; lower these caps or enable the queue to reduce it.
diff --git a/src/modules/webhook/webhook-delivery.service.spec.ts b/src/modules/webhook/webhook-delivery.service.spec.ts
index ddf085be8..2f37a4155 100644
--- a/src/modules/webhook/webhook-delivery.service.spec.ts
+++ b/src/modules/webhook/webhook-delivery.service.spec.ts
@@ -2991,6 +2991,36 @@ describe('WebhookDeliveryService', () => {
await Promise.all(failing);
});
+ // One event is a delivery to each webhook that names it, so the stage's share of the pool is divided among them:
+ // otherwise a session with several receipt webhooks, all with receivers that never answer, would hold a
+ // multiple of its share.
+ it("shares a session's quarter of the pool among the webhooks that name the event", async () => {
+ internals().dispatchLimiter = new ConcurrencyLimiter(16, 8); // a quarter of 16 slots is 4 deliveries
+ hooksBySession['sess-m'] = [
+ // The API does not refuse a repeated event name, and a webhook that lists it twice is still one delivery.
+ createMockWebhook({
+ id: 'wh-m1',
+ url: 'https://receipt.one.example/hook',
+ events: ['message.receipt', 'message.receipt'],
+ }),
+ createMockWebhook({ id: 'wh-m2', url: 'https://receipt.two.example/hook', events: ['message.receipt'] }),
+ // These two do not receive receipts: one names another event, the other is a catch-all.
+ createMockWebhook({ id: 'wh-m3', url: 'https://other.example/hook', events: ['message.received'] }),
+ createMockWebhook({ id: 'wh-m4', url: 'https://other.example/all', events: ['*'] }),
+ ];
+ const receipts = burst(10, 'sess-m');
+ await settle(() => hangingReceipts.length > 4);
+
+ // Two events at a time, each to its two receipt webhooks: 4 deliveries, not 8 and not 2.
+ expect(hangingReceipts).toHaveLength(4);
+ expect(outboxService.open).toHaveBeenCalledTimes(4);
+
+ await drain(receipts);
+
+ // Two running and six waiting events were admitted, each delivered to both webhooks; two were dropped.
+ expect(urlsCalled()).toHaveLength(16);
+ });
+
it('forgets a session once its events have settled, and counts the next burst afresh', async () => {
const warn = warnSpy();
for (let round = 0; round < 2; round++) {
diff --git a/src/modules/webhook/webhook-delivery.service.ts b/src/modules/webhook/webhook-delivery.service.ts
index 5b5530730..80ea4911f 100644
--- a/src/modules/webhook/webhook-delivery.service.ts
+++ b/src/modules/webhook/webhook-delivery.service.ts
@@ -194,18 +194,22 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {
*/
private readonly locallyPending = new Map<string, number>();
/**
- * Per session, the opt-in events (WILDCARD_EXCLUDED_EVENTS) that at least one active webhook names.
- * The promise is cached, not its result, so a burst of receipts on a cold entry shares one lookup.
+ * Per session, the opt-in events (WILDCARD_EXCLUDED_EVENTS) that active webhooks name, with how many
+ * name each. The promise is cached, not its result, so a burst of receipts on a cold entry shares one lookup.
*/
- private readonly optInSubscriptions = new Map<string, { events: Promise<ReadonlySet<string>>; expiresAt: number }>();
+ private readonly optInSubscriptions = new Map<
+ string,
+ { named: Promise<ReadonlyMap<string, number>>; expiresAt: number }
+ >();
/**
* Per session, the stage its opt-in events (WILDCARD_EXCLUDED_EVENTS) pass before the shared dispatch
* pool. One flush of buffered receipts is thousands of dispatches in a tick: a quarter of the dispatch
- * concurrency of them run at once, up to three quarters of WEBHOOK_DISPATCH_MAX_QUEUED wait their turn
- * (holding only the event), and the rest are dropped before they open an outbox row, a failure row or a
- * hook call. The pool never sees more than the running ones, so a flood, or a receiver that never
- * answers, cannot shed or hold back what another session sends. An entry lives while its session has
- * events in the stage; `dropped` counts those turned away since it was created.
+ * concurrency runs at once, shared by the webhooks that name the event since one event is a delivery to
+ * each, up to three quarters of WEBHOOK_DISPATCH_MAX_QUEUED events wait their turn (holding only the
+ * event), and the rest are dropped before they open an outbox row, a failure row or a hook call. The pool
+ * never sees more than a session's running deliveries, so a flood, or a receiver that never answers,
+ * cannot shed or hold back what another session sends. An entry lives while its session has events in
+ * the stage; `dropped` counts those turned away since it was created.
*/
private readonly optInStages = new Map<string, { stage: ConcurrencyLimiter; dropped: number }>();
@@ -305,14 +309,15 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {
if (!WILDCARD_EXCLUDED_EVENTS.includes(event)) return this.fanOut(sessionId, event, data);
// An opt-in event fires per recipient (about 2,000 per message to a 1,000-member group): skip it
// before the per-dispatch webhook read when no active webhook of the session names it.
- if (!(await this.sessionNamesOptInEvent(sessionId, event))) return;
+ const named = await this.webhooksNamingOptInEvent(sessionId, event);
+ if (named === 0) return;
let entry = this.optInStages.get(sessionId);
if (!entry) {
- // A quarter of the pool runs at once (the limiter keeps at least one) and three quarters of the park bound
- // wait, at least one place.
+ // A quarter of the pool runs at once, divided among the webhooks that name the event (the limiter keeps at
+ // least one event running), and three quarters of the park bound wait, at least one place.
entry = {
stage: new ConcurrencyLimiter(
- Math.floor(this.dispatchLimiter.maxCount / 4),
+ Math.floor(this.dispatchLimiter.maxCount / 4 / named),
Math.max(1, Math.floor((this.dispatchMaxQueued * 3) / 4)),
),
dropped: 0,
@@ -396,38 +401,40 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {
}
/**
- * Whether an active webhook of the session names `event`, read at most once per
- * OPT_IN_SUBSCRIPTIONS_TTL_MS. A failed lookup is not cached and lets the event through, so
- * dispatch's own lookup reports the failure as before.
+ * How many active webhooks of the session name `event`, read at most once per
+ * OPT_IN_SUBSCRIPTIONS_TTL_MS. A failed lookup is not cached and counts as one, which lets the event
+ * through, so dispatch's own lookup reports the failure as before.
*/
- private async sessionNamesOptInEvent(sessionId: string, event: string): Promise<boolean> {
+ private async webhooksNamingOptInEvent(sessionId: string, event: string): Promise<number> {
const now = Date.now();
let entry = this.optInSubscriptions.get(sessionId);
if (!entry || entry.expiresAt <= now) {
const created = {
- events: this.webhookRepository
+ named: this.webhookRepository
.find({ select: { id: true, events: true }, where: { sessionId, active: true } })
.then(rows => {
- const named = new Set<string>();
+ const named = new Map<string, number>();
for (const row of rows) {
if (!Array.isArray(row.events)) continue;
- for (const e of row.events) if (WILDCARD_EXCLUDED_EVENTS.includes(e)) named.add(e);
+ for (const e of new Set(row.events)) {
+ if (WILDCARD_EXCLUDED_EVENTS.includes(e)) named.set(e, (named.get(e) ?? 0) + 1);
+ }
}
return named;
}),
expiresAt: now + OPT_IN_SUBSCRIPTIONS_TTL_MS,
};
// Identity check: an invalidation (or a newer entry) since this lookup started must not be undone.
- created.events.catch(() => {
+ created.named.catch(() => {
if (this.optInSubscriptions.get(sessionId) === created) this.optInSubscriptions.delete(sessionId);
});
this.optInSubscriptions.set(sessionId, created);
entry = created;
}
try {
- return (await entry.events).has(event);
+ return (await entry.named).get(event) ?? 0;
} catch {
- return true;
+ return 1;
}
}