Free, Open Source, Self-Hosted WhatsApp API Gateway
TypeScript apibotgatewayself-hostedwhatsappwhatsapp-apiwhatsapp-automationwhatsapp-botwhatsapp-gatewaywhatsapp-gateway-apiwhatsapp-webwhatsapp-web-api
docs(webhook): describe the opt-in bound per webhook
Yudhi Armyndharis061446c7805f
diff --git a/.env.example b/.env.exampleindex f21bba4ac..084b8e3a4 100644--- a/.env.example+++ b/.env.example@@ -465,11 +465,18 @@ WEBHOOK_RETRY_DELAY=5000 # Base delay between retries in ms (doubles# plus this; more are shed to the outbox.# Deliveries to one session's failing receivers park at most# a quarter of it, so other sessions keep room to wait.- # Opt-in events such as message.receipt are bounded per session- # instead: a quarter of WEBHOOK_DISPATCH_CONCURRENCY deliveries- # run at once and three quarters of this many events wait, and- # further ones are dropped, not replayed, and counted in- # openwa_webhook_delivery_failures_total.+ # Opt-in events such as message.receipt are bounded per receiving+ # webhook instead: a quarter of WEBHOOK_DISPATCH_CONCURRENCY+ # deliveries run at once, divided among the session's webhooks+ # that name the event and one at least for each, and three+ # quarters of this many events wait; further ones are dropped,+ # not replayed, and counted in+ # openwa_webhook_delivery_failures_total. A webhook whose+ # receiver never answers holds only its own places, but the pool+ # is shared: a session holds up to four at once (one per webhook+ # past four), and another session's events wait for a slot once+ # such sessions together hold all of them, four sessions at the+ # defaults. The events waiting are not capped process-wide.# Each inline delivery holds its event's inline media (up to# 4/3 x WEBHOOK_MEDIA_INLINE_MAX_BYTES as base64), up to about# 1.3 GiB per process at the defaults; QUEUE_ENABLED=true keepsdiff --git a/docs/06-api-specification.md b/docs/06-api-specification.mdindex 4f19288d0..1fe4aef9d 100644--- a/docs/06-api-specification.md+++ b/docs/06-api-specification.md@@ -7229,7 +7229,7 @@ These are the events OpenWA actually emits. A webhook is registered with an `eve| `message.received` | An inbound message arrives | The full message object: `id`, `from`, `to`, `body`, `type`, `timestamp` (epoch **seconds**), `isGroup`, `kind` (user-facing chat discriminator of `chatId` — `individual\|group\|channel\|status\|broadcast\|unknown`), `fromMe`, `chatId`, `media{mimetype,filename?,data?,omitted?,sizeBytes?}` (media messages only), `contact{…}` (plus optional `senderPhone` for `@lid` senders, `order{orderId,token?}` / `product{productId,…}` on `order` / `product` messages, and `poll{name,options,allowMultipleAnswers}` on a poll creation message; on Baileys also `buttons:[{id,text}]` on a WhatsApp Business prompt that offers choices, and `button{id,text?}` when the sender tapped one) || `message.sent` | An outbound message is created/sent from this session | Same message object shape as `message.received` || `message.ack` | A delivery/read receipt updates an outbound message. On Baileys only 1:1 receipts advance it; opt-in `message.receipt` reports group and status receipts per participant and leaves the message at `sent`, see below | `{ id, messageId, status, ack, chatId? }` — `status` is the canonical state (`pending`/`sent`/`delivered`/`read`/`failed`); `ack` is the deprecated legacy integer derived from it; `chatId` is the conversation the acked message belongs to, present whenever the engine's update names the chat (both engines do on the live path) |-| `message.receipt` | One recipient receives, reads or plays a group or status message the account sent (**Baileys only**; not included in `*`) | `{ id, messageId, chatId, participant, status, timestamp }` ; one event per recipient and status; `status` is `delivered`/`read`/`played`, `timestamp` is epoch **seconds**. Receipts from the account's own linked devices are not reported. On Baileys 7.0.0-rc14 a played voice note reports `read`. Nothing is stored and `message.ack` is not advanced by it; per session a quarter of `WEBHOOK_DISPATCH_CONCURRENCY` deliveries run at once, three quarters of `WEBHOOK_DISPATCH_MAX_QUEUED` events wait, and further ones are dropped, not replayed (see Delivery semantics) |+| `message.receipt` | One recipient receives, reads or plays a group or status message the account sent (**Baileys only**; not included in `*`) | `{ id, messageId, chatId, participant, status, timestamp }` ; one event per recipient and status; `status` is `delivered`/`read`/`played`, `timestamp` is epoch **seconds**. Receipts from the account's own linked devices are not reported. On Baileys 7.0.0-rc14 a played voice note reports `read`. Nothing is stored and `message.ack` is not advanced by it; per session a quarter of `WEBHOOK_DISPATCH_CONCURRENCY` deliveries run at once, split among its receipt webhooks; per webhook three quarters of `WEBHOOK_DISPATCH_MAX_QUEUED` events wait and more are dropped, not replayed (see Delivery semantics) || `message.failed` | A receipt resolves to `failed` (dispatched in addition to `message.ack`) | `{ id, messageId, status: "failed", ack: -1, chatId? }`; `chatId` follows the same rules as `message.ack` || `message.revoked` | A message is deleted/recalled | `{ id, revokedId?, chatId, from, to, type: "revoked", body: "", timestamp }` — **reconcile on `revokedId`** (the original deleted message's id), falling back to `id`. On whatsapp-web.js `id` is the _revocation notification_ (a distinct message that won't match a stored id) and `revokedId` may be absent when the original isn't cached locally; on Baileys the two coincide || `message.reaction` | A reaction is added, changed, or removed | `{ messageId, chatId, reaction, senderId, reactions? }` — `reactions` is the post-apply `{ senderId: emoji }` snapshot, omitted when the gateway holds no stored copy of the message to compute it from (an ephemeral message, or one predating the session going live); treat it as unknown rather than empty and keep the map you already hold. `reaction` is empty when removed |@@ -7301,7 +7301,7 @@ Webhook delivery is **at-least-once**. A consumer can legitimately receive the sOutbox creation is best effort, and message persistence and outbox insertion are separate transactions. A write failure or crash between them can still lose the webhook delivery. Failure records with zero attempts are excluded from operator redrive. Capacity and shutdown rejections can still be recovered from an unsettled outbox record; shutdown during retry backoff may follow earlier POSTs. The direct dispatcher leaves its local in-flight work alone during reconciliation. Graceful shutdown drains current queue jobs for at most 10 s; direct sends, an operator redrive replay in flight, and an outbox replay in flight with its plugin hooks, get up to `WEBHOOK_SHUTDOWN_DRAIN_MS` (default 5s). That replay sends at most one more POST, bounded by `WEBHOOK_TIMEOUT`, and no further row is replayed; the next start replays what it left pending.-**Opt-in events are bounded, not replayed.** `message.receipt` fires once per recipient and status, so one flush of buffered receipts can be thousands of events in a tick. Each session passes a queue of its own before the shared dispatch pool: a quarter of `WEBHOOK_DISPATCH_CONCURRENCY` deliveries of its receipts run at once, shared by the webhooks that name `message.receipt` since one receipt is a delivery to each, and up to three quarters of `WEBHOOK_DISPATCH_MAX_QUEUED` more receipts wait their turn, which with the defaults delivers a flush of 500 in full and at least 750 of a flush of 2000. Any further receipt is dropped: it leaves no outbox or failure row and is not replayed, and it is counted in `openwa_webhook_delivery_failures_total` and logged once per burst (`webhook_opt_in_dropping` when the session starts dropping, then `webhook_opt_in_dropped` with the count). Because the queue is per session, one session's flood, or a receiver of its that never answers, does not shed or delay another session's events. Receipts still waiting at shutdown are dropped without being counted.+**Opt-in events are bounded, not replayed.** `message.receipt` fires once per recipient and status, so one flush of buffered receipts can be thousands of events in a tick. Each webhook that names it has a queue of its own before the shared dispatch pool: a quarter of `WEBHOOK_DISPATCH_CONCURRENCY` deliveries run at once, divided among the session's webhooks that name `message.receipt` and never fewer than one running delivery per webhook, and up to three quarters of `WEBHOOK_DISPATCH_MAX_QUEUED` more receipts wait their turn. With the defaults a flush of 500 is delivered in full and at least 750 of a flush of 2000 (754 with one webhook, 751 with four). A flush drains at that quarter of the pool: 750 receipts to a receiver that takes 30 ms each take about 6 s, or about 24 s when four webhooks share the quarter. Any further receipt is dropped for that webhook: it leaves no outbox or failure row, is not replayed, and is counted in `openwa_webhook_delivery_failures_total` once for each webhook it is dropped for. It is logged once per burst and webhook (`webhook_opt_in_dropping` when the webhook starts dropping, then `webhook_opt_in_dropped` with the count). A webhook's queue is its own, so a receiver of it that never answers holds only that webhook's places: the session's other webhooks and its other events are not slowed. Between sessions the isolation has a limit, because the dispatch pool is shared. A receiver that never answers holds its places for a `WEBHOOK_TIMEOUT`; a session holds up to four at once, or one per webhook when it has more than four. An event of another session waits for a slot once such sessions together hold all `WEBHOOK_DISPATCH_CONCURRENCY` of them, which at the defaults is four sessions: it then waits one `WEBHOOK_TIMEOUT` with four or five, two with eight. The receipts waiting in all queues together are not capped. Receipts still waiting at shutdown are dropped without being counted.Before downgrading to a version that does not recognize the `queued` outbox state, drain or recover those records and take a backup. Its cleanup may otherwise treat an unsettled queued record as settled. Reverting the terminal-identity migration removes its index but does not restore historical duplicates.diff --git a/docs/10-devops-infrastructure.md b/docs/10-devops-infrastructure.mdindex 17724099f..f36c6705a 100644--- a/docs/10-devops-infrastructure.md+++ b/docs/10-devops-infrastructure.md@@ -503,10 +503,14 @@ WEBHOOK_DISPATCH_MAX_QUEUED=1000# The failing state and the cap are held per process, not per cluster.# The first 2xx from the webhook lifts it. With the queue disabled, a session parks at most a# quarter of WEBHOOK_DISPATCH_MAX_QUEUED behind that limit and sheds the rest, so other sessions-# keep room. Opt-in events such as message.receipt are bounded per session instead: a quarter of-# WEBHOOK_DISPATCH_CONCURRENCY deliveries run at once and three quarters of-# WEBHOOK_DISPATCH_MAX_QUEUED events wait, and further ones are dropped, not replayed, and counted-# in openwa_webhook_delivery_failures_total.+# keep room. Opt-in events such as message.receipt are bounded per receiving webhook instead: a+# quarter of WEBHOOK_DISPATCH_CONCURRENCY deliveries run at once, divided among the session's+# webhooks that name the event and one at least for each, and three quarters of+# WEBHOOK_DISPATCH_MAX_QUEUED events wait; further ones are dropped, not replayed, and counted in+# openwa_webhook_delivery_failures_total. A webhook whose receiver never answers holds only its own+# places, but the pool is shared: a session holds up to four at once (one per webhook past four), and+# another session's events wait for a slot once such sessions together hold all of them, which+# is four sessions at the defaults. There is no process-wide cap on the events waiting.# Each running, parked or backoff-waiting inline delivery holds its event's inline media# (up to 4/3 x WEBHOOK_MEDIA_INLINE_MAX_BYTES as base64), about 1.3 GiB per process at the defaults# when every slot is taken; lower these caps or enable the queue to reduce it.diff --git a/src/common/metrics/webhook-delivery-metrics.ts b/src/common/metrics/webhook-delivery-metrics.tsindex 8f6afc3e1..44acdda4a 100644--- a/src/common/metrics/webhook-delivery-metrics.ts+++ b/src/common/metrics/webhook-delivery-metrics.ts@@ -4,7 +4,7 @@* deliveries stopped by shutdown between retries after earlier attempts were sent. It is* incremented once per row inserted into the durable `webhook_delivery_failures` dead-letter table;* a later replay may replace or remove that row, and the counter keeps the failure. It is also- * incremented once for each opt-in event dropped past the bound of its session's queue, which files+ * incremented once for each opt-in event dropped past the bound of its webhook's queue, which files* no row because there is nothing to replay. Kept as a plain* in-process counter rather than a `COUNT(*)` over that table because the table is pruned on a* retention schedule, which would make its count non-monotonic and therefore invalid as adiff --git a/src/modules/metrics/metrics.service.ts b/src/modules/metrics/metrics.service.tsindex 8ed39f400..6db07712c 100644--- a/src/modules/metrics/metrics.service.ts+++ b/src/modules/metrics/metrics.service.ts@@ -221,7 +221,7 @@ export class MetricsService implements OnModuleDestroy {}lines.push(- '# HELP openwa_webhook_delivery_failures_total Webhook delivery failures recorded since process start: retries exhausted, never sent (shed, refused at shutdown, rejected before sending, or an opt-in event dropped past its session bound), or stopped by shutdown between direct retries.',+ '# HELP openwa_webhook_delivery_failures_total Webhook delivery failures recorded since process start: retries exhausted, never sent (shed, refused at shutdown, rejected before sending, or an opt-in event dropped past the bound of its webhook), or stopped by shutdown between direct retries.',);lines.push('# TYPE openwa_webhook_delivery_failures_total counter');lines.push(`openwa_webhook_delivery_failures_total ${getWebhookDeliveryFailuresTotal()}`);