Free, Open Source, Self-Hosted WhatsApp API Gateway
TypeScript apibotgatewayself-hostedwhatsappwhatsapp-apiwhatsapp-automationwhatsapp-botwhatsapp-gatewaywhatsapp-gateway-apiwhatsapp-webwhatsapp-web-api
docs(webhooks): align the outbox and redrive lifecycle text (#1841)
Yudhi Armyndharisa7fca2855a4c
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex c66d61f2e..5ace25751 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -99,6 +99,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Documentation+- The API reference, database design, runbook and glossary describe when the webhook outbox copy is retired after a sweep replay or a redrive, and which rows neither path recovers.- Document the memory media messages and parked webhook deliveries hold during a media burst, and the settings that limit it.- The README and architecture docs describe the Local/S3 storage backend as the live media store, not a backup target.- Document how to recover a lost admin API key without revoking the other keys.diff --git a/docs/05-database-design.md b/docs/05-database-design.mdindex 7c5f44b59..a652d920a 100644--- a/docs/05-database-design.md+++ b/docs/05-database-design.md@@ -709,7 +709,7 @@ The data connection also owns:- **`templates`** — reusable message templates (`src/modules/template/entities/template.entity.ts`), with a unique constraint on `(sessionId, name)` — one template name per session.- **`status_updates`** — inbound status/story broadcasts with a 24-hour TTL (`src/modules/status-store/entities/status-update.entity.ts`); unique on `(sessionId, waStatusId)`. Attached media is stored via `StorageService`, not in the row.- **`webhook_delivery_failures`**: durable record of a webhook delivery that exhausted its retries, or was not sent (attempts 0) (`src/modules/webhook/entities/webhook-delivery-failure.entity.ts`), surfaced via the ADMIN `GET /webhooks/delivery-failures`. A shed or shutdown-refused delivery is replayed by the outbox until its replay budget runs out; an oversize payload or a preflight failure on first dispatch is retired after its failure record is stored. A later successful delivery removes the row. While `WEBHOOK_FAILURE_PAYLOAD_RETENTION_HOURS` > 0 (default 0), a terminal row (attempts > 0) also keeps the pre-hook event data in `payload` (never selected by the listing, which reports `replayable` instead) so ADMIN `POST /webhooks/delivery-failures/redrive` can replay it with the stored idempotency key; the payload is cleared once that window passes and the row stays.-- **`webhook_outbox_events`**: the outbound delivery record (`src/modules/webhook/entities/webhook-outbox-event.entity.ts`). A successfully written `pending` row holds pre-hook event data before the attempt. Enqueue changes it to `queued` without dropping the payload. Delivery success, subscription invalidation or a successfully recorded terminal failure retires the copy; failed terminal recording leaves it available for reconciliation. The reconciler checks the stored job before replaying queued work and reuses its idempotency key. Settled rows drop their payload and are pruned on age (§5.7). Message persistence and outbox insertion remain separate transactions.+- **`webhook_outbox_events`**: the outbound delivery record (`src/modules/webhook/entities/webhook-outbox-event.entity.ts`). A successfully written `pending` row holds pre-hook event data before the attempt. Enqueue changes it to `queued` without dropping the payload. For an ordinary dispatch, delivery success, subscription invalidation or a successfully recorded terminal failure retires the copy; failed terminal recording leaves it available for reconciliation. A sweep replay sent directly (the queue off, or its enqueue failed) that fails keeps the copy pending until the replay budget is spent, and a delivered redrive of the event also retires it. The reconciler checks the stored job before replaying queued work and reuses its idempotency key. Settled rows drop their payload and are pruned on age (§5.7). Message persistence and outbox insertion remain separate transactions.- **`send_idempotency_keys`**: one claimed `Idempotency-Key` per `(sessionId, idempotencyKey)` (unique) from the opt-in header on the message send routes (`src/modules/message/entities/send-idempotency-key.entity.ts`). The insert that wins the unique key runs the send; the row then holds `state` (`pending`, `completed` with the `responseBody` replayed to retries, or `failed` when the outcome is unknown), the `route` and a SHA-256 `requestHash` of route and body. Rows expire 24 hours after the claim (`expiresAt`) and are pruned hourly (§5.7). Not exported by `GET /infra/export-data`.- **`plugin_instances`** — one configured instance of an adapter plugin, keyed `${pluginId}:${instanceId}` (`src/modules/integration/entities/plugin-instance.entity.ts`); holds the host-minted ingress HMAC secret, masked on API reads.- **`ingress_events`** — persist-before-ack durable row and inbound dedup oracle, unique on `(pluginId, instanceId, providerDeliveryId)` (`src/modules/integration/entities/ingress-event.entity.ts`). The full payload is retired to `NULL` once dispatch is settled, leaving a slim dedup marker.diff --git a/docs/06-api-specification.md b/docs/06-api-specification.mdindex 397655f43..718dab5d0 100644--- a/docs/06-api-specification.md+++ b/docs/06-api-specification.md@@ -4745,7 +4745,7 @@ Queued deliveries retain their pre-hook outbox copy until the worker completes dReplay recorded webhook deliveries that still hold their event data, in one bounded batch, with the lowest attempt counts first, then oldest first. A row is replayable when it is terminal (`attempts` > 0; an `attempts: 0` row is the outbox's to replay) and was recorded while `WEBHOOK_FAILURE_PAYLOAD_RETENTION_HOURS` > 0, until that window passes. With the default `0` nothing is kept and this call replays nothing.-Each replay reuses the row's stored `idempotencyKey` (sent as `X-OpenWA-Idempotency-Key`), so receivers can deduplicate a POST that timed out after processing. `webhook:before` hooks run again on the stored pre-hook data. Each row gets **one direct POST** inside the request, including when ordinary dispatch uses the queue. Success removes its row; another failure keeps it and raises its attempt count. Calls on one node run serially. Graceful shutdown stops a batch at its next row and waits for the replays still running (at most four), including their `webhook:before` hooks; each POST is bounded by `WEBHOOK_TIMEOUT`. Rows it did not reach stay for a later call, and a call queued behind it replays nothing. The audit log records counts without payload content.+Each replay reuses the row's stored `idempotencyKey` (sent as `X-OpenWA-Idempotency-Key`), so receivers can deduplicate a POST that timed out after processing. `webhook:before` hooks run again on the stored pre-hook data. Each row gets **one direct POST** inside the request, including when ordinary dispatch uses the queue. Success removes its row and closes any outbox copy the sweep still holds for that event, so the sweep does not send it again; another failure keeps it and raises its attempt count. A sweep replay of the same event already under way can still POST it again under the same key. Calls on one node run serially. Graceful shutdown stops a batch at its next row and waits for the replays still running (at most four), including their `webhook:before` hooks; each POST is bounded by `WEBHOOK_TIMEOUT`. Rows it did not reach stay for a later call, and a call queued behind it replays nothing. The audit log records counts without payload content.Choose a batch limit that fits the client's timeout, especially for unavailable receivers. A clienttimeout does not cancel the batch; inspect the failure log before retrying it.@@ -4774,7 +4774,7 @@ timeout does not cancel the batch; inspect the failure log before retrying it.}```-`redriven` equals `delivered`; `enqueued` is retained for response compatibility and is always `0`. Removed, disabled, unsubscribed, expired and out-of-scope rows are excluded before applying the limit. `skipped` counts selected rows cancelled by a hook or invalidated during the call. `remaining` counts eligible rows in the same requested session, webhook and IDs after the call. Retry another batch only after resolving persistent failures or hook cancellations; `remaining > 0` alone does not mean another call will succeed.+`redriven` equals `delivered`; `enqueued` is retained for response compatibility and is always `0`. A row is excluded before applying the limit when it has `attempts: 0`, holds no event data, is past the retention window or out of scope, or its webhook is removed, disabled, unsubscribed from the event or no longer in the row's session. `skipped` counts selected rows cancelled by a hook or invalidated during the call. `remaining` counts eligible rows in the same requested session, webhook and IDs after the call. Retry another batch only after resolving persistent failures or hook cancellations; `remaining > 0` alone does not mean another call will succeed.**Errors:** `400` body fails validation · `401` missing/invalid API key · `403` key role below ADMIN@@ -7244,7 +7244,7 @@ Webhook delivery is **at-least-once**. A consumer can legitimately receive the s- The underlying WhatsApp engine can re-fire an event for a single message.- A failed delivery (non-2xx response, timeout, or network error) is retried.-**Crash boundary.** A successfully written outbox record retains the pre-hook event data until delivery succeeds, the subscription becomes ineligible, or the terminal failure is stored. Queued records keep this copy even after enqueue; the worker retires it only after one of those outcomes. If failure recording fails, the copy remains eligible for recovery. The bounded sweep (`WEBHOOK_RECONCILE_INTERVAL_MS`, default 60s) checks the stored job before replaying, leaves active jobs alone, and reuses the stored `X-OpenWA-Idempotency-Key`. If Redis cannot confirm the job state, replay waits. A missing or failed job can be replayed within `WEBHOOK_RECONCILE_MAX_ATTEMPTS`; exhausting that budget retires the outbox only after terminal failure recording succeeds. Receiver deduplication remains necessary because a crash after receiver processing can cause another POST.+**Crash boundary.** For an ordinary dispatch, a successfully written outbox record retains the pre-hook event data until delivery succeeds, the subscription becomes ineligible, or the terminal failure is stored. Queued records keep this copy even after enqueue; the worker retires it only after one of those outcomes. If failure recording fails, the copy remains eligible for recovery. The bounded sweep (`WEBHOOK_RECONCILE_INTERVAL_MS`, default 60s) checks the stored job before replaying, leaves active jobs alone, and reuses the stored `X-OpenWA-Idempotency-Key`. If Redis cannot confirm the job state, replay waits. A missing or failed job, or a stranded direct delivery, can be replayed within `WEBHOOK_RECONCILE_MAX_ATTEMPTS`. A replay sent directly (the queue off, or its enqueue failed) that exhausts its retries stores its terminal failure row but keeps the outbox copy pending, so later sweeps replay it until that budget is spent; exhausting it retires the outbox only after terminal failure recording succeeds. Receiver deduplication remains necessary because a crash after receiver processing can cause another POST.Outbox creation is best effort, and message persistence and outbox insertion are separate transactions. A write failure or crash between them can still lose the webhook delivery. Failure records with zero attempts are excluded from operator redrive. Capacity and shutdown rejections can still be recovered from an unsettled outbox record; shutdown during retry backoff may follow earlier POSTs. The direct dispatcher leaves its local in-flight work alone during reconciliation. Graceful shutdown drains current queue jobs; direct sends, and an outbox replay in flight with its plugin hooks, get up to `WEBHOOK_SHUTDOWN_DRAIN_MS` (default 5s). That replay sends at most one more POST, bounded by `WEBHOOK_TIMEOUT`, and no further row is replayed; the next start replays what it left pending.diff --git a/docs/11-operational-runbooks.md b/docs/11-operational-runbooks.mdindex b7b4ca602..386d54f8e 100644--- a/docs/11-operational-runbooks.md+++ b/docs/11-operational-runbooks.md@@ -346,7 +346,7 @@ curl -X PUT http://localhost:2785/api/sessions/{sessionId}/webhooks/{webhookId}# it. The outbox sweep runs only while WEBHOOK_RECONCILE_INTERVAL_MS > 0 (default 60000). It# sends an event again only if it was shed, refused at shutdown, or cut off by a restart or a# database fault before its dispatch settled, and then only until WEBHOOK_RECONCILE_MAX_ATTEMPTS-# sweeps are spent, so such an event can show an attempts > 0 row while the sweep still holds it.+# replays are spent, so such an event can show an attempts > 0 row while the sweep still holds it.# A row can be replayed only while step 2 lists it with "replayable": true, that is, it was# recorded while WEBHOOK_FAILURE_PAYLOAD_RETENTION_HOURS > 0 (with the queue on, also when the# event was queued) and that window has not passed. With the default 0 no event data is keptdiff --git a/docs/21-glossary.md b/docs/21-glossary.mdindex 611083ef8..9f85b58da 100644--- a/docs/21-glossary.md+++ b/docs/21-glossary.md@@ -73,7 +73,7 @@ Web interface to manage OpenWA without using the API directly. Built with React### Dead Letter Queue (DLQ)-The durable record of deliveries abandoned after every retry; an outbound-webhook row with `attempts: 0` was instead shed, oversize, failed preflight or refused at shutdown, and a later successful delivery of the event removes its row. In OpenWA it is a **database table**, not a Redis queue: `webhook_delivery_failures` for outbound webhooks and `integration_delivery_failures` for plugin ingress. Both support an ADMIN redrive: plugin ingress with `POST /api/integration/instances/:pluginId/:instanceId/redrive`, and outbound webhooks with `POST /api/webhooks/delivery-failures/redrive`, which replays only terminal rows recorded while `WEBHOOK_FAILURE_PAYLOAD_RETENTION_HOURS` > 0 (default 0 keeps no event data) and still inside that window. While `WEBHOOK_RECONCILE_INTERVAL_MS` > 0 the gateway also replays a shed or shutdown-refused delivery from its outbox on its own. Neither path covers a delivery that was over the size cap or failed preflight, nor an exhausted one recorded without event data or past its retention window: the provider or your own tooling must re-send that event.+The durable record of deliveries abandoned after every retry; an outbound-webhook row with `attempts: 0` was instead shed, oversize, failed preflight or refused at shutdown, and a later successful delivery of the event removes its row. In OpenWA it is a **database table**, not a Redis queue: `webhook_delivery_failures` for outbound webhooks and `integration_delivery_failures` for plugin ingress. Both support an ADMIN redrive: plugin ingress with `POST /api/integration/instances/:pluginId/:instanceId/redrive`, and outbound webhooks with `POST /api/webhooks/delivery-failures/redrive`, which replays only terminal rows recorded while `WEBHOOK_FAILURE_PAYLOAD_RETENTION_HOURS` > 0 (default 0 keeps no event data) and still inside that window. While `WEBHOOK_RECONCILE_INTERVAL_MS` > 0 the gateway also replays a shed or shutdown-refused delivery from its outbox on its own; with the sweep off such an event is not replayed, but turning the sweep back on replays it while its outbox record is pending. Neither path covers a delivery that was over the size cap or failed preflight, a shed or shutdown-refused one the sweep dropped because its webhook was removed, disabled or unsubscribed, nor an exhausted one recorded without event data or past its retention window: the provider or your own tooling must re-send that event.### Dockerdiff --git a/src/common/docs-governance.spec.ts b/src/common/docs-governance.spec.tsindex 3f61be402..f40004d66 100644--- a/src/common/docs-governance.spec.ts+++ b/src/common/docs-governance.spec.ts@@ -224,6 +224,37 @@ describe('governance docs match the repository', () => {expect(glossary).not.toMatch(staleClaim);});+ // A direct sweep replay that fails keeps its outbox copy pending, a delivered redrive closes it,+ // and the replay budget counts replays, not sweeps. An attempts-0 row the sweep never replays has+ // no other way back, since redrive takes only terminal rows.+ it('describes the webhook outbox copy after a sweep replay or a redrive', () => {+ const flat = (text: string): string => text.replace(/\n#?\s*/g, ' ').replace(/\s+/g, ' ');+ const doc06 = read('docs/06-api-specification.md');+ const redrive = between(doc06, '#### POST /api/webhooks/delivery-failures/redrive', '\n#### ');+ expect(redrive).toMatch(/Success removes its row and closes any outbox copy/);+ expect(redrive).toMatch(/sweep replay of the same event already under way can still POST/);+ expect(redrive).toMatch(/excluded before applying the limit when it has `attempts: 0`/);+ const crash = between(doc06, '**Crash boundary.**', '\n\n');+ expect(crash).toMatch(/^\*\*Crash boundary\.\*\* For an ordinary dispatch, a successfully written outbox record/);+ expect(crash).toMatch(+ /\(the queue off, or its enqueue failed\) that exhausts its retries stores its terminal failure row but keeps the outbox copy pending/,+ );+ const runbook = flat(between(read('docs/11-operational-runbooks.md'), '# 6. Replay deliveries', '```'));+ expect(runbook).toContain('until WEBHOOK_RECONCILE_MAX_ATTEMPTS replays are spent');+ const glossary = between(read('docs/21-glossary.md'), '### Dead Letter Queue (DLQ)', '\n### ');+ expect(glossary).toMatch(+ /with the sweep off such an event is not replayed, but turning the sweep back on replays it/,+ );+ expect(glossary).toMatch(+ /shed or shutdown-refused one the sweep dropped because its webhook was removed, disabled or unsubscribed, nor/,+ );+ expect(doc06).not.toMatch(/Removed, disabled, unsubscribed, expired and out-of-scope rows are excluded/);+ const outbox = between(read('docs/05-database-design.md'), '- **`webhook_outbox_events`**', '\n');+ expect(outbox).toMatch(/For an ordinary dispatch, delivery success, subscription invalidation/);+ expect(outbox).toMatch(/that fails keeps the copy pending until the replay budget is spent/);+ expect(outbox).toMatch(/a delivered redrive of the event also retires it/);+ });+// Status media lives only in the storage backend, and backup.sh never copies an S3 bucket.it('describes the storage backend as the live media store, not a backup target', () => {expect(read('scripts/backup.sh')).toMatch(/the bucket's contents are not archived/);