Free, Open Source, Self-Hosted WhatsApp API Gateway
TypeScript apibotgatewayself-hostedwhatsappwhatsapp-apiwhatsapp-automationwhatsapp-botwhatsapp-gatewaywhatsapp-gateway-apiwhatsapp-webwhatsapp-web-api
fix(webhook): keep the record of an unattempted delivery
Yudhi Armyndharis85f9c82914ed
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 6eaabd5b8..1cc9a7200 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -64,7 +64,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0- `GET /api/sessions/:sessionId/presence/:chatId` returns `null` once the session has no running engine, instead of the last presence reported before a stop, logout, force-kill or failure.- An explicit `maxReconnectAttempts` is honoured and the reconnect delay is capped at 5 minutes; the budget used to restart once the backoff passed 5 minutes, so a limit above about 6 attempts was never reached and the documented 1-hour cap never applied.- The webhook delivery reconciler no longer replays a delivery that is still waiting for a dispatch slot or retrying on the node that dispatched it, which sent a duplicate outside `WEBHOOK_DISPATCH_CONCURRENCY` and could close a slow delivery as failed while it was still running.-- A webhook delivery shed at `WEBHOOK_DISPATCH_MAX_QUEUED` or refused during shutdown is no longer replayed by the delivery reconciler, so its delivery-failure row no longer reports an event that was delivered after all.- `session.reconnect_loop` webhooks carry an idempotency key salted per occurrence, so an alert from a later outage that reaches the same attempt count is no longer deduplicated onto the earlier one or left without a delivery record.- A bulk batch sends each message through the session's current engine, so a reconnect or restart mid-batch no longer fails every remaining message.- `POST /messages/send-bulk` answers 400 for an item with an empty `chatId`, a text item without text, or a media item without a `url` or `base64` under its type; such items used to be accepted with 202 and fail later.diff --git a/docs/06-api-specification.md b/docs/06-api-specification.mdindex f980d7b93..752b28c01 100644--- a/docs/06-api-specification.md+++ b/docs/06-api-specification.md@@ -6868,7 +6868,7 @@ Webhook delivery is **at-least-once**. A consumer can legitimately receive the s- The underlying WhatsApp engine can re-fire an event for a single message.- A failed delivery (non-2xx response, timeout, or network error) is retried.-**Crash boundary.** Every delivery is recorded before it is attempted, and the record is retired once something durable owns it: the queue job in queued mode, the completed send in direct mode. A hard crash (SIGKILL, OOM) therefore leaves the record behind, and a bounded sweep (`WEBHOOK_RECONCILE_INTERVAL_MS`, default 60s) replays whatever is still stranded, reusing the stored `X-OpenWA-Idempotency-Key` so the retry stays deduplicable at your receiver. A delivery that keeps failing exhausts `WEBHOOK_RECONCILE_MAX_ATTEMPTS` and goes terminal rather than replaying forever. One window remains open: a crash between persisting the message and writing that record loses the delivery, because the two are not yet one transaction. The failure table records exhausted retries, plus over-budget, dispatch-capacity-exceeded, and shutdown-rejected deliveries with attempts 0. A delivery the dispatcher shed for capacity or refused during the drain is not replayed: its failure row is the only record, so recover it from there. A node's sweep also leaves alone a delivery that the same node is still waiting to dispatch or retrying, however long that takes; with several nodes on one database, another node can still replay it, and the stored idempotency key keeps that duplicate deduplicable. Enabling the queue (`QUEUE_ENABLED=true`, needs Redis) makes the dispatch durable from the enqueue onward. In both modes, a graceful shutdown drains in-flight deliveries first: the queued path waits for each worker's current job, and the direct path waits up to `WEBHOOK_SHUTDOWN_DRAIN_MS` (default 5s; raise it to at least `WEBHOOK_TIMEOUT`, default 10s, if a slow receiver must finish).+**Crash boundary.** Every delivery is recorded before it is attempted, and the record is retired once something durable owns it: the queue job in queued mode, the completed send in direct mode. A hard crash (SIGKILL, OOM) therefore leaves the record behind, and a bounded sweep (`WEBHOOK_RECONCILE_INTERVAL_MS`, default 60s) replays whatever is still stranded, reusing the stored `X-OpenWA-Idempotency-Key` so the retry stays deduplicable at your receiver. A delivery that keeps failing exhausts `WEBHOOK_RECONCILE_MAX_ATTEMPTS` and goes terminal rather than replaying forever. One window remains open: a crash between persisting the message and writing that record loses the delivery, because the two are not yet one transaction. The failure table records exhausted retries, plus over-budget, dispatch-capacity-exceeded, and shutdown-rejected deliveries with attempts 0. Read the last two as a report rather than a verdict: a delivery the dispatcher shed for capacity or refused during the drain was rejected before its POST, so it keeps its record and is replayed by the same sweep, and a row there can belong to an event that was later delivered. A node's sweep also leaves alone a delivery that the same node is still waiting to dispatch or retrying, however long that takes; with several nodes on one database, another node can still replay it, and the stored idempotency key keeps that duplicate deduplicable. Enabling the queue (`QUEUE_ENABLED=true`, needs Redis) makes the dispatch durable from the enqueue onward. In both modes, a graceful shutdown drains in-flight deliveries first: the queued path waits for each worker's current job, and the direct path waits up to `WEBHOOK_SHUTDOWN_DRAIN_MS` (default 5s; raise it to at least `WEBHOOK_TIMEOUT`, default 10s, if a slow receiver must finish).**Design your handler to be idempotent**, keyed on the `X-OpenWA-Idempotency-Key` header (see below). As a server-side safety net, OpenWA de-duplicates inbound `message.received` before dispatch (a re-fired event for an already-persisted message is dropped), so one webhook normally sees each inbound message once — but this is best-effort defense-in-depth and does not remove the need for consumer-side idempotency.diff --git a/src/modules/webhook/webhook-delivery.service.spec.ts b/src/modules/webhook/webhook-delivery.service.spec.tsindex badecce4c..de7a53655 100644--- a/src/modules/webhook/webhook-delivery.service.spec.ts+++ b/src/modules/webhook/webhook-delivery.service.spec.ts@@ -349,11 +349,15 @@ describe('WebhookDeliveryService', () => {}),);expect(mockFetch).toHaveBeenCalledTimes(1);- // The shed delivery's record is retired, so the reconciler does not replay what was shed on- // purpose and leave a failure row behind for an event that was delivered after all.- expect(outboxService.close).toHaveBeenCalledWith('wh-b', expect.stringMatching(/_wh-b$/), 'failed');+ // The shed delivery was rejected before its POST, so only the delivered one is retired: wh-b's+ // row stays pending and the reconciler is what still gets that event to the receiver.expect(outboxService.close).toHaveBeenCalledWith('wh-a', expect.stringMatching(/_wh-a$/), 'dispatched');- expect(service.isLocallyPending((outboxService.close.mock.calls as string[][])[0][1])).toBe(false);+ expect((outboxService.close.mock.calls as string[][]).map(c => c[0])).toEqual(['wh-a']);+ // Both keys, the shed one included, are released: the reconciler only replays a pending row+ // once no dispatch on this node still owns it, so a leaked key would strand wh-b for good.+ for (const c of outboxService.open.mock.calls as Array<[{ idempotencyKey: string }]>) {+ expect(service.isLocallyPending(c[0].idempotencyKey)).toBe(false);+ }});it('reports a delivery as locally pending while it is parked or in flight, and releases it once settled', async () => {@@ -871,9 +875,9 @@ describe('WebhookDeliveryService', () => {const recorded = shutdownInserts.map(c => c[0]);expect(recorded.map(r => r.webhookId)).toEqual(expect.arrayContaining(['wh-b', 'wh-c']));for (const r of recorded) expect(r.lastError).toBe('ConcurrencyLimiter closed');- // Their records are retired too, so the next start does not replay what was dead-lettered here.- expect(outboxService.close).toHaveBeenCalledWith('wh-b', expect.stringMatching(/_wh-b$/), 'failed');- expect(outboxService.close).toHaveBeenCalledWith('wh-c', expect.stringMatching(/_wh-c$/), 'failed');+ // Their outbox rows are left pending: the POST never happened, so the next start's sweep is+ // the only thing that can still deliver those two events.+ expect(outboxService.close).not.toHaveBeenCalled();// The in-flight delivery was NOT falsely dead-lettered (the receiver may have it); it is// logged as abandoned so the loss is still operator-visible.expect(@@ -888,7 +892,7 @@ describe('WebhookDeliveryService', () => {loggerSpy.mockRestore();});- it('waits for the outbox close of a delivery refused during shutdown before returning', async () => {+ it('waits for the failure record of a delivery refused during shutdown before returning', async () => {const hooks = [createMockWebhook({ id: 'wh-a', url: 'https://a.example/hook', events: ['message.received'] }),createMockWebhook({ id: 'wh-b', url: 'https://b.example/hook', events: ['message.received'] }),@@ -907,10 +911,10 @@ describe('WebhookDeliveryService', () => {let releaseActive: (value: unknown) => void = () => undefined;mockFetch.mockImplementation(() => new Promise(resolve => (releaseActive = resolve)));- // The close of the refused delivery stays open until the test releases it.- let releaseClose: () => void = () => undefined;- outboxService.close.mockImplementation((_id: string, _key: string, state: string) =>- state === 'failed' ? new Promise<void>(resolve => (releaseClose = resolve)) : Promise.resolve(),+ // The failure record of the refused delivery stays open until the test releases it.+ let releaseInsert: () => void = () => undefined;+ (failureRepository.insert as jest.Mock).mockImplementation(+ () => new Promise<void>(resolve => (releaseInsert = resolve)),);const dispatchP = service.dispatch('sess-1', 'message.received', { from: '[email protected]' });@@ -920,19 +924,21 @@ describe('WebhookDeliveryService', () => {const destroyP = service.onModuleDestroy().then(() => (destroyed = true));// The in-flight delivery finishes, so only the refused delivery's bookkeeping keeps the drain open.releaseActive({ ok: true, status: 200 });- const closedFailed = (): boolean => (outboxService.close.mock.calls as string[][]).some(c => c[2] === 'failed');- for (let i = 0; i < 40 && !destroyed && !closedFailed(); i++) await new Promise(r => setTimeout(r, 10));+ const recording = (): boolean => (failureRepository.insert as jest.Mock).mock.calls.length > 0;+ for (let i = 0; i < 40 && !destroyed && !recording(); i++) await new Promise(r => setTimeout(r, 10));await new Promise(r => setTimeout(r, 120));- // A close outside the tracked bookkeeping would let teardown finish first, closing the database- // under a row that then stays pending and is replayed after restart.- expect(closedFailed()).toBe(true);+ // A write outside the tracked bookkeeping would let teardown finish first, closing the database+ // under the failure record of a delivery the receiver never got.+ expect(recording()).toBe(true);expect(destroyed).toBe(false);- releaseClose();+ releaseInsert();await destroyP;await dispatchP;expect(destroyed).toBe(true);+ // The refused delivery keeps its pending outbox row, so the next start's sweep replays it.+ expect(outboxService.close).not.toHaveBeenCalledWith('wh-b', expect.anything(), 'failed');});it('queued mode: parked enqueues drain to the queue on shutdown instead of dead-lettering', async () => {diff --git a/src/modules/webhook/webhook-delivery.service.ts b/src/modules/webhook/webhook-delivery.service.tsindex 5b92f40d8..a3afb9a85 100644--- a/src/modules/webhook/webhook-delivery.service.ts+++ b/src/modules/webhook/webhook-delivery.service.ts@@ -665,6 +665,9 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {}).catch(async error => {if (error instanceof Error && error.message === 'ConcurrencyLimiter queue full') {+ // Shed before the task ran, so nothing was POSTed. The failure row reports the shed, but+ // the outbox row stays 'pending' on purpose: retiring it would drop the only copy of an+ // event the receiver provably never got. The sweep replays it once the backlog clears.await this.recordUndelivered(webhook,deliveryId,@@ -673,15 +676,13 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {'webhook_dispatch_capacity_exceeded',ctx,);- // Shed on purpose: the failure row is the record, so a replay must not undo the shedding.- await this.outbox.close(webhook.id, idempotencyKey, 'failed');return;}if (error instanceof Error && error.message === 'ConcurrencyLimiter closed') {// Rejected by the shutdown drain before dispatching — record it like any other- // undelivered delivery, retire its outbox row so it is not replayed after restart, and- // track both writes so onModuleDestroy can await them (the- // limiter slot bookkeeping no longer covers this task).+ // undelivered delivery, and track the write so onModuleDestroy can await it (the+ // limiter slot bookkeeping no longer covers this task). Its outbox row stays 'pending':+ // the POST never happened, so the next start's sweep is what finally delivers the event.const record = this.recordUndelivered(webhook,deliveryId,@@ -689,7 +690,7 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {error,'webhook_dispatch_shutdown',ctx,- ).then(() => this.outbox.close(webhook.id, idempotencyKey, 'failed'));+ );this.pendingBookkeeping.add(record);try {await record;