Free, Open Source, Self-Hosted WhatsApp API Gateway
TypeScript apibotgatewayself-hostedwhatsappwhatsapp-apiwhatsapp-automationwhatsapp-botwhatsapp-gatewaywhatsapp-gateway-apiwhatsapp-webwhatsapp-web-api
fix(webhook): retire the outbox row of a shed or shutdown delivery
Yudhi Armyndharisc0706e9e124e
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 88cc55802..980b2a1b6 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -20,6 +20,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0- A session whose automatic reconnect fails to relaunch the engine (a network, DNS or browser launch error) keeps retrying with backoff instead of stopping in `failed` until restarted by hand; an authentication failure or a stale browser profile still ends in `failed` ([#1580](https://github.com/rmyndharis/OpenWA/issues/1580)).- A session that runs out of reconnect attempts fires the `session:error` plugin hook when it lands in `failed`.- An explicit `maxReconnectAttempts` is honoured and the reconnect delay is capped at 5 minutes; the budget used to restart once the backoff passed 5 minutes, so the limit was never reached and the documented 1-hour cap never applied.+- A webhook delivery shed at `WEBHOOK_DISPATCH_MAX_QUEUED` or refused during shutdown is no longer replayed by the delivery reconciler, so its delivery-failure row no longer reports an event that was delivered after all.- `session.reconnect_loop` webhooks carry an idempotency key salted per occurrence, so an alert from a later outage that reaches the same attempt count is no longer deduplicated onto the earlier one or left without a delivery record.## [0.23.5] - 2026-09-14diff --git a/docs/06-api-specification.md b/docs/06-api-specification.mdindex 3470be653..2b3e9712a 100644--- a/docs/06-api-specification.md+++ b/docs/06-api-specification.md@@ -6859,7 +6859,7 @@ Webhook delivery is **at-least-once**. A consumer can legitimately receive the s- The underlying WhatsApp engine can re-fire an event for a single message.- A failed delivery (non-2xx response, timeout, or network error) is retried.-**Crash boundary.** Every delivery is recorded before it is attempted, and the record is retired once something durable owns it: the queue job in queued mode, the completed send in direct mode. A hard crash (SIGKILL, OOM) therefore leaves the record behind, and a bounded sweep (`WEBHOOK_RECONCILE_INTERVAL_MS`, default 60s) replays whatever is still stranded, reusing the stored `X-OpenWA-Idempotency-Key` so the retry stays deduplicable at your receiver. A delivery that keeps failing exhausts `WEBHOOK_RECONCILE_MAX_ATTEMPTS` and goes terminal rather than replaying forever. One window remains open: a crash between persisting the message and writing that record loses the delivery, because the two are not yet one transaction. The failure table records exhausted retries, plus over-budget, dispatch-capacity-exceeded, and shutdown-rejected deliveries with attempts 0. Read the last two as a report rather than a verdict: a delivery the dispatcher shed for capacity or refused during the drain keeps its record and is replayed by the same sweep, so a row there can belong to an event that was later delivered. Enabling the queue (`QUEUE_ENABLED=true`, needs Redis) makes the dispatch durable from the enqueue onward. In both modes, a graceful shutdown drains in-flight deliveries first: the queued path waits for each worker's current job, and the direct path waits up to `WEBHOOK_SHUTDOWN_DRAIN_MS` (default 5s; raise it to at least `WEBHOOK_TIMEOUT`, default 10s, if a slow receiver must finish).+**Crash boundary.** Every delivery is recorded before it is attempted, and the record is retired once something durable owns it: the queue job in queued mode, the completed send in direct mode. A hard crash (SIGKILL, OOM) therefore leaves the record behind, and a bounded sweep (`WEBHOOK_RECONCILE_INTERVAL_MS`, default 60s) replays whatever is still stranded, reusing the stored `X-OpenWA-Idempotency-Key` so the retry stays deduplicable at your receiver. A delivery that keeps failing exhausts `WEBHOOK_RECONCILE_MAX_ATTEMPTS` and goes terminal rather than replaying forever. One window remains open: a crash between persisting the message and writing that record loses the delivery, because the two are not yet one transaction. The failure table records exhausted retries, plus over-budget, dispatch-capacity-exceeded, and shutdown-rejected deliveries with attempts 0. A delivery the dispatcher shed for capacity or refused during the drain is not replayed: its failure row is the only record, so recover it from there. Enabling the queue (`QUEUE_ENABLED=true`, needs Redis) makes the dispatch durable from the enqueue onward. In both modes, a graceful shutdown drains in-flight deliveries first: the queued path waits for each worker's current job, and the direct path waits up to `WEBHOOK_SHUTDOWN_DRAIN_MS` (default 5s; raise it to at least `WEBHOOK_TIMEOUT`, default 10s, if a slow receiver must finish).**Design your handler to be idempotent**, keyed on the `X-OpenWA-Idempotency-Key` header (see below). As a server-side safety net, OpenWA de-duplicates inbound `message.received` before dispatch (a re-fired event for an already-persisted message is dropped), so one webhook normally sees each inbound message once — but this is best-effort defense-in-depth and does not remove the need for consumer-side idempotency.diff --git a/src/modules/webhook/webhook-delivery.service.spec.ts b/src/modules/webhook/webhook-delivery.service.spec.tsindex 237c2cd9b..a81867f57 100644--- a/src/modules/webhook/webhook-delivery.service.spec.ts+++ b/src/modules/webhook/webhook-delivery.service.spec.ts@@ -349,6 +349,10 @@ describe('WebhookDeliveryService', () => {}),);expect(mockFetch).toHaveBeenCalledTimes(1);+ // The shed delivery's record is retired, so the reconciler does not replay what was shed on+ // purpose and leave a failure row behind for an event that was delivered after all.+ expect(outboxService.close).toHaveBeenCalledWith('wh-b', expect.stringMatching(/_wh-b$/), 'failed');+ expect(outboxService.close).toHaveBeenCalledWith('wh-a', expect.stringMatching(/_wh-a$/), 'dispatched');});it('salts each sibling webhook with a distinct idempotency key so one receiver cannot dedupe out another', async () => {@@ -837,6 +841,9 @@ describe('WebhookDeliveryService', () => {const recorded = shutdownInserts.map(c => c[0]);expect(recorded.map(r => r.webhookId)).toEqual(expect.arrayContaining(['wh-b', 'wh-c']));for (const r of recorded) expect(r.lastError).toBe('ConcurrencyLimiter closed');+ // Their records are retired too, so the next start does not replay what was dead-lettered here.+ expect(outboxService.close).toHaveBeenCalledWith('wh-b', expect.stringMatching(/_wh-b$/), 'failed');+ expect(outboxService.close).toHaveBeenCalledWith('wh-c', expect.stringMatching(/_wh-c$/), 'failed');// The in-flight delivery was NOT falsely dead-lettered (the receiver may have it); it is// logged as abandoned so the loss is still operator-visible.expect(@@ -851,6 +858,53 @@ describe('WebhookDeliveryService', () => {loggerSpy.mockRestore();});+ it('waits for the outbox close of a delivery refused during shutdown before returning', async () => {+ const hooks = [+ createMockWebhook({ id: 'wh-a', url: 'https://a.example/hook', events: ['message.received'] }),+ createMockWebhook({ id: 'wh-b', url: 'https://b.example/hook', events: ['message.received'] }),+ ];+ (repository.find as jest.Mock).mockResolvedValue(hooks);+ (repository.update as jest.Mock).mockResolvedValue({ affected: 1 });+ (hookManager.execute as jest.Mock).mockImplementation((_event: string, data: unknown) =>+ Promise.resolve({ continue: true, data }),+ );+ (configService.get as jest.Mock).mockImplementation(<T>(key: string, def?: T): T | boolean | number => {+ if (key === 'queue.enabled') return false;+ if (key === 'webhook.shutdownDrainMs') return 2000;+ return def as T;+ });+ (service as unknown as { dispatchLimiter: ConcurrencyLimiter }).dispatchLimiter = new ConcurrencyLimiter(1, 1000);++ let releaseActive: (value: unknown) => void = () => undefined;+ mockFetch.mockImplementation(() => new Promise(resolve => (releaseActive = resolve)));+ // The close of the refused delivery stays open until the test releases it.+ let releaseClose: () => void = () => undefined;+ outboxService.close.mockImplementation((_id: string, _key: string, state: string) =>+ state === 'failed' ? new Promise<void>(resolve => (releaseClose = resolve)) : Promise.resolve(),+ );++ const dispatchP = service.dispatch('sess-1', 'message.received', { from: '[email protected]' });+ for (let i = 0; i < 20 && mockFetch.mock.calls.length === 0; i++) await new Promise(r => setImmediate(r));++ let destroyed = false;+ const destroyP = service.onModuleDestroy().then(() => (destroyed = true));+ // The in-flight delivery finishes, so only the refused delivery's bookkeeping keeps the drain open.+ releaseActive({ ok: true, status: 200 });+ const closedFailed = (): boolean => (outboxService.close.mock.calls as string[][]).some(c => c[2] === 'failed');+ for (let i = 0; i < 40 && !destroyed && !closedFailed(); i++) await new Promise(r => setTimeout(r, 10));+ await new Promise(r => setTimeout(r, 120));++ // A close outside the tracked bookkeeping would let teardown finish first, closing the database+ // under a row that then stays pending and is replayed after restart.+ expect(closedFailed()).toBe(true);+ expect(destroyed).toBe(false);++ releaseClose();+ await destroyP;+ await dispatchP;+ expect(destroyed).toBe(true);+ });+it('queued mode: parked enqueues drain to the queue on shutdown instead of dead-lettering', async () => {(service as unknown as { queueEnabled: boolean }).queueEnabled = true;const hooks = [diff --git a/src/modules/webhook/webhook-delivery.service.ts b/src/modules/webhook/webhook-delivery.service.tsindex 898382b7d..0bf930fc9 100644--- a/src/modules/webhook/webhook-delivery.service.ts+++ b/src/modules/webhook/webhook-delivery.service.ts@@ -644,11 +644,14 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {'webhook_dispatch_capacity_exceeded',ctx,);+ // Shed on purpose: the failure row is the record, so a replay must not undo the shedding.+ await this.outbox.close(webhook.id, idempotencyKey, 'failed');return;}if (error instanceof Error && error.message === 'ConcurrencyLimiter closed') {// Rejected by the shutdown drain before dispatching — record it like any other- // undelivered delivery, and track the write so onModuleDestroy can await it (the+ // undelivered delivery, retire its outbox row so it is not replayed after restart, and+ // track both writes so onModuleDestroy can await them (the// limiter slot bookkeeping no longer covers this task).const record = this.recordUndelivered(webhook,@@ -657,7 +660,7 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {error,'webhook_dispatch_shutdown',ctx,- );+ ).then(() => this.outbox.close(webhook.id, idempotencyKey, 'failed'));this.pendingBookkeeping.add(record);try {await record;