Free, Open Source, Self-Hosted WhatsApp API Gateway
TypeScript apibotgatewayself-hostedwhatsappwhatsapp-apiwhatsapp-automationwhatsapp-botwhatsapp-gatewaywhatsapp-gateway-apiwhatsapp-webwhatsapp-web-api
fix(webhook): skip outbox rows still being dispatched on this node
Yudhi Armyndharisd736c957b33c
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 980b2a1b6..c9bb7d15d 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -20,6 +20,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0- A session whose automatic reconnect fails to relaunch the engine (a network, DNS or browser launch error) keeps retrying with backoff instead of stopping in `failed` until restarted by hand; an authentication failure or a stale browser profile still ends in `failed` ([#1580](https://github.com/rmyndharis/OpenWA/issues/1580)).- A session that runs out of reconnect attempts fires the `session:error` plugin hook when it lands in `failed`.- An explicit `maxReconnectAttempts` is honoured and the reconnect delay is capped at 5 minutes; the budget used to restart once the backoff passed 5 minutes, so the limit was never reached and the documented 1-hour cap never applied.+- The webhook delivery reconciler no longer replays a delivery that is still waiting for a dispatch slot or retrying on the node that dispatched it, which sent a duplicate outside `WEBHOOK_DISPATCH_CONCURRENCY` and could close a slow delivery as failed while it was still running.- A webhook delivery shed at `WEBHOOK_DISPATCH_MAX_QUEUED` or refused during shutdown is no longer replayed by the delivery reconciler, so its delivery-failure row no longer reports an event that was delivered after all.- `session.reconnect_loop` webhooks carry an idempotency key salted per occurrence, so an alert from a later outage that reaches the same attempt count is no longer deduplicated onto the earlier one or left without a delivery record.diff --git a/docs/06-api-specification.md b/docs/06-api-specification.mdindex 2b3e9712a..75408e673 100644--- a/docs/06-api-specification.md+++ b/docs/06-api-specification.md@@ -6859,7 +6859,7 @@ Webhook delivery is **at-least-once**. A consumer can legitimately receive the s- The underlying WhatsApp engine can re-fire an event for a single message.- A failed delivery (non-2xx response, timeout, or network error) is retried.-**Crash boundary.** Every delivery is recorded before it is attempted, and the record is retired once something durable owns it: the queue job in queued mode, the completed send in direct mode. A hard crash (SIGKILL, OOM) therefore leaves the record behind, and a bounded sweep (`WEBHOOK_RECONCILE_INTERVAL_MS`, default 60s) replays whatever is still stranded, reusing the stored `X-OpenWA-Idempotency-Key` so the retry stays deduplicable at your receiver. A delivery that keeps failing exhausts `WEBHOOK_RECONCILE_MAX_ATTEMPTS` and goes terminal rather than replaying forever. One window remains open: a crash between persisting the message and writing that record loses the delivery, because the two are not yet one transaction. The failure table records exhausted retries, plus over-budget, dispatch-capacity-exceeded, and shutdown-rejected deliveries with attempts 0. A delivery the dispatcher shed for capacity or refused during the drain is not replayed: its failure row is the only record, so recover it from there. Enabling the queue (`QUEUE_ENABLED=true`, needs Redis) makes the dispatch durable from the enqueue onward. In both modes, a graceful shutdown drains in-flight deliveries first: the queued path waits for each worker's current job, and the direct path waits up to `WEBHOOK_SHUTDOWN_DRAIN_MS` (default 5s; raise it to at least `WEBHOOK_TIMEOUT`, default 10s, if a slow receiver must finish).+**Crash boundary.** Every delivery is recorded before it is attempted, and the record is retired once something durable owns it: the queue job in queued mode, the completed send in direct mode. A hard crash (SIGKILL, OOM) therefore leaves the record behind, and a bounded sweep (`WEBHOOK_RECONCILE_INTERVAL_MS`, default 60s) replays whatever is still stranded, reusing the stored `X-OpenWA-Idempotency-Key` so the retry stays deduplicable at your receiver. A delivery that keeps failing exhausts `WEBHOOK_RECONCILE_MAX_ATTEMPTS` and goes terminal rather than replaying forever. One window remains open: a crash between persisting the message and writing that record loses the delivery, because the two are not yet one transaction. The failure table records exhausted retries, plus over-budget, dispatch-capacity-exceeded, and shutdown-rejected deliveries with attempts 0. A delivery the dispatcher shed for capacity or refused during the drain is not replayed: its failure row is the only record, so recover it from there. A node's sweep also leaves alone a delivery that the same node is still waiting to dispatch or retrying, however long that takes; with several nodes on one database, another node can still replay it, and the stored idempotency key keeps that duplicate deduplicable. Enabling the queue (`QUEUE_ENABLED=true`, needs Redis) makes the dispatch durable from the enqueue onward. In both modes, a graceful shutdown drains in-flight deliveries first: the queued path waits for each worker's current job, and the direct path waits up to `WEBHOOK_SHUTDOWN_DRAIN_MS` (default 5s; raise it to at least `WEBHOOK_TIMEOUT`, default 10s, if a slow receiver must finish).**Design your handler to be idempotent**, keyed on the `X-OpenWA-Idempotency-Key` header (see below). As a server-side safety net, OpenWA de-duplicates inbound `message.received` before dispatch (a re-fired event for an already-persisted message is dropped), so one webhook normally sees each inbound message once — but this is best-effort defense-in-depth and does not remove the need for consumer-side idempotency.diff --git a/src/modules/webhook/webhook-delivery.service.spec.ts b/src/modules/webhook/webhook-delivery.service.spec.tsindex a81867f57..badecce4c 100644--- a/src/modules/webhook/webhook-delivery.service.spec.ts+++ b/src/modules/webhook/webhook-delivery.service.spec.ts@@ -353,6 +353,36 @@ describe('WebhookDeliveryService', () => {// purpose and leave a failure row behind for an event that was delivered after all.expect(outboxService.close).toHaveBeenCalledWith('wh-b', expect.stringMatching(/_wh-b$/), 'failed');expect(outboxService.close).toHaveBeenCalledWith('wh-a', expect.stringMatching(/_wh-a$/), 'dispatched');+ expect(service.isLocallyPending((outboxService.close.mock.calls as string[][])[0][1])).toBe(false);+ });++ it('reports a delivery as locally pending while it is parked or in flight, and releases it once settled', async () => {+ const wA = createMockWebhook({ id: 'wh-a', url: 'https://a.example/hook', events: ['message.received'] });+ const wB = createMockWebhook({ id: 'wh-b', url: 'https://b.example/hook', events: ['message.received'] });+ (repository.find as jest.Mock).mockResolvedValue([wA, wB]);+ (repository.update as jest.Mock).mockResolvedValue({ affected: 1 });+ (hookManager.execute as jest.Mock).mockImplementation((_event: string, data: unknown) =>+ Promise.resolve({ continue: true, data }),+ );+ // One slot: A holds it, B parks behind it.+ (service as unknown as { dispatchLimiter: ConcurrencyLimiter }).dispatchLimiter = new ConcurrencyLimiter(1, 10);++ let release: (value: unknown) => void = () => undefined;+ mockFetch.mockImplementation(() => new Promise(resolve => (release = resolve)));++ const pending = service.dispatch('sess-1', 'message.received', { from: '[email protected]' });+ for (let i = 0; i < 20 && mockFetch.mock.calls.length === 0; i++) await new Promise(r => setImmediate(r));++ const keys = (outboxService.open.mock.calls as Array<[{ idempotencyKey: string }]>).map(c => c[0].idempotencyKey);+ expect(keys).toHaveLength(2);+ for (const key of keys) expect(service.isLocallyPending(key)).toBe(true);++ release({ ok: true, status: 200 });+ for (let i = 0; i < 20 && mockFetch.mock.calls.length < 2; i++) await new Promise(r => setImmediate(r));+ release({ ok: true, status: 200 });+ await pending;++ for (const key of keys) expect(service.isLocallyPending(key)).toBe(false);});it('salts each sibling webhook with a distinct idempotency key so one receiver cannot dedupe out another', async () => {diff --git a/src/modules/webhook/webhook-delivery.service.ts b/src/modules/webhook/webhook-delivery.service.tsindex 0bf930fc9..5b92f40d8 100644--- a/src/modules/webhook/webhook-delivery.service.ts+++ b/src/modules/webhook/webhook-delivery.service.ts@@ -97,6 +97,13 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {>();/** Late bookkeeping (dead-letter rows) written by tasks the limiter already released — awaited on shutdown. */private readonly pendingBookkeeping = new Set<Promise<void>>();+ /**+ * Outbox rows this node still owns, by idempotency key and counted (the same key can be dispatched+ * twice), from the moment the row is opened until its dispatch settles: parked in the limiter,+ * holding a slot, or inside a direct retry loop. The reconciler skips these so a slow delivery is+ * not replayed alongside itself.+ */+ private readonly locallyPending = new Map<string, number>();constructor(@InjectRepository(Webhook, 'data')@@ -616,6 +623,28 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {deliveryId,payload: ctx.baseData,});+ this.locallyPending.set(idempotencyKey, (this.locallyPending.get(idempotencyKey) ?? 0) + 1);+ try {+ await this.runLimited(webhook, deliveryId, idempotencyKey, ctx);+ } finally {+ const left = (this.locallyPending.get(idempotencyKey) ?? 1) - 1;+ if (left > 0) this.locallyPending.set(idempotencyKey, left);+ else this.locallyPending.delete(idempotencyKey);+ }+ }++ /** True while a dispatch on this node still owns the outbox row for this key. */+ isLocallyPending(idempotencyKey: string): boolean {+ return this.locallyPending.has(idempotencyKey);+ }++ private async runLimited(+ webhook: Webhook,+ deliveryId: string,+ idempotencyKey: string,+ ctx: DispatchEventContext,+ ): Promise<void> {+ const { sessionId, event } = ctx;await this.dispatchLimiter.run(async () => {this.inFlightDeliveries.set(deliveryId, {diff --git a/src/modules/webhook/webhook-outbox.service.spec.ts b/src/modules/webhook/webhook-outbox.service.spec.tsindex 01f0fb78b..f8a2158d1 100644--- a/src/modules/webhook/webhook-outbox.service.spec.ts+++ b/src/modules/webhook/webhook-outbox.service.spec.ts@@ -45,7 +45,8 @@ describe('WebhookOutboxService', () => {repo.update.mockRejectedValue(new Error('disk full'));await expect(service.close('wh-1', 'key_wh-1', 'dispatched')).resolves.toBeUndefined();- await expect(service.countAttempt('row-1', 0)).resolves.toBeUndefined();+ // A failed count must not block the replay: the write is best-effort like the rest.+ await expect(service.countAttempt('row-1', 0)).resolves.toBe(true);});it('retires the payload when recording an outcome, so a settled row stops carrying a message body', async () => {@@ -75,9 +76,20 @@ describe('WebhookOutboxService', () => {});it('counts an attempt by incrementing the stored number, not by recomputing it', async () => {- await service.countAttempt('row-1', 2);+ repo.update.mockResolvedValue({ affected: 1 });- expect(repo.update).toHaveBeenCalledWith({ id: 'row-1' }, expect.objectContaining({ attempts: 3 }));+ await expect(service.countAttempt('row-1', 2)).resolves.toBe(true);++ expect(repo.update).toHaveBeenCalledWith(+ { id: 'row-1', state: 'pending' },+ expect.objectContaining({ attempts: 3 }),+ );+ });++ it('reports a row that is no longer pending, so a stale copy is not replayed', async () => {+ repo.update.mockResolvedValue({ affected: 0 });++ await expect(service.countAttempt('row-1', 2)).resolves.toBe(false);});});diff --git a/src/modules/webhook/webhook-outbox.service.ts b/src/modules/webhook/webhook-outbox.service.tsindex f1baf88d8..4a84a4556 100644--- a/src/modules/webhook/webhook-outbox.service.ts+++ b/src/modules/webhook/webhook-outbox.service.ts@@ -148,12 +148,23 @@ export class WebhookOutboxService implements OnModuleInit, OnModuleDestroy {}));}- /** Count one replay attempt against the row's budget. */- async countAttempt(id: string, attempts: number): Promise<void> {+ /**+ * Count one replay attempt against the row's budget.+ *+ * Returns false when the row is no longer pending: the sweep reads its batch once, so a delivery+ * that settled after that read must not be replayed from the stale copy. A failed write returns+ * true, keeping the replay best-effort like every other write here.+ */+ async countAttempt(id: string, attempts: number): Promise<boolean> {try {- await this.outbox.update({ id }, { attempts: attempts + 1, lastAttemptAt: new Date() });+ const result = await this.outbox.update(+ { id, state: 'pending' },+ { attempts: attempts + 1, lastAttemptAt: new Date() },+ );+ return (result.affected ?? 0) > 0;} catch (error) {this.logger.warn(`Could not count a replay attempt: ${String(error)}`);+ return true;}}}diff --git a/src/modules/webhook/webhook-reconciler.service.spec.ts b/src/modules/webhook/webhook-reconciler.service.spec.tsindex 23b6cf0ee..5a8637675 100644--- a/src/modules/webhook/webhook-reconciler.service.spec.ts+++ b/src/modules/webhook/webhook-reconciler.service.spec.ts@@ -37,13 +37,20 @@ describe('resolveWebhookReconcilerOptions', () => {describe('WebhookReconcilerService', () => {let outbox: { findStale: jest.Mock; close: jest.Mock; countAttempt: jest.Mock };- let delivery: { redeliver: jest.Mock };+ let delivery: { redeliver: jest.Mock; isLocallyPending: jest.Mock };let webhooks: { findOne: jest.Mock };let service: WebhookReconcilerService;beforeEach(() => {- outbox = { findStale: jest.fn().mockResolvedValue([]), close: jest.fn(), countAttempt: jest.fn() };- delivery = { redeliver: jest.fn().mockResolvedValue('delivered') };+ outbox = {+ findStale: jest.fn().mockResolvedValue([]),+ close: jest.fn(),+ countAttempt: jest.fn().mockResolvedValue(true),+ };+ delivery = {+ redeliver: jest.fn().mockResolvedValue('delivered'),+ isLocallyPending: jest.fn().mockReturnValue(false),+ };webhooks = { findOne: jest.fn().mockResolvedValue({ id: 'wh-1', active: true }) };service = new WebhookReconcilerService(webhooks as never, outbox as never, delivery as never);});@@ -140,6 +147,33 @@ describe('WebhookReconcilerService', () => {expect(stats).toMatchObject({ skipped: 1 });});+ it('leaves a row alone while this node is still dispatching it', async () => {+ // A direct delivery with retries, or one parked behind slow receivers, can stay pending past the+ // grace window. Replaying it would POST alongside the original and spend its budget mid-flight.+ outbox.findStale.mockResolvedValue([row({ attempts: 3 })]);+ delivery.isLocallyPending.mockImplementation((key: string) => key === 'stored-key_wh-1');++ const stats = await service.sweep(OPTS);++ expect(delivery.redeliver).not.toHaveBeenCalled();+ expect(outbox.countAttempt).not.toHaveBeenCalled();+ expect(outbox.close).not.toHaveBeenCalled();+ expect(stats).toMatchObject({ scanned: 1, skipped: 1, replayed: 0, failed: 0 });+ });++ it('does not replay a row that settled after the batch was read', async () => {+ // The original dispatch finished while an earlier row in the pass was replaying: no longer+ // locally pending, but no longer pending in the database either.+ outbox.findStale.mockResolvedValue([row({ attempts: 1 })]);+ outbox.countAttempt.mockResolvedValue(false);++ const stats = await service.sweep(OPTS);++ expect(delivery.redeliver).not.toHaveBeenCalled();+ expect(outbox.close).not.toHaveBeenCalled();+ expect(stats).toMatchObject({ scanned: 1, skipped: 1, replayed: 0, failed: 0 });+ });+it('never stacks a second pass on top of a slow one', async () => {let release: () => void = () => {};outbox.findStale.mockImplementation(diff --git a/src/modules/webhook/webhook-reconciler.service.ts b/src/modules/webhook/webhook-reconciler.service.tsindex b1360f184..8eb9b47ee 100644--- a/src/modules/webhook/webhook-reconciler.service.ts+++ b/src/modules/webhook/webhook-reconciler.service.ts@@ -94,6 +94,13 @@ export class WebhookReconcilerService implements OnModuleInit, OnModuleDestroy {const rows = await this.outbox.findStale(new Date(now.getTime() - opts.graceMs), opts.batchSize);stats.scanned = rows.length;for (const row of rows) {+ if (this.delivery.isLocallyPending(row.idempotencyKey)) {+ // Still owned by a dispatch on this node (parked in the limiter or mid retry loop), so it+ // is slow rather than stranded. Replaying it would POST alongside the original and outside+ // the dispatch concurrency bound, and would spend its budget while it is still running.+ stats.skipped++;+ continue;+ }if (row.attempts >= opts.maxAttempts) {// Budget spent: stop replaying and leave the failure row as the recovery path.await this.outbox.close(row.webhookId, row.idempotencyKey, 'failed');@@ -108,7 +115,12 @@ export class WebhookReconcilerService implements OnModuleInit, OnModuleDestroy {stats.skipped++;continue;}- await this.outbox.countAttempt(row.id, row.attempts);+ if (!(await this.outbox.countAttempt(row.id, row.attempts))) {+ // Settled since the batch was read, typically a local dispatch that finished while an+ // earlier row in this pass was replaying. The copy in hand is stale; replaying it duplicates.+ stats.skipped++;+ continue;+ }try {// The outcome is a RETURN VALUE, not an exception. Every delivery failure is handled in// place (dead-letter row, hook, log), so redeliver resolves either way and a catch here