Free, Open Source, Self-Hosted WhatsApp API Gateway
TypeScript apibotgatewayself-hostedwhatsappwhatsapp-apiwhatsapp-automationwhatsapp-botwhatsapp-gatewaywhatsapp-gateway-apiwhatsapp-webwhatsapp-web-api
fix(webhook): replay once per stranded row after a receiver failed
Yudhi Armyndharisf45b32ae2594
diff --git a/docs/06-api-specification.md b/docs/06-api-specification.mdindex 20025c845..dae95be3c 100644--- a/docs/06-api-specification.md+++ b/docs/06-api-specification.md@@ -7290,7 +7290,7 @@ Webhook delivery is **at-least-once**. A consumer can legitimately receive the s- The underlying WhatsApp engine can re-fire an event for a single message.- A failed delivery (non-2xx response, timeout, or network error) is retried.-**Crash boundary.** For an ordinary dispatch, a successfully written outbox record retains the pre-hook event data until delivery succeeds, the subscription becomes ineligible, or the terminal failure is stored. Queued records keep this copy even after enqueue; the worker retires it only after one of those outcomes. If failure recording fails, the copy remains eligible for recovery. The bounded sweep (`WEBHOOK_RECONCILE_INTERVAL_MS`, default 60s) checks the stored job before replaying, leaves active jobs alone, retires the row of a completed job without a replay for as long as the queue still keeps that job (completed jobs are kept for an hour, at most the 1000 most recent), and reuses the stored `X-OpenWA-Idempotency-Key`. If Redis cannot confirm the job state, replay waits. A missing or failed job, or a stranded direct delivery, can be replayed within `WEBHOOK_RECONCILE_MAX_ATTEMPTS`; a completed job already evicted from the queue reads as missing. A replay sent directly (the queue off, or its enqueue failed) that exhausts its retries stores its terminal failure row but keeps the outbox copy pending, so later sweeps replay it until that budget is spent; exhausting it retires the outbox only after terminal failure recording succeeds. Receiver deduplication remains necessary because a crash after receiver processing can cause another POST.+**Crash boundary.** For an ordinary dispatch, a successfully written outbox record retains the pre-hook event data until delivery succeeds, the subscription becomes ineligible, or the terminal failure is stored. Queued records keep this copy even after enqueue; the worker retires it only after one of those outcomes. If failure recording fails, the copy remains eligible for recovery. The bounded sweep (`WEBHOOK_RECONCILE_INTERVAL_MS`, default 60s) checks the stored job before replaying, leaves active jobs alone, retires the row of a completed job without a replay for as long as the queue still keeps that job (completed jobs are kept for an hour, at most the 1000 most recent), and reuses the stored `X-OpenWA-Idempotency-Key`. If Redis cannot confirm the job state, replay waits. A missing or failed job, or a stranded direct delivery, can be replayed within `WEBHOOK_RECONCILE_MAX_ATTEMPTS`; a completed job already evicted from the queue reads as missing. A replay sent directly (the queue off, or its enqueue failed) that exhausts its retries stores its terminal failure row but keeps the outbox copy pending, so later sweeps replay it until that budget is spent; exhausting it retires the outbox only after terminal failure recording succeeds. Once a replay to a webhook has failed, that webhook's remaining rows in the same sweep make one direct attempt each, so a receiver that is down does not hold the sweep for a full retry schedule per row. Receiver deduplication remains necessary because a crash after receiver processing can cause another POST.Outbox creation is best effort, and message persistence and outbox insertion are separate transactions. A write failure or crash between them can still lose the webhook delivery. Failure records with zero attempts are excluded from operator redrive. Capacity and shutdown rejections can still be recovered from an unsettled outbox record; shutdown during retry backoff may follow earlier POSTs. The direct dispatcher leaves its local in-flight work alone during reconciliation. Graceful shutdown drains current queue jobs for at most 10 s; direct sends, an operator redrive replay in flight, and an outbox replay in flight with its plugin hooks, get up to `WEBHOOK_SHUTDOWN_DRAIN_MS` (default 5s). That replay sends at most one more POST, bounded by `WEBHOOK_TIMEOUT`, and no further row is replayed; the next start replays what it left pending.diff --git a/src/modules/webhook/webhook-delivery.service.spec.ts b/src/modules/webhook/webhook-delivery.service.spec.tsindex 552396f89..90f3a963a 100644--- a/src/modules/webhook/webhook-delivery.service.spec.ts+++ b/src/modules/webhook/webhook-delivery.service.spec.ts@@ -1760,6 +1760,39 @@ describe('WebhookDeliveryService', () => {expect(undiciFetch as unknown as jest.Mock).toHaveBeenCalledTimes(2);});+ it('still enqueues a reconciler replay when the queue is enabled', async () => {+ const queueService = await buildQueueService((key: string, def?: unknown) =>+ key === 'queue.enabled' ? true : def,+ );+ const webhook = createMockWebhook({ events: ['message.received'], retryCount: 4 });+ (hookManager.execute as jest.Mock).mockResolvedValue({ continue: true, data: {} });++ await expect(+ queueService.redeliver(webhook, 'sess-1', 'message.received', 'stored-key', {}, { oneDirectAttempt: true }),+ ).resolves.toBe('enqueued');++ expect(webhookQueue.add).toHaveBeenCalledTimes(1);+ expect(undiciFetch as unknown as jest.Mock).not.toHaveBeenCalled();+ });++ it('limits the direct fallback of a reconciler replay to one attempt when the enqueue fails', async () => {+ const queueService = await buildQueueService((key: string, def?: unknown) => {+ if (key === 'queue.enabled') return true;+ if (key === 'webhook.retryDelay') return 1;+ return def;+ });+ const webhook = createMockWebhook({ events: ['message.received'], retryCount: 4 });+ (hookManager.execute as jest.Mock).mockResolvedValue({ continue: true, data: {} });+ (webhookQueue.add as jest.Mock).mockRejectedValueOnce(new Error('Redis unreachable'));+ (undiciFetch as unknown as jest.Mock).mockRejectedValue(new Error('receiver down'));++ await expect(+ queueService.redeliver(webhook, 'sess-1', 'message.received', 'stored-key', {}, { oneDirectAttempt: true }),+ ).resolves.toBe('failed');++ expect(undiciFetch as unknown as jest.Mock).toHaveBeenCalledTimes(1);+ });+it('keys the queue job by the delivery id, so a retried enqueue cannot double-deliver', async () => {const queueService = await buildQueueService((key: string, def?: unknown) => {if (key === 'queue.enabled') return true;@@ -2260,6 +2293,20 @@ describe('WebhookDeliveryService', () => {expect(insertedFailures).toEqual([expect.objectContaining({ attempts: 1, idempotencyKey: 'k' })]);});+ it('makes one direct attempt per reconciler replay (oneDirectAttempt), with no backoff', async () => {+ const webhook = createMockWebhook({ retryCount: 4 });+ mockFetch.mockRejectedValue(new Error('receiver down'));+ (sleep as unknown as jest.Mock).mockClear();++ await expect(+ service.redeliver(webhook, 'sess-1', 'message.received', 'k', {}, { oneDirectAttempt: true }),+ ).resolves.toBe('failed');++ expect(mockFetch).toHaveBeenCalledTimes(1);+ expect(sleep).not.toHaveBeenCalled();+ expect(insertedFailures).toEqual([expect.objectContaining({ attempts: 1, idempotencyKey: 'k' })]);+ });+it('keeps no replay payload on the terminal row while payload retention is off (the default)', async () => {const webhook = createMockWebhook({ retryCount: 1 });mockFetch.mockRejectedValue(new Error('receiver down'));diff --git a/src/modules/webhook/webhook-delivery.service.ts b/src/modules/webhook/webhook-delivery.service.tsindex 4eaf19dad..f126bee82 100644--- a/src/modules/webhook/webhook-delivery.service.ts+++ b/src/modules/webhook/webhook-delivery.service.ts@@ -106,6 +106,13 @@ interface DispatchEventContext {* which runs inside an HTTP request: the in-process backoff schedule would hold it for minutes.*/singleAttempt?: boolean;+ /**+ * Like `singleAttempt` for the direct path (the queue off, or the fallback for a failed enqueue), but+ * an enabled queue is still used. Set by the reconciler, which replays its backlog one row after+ * another: a retry schedule per row would hold every row behind an unreachable receiver for+ * minutes, and the next sweep is the retry.+ */+ oneDirectAttempt?: boolean;/*** Ends a replay's retry backoff early and gives no further attempt. A replay holds no dispatch* slot, so the shutdown drain's limiter close cannot stop it; its owner aborts this instead.@@ -1007,7 +1014,7 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {event: string,idempotencyKey: string,data: Record<string, unknown>,- options: { singleAttempt?: boolean; signal?: AbortSignal } = {},+ options: { singleAttempt?: boolean; oneDirectAttempt?: boolean; signal?: AbortSignal } = {},): Promise<WebhookDeliveryOutcome> {const deliveryId = generateDeliveryId();const outcome = await this.deliverOne(webhook, deliveryId, idempotencyKey, {@@ -1015,6 +1022,7 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {event,baseData: data,singleAttempt: options.singleAttempt,+ oneDirectAttempt: options.oneDirectAttempt,signal: options.signal,});// The receiver has the event. An operator redrive of a failure row can find the same delivery@@ -1095,7 +1103,7 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {return {yieldSlot: ctx.yieldSlot,replayData: this.keepsFailurePayload() ? ctx.baseData : undefined,- singleAttempt: ctx.singleAttempt,+ singleAttempt: ctx.singleAttempt || ctx.oneDirectAttempt,signal: ctx.signal,};}diff --git a/src/modules/webhook/webhook-reconciler.service.spec.ts b/src/modules/webhook/webhook-reconciler.service.spec.tsindex 66c8b3a14..dd38287fc 100644--- a/src/modules/webhook/webhook-reconciler.service.spec.ts+++ b/src/modules/webhook/webhook-reconciler.service.spec.ts@@ -81,7 +81,7 @@ describe('WebhookReconcilerService', () => {'message.received','stored-key_wh-1',{ from: '[email protected]' },- { signal: expect.any(AbortSignal) as AbortSignal },+ { signal: expect.any(AbortSignal) as AbortSignal, oneDirectAttempt: false },);// redeliver retires a delivered event's row itself; a second close here only repeats the write.expect(outbox.close).not.toHaveBeenCalled();@@ -107,6 +107,68 @@ describe('WebhookReconcilerService', () => {expect(stats).toMatchObject({ replayed: 0, failed: 1 });});+ // Direct mode replays one row at a time with the full retry schedule. A receiver that never answers+ // would cost that schedule per stranded row and hold every webhook queued behind it.+ describe('a receiver that already failed in this pass', () => {+ const replayedWith = (): Array<[string, boolean]> =>+ delivery.redeliver.mock.calls.map(+ ([wh, , , key, , options]: [+ { id: string },+ string,+ string,+ string,+ unknown,+ { oneDirectAttempt: boolean },+ ]) => [`${wh.id}/${key}`, options.oneDirectAttempt],+ );++ beforeEach(() => {+ webhooks.findOne.mockImplementation(({ where }: { where: { id: string } }) =>+ Promise.resolve({ id: where.id, sessionId: 'sess-1', active: true, events: ['*'] }),+ );+ outbox.findStale.mockResolvedValue([+ row({ id: 'r1', webhookId: 'wh-down', idempotencyKey: 'k1' }),+ row({ id: 'r2', webhookId: 'wh-down', idempotencyKey: 'k2' }),+ row({ id: 'r3', webhookId: 'wh-up', idempotencyKey: 'k3' }),+ row({ id: 'r4', webhookId: 'wh-down', idempotencyKey: 'k4' }),+ ]);+ });++ it('gets one attempt per further row, while other webhooks keep the full schedule', async () => {+ delivery.redeliver.mockImplementation((wh: { id: string }) =>+ Promise.resolve(wh.id === 'wh-down' ? 'failed' : 'delivered'),+ );++ const stats = await service.sweep(OPTS);++ expect(replayedWith()).toEqual([+ ['wh-down/k1', false],+ ['wh-down/k2', true],+ ['wh-up/k3', false],+ ['wh-down/k4', true],+ ]);+ expect(stats).toMatchObject({ replayed: 1, failed: 3 });+ });++ it('is forgotten by the next sweep, which gives the receiver its full schedule again', async () => {+ delivery.redeliver.mockResolvedValue('failed');+ await service.sweep(OPTS);+ delivery.redeliver.mockClear();++ await service.sweep(OPTS);++ expect(replayedWith()[0]).toEqual(['wh-down/k1', false]);+ });++ it('does not count an unexpected fault as the receiver being down', async () => {+ delivery.redeliver.mockRejectedValue(new Error('boom'));++ await service.sweep(OPTS);++ expect(replayedWith().every(([, once]) => !once)).toBe(true);+ });+ });+it('leaves retirement of an enqueued replay to the worker', async () => {outbox.findStale.mockResolvedValue([row({ attempts: 1 })]);delivery.redeliver.mockResolvedValue('enqueued');diff --git a/src/modules/webhook/webhook-reconciler.service.ts b/src/modules/webhook/webhook-reconciler.service.tsindex 4b119b103..1e983b4f0 100644--- a/src/modules/webhook/webhook-reconciler.service.ts+++ b/src/modules/webhook/webhook-reconciler.service.ts@@ -147,6 +147,11 @@ export class WebhookReconcilerService implements OnModuleInit, OnModuleDestroy {// Live jobs may occupy a whole page. Advance past them without spending their replay budget.this.cursor = scannedThrough ?? rows.at(-1)?.id;stats.scanned = rows.length;+ // Webhooks whose replay did not deliver in this pass. Direct mode replays one row at a time with+ // the full retry schedule, so a receiver that never answers would cost that schedule per stranded+ // row and hold every webhook behind it. The first failure has shown it is down: its further rows+ // make one attempt each, and the next sweep is their retry.+ const failedWebhooks = new Set<string>();for (const row of rows) {if (this.stop.signal.aborted) break;if (this.delivery.isLocallyPending(row.idempotencyKey)) {@@ -205,11 +210,12 @@ export class WebhookReconcilerService implements OnModuleInit, OnModuleDestroy {row.event,row.idempotencyKey,row.payload,- { signal: this.stop.signal },+ { signal: this.stop.signal, oneDirectAttempt: failedWebhooks.has(row.webhookId) },);if (outcome === 'failed' || outcome === 'unrecorded') {// Left 'pending' on purpose: the next sweep retries it until the budget is spent.this.logger.warn(`Replay of ${row.event} to webhook ${row.webhookId} did not deliver`);+ failedWebhooks.add(row.webhookId);stats.failed++;continue;}