Free, Open Source, Self-Hosted WhatsApp API Gateway
TypeScript apibotgatewayself-hostedwhatsappwhatsapp-apiwhatsapp-automationwhatsapp-botwhatsapp-gatewaywhatsapp-gateway-apiwhatsapp-webwhatsapp-web-api
fix(webhook): retire the outbox row of a completed queue job
Yudhi Armyndharis5297bf09f138
diff --git a/docs/06-api-specification.md b/docs/06-api-specification.mdindex a1ff1fa67..b4ffede49 100644--- a/docs/06-api-specification.md+++ b/docs/06-api-specification.md@@ -4739,7 +4739,7 @@ A delivery shed because the dispatch queue was full, or refused during shutdown,**Errors:** `401` missing/invalid API key · `403` key role below ADMIN-Queued deliveries retain their pre-hook outbox copy until the worker completes delivery or stores a terminal failure. The reconciler checks the current job before replaying and keeps rows while Redis cannot be inspected. If failure persistence is unavailable after the replay budget is spent, it retries the storage handoff without another POST. This recovery boundary is best effort before the outbox row is created; receivers must still deduplicate repeated POSTs.+Queued deliveries retain their pre-hook outbox copy until the worker completes delivery or stores a terminal failure. The reconciler checks the current job before replaying, retires the row of a completed job without a replay, and keeps rows while Redis cannot be inspected. If failure persistence is unavailable after the replay budget is spent, it retries the storage handoff without another POST. This recovery boundary is best effort before the outbox row is created; receivers must still deduplicate repeated POSTs.#### POST /api/webhooks/delivery-failures/redrive@@ -7268,7 +7268,7 @@ Webhook delivery is **at-least-once**. A consumer can legitimately receive the s- The underlying WhatsApp engine can re-fire an event for a single message.- A failed delivery (non-2xx response, timeout, or network error) is retried.-**Crash boundary.** For an ordinary dispatch, a successfully written outbox record retains the pre-hook event data until delivery succeeds, the subscription becomes ineligible, or the terminal failure is stored. Queued records keep this copy even after enqueue; the worker retires it only after one of those outcomes. If failure recording fails, the copy remains eligible for recovery. The bounded sweep (`WEBHOOK_RECONCILE_INTERVAL_MS`, default 60s) checks the stored job before replaying, leaves active jobs alone, and reuses the stored `X-OpenWA-Idempotency-Key`. If Redis cannot confirm the job state, replay waits. A missing or failed job, or a stranded direct delivery, can be replayed within `WEBHOOK_RECONCILE_MAX_ATTEMPTS`. A replay sent directly (the queue off, or its enqueue failed) that exhausts its retries stores its terminal failure row but keeps the outbox copy pending, so later sweeps replay it until that budget is spent; exhausting it retires the outbox only after terminal failure recording succeeds. Receiver deduplication remains necessary because a crash after receiver processing can cause another POST.+**Crash boundary.** For an ordinary dispatch, a successfully written outbox record retains the pre-hook event data until delivery succeeds, the subscription becomes ineligible, or the terminal failure is stored. Queued records keep this copy even after enqueue; the worker retires it only after one of those outcomes. If failure recording fails, the copy remains eligible for recovery. The bounded sweep (`WEBHOOK_RECONCILE_INTERVAL_MS`, default 60s) checks the stored job before replaying, leaves active jobs alone, retires the row of a completed job without a replay, and reuses the stored `X-OpenWA-Idempotency-Key`. If Redis cannot confirm the job state, replay waits. A missing or failed job, or a stranded direct delivery, can be replayed within `WEBHOOK_RECONCILE_MAX_ATTEMPTS`. A replay sent directly (the queue off, or its enqueue failed) that exhausts its retries stores its terminal failure row but keeps the outbox copy pending, so later sweeps replay it until that budget is spent; exhausting it retires the outbox only after terminal failure recording succeeds. Receiver deduplication remains necessary because a crash after receiver processing can cause another POST.Outbox creation is best effort, and message persistence and outbox insertion are separate transactions. A write failure or crash between them can still lose the webhook delivery. Failure records with zero attempts are excluded from operator redrive. Capacity and shutdown rejections can still be recovered from an unsettled outbox record; shutdown during retry backoff may follow earlier POSTs. The direct dispatcher leaves its local in-flight work alone during reconciliation. Graceful shutdown drains current queue jobs for at most 10 s; direct sends, an operator redrive replay in flight, and an outbox replay in flight with its plugin hooks, get up to `WEBHOOK_SHUTDOWN_DRAIN_MS` (default 5s). That replay sends at most one more POST, bounded by `WEBHOOK_TIMEOUT`, and no further row is replayed; the next start replays what it left pending.diff --git a/src/modules/webhook/webhook-delivery.service.spec.ts b/src/modules/webhook/webhook-delivery.service.spec.tsindex ad3d8b519..4bd8d483b 100644--- a/src/modules/webhook/webhook-delivery.service.spec.ts+++ b/src/modules/webhook/webhook-delivery.service.spec.ts@@ -1998,6 +1998,56 @@ describe('WebhookDeliveryService', () => {expect.objectContaining({ method: 'POST' }),);});++ // The reconciler replays a queued outbox row only when the queue no longer owns it. A live job owns it+ // (and so does a queue that cannot be read: a replay beside it would POST twice), a completed job means+ // a worker already settled it, and only a failed, unknown or missing job leaves a replay to do.+ describe('queued job state', () => {+ const states: Array<[string, boolean, boolean]> = [+ ['waiting', true, false],+ ['active', true, false],+ ['delayed', true, false],+ ['prioritized', true, false],+ ['waiting-children', true, false],+ ['completed', false, true],+ ['failed', false, false],+ ['unknown', false, false],+ ];+ const queueOn = (key: string, def?: unknown) => (key === 'queue.enabled' ? true : def);++ it.each(states)('reads a %s job as pending=%s, completed=%s', async (state, pending, completed) => {+ const queueService = await buildQueueService(queueOn);+ webhookQueue.getJob = jest.fn().mockResolvedValue({ getState: jest.fn().mockResolvedValue(state) });+ expect(await queueService.isQueueJobPending('job-1')).toBe(pending);+ expect(await queueService.hasQueueJobCompleted('job-1')).toBe(completed);+ });++ it('reads a missing job as neither pending nor completed', async () => {+ const queueService = await buildQueueService(queueOn);+ webhookQueue.getJob = jest.fn().mockResolvedValue(undefined);+ expect(await queueService.isQueueJobPending('job-1')).toBe(false);+ expect(await queueService.hasQueueJobCompleted('job-1')).toBe(false);+ });++ it('reads an unreadable queue as pending, and never as completed', async () => {+ const queueService = await buildQueueService(queueOn);+ webhookQueue.getJob = jest.fn().mockRejectedValue(new Error('redis down'));+ expect(await queueService.isQueueJobPending('job-1')).toBe(true);+ expect(await queueService.hasQueueJobCompleted('job-1')).toBe(false);+ });++ it('has no job to inspect without a queue', async () => {+ const withoutQueue = new WebhookDeliveryService(+ repository as never,+ failureRepository as never,+ { get: jest.fn((_key: string, def?: unknown) => def) } as never,+ hookManager as never,+ outboxService as never,+ );+ expect(await withoutQueue.isQueueJobPending('job-1')).toBe(false);+ expect(await withoutQueue.hasQueueJobCompleted('job-1')).toBe(false);+ });+ });});describe('direct retries', () => {diff --git a/src/modules/webhook/webhook-delivery.service.ts b/src/modules/webhook/webhook-delivery.service.tsindex 425356dc2..74143c7b0 100644--- a/src/modules/webhook/webhook-delivery.service.ts+++ b/src/modules/webhook/webhook-delivery.service.ts@@ -979,6 +979,23 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {}}+ /**+ * A finished job: its worker settled the delivery, so an outbox row still queued for it is a leftover and+ * not a stranded send. Anything unreadable answers false here; isQueueJobPending already holds the+ * conservative answer for that case.+ */+ async hasQueueJobCompleted(deliveryId: string): Promise<boolean> {+ if (!this.webhookQueue) return false;+ try {+ await producerReady(this.webhookQueue);+ const job = await this.webhookQueue.getJob(deliveryId);+ return job ? (await job.getState()) === 'completed' : false;+ } catch (error) {+ this.logger.warn('Could not inspect queued webhook delivery', { deliveryId, error: String(error) });+ return false;+ }+ }+/** Finish a spent outbox replay budget only after its failure record is durable. */async recordReplayExhaustion(row: ReplayableDelivery, url: string): Promise<boolean> {const existing = await this.failureRepository.count({diff --git a/src/modules/webhook/webhook-reconciler.service.spec.ts b/src/modules/webhook/webhook-reconciler.service.spec.tsindex 1b169047d..0c130c0a4 100644--- a/src/modules/webhook/webhook-reconciler.service.spec.ts+++ b/src/modules/webhook/webhook-reconciler.service.spec.ts@@ -44,6 +44,7 @@ describe('WebhookReconcilerService', () => {isLocallyPending: jest.Mock;recordReplayExhaustion: jest.Mock;isQueueJobPending: jest.Mock;+ hasQueueJobCompleted: jest.Mock;};let webhooks: { findOne: jest.Mock };let service: WebhookReconcilerService;@@ -58,6 +59,7 @@ describe('WebhookReconcilerService', () => {redeliver: jest.fn().mockResolvedValue('delivered'),recordReplayExhaustion: jest.fn().mockResolvedValue(true),isQueueJobPending: jest.fn().mockResolvedValue(false),+ hasQueueJobCompleted: jest.fn().mockResolvedValue(false),isLocallyPending: jest.fn().mockReturnValue(false),};webhooks = {@@ -165,6 +167,22 @@ describe('WebhookReconcilerService', () => {expect(outbox.close).not.toHaveBeenCalled();});+ it('closes a queued row whose job already completed instead of replaying it', async () => {+ // A worker settled the job but its outbox row was left behind (a worker that never closes the outbox, or a+ // failed close). A replay would POST an event the receiver already has, up to the budget, and then record+ // a failure for a delivery that succeeded.+ outbox.findStale.mockResolvedValue([row({ state: 'queued', deliveryId: 'job' })]);+ delivery.hasQueueJobCompleted.mockResolvedValue(true);++ const stats = await service.sweep(OPTS);++ expect(outbox.close).toHaveBeenCalledWith('wh-1', 'stored-key_wh-1', 'dispatched');+ expect(delivery.redeliver).not.toHaveBeenCalled();+ expect(outbox.countAttempt).not.toHaveBeenCalled();+ expect(delivery.recordReplayExhaustion).not.toHaveBeenCalled();+ expect(stats).toMatchObject({ skipped: 1, replayed: 0, failed: 0 });+ });+it('does not replay to a subscription that is gone or switched off', async () => {outbox.findStale.mockResolvedValue([row()]);webhooks.findOne.mockResolvedValue({ id: 'wh-1', active: false });diff --git a/src/modules/webhook/webhook-reconciler.service.ts b/src/modules/webhook/webhook-reconciler.service.tsindex 4caa87545..41827c2d7 100644--- a/src/modules/webhook/webhook-reconciler.service.ts+++ b/src/modules/webhook/webhook-reconciler.service.ts@@ -163,6 +163,14 @@ export class WebhookReconcilerService implements OnModuleInit, OnModuleDestroy {stats.skipped++;continue;}+ if (row.deliveryId && (await this.delivery.hasQueueJobCompleted(row.deliveryId))) {+ // A worker settled this job and its outbox row was left behind (a worker that never closes the+ // outbox, or a close that failed). It is not stranded: a replay would POST an event the receiver+ // already has, up to the budget, and then record a failure for a delivery that succeeded.+ await this.outbox.close(row.webhookId, row.idempotencyKey, 'dispatched');+ stats.skipped++;+ continue;+ }if (row.attempts >= opts.maxAttempts) {// A database fault may have prevented every terminal failure write. Keep the outbox// payload until this handoff succeeds, without sending beyond the replay budget.diff --git a/src/modules/webhook/webhook-recovery-boundaries.spec.ts b/src/modules/webhook/webhook-recovery-boundaries.spec.tsindex c1324ebb1..64180901f 100644--- a/src/modules/webhook/webhook-recovery-boundaries.spec.ts+++ b/src/modules/webhook/webhook-recovery-boundaries.spec.ts@@ -60,6 +60,7 @@ describe('webhook recovery across live ownership and configuration changes', ()const delivery = {isLocallyPending: jest.fn((key: string) => key === 'key0'),isQueueJobPending: jest.fn((id: string) => Promise.resolve(id === 'job1' || id === 'job2')),+ hasQueueJobCompleted: jest.fn().mockResolvedValue(false),// Like the real redeliver, a delivered replay retires its own outbox row.redeliver: jest.fn(async (webhook: Webhook, _sessionId: string, _event: string, key: string) => {await outbox.close(webhook.id, key, 'dispatched');