Free, Open Source, Self-Hosted WhatsApp API Gateway
TypeScript apibotgatewayself-hostedwhatsappwhatsapp-apiwhatsapp-automationwhatsapp-botwhatsapp-gatewaywhatsapp-gateway-apiwhatsapp-webwhatsapp-web-api
fix(webhook): keep failure rows in step with replayed deliveries
Yudhi Armyndharis0f87900a1b1f
diff --git a/docs/06-api-specification.md b/docs/06-api-specification.mdindex bd05093b5..cf7148fa2 100644--- a/docs/06-api-specification.md+++ b/docs/06-api-specification.md@@ -4598,6 +4598,8 @@ List webhook deliveries that exhausted every retry, most recent first. This is tBare array of `WebhookDeliveryFailure` rows, ordered by `createdAt` descending. `lastStatusCode` is `null` when the failure was a network/timeout/SSRF error rather than a non-2xx response; `idempotencyKey`/`deliveryId` let you correlate the lost event with your own receiver logs.+A delivery shed because the dispatch queue was full, or refused during shutdown, is recorded here with `attempts: 0` while its outbox row stays pending, so the outbox sweep replays it. Any delivery the receiver accepts with a `2xx`, inline or from a queued job, removes the rows filed under its idempotency key, so a replay that succeeds leaves none. A replay that exhausts its retries files a row carrying the real error and attempt count and only then removes the `attempts: 0` row, so the event stays on record throughout, across a restart or a lost queue job too. A replay that fails before sending (a payload over the size cap after the `webhook:before` hooks, or one that cannot be serialized) gives the `attempts: 0` row that reason instead. A replay that keeps failing files no second row. `openwa_webhook_delivery_failures_total` counts the original shed, and counts once more if a replay also exhausts its retries.+**Errors:** `401` missing/invalid API key · `403` key role below ADMIN#### POST /api/sessions/:sessionId/webhooks@@ -7040,7 +7042,7 @@ Webhook delivery is **at-least-once**. A consumer can legitimately receive the s- The underlying WhatsApp engine can re-fire an event for a single message.- A failed delivery (non-2xx response, timeout, or network error) is retried.-**Crash boundary.** Every delivery is recorded before it is attempted, and the record is retired once something durable owns it: the queue job in queued mode, the completed send in direct mode. A hard crash (SIGKILL, OOM) therefore leaves the record behind, and a bounded sweep (`WEBHOOK_RECONCILE_INTERVAL_MS`, default 60s) replays whatever is still stranded, reusing the stored `X-OpenWA-Idempotency-Key` so the retry stays deduplicable at your receiver. A delivery that keeps failing exhausts `WEBHOOK_RECONCILE_MAX_ATTEMPTS` and goes terminal rather than replaying forever. One window remains open: a crash between persisting the message and writing that record loses the delivery, because the two are not yet one transaction. The failure table records exhausted retries, plus over-budget, dispatch-capacity-exceeded, and shutdown-rejected deliveries with attempts 0. Read the last two as a report rather than a verdict: a delivery the dispatcher shed for capacity or refused during the drain was rejected before its POST, so it keeps its record and is replayed by the same sweep, and a row there can belong to an event that was later delivered. A node's sweep also leaves alone a delivery that the same node is still waiting to dispatch or retrying, however long that takes; with several nodes on one database, another node can still replay it, and the stored idempotency key keeps that duplicate deduplicable. Enabling the queue (`QUEUE_ENABLED=true`, needs Redis) makes the dispatch durable from the enqueue onward. In both modes, a graceful shutdown drains in-flight deliveries first: the queued path waits for each worker's current job, and the direct path waits up to `WEBHOOK_SHUTDOWN_DRAIN_MS` (default 5s; raise it to at least `WEBHOOK_TIMEOUT`, default 10s, if a slow receiver must finish).+**Crash boundary.** Every delivery is recorded before it is attempted, and the record is retired once something durable owns it: the queue job in queued mode, the completed send in direct mode. A hard crash (SIGKILL, OOM) therefore leaves the record behind, and a bounded sweep (`WEBHOOK_RECONCILE_INTERVAL_MS`, default 60s) replays whatever is still stranded, reusing the stored `X-OpenWA-Idempotency-Key` so the retry stays deduplicable at your receiver. A delivery that keeps failing exhausts `WEBHOOK_RECONCILE_MAX_ATTEMPTS` and goes terminal rather than replaying forever. One window remains open: a crash between persisting the message and writing that record loses the delivery, because the two are not yet one transaction. The failure table records exhausted retries, plus over-budget, dispatch-capacity-exceeded, and shutdown-rejected deliveries with attempts 0. Read the last two as a report rather than a verdict: a delivery the dispatcher shed for capacity or refused during the drain was rejected before its POST, so it keeps its record until the same sweep replays it; a replay whose retries run out replaces that record with its own row, one that fails before sending gives it that reason, and one that delivers removes it. A node's sweep also leaves alone a delivery that the same node is still waiting to dispatch or retrying, however long that takes; with several nodes on one database, another node can still replay it, and the stored idempotency key keeps that duplicate deduplicable. Enabling the queue (`QUEUE_ENABLED=true`, needs Redis) makes the dispatch durable from the enqueue onward. In both modes, a graceful shutdown drains in-flight deliveries first: the queued path waits for each worker's current job, and the direct path waits up to `WEBHOOK_SHUTDOWN_DRAIN_MS` (default 5s; raise it to at least `WEBHOOK_TIMEOUT`, default 10s, if a slow receiver must finish).**Design your handler to be idempotent**, keyed on the `X-OpenWA-Idempotency-Key` header (see below). As a server-side safety net, OpenWA de-duplicates inbound `message.received` before dispatch (a re-fired event for an already-persisted message is dropped), so one webhook normally sees each inbound message once — but this is best-effort defense-in-depth and does not remove the need for consumer-side idempotency.diff --git a/src/modules/queue/processors/webhook.processor.spec.ts b/src/modules/queue/processors/webhook.processor.spec.tsindex 277845803..a200bcc63 100644--- a/src/modules/queue/processors/webhook.processor.spec.ts+++ b/src/modules/queue/processors/webhook.processor.spec.ts@@ -1,5 +1,5 @@import { Job } from 'bullmq';-import { Repository } from 'typeorm';+import { FindOperator, Repository } from 'typeorm';import { ConfigService } from '@nestjs/config';import { WebhookProcessor } from './webhook.processor';import { Webhook } from '../../webhook/entities/webhook.entity';@@ -24,7 +24,8 @@ jest.mock('undici', () => {describe('WebhookProcessor', () => {let processor: WebhookProcessor;let repo: { update: jest.Mock; findOne: jest.Mock };- let failureRepo: { insert: jest.Mock; count: jest.Mock };+ let failureRepo: { insert: jest.Mock; count: jest.Mock; delete: jest.Mock };+ let failureRows: Array<{ webhookId?: string; idempotencyKey?: string | null; attempts?: number }>;let hookManager: { execute: jest.Mock };let configService: { get: jest.Mock };let mockFetch: jest.Mock;@@ -60,18 +61,38 @@ describe('WebhookProcessor', () => {};// Stateful like the real table: the recorder counts existing rows for the delivery before it// inserts, so a constant would leave that guard unexercised here and let a duplicated row pass.- const insertedFailures: Array<{ webhookId?: string; idempotencyKey?: string | null }> = [];+ failureRows = [];+ const insertedFailures = failureRows;failureRepo = {insert: jest.fn().mockImplementation((rowToInsert: { webhookId?: string; idempotencyKey?: string | null }) => {insertedFailures.push(rowToInsert);return Promise.resolve({});}),+ delete: jest+ .fn()+ .mockImplementation((where: { webhookId?: string; idempotencyKey?: string; attempts?: number }) => {+ const keep = failureRows.filter(+ r =>+ r.webhookId !== where.webhookId ||+ r.idempotencyKey !== where.idempotencyKey ||+ (where.attempts !== undefined && r.attempts !== where.attempts),+ );+ const affected = failureRows.length - keep.length;+ failureRows.splice(0, failureRows.length, ...keep);+ return Promise.resolve({ affected });+ }),count: jest.fn()- .mockImplementation((opts: { where: { webhookId?: string; idempotencyKey?: string } }) =>+ .mockImplementation((opts: { where: { webhookId?: string; idempotencyKey?: string; attempts?: unknown } }) =>Promise.resolve(insertedFailures.filter(- r => r.webhookId === opts.where.webhookId && r.idempotencyKey === opts.where.idempotencyKey,+ r =>+ r.webhookId === opts.where.webhookId &&+ r.idempotencyKey === opts.where.idempotencyKey &&+ (opts.where.attempts === undefined ||+ (opts.where.attempts instanceof FindOperator && opts.where.attempts.type === 'moreThan'+ ? ((r as { attempts?: number }).attempts ?? 0) > (opts.where.attempts.value as number)+ : (r as { attempts?: number }).attempts === opts.where.attempts)),).length,),),@@ -164,6 +185,54 @@ describe('WebhookProcessor', () => {);});+ it("replaces a replayed delivery's attempts-0 shed row with its own final-attempt row", async () => {+ // The shed row's dedup entry would otherwise suppress this row and keep only the capacity reason.+ failureRows.push({ webhookId: 'wh-1', idempotencyKey: 'k', attempts: 0 });+ mockFetch.mockResolvedValue({ ok: false, status: 503, statusText: 'Service Unavailable' });++ await expect(processor.process(makeJob({ maxRetries: 3 }, 2))).rejects.toThrow();++ expect(failureRepo.delete).toHaveBeenCalledWith({ webhookId: 'wh-1', idempotencyKey: 'k', attempts: 0 });+ expect(failureRows).toEqual([expect.objectContaining({ idempotencyKey: 'k', attempts: 3, lastStatusCode: 503 })]);+ });++ it('keeps the delivery error and both rows when the shed-row delete fails', async () => {+ // The final-attempt row is written before the shed row is removed, so a failed delete leaves+ // the event recorded twice rather than not at all. It is logged, not thrown: process() still+ // rejects with the HTTP error.+ failureRows.push({ webhookId: 'wh-1', idempotencyKey: 'k', attempts: 0 });+ failureRepo.delete.mockRejectedValueOnce(new Error('db down'));+ mockFetch.mockResolvedValue({ ok: false, status: 503, statusText: 'Service Unavailable' });++ await expect(processor.process(makeJob({ maxRetries: 3 }, 2))).rejects.toThrow('HTTP 503');++ expect(failureRows).toEqual([+ expect.objectContaining({ idempotencyKey: 'k', attempts: 0 }),+ expect.objectContaining({ idempotencyKey: 'k', attempts: 3, lastStatusCode: 503 }),+ ]);+ });++ it('deletes nothing for a job whose payload carries no idempotency key', async () => {+ mockFetch.mockResolvedValue({ ok: false, status: 503, statusText: 'Service Unavailable' });+ const job = makeJob({ maxRetries: 3 }, 2);+ (job.data.payload as { idempotencyKey?: string }).idempotencyKey = undefined;++ await expect(processor.process(job)).rejects.toThrow();++ expect(failureRepo.delete).not.toHaveBeenCalled();+ expect(failureRepo.insert).toHaveBeenCalledTimes(1);+ });++ it('leaves the shed row alone on an attempt that is not final', async () => {+ failureRows.push({ webhookId: 'wh-1', idempotencyKey: 'k', attempts: 0 });+ mockFetch.mockResolvedValue({ ok: false, status: 500, statusText: 'Server Error' });++ await expect(processor.process(makeJob({ maxRetries: 3 }, 0))).rejects.toThrow();++ expect(failureRepo.delete).not.toHaveBeenCalled();+ expect(failureRows).toEqual([expect.objectContaining({ idempotencyKey: 'k', attempts: 0 })]);+ });+it('does NOT persist a delivery-failure record before the final attempt', async () => {mockFetch.mockResolvedValue({ ok: false, status: 500, statusText: 'Server Error' });@@ -203,6 +272,31 @@ describe('WebhookProcessor', () => {expect(inserted.lastError).not.toMatch(/169\.254\.169\.254/);});+ it('clears every failure row of the delivery once the job delivers it', async () => {+ // An earlier inline replay failed (attempts > 0) after a shed (attempts 0); this queued replay+ // then delivered. Both rows would list a delivered event as lost. Another delivery's row stays.+ failureRows.push(+ { webhookId: 'wh-1', idempotencyKey: 'k', attempts: 0 },+ { webhookId: 'wh-1', idempotencyKey: 'k', attempts: 3 },+ { webhookId: 'wh-1', idempotencyKey: 'other', attempts: 3 },+ );+ mockFetch.mockResolvedValue({ ok: true, status: 200 });++ await expect(processor.process(makeJob())).resolves.toMatchObject({ success: true });++ expect(failureRows).toEqual([expect.objectContaining({ idempotencyKey: 'other' })]);+ });++ it('keeps the success outcome when clearing the failure rows fails', async () => {+ failureRepo.delete.mockRejectedValueOnce(new Error('db down'));+ mockFetch.mockResolvedValue({ ok: true, status: 200 });++ await expect(processor.process(makeJob({ maxRetries: 3 }, 2))).resolves.toMatchObject({ success: true });++ expect(failureRepo.delete).toHaveBeenCalledTimes(1);+ expect(failureRepo.insert).not.toHaveBeenCalled();+ });+it('keeps the success outcome when post-delivery bookkeeping fails after a 2xx (no retry, no DLQ row)', async () => {mockFetch.mockResolvedValue({ ok: true, status: 200 });repo.update.mockRejectedValue(new Error('db down'));@@ -331,6 +425,28 @@ describe('WebhookProcessor', () => {expect(hookManager.execute).not.toHaveBeenCalled();});+ it("replaces a replayed delivery's attempts-0 shed row with the stall row", async () => {+ failureRows.push({ webhookId: 'wh-1', idempotencyKey: 'k', attempts: 0 });++ await processor.onWorkerFailed(makeJob({}, 1), new Error('job stalled more than allowable limit'));++ expect(failureRows).toEqual([+ expect.objectContaining({+ idempotencyKey: 'k',+ attempts: 1,+ lastError: 'job stalled more than allowable limit',+ }),+ ]);+ });++ it('still files the stall row when the shed-row delete fails', async () => {+ failureRepo.delete.mockRejectedValueOnce(new Error('db down'));++ await processor.onWorkerFailed(makeJob({}, 1), new Error('job stalled more than allowable limit'));++ expect(failureRepo.insert).toHaveBeenCalledWith(expect.objectContaining({ idempotencyKey: 'k', attempts: 1 }));+ });+it('still records the stall against the enqueue-time URL when the webhook row cannot be read', async () => {repo.findOne.mockRejectedValue(new Error('db down'));diff --git a/src/modules/queue/processors/webhook.processor.ts b/src/modules/queue/processors/webhook.processor.tsindex 96d07d319..bff444f94 100644--- a/src/modules/queue/processors/webhook.processor.ts+++ b/src/modules/queue/processors/webhook.processor.ts@@ -9,7 +9,11 @@ import { workerConnectionOptions, webhookWorkerConcurrency } from '../redis-connimport { WebhookJobData, WebhookPayload } from '../../webhook/webhook.service';import { Webhook } from '../../webhook/entities/webhook.entity';import { WebhookDeliveryFailure } from '../../webhook/entities/webhook-delivery-failure.entity';-import { recordWebhookDeliveryFailure, statusCodeFromError } from '../../webhook/utils/record-delivery-failure';+import {+ clearDeliveryFailureRows,+ recordWebhookDeliveryFailure,+ statusCodeFromError,+} from '../../webhook/utils/record-delivery-failure';import { buildDeliveryHeaders, postWebhookPayload } from '../../webhook/utils/deliver-once';import { HookManager } from '../../../core/hooks';import { redactSsrfError } from '../../../common/security/ssrf-guard';@@ -186,6 +190,11 @@ export class WebhookProcessor extends WorkerHost {);}+ // A delivered event must not stay listed as lost. A failure row exists for it only when an+ // earlier dispatch of the same delivery was shed, refused or failed before this job ran. An+ // indexed delete that usually matches nothing.+ await clearDeliveryFailureRows(this.failureRepository, this.logger, webhookId, payload.idempotencyKey);+// Execute hook after successful deliveryawait this.hookManager.execute('webhook:delivered',diff --git a/src/modules/webhook/utils/record-delivery-failure.spec.ts b/src/modules/webhook/utils/record-delivery-failure.spec.tsindex 005d3ddd3..b4f901f73 100644--- a/src/modules/webhook/utils/record-delivery-failure.spec.ts+++ b/src/modules/webhook/utils/record-delivery-failure.spec.ts@@ -1,6 +1,6 @@-import { Repository } from 'typeorm';+import { MoreThan, Repository } from 'typeorm';import { WebhookDeliveryFailure } from '../entities/webhook-delivery-failure.entity';-import { recordWebhookDeliveryFailure, statusCodeFromError } from './record-delivery-failure';+import { clearDeliveryFailureRows, recordWebhookDeliveryFailure, statusCodeFromError } from './record-delivery-failure';describe('statusCodeFromError', () => {it('parses the status from an "HTTP <code>: ..." message', () => {@@ -28,7 +28,12 @@ describe('recordWebhookDeliveryFailure', () => {};const repoWith = (insert: jest.Mock, existing = 0): Repository<WebhookDeliveryFailure> =>- ({ insert, count: jest.fn().mockResolvedValue(existing) }) as unknown as Repository<WebhookDeliveryFailure>;+ ({+ insert,+ count: jest.fn().mockResolvedValue(existing),+ delete: jest.fn().mockResolvedValue({}),+ update: jest.fn().mockResolvedValue({}),+ }) as unknown as Repository<WebhookDeliveryFailure>;it('inserts the failure record, defaulting a missing lastStatusCode to null', async () => {const insert = jest.fn().mockResolvedValue({});@@ -54,6 +59,56 @@ describe('recordWebhookDeliveryFailure', () => {expect(logger.error).not.toHaveBeenCalled();});+ it('gives the attempts-0 row the reason of an unsent delivery recorded again, without a second row', async () => {+ // A shed delivery whose replay then fails before sending: the operator has to act on the new+ // reason, not on the shed. Only attempts-0 rows take it, and the metric does not count it again.+ const insert = jest.fn().mockResolvedValue({});+ const update = jest.fn().mockResolvedValue({ affected: 1 });+ const repo = {+ insert,+ update,+ count: jest.fn().mockResolvedValue(1),+ } as unknown as Repository<WebhookDeliveryFailure>;+ const logger = { error: jest.fn() };++ await expect(+ recordWebhookDeliveryFailure(repo, logger, { ...input, attempts: 0, lastStatusCode: null, lastError: 'too big' }),+ ).resolves.toBe(false);++ expect(insert).not.toHaveBeenCalled();+ expect(update).toHaveBeenCalledWith(+ { webhookId: 'wh-1', idempotencyKey: 'k', attempts: 0 },+ { lastError: 'too big', url: 'https://r.example/h', deliveryId: 'd' },+ );+ });++ it('leaves the recorded row alone for a terminal failure recorded again', async () => {+ const update = jest.fn();+ const repo = {+ insert: jest.fn(),+ update,+ count: jest.fn().mockResolvedValue(1),+ } as unknown as Repository<WebhookDeliveryFailure>;++ await expect(recordWebhookDeliveryFailure(repo, { error: jest.fn() }, input)).resolves.toBe(false);++ expect(update).not.toHaveBeenCalled();+ });++ it('still reports an unsent duplicate as not recorded when refreshing its reason fails', async () => {+ // A true here would bump the failure metric for a loss that is already counted.+ const repo = {+ insert: jest.fn(),+ update: jest.fn().mockRejectedValue(new Error('db down')),+ count: jest.fn().mockResolvedValue(1),+ } as unknown as Repository<WebhookDeliveryFailure>;+ const logger = { error: jest.fn() };++ await expect(recordWebhookDeliveryFailure(repo, logger, { ...input, attempts: 0 })).resolves.toBe(false);++ expect(logger.error).toHaveBeenCalledTimes(1);+ });+it('still records a failure that carries no idempotency key', async () => {// Control: without a key there is no identity to dedupe on, so the guard must not swallow the// row. Otherwise "skips a duplicate" would be satisfied by a helper that stopped inserting.@@ -67,6 +122,58 @@ describe('recordWebhookDeliveryFailure', () => {expect(insert).toHaveBeenCalled();});+ it('dedupes a terminal row only against terminal rows, then removes the attempts-0 row it replaces', async () => {+ // A shed or shutdown-refused dispatch left an attempts-0 row. Counted as a duplicate, it would+ // keep the capacity reason as the only record and drop the receiver's actual answer.+ const calls: string[] = [];+ const insert = jest.fn().mockImplementation(() => (calls.push('insert'), Promise.resolve({})));+ const count = jest.fn().mockResolvedValue(0);+ const del = jest.fn().mockImplementation(() => (calls.push('delete'), Promise.resolve({})));+ const repo = { insert, count, delete: del } as unknown as Repository<WebhookDeliveryFailure>;+ const logger = { error: jest.fn() };++ await expect(recordWebhookDeliveryFailure(repo, logger, input)).resolves.toBe(true);++ expect(count).toHaveBeenCalledWith({+ where: { webhookId: 'wh-1', idempotencyKey: 'k', attempts: MoreThan(0) },+ });+ expect(del).toHaveBeenCalledWith({ webhookId: 'wh-1', idempotencyKey: 'k', attempts: 0 });+ // Written first, so a crash in between leaves two rows for the event rather than none.+ expect(calls).toEqual(['insert', 'delete']);+ });++ it('dedupes an attempts-0 row against every row of the delivery and removes nothing', async () => {+ const insert = jest.fn().mockResolvedValue({});+ const count = jest.fn().mockResolvedValue(0);+ const del = jest.fn().mockResolvedValue({});+ const repo = { insert, count, delete: del } as unknown as Repository<WebhookDeliveryFailure>;+ const logger = { error: jest.fn() };++ await expect(recordWebhookDeliveryFailure(repo, logger, { ...input, attempts: 0 })).resolves.toBe(true);++ expect(count).toHaveBeenCalledWith({ where: { webhookId: 'wh-1', idempotencyKey: 'k' } });+ expect(del).not.toHaveBeenCalled();+ });++ it('keeps the terminal row and the result when removing the attempts-0 row fails', async () => {+ const insert = jest.fn().mockResolvedValue({});+ const repo = {+ insert,+ count: jest.fn().mockResolvedValue(0),+ delete: jest.fn().mockRejectedValue(new Error('db down')),+ } as unknown as Repository<WebhookDeliveryFailure>;+ const logger = { error: jest.fn() };++ await expect(recordWebhookDeliveryFailure(repo, logger, input)).resolves.toBe(true);++ expect(insert).toHaveBeenCalledTimes(1);+ expect(logger.error).toHaveBeenCalledWith(+ expect.any(String),+ 'db down',+ expect.objectContaining({ action: 'webhook_failure_clear_error' }),+ );+ });+it('swallows a repository error so a logging hiccup cannot re-poison the delivery', async () => {const insert = jest.fn().mockRejectedValue(new Error('db down'));const logger = { error: jest.fn() };@@ -77,3 +184,40 @@ describe('recordWebhookDeliveryFailure', () => {expect(logger.error).toHaveBeenCalled();});});++describe('clearDeliveryFailureRows', () => {+ const repoWithDelete = (del: jest.Mock): Repository<WebhookDeliveryFailure> =>+ ({ delete: del }) as unknown as Repository<WebhookDeliveryFailure>;++ it('deletes every row of one delivery, or only its attempts-0 rows', async () => {+ const del = jest.fn().mockResolvedValue({});+ const logger = { error: jest.fn() };++ await clearDeliveryFailureRows(repoWithDelete(del), logger, 'wh-1', 'k');+ await clearDeliveryFailureRows(repoWithDelete(del), logger, 'wh-1', 'k', true);++ expect(del.mock.calls).toEqual([+ [{ webhookId: 'wh-1', idempotencyKey: 'k' }],+ [{ webhookId: 'wh-1', idempotencyKey: 'k', attempts: 0 }],+ ]);+ });++ it('deletes nothing without an idempotency key', async () => {+ // Matching on webhookId alone would erase the rows of every other lost event of that webhook.+ const del = jest.fn().mockResolvedValue({});++ await clearDeliveryFailureRows(repoWithDelete(del), { error: jest.fn() }, 'wh-1', undefined);++ expect(del).not.toHaveBeenCalled();+ });++ it('logs a failed delete instead of throwing', async () => {+ const logger = { error: jest.fn() };++ await expect(+ clearDeliveryFailureRows(repoWithDelete(jest.fn().mockRejectedValue(new Error('db down'))), logger, 'wh-1', 'k'),+ ).resolves.toBeUndefined();++ expect(logger.error).toHaveBeenCalledTimes(1);+ });+});diff --git a/src/modules/webhook/utils/record-delivery-failure.ts b/src/modules/webhook/utils/record-delivery-failure.tsindex ee9a1f921..b7556306a 100644--- a/src/modules/webhook/utils/record-delivery-failure.ts+++ b/src/modules/webhook/utils/record-delivery-failure.ts@@ -1,4 +1,4 @@-import { Repository } from 'typeorm';+import { MoreThan, Repository } from 'typeorm';import { WebhookDeliveryFailure } from '../entities/webhook-delivery-failure.entity';export interface WebhookDeliveryFailureInput {@@ -24,6 +24,34 @@ export function statusCodeFromError(message: string): number | null {return m ? Number(m[1]) : null;}+/**+ * Remove the dead-letter rows of one delivery. Every successful POST calls this for its key, so a+ * delivered event is never listed as lost, whichever earlier dispatch of it was shed, refused or+ * failed. With `onlyUnattempted` it removes only the attempts-0 rows (shed, refused, preflight),+ * which the recorder does right after writing a terminal row, so the event always keeps a record.+ * Best-effort: a failed delete is logged and never fails the delivery that triggered it.+ */+export async function clearDeliveryFailureRows(+ repo: Repository<WebhookDeliveryFailure>,+ logger: ErrorLogger,+ webhookId: string,+ idempotencyKey: string | undefined,+ onlyUnattempted = false,+): Promise<void> {+ // Without a key there is no delivery identity to match, and deleting on webhookId alone would+ // erase the rows of every other lost event of that webhook.+ if (!idempotencyKey) return;+ try {+ await repo.delete(onlyUnattempted ? { webhookId, idempotencyKey, attempts: 0 } : { webhookId, idempotencyKey });+ } catch (err) {+ logger.error('Failed to clear webhook delivery-failure rows', err instanceof Error ? err.message : String(err), {+ webhookId,+ idempotencyKey,+ action: 'webhook_failure_clear_error',+ });+ }+}+/*** Append a durable record of a webhook delivery that exhausted all retries. Called from BOTH terminal* paths — the BullMQ processor's final attempt and the direct-fallback's last attempt. Wrapped in its@@ -40,15 +68,50 @@ export async function recordWebhookDeliveryFailure(// up to its budget and every failed replay arrives here, so without this guard a single lost// event is reported to the operator as several and counted that many times in the failure// metric. An absent idempotencyKey carries no identity to dedupe on, so it is always appended.+ // A terminal row (attempts > 0) is deduplicated only against other terminal rows: an attempts-0+ // row (a shed or shutdown-refused dispatch) is replaced by it below instead of suppressing it.+ const terminal = input.attempts > 0;if (input.idempotencyKey) {const existing = await repo.count({- where: { webhookId: input.webhookId, idempotencyKey: input.idempotencyKey },+ where: terminal+ ? { webhookId: input.webhookId, idempotencyKey: input.idempotencyKey, attempts: MoreThan(0) }+ : { webhookId: input.webhookId, idempotencyKey: input.idempotencyKey },});if (existing > 0) {+ if (!terminal) {+ // An unsent delivery recorded again: typically a replay of a shed or refused dispatch that+ // then failed before sending (a payload over the size cap, or one that cannot be+ // serialized). Its reason is the one the operator has to act on now, so the attempts-0 row+ // takes it. A terminal row of the same delivery is left as it is.+ await repo+ .update(+ { webhookId: input.webhookId, idempotencyKey: input.idempotencyKey, attempts: 0 },+ {+ lastError: input.lastError,+ url: input.url,+ ...(input.deliveryId ? { deliveryId: input.deliveryId } : {}),+ },+ )+ .catch((err: unknown) =>+ logger.error(+ 'Failed to refresh the reason of an unsent webhook delivery',+ err instanceof Error ? err.message : String(err),+ {+ webhookId: input.webhookId,+ idempotencyKey: input.idempotencyKey,+ action: 'webhook_failure_record_error',+ },+ ),+ );+ }return false;}}await repo.insert({ ...input, lastStatusCode: input.lastStatusCode ?? null });+ if (terminal) {+ // Only after the insert, so a crash in between leaves both rows, never none.+ await clearDeliveryFailureRows(repo, logger, input.webhookId, input.idempotencyKey, true);+ }return true;} catch (err) {logger.error(diff --git a/src/modules/webhook/webhook-delivery.service.spec.ts b/src/modules/webhook/webhook-delivery.service.spec.tsindex 1f1c6bf2a..a1ad31aec 100644--- a/src/modules/webhook/webhook-delivery.service.spec.ts+++ b/src/modules/webhook/webhook-delivery.service.spec.ts@@ -13,7 +13,7 @@ jest.mock('undici', () => {import { Test, TestingModule } from '@nestjs/testing';import { getRepositoryToken } from '@nestjs/typeorm';import { getQueueToken } from '@nestjs/bullmq';-import { Repository } from 'typeorm';+import { FindOperator, Repository } from 'typeorm';import { ConfigService } from '@nestjs/config';import * as crypto from 'crypto';import { fetch as undiciFetch } from 'undici';@@ -49,6 +49,16 @@ function createMockWebhook(overrides: Partial<Webhook> = {}): Webhook {};}+// The recorder counts terminal rows with `attempts: MoreThan(0)`; honour that operator like the+// real table so the fake cannot count an attempts-0 row as a terminal duplicate.+function attemptsMatch(actual: number | undefined, filter: unknown): boolean {+ if (filter === undefined) return true;+ if (filter instanceof FindOperator) {+ return filter.type === 'moreThan' ? (actual ?? 0) > (filter.value as number) : false;+ }+ return actual === filter;+}+describe('WebhookDeliveryService', () => {let service: WebhookDeliveryService;let repository: jest.Mocked<Partial<Repository<Webhook>>>;@@ -58,6 +68,7 @@ describe('WebhookDeliveryService', () => {let webhookQueue: jest.Mocked<Record<string, jest.Mock>>;let lidStore: { getCached: jest.Mock; resolveLid: jest.Mock };let outboxService: { open: jest.Mock; close: jest.Mock };+ let insertedFailures: Array<Partial<WebhookDeliveryFailure>>;beforeEach(async () => {repository = {@@ -68,23 +79,44 @@ describe('WebhookDeliveryService', () => {// Backed by the rows it accepts, not a constant: the dedupe guard reads count() before every// insert, and a jest.fn() returning undefined would make the guard silently inert here while// still passing every assertion.- const insertedFailures: Array<{ webhookId?: string; idempotencyKey?: string | null }> = [];+ insertedFailures = [];failureRepository = {- insert: jest.fn().mockImplementation((rowToInsert: { webhookId?: string; idempotencyKey?: string | null }) => {+ insert: jest.fn().mockImplementation((rowToInsert: Partial<WebhookDeliveryFailure>) => {insertedFailures.push(rowToInsert);return Promise.resolve({});}),count: jest.fn()- .mockImplementation((opts: { where: { webhookId?: string; idempotencyKey?: string } }) =>+ .mockImplementation((opts: { where: { webhookId?: string; idempotencyKey?: string; attempts?: unknown } }) =>Promise.resolve(insertedFailures.filter(- r => r.webhookId === opts.where.webhookId && r.idempotencyKey === opts.where.idempotencyKey,+ r =>+ r.webhookId === opts.where.webhookId &&+ r.idempotencyKey === opts.where.idempotencyKey &&+ attemptsMatch(r.attempts, opts.where.attempts),).length,),),find: jest.fn().mockResolvedValue([]),- delete: jest.fn().mockResolvedValue({ affected: 0 }),+ // Also backed by the rows, so a delete that runs too late (after the dedupe guard) shows up.+ delete: jest.fn().mockImplementation((where: Partial<WebhookDeliveryFailure>) => {+ const before = insertedFailures.length;+ insertedFailures = insertedFailures.filter(r =>+ Object.entries(where).some(([k, v]) => r[k as keyof WebhookDeliveryFailure] !== v),+ );+ return Promise.resolve({ affected: before - insertedFailures.length });+ }),+ update: jest+ .fn()+ .mockImplementation((where: Partial<WebhookDeliveryFailure>, set: Partial<WebhookDeliveryFailure>) => {+ let affected = 0;+ insertedFailures = insertedFailures.map(r => {+ if (Object.entries(where).some(([k, v]) => r[k as keyof WebhookDeliveryFailure] !== v)) return r;+ affected++;+ return { ...r, ...set };+ });+ return Promise.resolve({ affected });+ }),};configService = {@@ -781,6 +813,86 @@ describe('WebhookDeliveryService', () => {expect(getWebhookDeliveryFailuresTotal()).toBe(failuresBefore + 2);});+ it('replaces the attempts-0 shed row with the real error when a direct replay fails', async () => {+ // The shed row's dedup entry used to block the replay's own row, so the only record of a+ // lost event said "queue full" with no attempts, never the receiver's actual answer.+ const webhook = createMockWebhook({ id: 'wh-1', events: ['message.received'], retryCount: 1 });+ (hookManager.execute as jest.Mock).mockResolvedValue({ continue: true, data: {} });+ await failureRepository.insert!({+ webhookId: 'wh-1',+ idempotencyKey: 'shed-key',+ attempts: 0,+ lastError: 'ConcurrencyLimiter queue full',+ });+ mockFetch.mockReset();+ mockFetch.mockResolvedValue({ ok: false, status: 500, statusText: 'Server Error' });++ await expect(+ service.redeliver(webhook, 'sess-1', 'message.received', 'shed-key', { from: '[email protected]' }),+ ).resolves.toBe('failed');++ expect(insertedFailures).toEqual([+ expect.objectContaining({+ webhookId: 'wh-1',+ idempotencyKey: 'shed-key',+ attempts: 1,+ lastStatusCode: 500,+ lastError: 'HTTP 500: Server Error',+ }),+ ]);+ });++ it('keeps the shed row while a direct replay is still in flight, so a restart then cannot lose it', async () => {+ // Replays are not drained at shutdown. Deleting the shed row before the POST left an event+ // whose final replay was cut short with no failure record at all.+ const webhook = createMockWebhook({ id: 'wh-1', events: ['message.received'], retryCount: 1 });+ (hookManager.execute as jest.Mock).mockResolvedValue({ continue: true, data: {} });+ await failureRepository.insert!({ webhookId: 'wh-1', idempotencyKey: 'shed-key', attempts: 0 });+ mockFetch.mockReset();+ let release!: (v: unknown) => void;+ mockFetch.mockImplementation(() => new Promise(resolve => (release = resolve)));++ const replay = service.redeliver(webhook, 'sess-1', 'message.received', 'shed-key', { from: '[email protected]' });+ for (let i = 0; i < 20 && mockFetch.mock.calls.length === 0; i++) await new Promise(r => setImmediate(r));+ expect(mockFetch).toHaveBeenCalledTimes(1);+ expect(insertedFailures).toEqual([expect.objectContaining({ idempotencyKey: 'shed-key', attempts: 0 })]);++ release({ ok: false, status: 503, statusText: 'Unavailable' });+ await expect(replay).resolves.toBe('failed');+ expect(insertedFailures).toEqual([expect.objectContaining({ idempotencyKey: 'shed-key', attempts: 1 })]);+ });++ it('keeps one row for a replay that preflight rejects, carrying the preflight reason', async () => {+ // Every sweep replays it again, so a second row would list one lost event several times. The+ // shed reason is stale, though: the operator has to fix the payload size, not the capacity.+ (configService.get as jest.Mock).mockImplementation(<T>(key: string, def?: T): T | boolean | number => {+ if (key === 'queue.enabled') return false;+ if (key === 'webhook.maxPayloadBytes') return 1024;+ return def as T;+ });+ const webhook = createMockWebhook({ id: 'wh-1', events: ['message.received'], retryCount: 1 });+ (hookManager.execute as jest.Mock).mockResolvedValue({ continue: true, data: {} });+ await failureRepository.insert!({+ webhookId: 'wh-1',+ idempotencyKey: 'shed-key',+ attempts: 0,+ lastError: 'ConcurrencyLimiter queue full',+ });+ mockFetch.mockReset();+ const failuresBefore = getWebhookDeliveryFailuresTotal();++ await expect(+ service.redeliver(webhook, 'sess-1', 'message.received', 'shed-key', { big: 'x'.repeat(64 * 1024) }),+ ).resolves.toBe('failed');++ expect(mockFetch).not.toHaveBeenCalled();+ expect(failureRepository.delete).not.toHaveBeenCalled();+ expect(failureRepository.insert).toHaveBeenCalledTimes(1);+ expect(insertedFailures).toEqual([expect.objectContaining({ idempotencyKey: 'shed-key', attempts: 0 })]);+ expect(insertedFailures[0].lastError).toContain('exceeding the 1024-byte cap');+ expect(getWebhookDeliveryFailuresTotal()).toBe(failuresBefore);+ });+it("isolates each webhook's data so an in-place before-hook mutation cannot bleed across webhooks", async () => {const a = createMockWebhook({ id: 'wh-a', events: ['message.received'] });const b = createMockWebhook({ id: 'wh-b', events: ['message.received'] });@@ -1453,6 +1565,87 @@ describe('WebhookDeliveryService', () => {expect(jobData).not.toHaveProperty('headers');});+ it('leaves the shed row of a replay in place when its job is queued', async () => {+ // The job resolves it: a delivery clears it and a terminal failure replaces it. Deleting it+ // here would leave no record if the job is then lost, or the add fails and falls back.+ const queueService = await buildQueueService((key: string, def?: unknown) => {+ if (key === 'queue.enabled') return true;+ if (key === 'webhook.retryDelay') return 5000;+ return def;+ });+ (hookManager.execute as jest.Mock).mockResolvedValue({ continue: true, data: {} });+ await failureRepository.insert!({ webhookId: 'wh-1', idempotencyKey: 'stored-key', attempts: 0 });+ const webhook = createMockWebhook({ id: 'wh-1', events: ['message.received'] });++ await expect(queueService.redeliver(webhook, 'sess-1', 'message.received', 'stored-key', {})).resolves.toBe(+ 'enqueued',+ );++ expect(failureRepository.delete).not.toHaveBeenCalled();+ expect(insertedFailures).toEqual([expect.objectContaining({ idempotencyKey: 'stored-key', attempts: 0 })]);+ });++ it('keeps the shed row while a replay whose queue add failed is still retrying directly', async () => {+ // The fallback runs the direct retry loop, which nothing drains at shutdown. The shed row is+ // replaced only when that loop files its own row, so a restart during it cannot lose the event.+ const queueService = await buildQueueService((key: string, def?: unknown) => {+ if (key === 'queue.enabled') return true;+ if (key === 'webhook.retryDelay') return 1;+ return def;+ });+ (hookManager.execute as jest.Mock).mockResolvedValue({ continue: true, data: {} });+ await failureRepository.insert!({ webhookId: 'wh-1', idempotencyKey: 'shed-key', attempts: 0 });+ (webhookQueue.add as jest.Mock).mockRejectedValueOnce(new Error('Redis unreachable'));+ const mockFetch = undiciFetch as jest.Mock;+ mockFetch.mockReset();+ let release!: (v: unknown) => void;+ mockFetch.mockImplementation(() => new Promise(resolve => (release = resolve)));+ const webhook = createMockWebhook({ id: 'wh-1', events: ['message.received'], retryCount: 1 });++ const replay = queueService.redeliver(webhook, 'sess-1', 'message.received', 'shed-key', { from: '[email protected]' });+ for (let i = 0; i < 20 && mockFetch.mock.calls.length === 0; i++) await new Promise(r => setImmediate(r));+ expect(mockFetch).toHaveBeenCalledTimes(1);+ expect(insertedFailures).toEqual([expect.objectContaining({ idempotencyKey: 'shed-key', attempts: 0 })]);++ release({ ok: false, status: 503, statusText: 'Unavailable' });+ await replay;+ expect(insertedFailures).toEqual([expect.objectContaining({ idempotencyKey: 'shed-key', attempts: 1 })]);+ });++ it('clears the failure rows of a replay that the fallback POST delivers after a failed queue add', async () => {+ const queueService = await buildQueueService((key: string, def?: unknown) => {+ if (key === 'queue.enabled') return true;+ if (key === 'webhook.retryDelay') return 1;+ return def;+ });+ (hookManager.execute as jest.Mock).mockResolvedValue({ continue: true, data: {} });+ await failureRepository.insert!({ webhookId: 'wh-1', idempotencyKey: 'shed-key', attempts: 0 });+ (webhookQueue.add as jest.Mock).mockRejectedValueOnce(new Error('Redis unreachable'));+ (undiciFetch as jest.Mock).mockResolvedValue({ ok: true, status: 200 });+ const webhook = createMockWebhook({ id: 'wh-1', events: ['message.received'] });++ await expect(queueService.redeliver(webhook, 'sess-1', 'message.received', 'shed-key', {})).resolves.toBe(+ 'delivered',+ );++ expect(insertedFailures).toEqual([]);+ });++ it('does not touch failure rows when queueing a first dispatch', async () => {+ const queueService = await buildQueueService((key: string, def?: unknown) => {+ if (key === 'queue.enabled') return true;+ if (key === 'webhook.retryDelay') return 5000;+ return def;+ });+ (repository.find as jest.Mock).mockResolvedValue([createMockWebhook({ events: ['message.received'] })]);+ (hookManager.execute as jest.Mock).mockResolvedValue({ continue: true, data: {} });++ await queueService.dispatch('sess-1', 'message.received', {});++ expect(webhookQueue.add).toHaveBeenCalledTimes(1);+ expect(failureRepository.delete).not.toHaveBeenCalled();+ });+it('gives sibling webhooks distinct job ids, so one event fanning out is not collapsed into one job', async () => {const queueService = await buildQueueService((key: string, def?: unknown) => {if (key === 'queue.enabled') return true;@@ -1597,5 +1790,40 @@ describe('WebhookDeliveryService', () => {expect(getWebhookDeliveryFailuresTotal()).toBe(failuresBefore + 1);mockFetch.mockReset();});++ it('clears every failure row of a delivery once a direct replay delivers it', async () => {+ // The shed row and an earlier failed replay's row both list the event as lost, and the+ // receiver now has it. A row of another delivery of the same webhook stays.+ const webhook = createMockWebhook({ id: 'wh-1', events: ['message.received'] });+ (hookManager.execute as jest.Mock).mockResolvedValue({ continue: true, data: {} });+ await failureRepository.insert!({ webhookId: 'wh-1', idempotencyKey: 'shed-key', attempts: 0 });+ await failureRepository.insert!({ webhookId: 'wh-1', idempotencyKey: 'shed-key', attempts: 3 });+ await failureRepository.insert!({ webhookId: 'wh-1', idempotencyKey: 'other-key', attempts: 3 });+ const mockFetch = undiciFetch as jest.Mock;+ mockFetch.mockReset();+ mockFetch.mockResolvedValue({ ok: true, status: 200 });++ await expect(service.redeliver(webhook, 'sess-1', 'message.received', 'shed-key', {})).resolves.toBe('delivered');++ expect(insertedFailures).toEqual([expect.objectContaining({ idempotencyKey: 'other-key' })]);+ mockFetch.mockReset();+ });++ it('keeps a delivered outcome when clearing its failure rows fails', async () => {+ const webhook = createMockWebhook({ id: 'wh-1', events: ['message.received'], retryCount: 3 });+ (hookManager.execute as jest.Mock).mockResolvedValue({ continue: true, data: {} });+ (failureRepository.delete as jest.Mock).mockRejectedValueOnce(new Error('db down'));+ const mockFetch = undiciFetch as jest.Mock;+ mockFetch.mockReset();+ mockFetch.mockResolvedValue({ ok: true, status: 200 });++ await expect(service.redeliver(webhook, 'sess-1', 'message.received', 'k', {})).resolves.toBe('delivered');++ // One POST, no retry, and no dead-letter row for an event the receiver accepted.+ expect(failureRepository.delete).toHaveBeenCalledTimes(1);+ expect(mockFetch).toHaveBeenCalledTimes(1);+ expect(failureRepository.insert).not.toHaveBeenCalled();+ mockFetch.mockReset();+ });});});diff --git a/src/modules/webhook/webhook-delivery.service.ts b/src/modules/webhook/webhook-delivery.service.tsindex 1f762e3e1..7e025e5b7 100644--- a/src/modules/webhook/webhook-delivery.service.ts+++ b/src/modules/webhook/webhook-delivery.service.ts@@ -8,7 +8,7 @@ import { setTimeout } from 'node:timers/promises';import { Webhook } from './entities/webhook.entity';import { WebhookOutboxService } from './webhook-outbox.service';import { WebhookDeliveryFailure } from './entities/webhook-delivery-failure.entity';-import { recordWebhookDeliveryFailure } from './utils/record-delivery-failure';+import { clearDeliveryFailureRows, recordWebhookDeliveryFailure } from './utils/record-delivery-failure';import {buildDeliveryHeaders,generateSignature,@@ -83,7 +83,7 @@ interface DispatchEventContext {* The webhook delivery engine: given an event occurrence, fan it out to the session's matching* webhooks — bounded by the dispatch limiter — through the BullMQ queue when enabled (with a* direct-delivery fallback when enqueue fails) or through direct inline delivery when not.- * Records terminally failed deliveries in webhook_delivery_failures. Webhook registration/CRUD+ * Records failed and unsent deliveries in webhook_delivery_failures. Webhook registration/CRUD* lives on WebhookService, which delegates dispatch here.*/const isPlainObject = (value: unknown): boolean => typeof value === 'object' && value !== null && !Array.isArray(value);@@ -456,6 +456,10 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {const { finalPayload, body, headers } = preflight;// Use queue if available, otherwise fallback to direct deliveryif (this.queueEnabled && this.webhookQueue) {+ // A replay's attempts-0 row is not cleared here. It stays until the delivery resolves: a+ // successful POST (the processor's, or the fallback's when the add fails) clears it, and a+ // terminal failure replaces it right after filing its own row. A restart or a lost job at+ // any point before that still leaves the event on record.return this.enqueueWithFallback(webhook, finalPayload, body, headers, deliveryId, idempotencyKey, ctx);}return this.deliverDirect(webhook, finalPayload, body, headers, deliveryId, ctx);@@ -769,6 +773,9 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {{ webhookId: webhook.id, deliveryId: payload.deliveryId, action: 'webhook_bookkeeping_failed' },);}+ // A delivered event must not stay listed as lost: a replay of a shed, refused or failed+ // dispatch, or a re-emitted event, reuses the key an earlier failure row was filed under.+ await clearDeliveryFailureRows(this.failureRepository, this.logger, webhook.id, payload.idempotencyKey);this.logger.debug(`Webhook delivered to ${webhook.id}`, {webhookId: webhook.id,@@ -790,6 +797,8 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {}// All direct-path retries exhausted — persist a durable failure record before giving up, mirroring// the queued processor's final-attempt path so the queue-disabled path isn't a blind spot.+ // The recorder writes this row first and only then removes an attempts-0 row of the same+ // delivery (a shed or refused dispatch), so a restart at any point keeps a record.const recorded = await recordTerminalFailure(this.failureRepository, this.logger, {webhookId: webhook.id,sessionId: payload.sessionId,