Skip to content

Free, Open Source, Self-Hosted WhatsApp API Gateway

fix(webhook): keep failure rows in step with replayed deliveries

Yudhi Armyndharis0f87900a1b1f
diff --git a/docs/06-api-specification.md b/docs/06-api-specification.md
index bd05093b5..cf7148fa2 100644
--- a/docs/06-api-specification.md
+++ b/docs/06-api-specification.md
@@ -4598,6 +4598,8 @@ List webhook deliveries that exhausted every retry, most recent first. This is t
Bare array of `WebhookDeliveryFailure` rows, ordered by `createdAt` descending. `lastStatusCode` is `null` when the failure was a network/timeout/SSRF error rather than a non-2xx response; `idempotencyKey`/`deliveryId` let you correlate the lost event with your own receiver logs.
+A delivery shed because the dispatch queue was full, or refused during shutdown, is recorded here with `attempts: 0` while its outbox row stays pending, so the outbox sweep replays it. Any delivery the receiver accepts with a `2xx`, inline or from a queued job, removes the rows filed under its idempotency key, so a replay that succeeds leaves none. A replay that exhausts its retries files a row carrying the real error and attempt count and only then removes the `attempts: 0` row, so the event stays on record throughout, across a restart or a lost queue job too. A replay that fails before sending (a payload over the size cap after the `webhook:before` hooks, or one that cannot be serialized) gives the `attempts: 0` row that reason instead. A replay that keeps failing files no second row. `openwa_webhook_delivery_failures_total` counts the original shed, and counts once more if a replay also exhausts its retries.
+
**Errors:** `401` missing/invalid API key · `403` key role below ADMIN
#### POST /api/sessions/:sessionId/webhooks
@@ -7040,7 +7042,7 @@ Webhook delivery is **at-least-once**. A consumer can legitimately receive the s
- The underlying WhatsApp engine can re-fire an event for a single message.
- A failed delivery (non-2xx response, timeout, or network error) is retried.
-**Crash boundary.** Every delivery is recorded before it is attempted, and the record is retired once something durable owns it: the queue job in queued mode, the completed send in direct mode. A hard crash (SIGKILL, OOM) therefore leaves the record behind, and a bounded sweep (`WEBHOOK_RECONCILE_INTERVAL_MS`, default 60s) replays whatever is still stranded, reusing the stored `X-OpenWA-Idempotency-Key` so the retry stays deduplicable at your receiver. A delivery that keeps failing exhausts `WEBHOOK_RECONCILE_MAX_ATTEMPTS` and goes terminal rather than replaying forever. One window remains open: a crash between persisting the message and writing that record loses the delivery, because the two are not yet one transaction. The failure table records exhausted retries, plus over-budget, dispatch-capacity-exceeded, and shutdown-rejected deliveries with attempts 0. Read the last two as a report rather than a verdict: a delivery the dispatcher shed for capacity or refused during the drain was rejected before its POST, so it keeps its record and is replayed by the same sweep, and a row there can belong to an event that was later delivered. A node's sweep also leaves alone a delivery that the same node is still waiting to dispatch or retrying, however long that takes; with several nodes on one database, another node can still replay it, and the stored idempotency key keeps that duplicate deduplicable. Enabling the queue (`QUEUE_ENABLED=true`, needs Redis) makes the dispatch durable from the enqueue onward. In both modes, a graceful shutdown drains in-flight deliveries first: the queued path waits for each worker's current job, and the direct path waits up to `WEBHOOK_SHUTDOWN_DRAIN_MS` (default 5s; raise it to at least `WEBHOOK_TIMEOUT`, default 10s, if a slow receiver must finish).
+**Crash boundary.** Every delivery is recorded before it is attempted, and the record is retired once something durable owns it: the queue job in queued mode, the completed send in direct mode. A hard crash (SIGKILL, OOM) therefore leaves the record behind, and a bounded sweep (`WEBHOOK_RECONCILE_INTERVAL_MS`, default 60s) replays whatever is still stranded, reusing the stored `X-OpenWA-Idempotency-Key` so the retry stays deduplicable at your receiver. A delivery that keeps failing exhausts `WEBHOOK_RECONCILE_MAX_ATTEMPTS` and goes terminal rather than replaying forever. One window remains open: a crash between persisting the message and writing that record loses the delivery, because the two are not yet one transaction. The failure table records exhausted retries, plus over-budget, dispatch-capacity-exceeded, and shutdown-rejected deliveries with attempts 0. Read the last two as a report rather than a verdict: a delivery the dispatcher shed for capacity or refused during the drain was rejected before its POST, so it keeps its record until the same sweep replays it; a replay whose retries run out replaces that record with its own row, one that fails before sending gives it that reason, and one that delivers removes it. A node's sweep also leaves alone a delivery that the same node is still waiting to dispatch or retrying, however long that takes; with several nodes on one database, another node can still replay it, and the stored idempotency key keeps that duplicate deduplicable. Enabling the queue (`QUEUE_ENABLED=true`, needs Redis) makes the dispatch durable from the enqueue onward. In both modes, a graceful shutdown drains in-flight deliveries first: the queued path waits for each worker's current job, and the direct path waits up to `WEBHOOK_SHUTDOWN_DRAIN_MS` (default 5s; raise it to at least `WEBHOOK_TIMEOUT`, default 10s, if a slow receiver must finish).
**Design your handler to be idempotent**, keyed on the `X-OpenWA-Idempotency-Key` header (see below). As a server-side safety net, OpenWA de-duplicates inbound `message.received` before dispatch (a re-fired event for an already-persisted message is dropped), so one webhook normally sees each inbound message once — but this is best-effort defense-in-depth and does not remove the need for consumer-side idempotency.
diff --git a/src/modules/queue/processors/webhook.processor.spec.ts b/src/modules/queue/processors/webhook.processor.spec.ts
index 277845803..a200bcc63 100644
--- a/src/modules/queue/processors/webhook.processor.spec.ts
+++ b/src/modules/queue/processors/webhook.processor.spec.ts
@@ -1,5 +1,5 @@
import { Job } from 'bullmq';
-import { Repository } from 'typeorm';
+import { FindOperator, Repository } from 'typeorm';
import { ConfigService } from '@nestjs/config';
import { WebhookProcessor } from './webhook.processor';
import { Webhook } from '../../webhook/entities/webhook.entity';
@@ -24,7 +24,8 @@ jest.mock('undici', () => {
describe('WebhookProcessor', () => {
let processor: WebhookProcessor;
let repo: { update: jest.Mock; findOne: jest.Mock };
- let failureRepo: { insert: jest.Mock; count: jest.Mock };
+ let failureRepo: { insert: jest.Mock; count: jest.Mock; delete: jest.Mock };
+ let failureRows: Array<{ webhookId?: string; idempotencyKey?: string | null; attempts?: number }>;
let hookManager: { execute: jest.Mock };
let configService: { get: jest.Mock };
let mockFetch: jest.Mock;
@@ -60,18 +61,38 @@ describe('WebhookProcessor', () => {
};
// Stateful like the real table: the recorder counts existing rows for the delivery before it
// inserts, so a constant would leave that guard unexercised here and let a duplicated row pass.
- const insertedFailures: Array<{ webhookId?: string; idempotencyKey?: string | null }> = [];
+ failureRows = [];
+ const insertedFailures = failureRows;
failureRepo = {
insert: jest.fn().mockImplementation((rowToInsert: { webhookId?: string; idempotencyKey?: string | null }) => {
insertedFailures.push(rowToInsert);
return Promise.resolve({});
}),
+ delete: jest
+ .fn()
+ .mockImplementation((where: { webhookId?: string; idempotencyKey?: string; attempts?: number }) => {
+ const keep = failureRows.filter(
+ r =>
+ r.webhookId !== where.webhookId ||
+ r.idempotencyKey !== where.idempotencyKey ||
+ (where.attempts !== undefined && r.attempts !== where.attempts),
+ );
+ const affected = failureRows.length - keep.length;
+ failureRows.splice(0, failureRows.length, ...keep);
+ return Promise.resolve({ affected });
+ }),
count: jest
.fn()
- .mockImplementation((opts: { where: { webhookId?: string; idempotencyKey?: string } }) =>
+ .mockImplementation((opts: { where: { webhookId?: string; idempotencyKey?: string; attempts?: unknown } }) =>
Promise.resolve(
insertedFailures.filter(
- r => r.webhookId === opts.where.webhookId && r.idempotencyKey === opts.where.idempotencyKey,
+ r =>
+ r.webhookId === opts.where.webhookId &&
+ r.idempotencyKey === opts.where.idempotencyKey &&
+ (opts.where.attempts === undefined ||
+ (opts.where.attempts instanceof FindOperator && opts.where.attempts.type === 'moreThan'
+ ? ((r as { attempts?: number }).attempts ?? 0) > (opts.where.attempts.value as number)
+ : (r as { attempts?: number }).attempts === opts.where.attempts)),
).length,
),
),
@@ -164,6 +185,54 @@ describe('WebhookProcessor', () => {
);
});
+ it("replaces a replayed delivery's attempts-0 shed row with its own final-attempt row", async () => {
+ // The shed row's dedup entry would otherwise suppress this row and keep only the capacity reason.
+ failureRows.push({ webhookId: 'wh-1', idempotencyKey: 'k', attempts: 0 });
+ mockFetch.mockResolvedValue({ ok: false, status: 503, statusText: 'Service Unavailable' });
+
+ await expect(processor.process(makeJob({ maxRetries: 3 }, 2))).rejects.toThrow();
+
+ expect(failureRepo.delete).toHaveBeenCalledWith({ webhookId: 'wh-1', idempotencyKey: 'k', attempts: 0 });
+ expect(failureRows).toEqual([expect.objectContaining({ idempotencyKey: 'k', attempts: 3, lastStatusCode: 503 })]);
+ });
+
+ it('keeps the delivery error and both rows when the shed-row delete fails', async () => {
+ // The final-attempt row is written before the shed row is removed, so a failed delete leaves
+ // the event recorded twice rather than not at all. It is logged, not thrown: process() still
+ // rejects with the HTTP error.
+ failureRows.push({ webhookId: 'wh-1', idempotencyKey: 'k', attempts: 0 });
+ failureRepo.delete.mockRejectedValueOnce(new Error('db down'));
+ mockFetch.mockResolvedValue({ ok: false, status: 503, statusText: 'Service Unavailable' });
+
+ await expect(processor.process(makeJob({ maxRetries: 3 }, 2))).rejects.toThrow('HTTP 503');
+
+ expect(failureRows).toEqual([
+ expect.objectContaining({ idempotencyKey: 'k', attempts: 0 }),
+ expect.objectContaining({ idempotencyKey: 'k', attempts: 3, lastStatusCode: 503 }),
+ ]);
+ });
+
+ it('deletes nothing for a job whose payload carries no idempotency key', async () => {
+ mockFetch.mockResolvedValue({ ok: false, status: 503, statusText: 'Service Unavailable' });
+ const job = makeJob({ maxRetries: 3 }, 2);
+ (job.data.payload as { idempotencyKey?: string }).idempotencyKey = undefined;
+
+ await expect(processor.process(job)).rejects.toThrow();
+
+ expect(failureRepo.delete).not.toHaveBeenCalled();
+ expect(failureRepo.insert).toHaveBeenCalledTimes(1);
+ });
+
+ it('leaves the shed row alone on an attempt that is not final', async () => {
+ failureRows.push({ webhookId: 'wh-1', idempotencyKey: 'k', attempts: 0 });
+ mockFetch.mockResolvedValue({ ok: false, status: 500, statusText: 'Server Error' });
+
+ await expect(processor.process(makeJob({ maxRetries: 3 }, 0))).rejects.toThrow();
+
+ expect(failureRepo.delete).not.toHaveBeenCalled();
+ expect(failureRows).toEqual([expect.objectContaining({ idempotencyKey: 'k', attempts: 0 })]);
+ });
+
it('does NOT persist a delivery-failure record before the final attempt', async () => {
mockFetch.mockResolvedValue({ ok: false, status: 500, statusText: 'Server Error' });
@@ -203,6 +272,31 @@ describe('WebhookProcessor', () => {
expect(inserted.lastError).not.toMatch(/169\.254\.169\.254/);
});
+ it('clears every failure row of the delivery once the job delivers it', async () => {
+ // An earlier inline replay failed (attempts > 0) after a shed (attempts 0); this queued replay
+ // then delivered. Both rows would list a delivered event as lost. Another delivery's row stays.
+ failureRows.push(
+ { webhookId: 'wh-1', idempotencyKey: 'k', attempts: 0 },
+ { webhookId: 'wh-1', idempotencyKey: 'k', attempts: 3 },
+ { webhookId: 'wh-1', idempotencyKey: 'other', attempts: 3 },
+ );
+ mockFetch.mockResolvedValue({ ok: true, status: 200 });
+
+ await expect(processor.process(makeJob())).resolves.toMatchObject({ success: true });
+
+ expect(failureRows).toEqual([expect.objectContaining({ idempotencyKey: 'other' })]);
+ });
+
+ it('keeps the success outcome when clearing the failure rows fails', async () => {
+ failureRepo.delete.mockRejectedValueOnce(new Error('db down'));
+ mockFetch.mockResolvedValue({ ok: true, status: 200 });
+
+ await expect(processor.process(makeJob({ maxRetries: 3 }, 2))).resolves.toMatchObject({ success: true });
+
+ expect(failureRepo.delete).toHaveBeenCalledTimes(1);
+ expect(failureRepo.insert).not.toHaveBeenCalled();
+ });
+
it('keeps the success outcome when post-delivery bookkeeping fails after a 2xx (no retry, no DLQ row)', async () => {
mockFetch.mockResolvedValue({ ok: true, status: 200 });
repo.update.mockRejectedValue(new Error('db down'));
@@ -331,6 +425,28 @@ describe('WebhookProcessor', () => {
expect(hookManager.execute).not.toHaveBeenCalled();
});
+ it("replaces a replayed delivery's attempts-0 shed row with the stall row", async () => {
+ failureRows.push({ webhookId: 'wh-1', idempotencyKey: 'k', attempts: 0 });
+
+ await processor.onWorkerFailed(makeJob({}, 1), new Error('job stalled more than allowable limit'));
+
+ expect(failureRows).toEqual([
+ expect.objectContaining({
+ idempotencyKey: 'k',
+ attempts: 1,
+ lastError: 'job stalled more than allowable limit',
+ }),
+ ]);
+ });
+
+ it('still files the stall row when the shed-row delete fails', async () => {
+ failureRepo.delete.mockRejectedValueOnce(new Error('db down'));
+
+ await processor.onWorkerFailed(makeJob({}, 1), new Error('job stalled more than allowable limit'));
+
+ expect(failureRepo.insert).toHaveBeenCalledWith(expect.objectContaining({ idempotencyKey: 'k', attempts: 1 }));
+ });
+
it('still records the stall against the enqueue-time URL when the webhook row cannot be read', async () => {
repo.findOne.mockRejectedValue(new Error('db down'));
diff --git a/src/modules/queue/processors/webhook.processor.ts b/src/modules/queue/processors/webhook.processor.ts
index 96d07d319..bff444f94 100644
--- a/src/modules/queue/processors/webhook.processor.ts
+++ b/src/modules/queue/processors/webhook.processor.ts
@@ -9,7 +9,11 @@ import { workerConnectionOptions, webhookWorkerConcurrency } from '../redis-conn
import { WebhookJobData, WebhookPayload } from '../../webhook/webhook.service';
import { Webhook } from '../../webhook/entities/webhook.entity';
import { WebhookDeliveryFailure } from '../../webhook/entities/webhook-delivery-failure.entity';
-import { recordWebhookDeliveryFailure, statusCodeFromError } from '../../webhook/utils/record-delivery-failure';
+import {
+ clearDeliveryFailureRows,
+ recordWebhookDeliveryFailure,
+ statusCodeFromError,
+} from '../../webhook/utils/record-delivery-failure';
import { buildDeliveryHeaders, postWebhookPayload } from '../../webhook/utils/deliver-once';
import { HookManager } from '../../../core/hooks';
import { redactSsrfError } from '../../../common/security/ssrf-guard';
@@ -186,6 +190,11 @@ export class WebhookProcessor extends WorkerHost {
);
}
+ // A delivered event must not stay listed as lost. A failure row exists for it only when an
+ // earlier dispatch of the same delivery was shed, refused or failed before this job ran. An
+ // indexed delete that usually matches nothing.
+ await clearDeliveryFailureRows(this.failureRepository, this.logger, webhookId, payload.idempotencyKey);
+
// Execute hook after successful delivery
await this.hookManager.execute(
'webhook:delivered',
diff --git a/src/modules/webhook/utils/record-delivery-failure.spec.ts b/src/modules/webhook/utils/record-delivery-failure.spec.ts
index 005d3ddd3..b4f901f73 100644
--- a/src/modules/webhook/utils/record-delivery-failure.spec.ts
+++ b/src/modules/webhook/utils/record-delivery-failure.spec.ts
@@ -1,6 +1,6 @@
-import { Repository } from 'typeorm';
+import { MoreThan, Repository } from 'typeorm';
import { WebhookDeliveryFailure } from '../entities/webhook-delivery-failure.entity';
-import { recordWebhookDeliveryFailure, statusCodeFromError } from './record-delivery-failure';
+import { clearDeliveryFailureRows, recordWebhookDeliveryFailure, statusCodeFromError } from './record-delivery-failure';
describe('statusCodeFromError', () => {
it('parses the status from an "HTTP <code>: ..." message', () => {
@@ -28,7 +28,12 @@ describe('recordWebhookDeliveryFailure', () => {
};
const repoWith = (insert: jest.Mock, existing = 0): Repository<WebhookDeliveryFailure> =>
- ({ insert, count: jest.fn().mockResolvedValue(existing) }) as unknown as Repository<WebhookDeliveryFailure>;
+ ({
+ insert,
+ count: jest.fn().mockResolvedValue(existing),
+ delete: jest.fn().mockResolvedValue({}),
+ update: jest.fn().mockResolvedValue({}),
+ }) as unknown as Repository<WebhookDeliveryFailure>;
it('inserts the failure record, defaulting a missing lastStatusCode to null', async () => {
const insert = jest.fn().mockResolvedValue({});
@@ -54,6 +59,56 @@ describe('recordWebhookDeliveryFailure', () => {
expect(logger.error).not.toHaveBeenCalled();
});
+ it('gives the attempts-0 row the reason of an unsent delivery recorded again, without a second row', async () => {
+ // A shed delivery whose replay then fails before sending: the operator has to act on the new
+ // reason, not on the shed. Only attempts-0 rows take it, and the metric does not count it again.
+ const insert = jest.fn().mockResolvedValue({});
+ const update = jest.fn().mockResolvedValue({ affected: 1 });
+ const repo = {
+ insert,
+ update,
+ count: jest.fn().mockResolvedValue(1),
+ } as unknown as Repository<WebhookDeliveryFailure>;
+ const logger = { error: jest.fn() };
+
+ await expect(
+ recordWebhookDeliveryFailure(repo, logger, { ...input, attempts: 0, lastStatusCode: null, lastError: 'too big' }),
+ ).resolves.toBe(false);
+
+ expect(insert).not.toHaveBeenCalled();
+ expect(update).toHaveBeenCalledWith(
+ { webhookId: 'wh-1', idempotencyKey: 'k', attempts: 0 },
+ { lastError: 'too big', url: 'https://r.example/h', deliveryId: 'd' },
+ );
+ });
+
+ it('leaves the recorded row alone for a terminal failure recorded again', async () => {
+ const update = jest.fn();
+ const repo = {
+ insert: jest.fn(),
+ update,
+ count: jest.fn().mockResolvedValue(1),
+ } as unknown as Repository<WebhookDeliveryFailure>;
+
+ await expect(recordWebhookDeliveryFailure(repo, { error: jest.fn() }, input)).resolves.toBe(false);
+
+ expect(update).not.toHaveBeenCalled();
+ });
+
+ it('still reports an unsent duplicate as not recorded when refreshing its reason fails', async () => {
+ // A true here would bump the failure metric for a loss that is already counted.
+ const repo = {
+ insert: jest.fn(),
+ update: jest.fn().mockRejectedValue(new Error('db down')),
+ count: jest.fn().mockResolvedValue(1),
+ } as unknown as Repository<WebhookDeliveryFailure>;
+ const logger = { error: jest.fn() };
+
+ await expect(recordWebhookDeliveryFailure(repo, logger, { ...input, attempts: 0 })).resolves.toBe(false);
+
+ expect(logger.error).toHaveBeenCalledTimes(1);
+ });
+
it('still records a failure that carries no idempotency key', async () => {
// Control: without a key there is no identity to dedupe on, so the guard must not swallow the
// row. Otherwise "skips a duplicate" would be satisfied by a helper that stopped inserting.
@@ -67,6 +122,58 @@ describe('recordWebhookDeliveryFailure', () => {
expect(insert).toHaveBeenCalled();
});
+ it('dedupes a terminal row only against terminal rows, then removes the attempts-0 row it replaces', async () => {
+ // A shed or shutdown-refused dispatch left an attempts-0 row. Counted as a duplicate, it would
+ // keep the capacity reason as the only record and drop the receiver's actual answer.
+ const calls: string[] = [];
+ const insert = jest.fn().mockImplementation(() => (calls.push('insert'), Promise.resolve({})));
+ const count = jest.fn().mockResolvedValue(0);
+ const del = jest.fn().mockImplementation(() => (calls.push('delete'), Promise.resolve({})));
+ const repo = { insert, count, delete: del } as unknown as Repository<WebhookDeliveryFailure>;
+ const logger = { error: jest.fn() };
+
+ await expect(recordWebhookDeliveryFailure(repo, logger, input)).resolves.toBe(true);
+
+ expect(count).toHaveBeenCalledWith({
+ where: { webhookId: 'wh-1', idempotencyKey: 'k', attempts: MoreThan(0) },
+ });
+ expect(del).toHaveBeenCalledWith({ webhookId: 'wh-1', idempotencyKey: 'k', attempts: 0 });
+ // Written first, so a crash in between leaves two rows for the event rather than none.
+ expect(calls).toEqual(['insert', 'delete']);
+ });
+
+ it('dedupes an attempts-0 row against every row of the delivery and removes nothing', async () => {
+ const insert = jest.fn().mockResolvedValue({});
+ const count = jest.fn().mockResolvedValue(0);
+ const del = jest.fn().mockResolvedValue({});
+ const repo = { insert, count, delete: del } as unknown as Repository<WebhookDeliveryFailure>;
+ const logger = { error: jest.fn() };
+
+ await expect(recordWebhookDeliveryFailure(repo, logger, { ...input, attempts: 0 })).resolves.toBe(true);
+
+ expect(count).toHaveBeenCalledWith({ where: { webhookId: 'wh-1', idempotencyKey: 'k' } });
+ expect(del).not.toHaveBeenCalled();
+ });
+
+ it('keeps the terminal row and the result when removing the attempts-0 row fails', async () => {
+ const insert = jest.fn().mockResolvedValue({});
+ const repo = {
+ insert,
+ count: jest.fn().mockResolvedValue(0),
+ delete: jest.fn().mockRejectedValue(new Error('db down')),
+ } as unknown as Repository<WebhookDeliveryFailure>;
+ const logger = { error: jest.fn() };
+
+ await expect(recordWebhookDeliveryFailure(repo, logger, input)).resolves.toBe(true);
+
+ expect(insert).toHaveBeenCalledTimes(1);
+ expect(logger.error).toHaveBeenCalledWith(
+ expect.any(String),
+ 'db down',
+ expect.objectContaining({ action: 'webhook_failure_clear_error' }),
+ );
+ });
+
it('swallows a repository error so a logging hiccup cannot re-poison the delivery', async () => {
const insert = jest.fn().mockRejectedValue(new Error('db down'));
const logger = { error: jest.fn() };
@@ -77,3 +184,40 @@ describe('recordWebhookDeliveryFailure', () => {
expect(logger.error).toHaveBeenCalled();
});
});
+
+describe('clearDeliveryFailureRows', () => {
+ const repoWithDelete = (del: jest.Mock): Repository<WebhookDeliveryFailure> =>
+ ({ delete: del }) as unknown as Repository<WebhookDeliveryFailure>;
+
+ it('deletes every row of one delivery, or only its attempts-0 rows', async () => {
+ const del = jest.fn().mockResolvedValue({});
+ const logger = { error: jest.fn() };
+
+ await clearDeliveryFailureRows(repoWithDelete(del), logger, 'wh-1', 'k');
+ await clearDeliveryFailureRows(repoWithDelete(del), logger, 'wh-1', 'k', true);
+
+ expect(del.mock.calls).toEqual([
+ [{ webhookId: 'wh-1', idempotencyKey: 'k' }],
+ [{ webhookId: 'wh-1', idempotencyKey: 'k', attempts: 0 }],
+ ]);
+ });
+
+ it('deletes nothing without an idempotency key', async () => {
+ // Matching on webhookId alone would erase the rows of every other lost event of that webhook.
+ const del = jest.fn().mockResolvedValue({});
+
+ await clearDeliveryFailureRows(repoWithDelete(del), { error: jest.fn() }, 'wh-1', undefined);
+
+ expect(del).not.toHaveBeenCalled();
+ });
+
+ it('logs a failed delete instead of throwing', async () => {
+ const logger = { error: jest.fn() };
+
+ await expect(
+ clearDeliveryFailureRows(repoWithDelete(jest.fn().mockRejectedValue(new Error('db down'))), logger, 'wh-1', 'k'),
+ ).resolves.toBeUndefined();
+
+ expect(logger.error).toHaveBeenCalledTimes(1);
+ });
+});
diff --git a/src/modules/webhook/utils/record-delivery-failure.ts b/src/modules/webhook/utils/record-delivery-failure.ts
index ee9a1f921..b7556306a 100644
--- a/src/modules/webhook/utils/record-delivery-failure.ts
+++ b/src/modules/webhook/utils/record-delivery-failure.ts
@@ -1,4 +1,4 @@
-import { Repository } from 'typeorm';
+import { MoreThan, Repository } from 'typeorm';
import { WebhookDeliveryFailure } from '../entities/webhook-delivery-failure.entity';
export interface WebhookDeliveryFailureInput {
@@ -24,6 +24,34 @@ export function statusCodeFromError(message: string): number | null {
return m ? Number(m[1]) : null;
}
+/**
+ * Remove the dead-letter rows of one delivery. Every successful POST calls this for its key, so a
+ * delivered event is never listed as lost, whichever earlier dispatch of it was shed, refused or
+ * failed. With `onlyUnattempted` it removes only the attempts-0 rows (shed, refused, preflight),
+ * which the recorder does right after writing a terminal row, so the event always keeps a record.
+ * Best-effort: a failed delete is logged and never fails the delivery that triggered it.
+ */
+export async function clearDeliveryFailureRows(
+ repo: Repository<WebhookDeliveryFailure>,
+ logger: ErrorLogger,
+ webhookId: string,
+ idempotencyKey: string | undefined,
+ onlyUnattempted = false,
+): Promise<void> {
+ // Without a key there is no delivery identity to match, and deleting on webhookId alone would
+ // erase the rows of every other lost event of that webhook.
+ if (!idempotencyKey) return;
+ try {
+ await repo.delete(onlyUnattempted ? { webhookId, idempotencyKey, attempts: 0 } : { webhookId, idempotencyKey });
+ } catch (err) {
+ logger.error('Failed to clear webhook delivery-failure rows', err instanceof Error ? err.message : String(err), {
+ webhookId,
+ idempotencyKey,
+ action: 'webhook_failure_clear_error',
+ });
+ }
+}
+
/**
* Append a durable record of a webhook delivery that exhausted all retries. Called from BOTH terminal
* paths — the BullMQ processor's final attempt and the direct-fallback's last attempt. Wrapped in its
@@ -40,15 +68,50 @@ export async function recordWebhookDeliveryFailure(
// up to its budget and every failed replay arrives here, so without this guard a single lost
// event is reported to the operator as several and counted that many times in the failure
// metric. An absent idempotencyKey carries no identity to dedupe on, so it is always appended.
+ // A terminal row (attempts > 0) is deduplicated only against other terminal rows: an attempts-0
+ // row (a shed or shutdown-refused dispatch) is replaced by it below instead of suppressing it.
+ const terminal = input.attempts > 0;
if (input.idempotencyKey) {
const existing = await repo.count({
- where: { webhookId: input.webhookId, idempotencyKey: input.idempotencyKey },
+ where: terminal
+ ? { webhookId: input.webhookId, idempotencyKey: input.idempotencyKey, attempts: MoreThan(0) }
+ : { webhookId: input.webhookId, idempotencyKey: input.idempotencyKey },
});
if (existing > 0) {
+ if (!terminal) {
+ // An unsent delivery recorded again: typically a replay of a shed or refused dispatch that
+ // then failed before sending (a payload over the size cap, or one that cannot be
+ // serialized). Its reason is the one the operator has to act on now, so the attempts-0 row
+ // takes it. A terminal row of the same delivery is left as it is.
+ await repo
+ .update(
+ { webhookId: input.webhookId, idempotencyKey: input.idempotencyKey, attempts: 0 },
+ {
+ lastError: input.lastError,
+ url: input.url,
+ ...(input.deliveryId ? { deliveryId: input.deliveryId } : {}),
+ },
+ )
+ .catch((err: unknown) =>
+ logger.error(
+ 'Failed to refresh the reason of an unsent webhook delivery',
+ err instanceof Error ? err.message : String(err),
+ {
+ webhookId: input.webhookId,
+ idempotencyKey: input.idempotencyKey,
+ action: 'webhook_failure_record_error',
+ },
+ ),
+ );
+ }
return false;
}
}
await repo.insert({ ...input, lastStatusCode: input.lastStatusCode ?? null });
+ if (terminal) {
+ // Only after the insert, so a crash in between leaves both rows, never none.
+ await clearDeliveryFailureRows(repo, logger, input.webhookId, input.idempotencyKey, true);
+ }
return true;
} catch (err) {
logger.error(
diff --git a/src/modules/webhook/webhook-delivery.service.spec.ts b/src/modules/webhook/webhook-delivery.service.spec.ts
index 1f1c6bf2a..a1ad31aec 100644
--- a/src/modules/webhook/webhook-delivery.service.spec.ts
+++ b/src/modules/webhook/webhook-delivery.service.spec.ts
@@ -13,7 +13,7 @@ jest.mock('undici', () => {
import { Test, TestingModule } from '@nestjs/testing';
import { getRepositoryToken } from '@nestjs/typeorm';
import { getQueueToken } from '@nestjs/bullmq';
-import { Repository } from 'typeorm';
+import { FindOperator, Repository } from 'typeorm';
import { ConfigService } from '@nestjs/config';
import * as crypto from 'crypto';
import { fetch as undiciFetch } from 'undici';
@@ -49,6 +49,16 @@ function createMockWebhook(overrides: Partial<Webhook> = {}): Webhook {
};
}
+// The recorder counts terminal rows with `attempts: MoreThan(0)`; honour that operator like the
+// real table so the fake cannot count an attempts-0 row as a terminal duplicate.
+function attemptsMatch(actual: number | undefined, filter: unknown): boolean {
+ if (filter === undefined) return true;
+ if (filter instanceof FindOperator) {
+ return filter.type === 'moreThan' ? (actual ?? 0) > (filter.value as number) : false;
+ }
+ return actual === filter;
+}
+
describe('WebhookDeliveryService', () => {
let service: WebhookDeliveryService;
let repository: jest.Mocked<Partial<Repository<Webhook>>>;
@@ -58,6 +68,7 @@ describe('WebhookDeliveryService', () => {
let webhookQueue: jest.Mocked<Record<string, jest.Mock>>;
let lidStore: { getCached: jest.Mock; resolveLid: jest.Mock };
let outboxService: { open: jest.Mock; close: jest.Mock };
+ let insertedFailures: Array<Partial<WebhookDeliveryFailure>>;
beforeEach(async () => {
repository = {
@@ -68,23 +79,44 @@ describe('WebhookDeliveryService', () => {
// Backed by the rows it accepts, not a constant: the dedupe guard reads count() before every
// insert, and a jest.fn() returning undefined would make the guard silently inert here while
// still passing every assertion.
- const insertedFailures: Array<{ webhookId?: string; idempotencyKey?: string | null }> = [];
+ insertedFailures = [];
failureRepository = {
- insert: jest.fn().mockImplementation((rowToInsert: { webhookId?: string; idempotencyKey?: string | null }) => {
+ insert: jest.fn().mockImplementation((rowToInsert: Partial<WebhookDeliveryFailure>) => {
insertedFailures.push(rowToInsert);
return Promise.resolve({});
}),
count: jest
.fn()
- .mockImplementation((opts: { where: { webhookId?: string; idempotencyKey?: string } }) =>
+ .mockImplementation((opts: { where: { webhookId?: string; idempotencyKey?: string; attempts?: unknown } }) =>
Promise.resolve(
insertedFailures.filter(
- r => r.webhookId === opts.where.webhookId && r.idempotencyKey === opts.where.idempotencyKey,
+ r =>
+ r.webhookId === opts.where.webhookId &&
+ r.idempotencyKey === opts.where.idempotencyKey &&
+ attemptsMatch(r.attempts, opts.where.attempts),
).length,
),
),
find: jest.fn().mockResolvedValue([]),
- delete: jest.fn().mockResolvedValue({ affected: 0 }),
+ // Also backed by the rows, so a delete that runs too late (after the dedupe guard) shows up.
+ delete: jest.fn().mockImplementation((where: Partial<WebhookDeliveryFailure>) => {
+ const before = insertedFailures.length;
+ insertedFailures = insertedFailures.filter(r =>
+ Object.entries(where).some(([k, v]) => r[k as keyof WebhookDeliveryFailure] !== v),
+ );
+ return Promise.resolve({ affected: before - insertedFailures.length });
+ }),
+ update: jest
+ .fn()
+ .mockImplementation((where: Partial<WebhookDeliveryFailure>, set: Partial<WebhookDeliveryFailure>) => {
+ let affected = 0;
+ insertedFailures = insertedFailures.map(r => {
+ if (Object.entries(where).some(([k, v]) => r[k as keyof WebhookDeliveryFailure] !== v)) return r;
+ affected++;
+ return { ...r, ...set };
+ });
+ return Promise.resolve({ affected });
+ }),
};
configService = {
@@ -781,6 +813,86 @@ describe('WebhookDeliveryService', () => {
expect(getWebhookDeliveryFailuresTotal()).toBe(failuresBefore + 2);
});
+ it('replaces the attempts-0 shed row with the real error when a direct replay fails', async () => {
+ // The shed row's dedup entry used to block the replay's own row, so the only record of a
+ // lost event said "queue full" with no attempts, never the receiver's actual answer.
+ const webhook = createMockWebhook({ id: 'wh-1', events: ['message.received'], retryCount: 1 });
+ (hookManager.execute as jest.Mock).mockResolvedValue({ continue: true, data: {} });
+ await failureRepository.insert!({
+ webhookId: 'wh-1',
+ idempotencyKey: 'shed-key',
+ attempts: 0,
+ lastError: 'ConcurrencyLimiter queue full',
+ });
+ mockFetch.mockReset();
+ mockFetch.mockResolvedValue({ ok: false, status: 500, statusText: 'Server Error' });
+
+ await expect(
+ service.redeliver(webhook, 'sess-1', 'message.received', 'shed-key', { from: '[email protected]' }),
+ ).resolves.toBe('failed');
+
+ expect(insertedFailures).toEqual([
+ expect.objectContaining({
+ webhookId: 'wh-1',
+ idempotencyKey: 'shed-key',
+ attempts: 1,
+ lastStatusCode: 500,
+ lastError: 'HTTP 500: Server Error',
+ }),
+ ]);
+ });
+
+ it('keeps the shed row while a direct replay is still in flight, so a restart then cannot lose it', async () => {
+ // Replays are not drained at shutdown. Deleting the shed row before the POST left an event
+ // whose final replay was cut short with no failure record at all.
+ const webhook = createMockWebhook({ id: 'wh-1', events: ['message.received'], retryCount: 1 });
+ (hookManager.execute as jest.Mock).mockResolvedValue({ continue: true, data: {} });
+ await failureRepository.insert!({ webhookId: 'wh-1', idempotencyKey: 'shed-key', attempts: 0 });
+ mockFetch.mockReset();
+ let release!: (v: unknown) => void;
+ mockFetch.mockImplementation(() => new Promise(resolve => (release = resolve)));
+
+ const replay = service.redeliver(webhook, 'sess-1', 'message.received', 'shed-key', { from: '[email protected]' });
+ for (let i = 0; i < 20 && mockFetch.mock.calls.length === 0; i++) await new Promise(r => setImmediate(r));
+ expect(mockFetch).toHaveBeenCalledTimes(1);
+ expect(insertedFailures).toEqual([expect.objectContaining({ idempotencyKey: 'shed-key', attempts: 0 })]);
+
+ release({ ok: false, status: 503, statusText: 'Unavailable' });
+ await expect(replay).resolves.toBe('failed');
+ expect(insertedFailures).toEqual([expect.objectContaining({ idempotencyKey: 'shed-key', attempts: 1 })]);
+ });
+
+ it('keeps one row for a replay that preflight rejects, carrying the preflight reason', async () => {
+ // Every sweep replays it again, so a second row would list one lost event several times. The
+ // shed reason is stale, though: the operator has to fix the payload size, not the capacity.
+ (configService.get as jest.Mock).mockImplementation(<T>(key: string, def?: T): T | boolean | number => {
+ if (key === 'queue.enabled') return false;
+ if (key === 'webhook.maxPayloadBytes') return 1024;
+ return def as T;
+ });
+ const webhook = createMockWebhook({ id: 'wh-1', events: ['message.received'], retryCount: 1 });
+ (hookManager.execute as jest.Mock).mockResolvedValue({ continue: true, data: {} });
+ await failureRepository.insert!({
+ webhookId: 'wh-1',
+ idempotencyKey: 'shed-key',
+ attempts: 0,
+ lastError: 'ConcurrencyLimiter queue full',
+ });
+ mockFetch.mockReset();
+ const failuresBefore = getWebhookDeliveryFailuresTotal();
+
+ await expect(
+ service.redeliver(webhook, 'sess-1', 'message.received', 'shed-key', { big: 'x'.repeat(64 * 1024) }),
+ ).resolves.toBe('failed');
+
+ expect(mockFetch).not.toHaveBeenCalled();
+ expect(failureRepository.delete).not.toHaveBeenCalled();
+ expect(failureRepository.insert).toHaveBeenCalledTimes(1);
+ expect(insertedFailures).toEqual([expect.objectContaining({ idempotencyKey: 'shed-key', attempts: 0 })]);
+ expect(insertedFailures[0].lastError).toContain('exceeding the 1024-byte cap');
+ expect(getWebhookDeliveryFailuresTotal()).toBe(failuresBefore);
+ });
+
it("isolates each webhook's data so an in-place before-hook mutation cannot bleed across webhooks", async () => {
const a = createMockWebhook({ id: 'wh-a', events: ['message.received'] });
const b = createMockWebhook({ id: 'wh-b', events: ['message.received'] });
@@ -1453,6 +1565,87 @@ describe('WebhookDeliveryService', () => {
expect(jobData).not.toHaveProperty('headers');
});
+ it('leaves the shed row of a replay in place when its job is queued', async () => {
+ // The job resolves it: a delivery clears it and a terminal failure replaces it. Deleting it
+ // here would leave no record if the job is then lost, or the add fails and falls back.
+ const queueService = await buildQueueService((key: string, def?: unknown) => {
+ if (key === 'queue.enabled') return true;
+ if (key === 'webhook.retryDelay') return 5000;
+ return def;
+ });
+ (hookManager.execute as jest.Mock).mockResolvedValue({ continue: true, data: {} });
+ await failureRepository.insert!({ webhookId: 'wh-1', idempotencyKey: 'stored-key', attempts: 0 });
+ const webhook = createMockWebhook({ id: 'wh-1', events: ['message.received'] });
+
+ await expect(queueService.redeliver(webhook, 'sess-1', 'message.received', 'stored-key', {})).resolves.toBe(
+ 'enqueued',
+ );
+
+ expect(failureRepository.delete).not.toHaveBeenCalled();
+ expect(insertedFailures).toEqual([expect.objectContaining({ idempotencyKey: 'stored-key', attempts: 0 })]);
+ });
+
+ it('keeps the shed row while a replay whose queue add failed is still retrying directly', async () => {
+ // The fallback runs the direct retry loop, which nothing drains at shutdown. The shed row is
+ // replaced only when that loop files its own row, so a restart during it cannot lose the event.
+ const queueService = await buildQueueService((key: string, def?: unknown) => {
+ if (key === 'queue.enabled') return true;
+ if (key === 'webhook.retryDelay') return 1;
+ return def;
+ });
+ (hookManager.execute as jest.Mock).mockResolvedValue({ continue: true, data: {} });
+ await failureRepository.insert!({ webhookId: 'wh-1', idempotencyKey: 'shed-key', attempts: 0 });
+ (webhookQueue.add as jest.Mock).mockRejectedValueOnce(new Error('Redis unreachable'));
+ const mockFetch = undiciFetch as jest.Mock;
+ mockFetch.mockReset();
+ let release!: (v: unknown) => void;
+ mockFetch.mockImplementation(() => new Promise(resolve => (release = resolve)));
+ const webhook = createMockWebhook({ id: 'wh-1', events: ['message.received'], retryCount: 1 });
+
+ const replay = queueService.redeliver(webhook, 'sess-1', 'message.received', 'shed-key', { from: '[email protected]' });
+ for (let i = 0; i < 20 && mockFetch.mock.calls.length === 0; i++) await new Promise(r => setImmediate(r));
+ expect(mockFetch).toHaveBeenCalledTimes(1);
+ expect(insertedFailures).toEqual([expect.objectContaining({ idempotencyKey: 'shed-key', attempts: 0 })]);
+
+ release({ ok: false, status: 503, statusText: 'Unavailable' });
+ await replay;
+ expect(insertedFailures).toEqual([expect.objectContaining({ idempotencyKey: 'shed-key', attempts: 1 })]);
+ });
+
+ it('clears the failure rows of a replay that the fallback POST delivers after a failed queue add', async () => {
+ const queueService = await buildQueueService((key: string, def?: unknown) => {
+ if (key === 'queue.enabled') return true;
+ if (key === 'webhook.retryDelay') return 1;
+ return def;
+ });
+ (hookManager.execute as jest.Mock).mockResolvedValue({ continue: true, data: {} });
+ await failureRepository.insert!({ webhookId: 'wh-1', idempotencyKey: 'shed-key', attempts: 0 });
+ (webhookQueue.add as jest.Mock).mockRejectedValueOnce(new Error('Redis unreachable'));
+ (undiciFetch as jest.Mock).mockResolvedValue({ ok: true, status: 200 });
+ const webhook = createMockWebhook({ id: 'wh-1', events: ['message.received'] });
+
+ await expect(queueService.redeliver(webhook, 'sess-1', 'message.received', 'shed-key', {})).resolves.toBe(
+ 'delivered',
+ );
+
+ expect(insertedFailures).toEqual([]);
+ });
+
+ it('does not touch failure rows when queueing a first dispatch', async () => {
+ const queueService = await buildQueueService((key: string, def?: unknown) => {
+ if (key === 'queue.enabled') return true;
+ if (key === 'webhook.retryDelay') return 5000;
+ return def;
+ });
+ (repository.find as jest.Mock).mockResolvedValue([createMockWebhook({ events: ['message.received'] })]);
+ (hookManager.execute as jest.Mock).mockResolvedValue({ continue: true, data: {} });
+
+ await queueService.dispatch('sess-1', 'message.received', {});
+
+ expect(webhookQueue.add).toHaveBeenCalledTimes(1);
+ expect(failureRepository.delete).not.toHaveBeenCalled();
+ });
+
it('gives sibling webhooks distinct job ids, so one event fanning out is not collapsed into one job', async () => {
const queueService = await buildQueueService((key: string, def?: unknown) => {
if (key === 'queue.enabled') return true;
@@ -1597,5 +1790,40 @@ describe('WebhookDeliveryService', () => {
expect(getWebhookDeliveryFailuresTotal()).toBe(failuresBefore + 1);
mockFetch.mockReset();
});
+
+ it('clears every failure row of a delivery once a direct replay delivers it', async () => {
+ // The shed row and an earlier failed replay's row both list the event as lost, and the
+ // receiver now has it. A row of another delivery of the same webhook stays.
+ const webhook = createMockWebhook({ id: 'wh-1', events: ['message.received'] });
+ (hookManager.execute as jest.Mock).mockResolvedValue({ continue: true, data: {} });
+ await failureRepository.insert!({ webhookId: 'wh-1', idempotencyKey: 'shed-key', attempts: 0 });
+ await failureRepository.insert!({ webhookId: 'wh-1', idempotencyKey: 'shed-key', attempts: 3 });
+ await failureRepository.insert!({ webhookId: 'wh-1', idempotencyKey: 'other-key', attempts: 3 });
+ const mockFetch = undiciFetch as jest.Mock;
+ mockFetch.mockReset();
+ mockFetch.mockResolvedValue({ ok: true, status: 200 });
+
+ await expect(service.redeliver(webhook, 'sess-1', 'message.received', 'shed-key', {})).resolves.toBe('delivered');
+
+ expect(insertedFailures).toEqual([expect.objectContaining({ idempotencyKey: 'other-key' })]);
+ mockFetch.mockReset();
+ });
+
+ it('keeps a delivered outcome when clearing its failure rows fails', async () => {
+ const webhook = createMockWebhook({ id: 'wh-1', events: ['message.received'], retryCount: 3 });
+ (hookManager.execute as jest.Mock).mockResolvedValue({ continue: true, data: {} });
+ (failureRepository.delete as jest.Mock).mockRejectedValueOnce(new Error('db down'));
+ const mockFetch = undiciFetch as jest.Mock;
+ mockFetch.mockReset();
+ mockFetch.mockResolvedValue({ ok: true, status: 200 });
+
+ await expect(service.redeliver(webhook, 'sess-1', 'message.received', 'k', {})).resolves.toBe('delivered');
+
+ // One POST, no retry, and no dead-letter row for an event the receiver accepted.
+ expect(failureRepository.delete).toHaveBeenCalledTimes(1);
+ expect(mockFetch).toHaveBeenCalledTimes(1);
+ expect(failureRepository.insert).not.toHaveBeenCalled();
+ mockFetch.mockReset();
+ });
});
});
diff --git a/src/modules/webhook/webhook-delivery.service.ts b/src/modules/webhook/webhook-delivery.service.ts
index 1f762e3e1..7e025e5b7 100644
--- a/src/modules/webhook/webhook-delivery.service.ts
+++ b/src/modules/webhook/webhook-delivery.service.ts
@@ -8,7 +8,7 @@ import { setTimeout } from 'node:timers/promises';
import { Webhook } from './entities/webhook.entity';
import { WebhookOutboxService } from './webhook-outbox.service';
import { WebhookDeliveryFailure } from './entities/webhook-delivery-failure.entity';
-import { recordWebhookDeliveryFailure } from './utils/record-delivery-failure';
+import { clearDeliveryFailureRows, recordWebhookDeliveryFailure } from './utils/record-delivery-failure';
import {
buildDeliveryHeaders,
generateSignature,
@@ -83,7 +83,7 @@ interface DispatchEventContext {
* The webhook delivery engine: given an event occurrence, fan it out to the session's matching
* webhooks — bounded by the dispatch limiter — through the BullMQ queue when enabled (with a
* direct-delivery fallback when enqueue fails) or through direct inline delivery when not.
- * Records terminally failed deliveries in webhook_delivery_failures. Webhook registration/CRUD
+ * Records failed and unsent deliveries in webhook_delivery_failures. Webhook registration/CRUD
* lives on WebhookService, which delegates dispatch here.
*/
const isPlainObject = (value: unknown): boolean => typeof value === 'object' && value !== null && !Array.isArray(value);
@@ -456,6 +456,10 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {
const { finalPayload, body, headers } = preflight;
// Use queue if available, otherwise fallback to direct delivery
if (this.queueEnabled && this.webhookQueue) {
+ // A replay's attempts-0 row is not cleared here. It stays until the delivery resolves: a
+ // successful POST (the processor's, or the fallback's when the add fails) clears it, and a
+ // terminal failure replaces it right after filing its own row. A restart or a lost job at
+ // any point before that still leaves the event on record.
return this.enqueueWithFallback(webhook, finalPayload, body, headers, deliveryId, idempotencyKey, ctx);
}
return this.deliverDirect(webhook, finalPayload, body, headers, deliveryId, ctx);
@@ -769,6 +773,9 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {
{ webhookId: webhook.id, deliveryId: payload.deliveryId, action: 'webhook_bookkeeping_failed' },
);
}
+ // A delivered event must not stay listed as lost: a replay of a shed, refused or failed
+ // dispatch, or a re-emitted event, reuses the key an earlier failure row was filed under.
+ await clearDeliveryFailureRows(this.failureRepository, this.logger, webhook.id, payload.idempotencyKey);
this.logger.debug(`Webhook delivered to ${webhook.id}`, {
webhookId: webhook.id,
@@ -790,6 +797,8 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {
}
// All direct-path retries exhausted — persist a durable failure record before giving up, mirroring
// the queued processor's final-attempt path so the queue-disabled path isn't a blind spot.
+ // The recorder writes this row first and only then removes an attempts-0 row of the same
+ // delivery (a shed or refused dispatch), so a restart at any point keeps a record.
const recorded = await recordTerminalFailure(this.failureRepository, this.logger, {
webhookId: webhook.id,
sessionId: payload.sessionId,