Skip to content

Free, Open Source, Self-Hosted WhatsApp API Gateway

fix(webhook): keep the record of an unattempted delivery

Yudhi Armyndharis85f9c82914ed
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 6eaabd5b8..1cc9a7200 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -64,7 +64,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- `GET /api/sessions/:sessionId/presence/:chatId` returns `null` once the session has no running engine, instead of the last presence reported before a stop, logout, force-kill or failure.
- An explicit `maxReconnectAttempts` is honoured and the reconnect delay is capped at 5 minutes; the budget used to restart once the backoff passed 5 minutes, so a limit above about 6 attempts was never reached and the documented 1-hour cap never applied.
- The webhook delivery reconciler no longer replays a delivery that is still waiting for a dispatch slot or retrying on the node that dispatched it, which sent a duplicate outside `WEBHOOK_DISPATCH_CONCURRENCY` and could close a slow delivery as failed while it was still running.
-- A webhook delivery shed at `WEBHOOK_DISPATCH_MAX_QUEUED` or refused during shutdown is no longer replayed by the delivery reconciler, so its delivery-failure row no longer reports an event that was delivered after all.
- `session.reconnect_loop` webhooks carry an idempotency key salted per occurrence, so an alert from a later outage that reaches the same attempt count is no longer deduplicated onto the earlier one or left without a delivery record.
- A bulk batch sends each message through the session's current engine, so a reconnect or restart mid-batch no longer fails every remaining message.
- `POST /messages/send-bulk` answers 400 for an item with an empty `chatId`, a text item without text, or a media item without a `url` or `base64` under its type; such items used to be accepted with 202 and fail later.
diff --git a/docs/06-api-specification.md b/docs/06-api-specification.md
index f980d7b93..752b28c01 100644
--- a/docs/06-api-specification.md
+++ b/docs/06-api-specification.md
@@ -6868,7 +6868,7 @@ Webhook delivery is **at-least-once**. A consumer can legitimately receive the s
- The underlying WhatsApp engine can re-fire an event for a single message.
- A failed delivery (non-2xx response, timeout, or network error) is retried.
-**Crash boundary.** Every delivery is recorded before it is attempted, and the record is retired once something durable owns it: the queue job in queued mode, the completed send in direct mode. A hard crash (SIGKILL, OOM) therefore leaves the record behind, and a bounded sweep (`WEBHOOK_RECONCILE_INTERVAL_MS`, default 60s) replays whatever is still stranded, reusing the stored `X-OpenWA-Idempotency-Key` so the retry stays deduplicable at your receiver. A delivery that keeps failing exhausts `WEBHOOK_RECONCILE_MAX_ATTEMPTS` and goes terminal rather than replaying forever. One window remains open: a crash between persisting the message and writing that record loses the delivery, because the two are not yet one transaction. The failure table records exhausted retries, plus over-budget, dispatch-capacity-exceeded, and shutdown-rejected deliveries with attempts 0. A delivery the dispatcher shed for capacity or refused during the drain is not replayed: its failure row is the only record, so recover it from there. A node's sweep also leaves alone a delivery that the same node is still waiting to dispatch or retrying, however long that takes; with several nodes on one database, another node can still replay it, and the stored idempotency key keeps that duplicate deduplicable. Enabling the queue (`QUEUE_ENABLED=true`, needs Redis) makes the dispatch durable from the enqueue onward. In both modes, a graceful shutdown drains in-flight deliveries first: the queued path waits for each worker's current job, and the direct path waits up to `WEBHOOK_SHUTDOWN_DRAIN_MS` (default 5s; raise it to at least `WEBHOOK_TIMEOUT`, default 10s, if a slow receiver must finish).
+**Crash boundary.** Every delivery is recorded before it is attempted, and the record is retired once something durable owns it: the queue job in queued mode, the completed send in direct mode. A hard crash (SIGKILL, OOM) therefore leaves the record behind, and a bounded sweep (`WEBHOOK_RECONCILE_INTERVAL_MS`, default 60s) replays whatever is still stranded, reusing the stored `X-OpenWA-Idempotency-Key` so the retry stays deduplicable at your receiver. A delivery that keeps failing exhausts `WEBHOOK_RECONCILE_MAX_ATTEMPTS` and goes terminal rather than replaying forever. One window remains open: a crash between persisting the message and writing that record loses the delivery, because the two are not yet one transaction. The failure table records exhausted retries, plus over-budget, dispatch-capacity-exceeded, and shutdown-rejected deliveries with attempts 0. Read the last two as a report rather than a verdict: a delivery the dispatcher shed for capacity or refused during the drain was rejected before its POST, so it keeps its record and is replayed by the same sweep, and a row there can belong to an event that was later delivered. A node's sweep also leaves alone a delivery that the same node is still waiting to dispatch or retrying, however long that takes; with several nodes on one database, another node can still replay it, and the stored idempotency key keeps that duplicate deduplicable. Enabling the queue (`QUEUE_ENABLED=true`, needs Redis) makes the dispatch durable from the enqueue onward. In both modes, a graceful shutdown drains in-flight deliveries first: the queued path waits for each worker's current job, and the direct path waits up to `WEBHOOK_SHUTDOWN_DRAIN_MS` (default 5s; raise it to at least `WEBHOOK_TIMEOUT`, default 10s, if a slow receiver must finish).
**Design your handler to be idempotent**, keyed on the `X-OpenWA-Idempotency-Key` header (see below). As a server-side safety net, OpenWA de-duplicates inbound `message.received` before dispatch (a re-fired event for an already-persisted message is dropped), so one webhook normally sees each inbound message once — but this is best-effort defense-in-depth and does not remove the need for consumer-side idempotency.
diff --git a/src/modules/webhook/webhook-delivery.service.spec.ts b/src/modules/webhook/webhook-delivery.service.spec.ts
index badecce4c..de7a53655 100644
--- a/src/modules/webhook/webhook-delivery.service.spec.ts
+++ b/src/modules/webhook/webhook-delivery.service.spec.ts
@@ -349,11 +349,15 @@ describe('WebhookDeliveryService', () => {
}),
);
expect(mockFetch).toHaveBeenCalledTimes(1);
- // The shed delivery's record is retired, so the reconciler does not replay what was shed on
- // purpose and leave a failure row behind for an event that was delivered after all.
- expect(outboxService.close).toHaveBeenCalledWith('wh-b', expect.stringMatching(/_wh-b$/), 'failed');
+ // The shed delivery was rejected before its POST, so only the delivered one is retired: wh-b's
+ // row stays pending and the reconciler is what still gets that event to the receiver.
expect(outboxService.close).toHaveBeenCalledWith('wh-a', expect.stringMatching(/_wh-a$/), 'dispatched');
- expect(service.isLocallyPending((outboxService.close.mock.calls as string[][])[0][1])).toBe(false);
+ expect((outboxService.close.mock.calls as string[][]).map(c => c[0])).toEqual(['wh-a']);
+ // Both keys, the shed one included, are released: the reconciler only replays a pending row
+ // once no dispatch on this node still owns it, so a leaked key would strand wh-b for good.
+ for (const c of outboxService.open.mock.calls as Array<[{ idempotencyKey: string }]>) {
+ expect(service.isLocallyPending(c[0].idempotencyKey)).toBe(false);
+ }
});
it('reports a delivery as locally pending while it is parked or in flight, and releases it once settled', async () => {
@@ -871,9 +875,9 @@ describe('WebhookDeliveryService', () => {
const recorded = shutdownInserts.map(c => c[0]);
expect(recorded.map(r => r.webhookId)).toEqual(expect.arrayContaining(['wh-b', 'wh-c']));
for (const r of recorded) expect(r.lastError).toBe('ConcurrencyLimiter closed');
- // Their records are retired too, so the next start does not replay what was dead-lettered here.
- expect(outboxService.close).toHaveBeenCalledWith('wh-b', expect.stringMatching(/_wh-b$/), 'failed');
- expect(outboxService.close).toHaveBeenCalledWith('wh-c', expect.stringMatching(/_wh-c$/), 'failed');
+ // Their outbox rows are left pending: the POST never happened, so the next start's sweep is
+ // the only thing that can still deliver those two events.
+ expect(outboxService.close).not.toHaveBeenCalled();
// The in-flight delivery was NOT falsely dead-lettered (the receiver may have it); it is
// logged as abandoned so the loss is still operator-visible.
expect(
@@ -888,7 +892,7 @@ describe('WebhookDeliveryService', () => {
loggerSpy.mockRestore();
});
- it('waits for the outbox close of a delivery refused during shutdown before returning', async () => {
+ it('waits for the failure record of a delivery refused during shutdown before returning', async () => {
const hooks = [
createMockWebhook({ id: 'wh-a', url: 'https://a.example/hook', events: ['message.received'] }),
createMockWebhook({ id: 'wh-b', url: 'https://b.example/hook', events: ['message.received'] }),
@@ -907,10 +911,10 @@ describe('WebhookDeliveryService', () => {
let releaseActive: (value: unknown) => void = () => undefined;
mockFetch.mockImplementation(() => new Promise(resolve => (releaseActive = resolve)));
- // The close of the refused delivery stays open until the test releases it.
- let releaseClose: () => void = () => undefined;
- outboxService.close.mockImplementation((_id: string, _key: string, state: string) =>
- state === 'failed' ? new Promise<void>(resolve => (releaseClose = resolve)) : Promise.resolve(),
+ // The failure record of the refused delivery stays open until the test releases it.
+ let releaseInsert: () => void = () => undefined;
+ (failureRepository.insert as jest.Mock).mockImplementation(
+ () => new Promise<void>(resolve => (releaseInsert = resolve)),
);
const dispatchP = service.dispatch('sess-1', 'message.received', { from: '[email protected]' });
@@ -920,19 +924,21 @@ describe('WebhookDeliveryService', () => {
const destroyP = service.onModuleDestroy().then(() => (destroyed = true));
// The in-flight delivery finishes, so only the refused delivery's bookkeeping keeps the drain open.
releaseActive({ ok: true, status: 200 });
- const closedFailed = (): boolean => (outboxService.close.mock.calls as string[][]).some(c => c[2] === 'failed');
- for (let i = 0; i < 40 && !destroyed && !closedFailed(); i++) await new Promise(r => setTimeout(r, 10));
+ const recording = (): boolean => (failureRepository.insert as jest.Mock).mock.calls.length > 0;
+ for (let i = 0; i < 40 && !destroyed && !recording(); i++) await new Promise(r => setTimeout(r, 10));
await new Promise(r => setTimeout(r, 120));
- // A close outside the tracked bookkeeping would let teardown finish first, closing the database
- // under a row that then stays pending and is replayed after restart.
- expect(closedFailed()).toBe(true);
+ // A write outside the tracked bookkeeping would let teardown finish first, closing the database
+ // under the failure record of a delivery the receiver never got.
+ expect(recording()).toBe(true);
expect(destroyed).toBe(false);
- releaseClose();
+ releaseInsert();
await destroyP;
await dispatchP;
expect(destroyed).toBe(true);
+ // The refused delivery keeps its pending outbox row, so the next start's sweep replays it.
+ expect(outboxService.close).not.toHaveBeenCalledWith('wh-b', expect.anything(), 'failed');
});
it('queued mode: parked enqueues drain to the queue on shutdown instead of dead-lettering', async () => {
diff --git a/src/modules/webhook/webhook-delivery.service.ts b/src/modules/webhook/webhook-delivery.service.ts
index 5b92f40d8..a3afb9a85 100644
--- a/src/modules/webhook/webhook-delivery.service.ts
+++ b/src/modules/webhook/webhook-delivery.service.ts
@@ -665,6 +665,9 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {
})
.catch(async error => {
if (error instanceof Error && error.message === 'ConcurrencyLimiter queue full') {
+ // Shed before the task ran, so nothing was POSTed. The failure row reports the shed, but
+ // the outbox row stays 'pending' on purpose: retiring it would drop the only copy of an
+ // event the receiver provably never got. The sweep replays it once the backlog clears.
await this.recordUndelivered(
webhook,
deliveryId,
@@ -673,15 +676,13 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {
'webhook_dispatch_capacity_exceeded',
ctx,
);
- // Shed on purpose: the failure row is the record, so a replay must not undo the shedding.
- await this.outbox.close(webhook.id, idempotencyKey, 'failed');
return;
}
if (error instanceof Error && error.message === 'ConcurrencyLimiter closed') {
// Rejected by the shutdown drain before dispatching — record it like any other
- // undelivered delivery, retire its outbox row so it is not replayed after restart, and
- // track both writes so onModuleDestroy can await them (the
- // limiter slot bookkeeping no longer covers this task).
+ // undelivered delivery, and track the write so onModuleDestroy can await it (the
+ // limiter slot bookkeeping no longer covers this task). Its outbox row stays 'pending':
+ // the POST never happened, so the next start's sweep is what finally delivers the event.
const record = this.recordUndelivered(
webhook,
deliveryId,
@@ -689,7 +690,7 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {
error,
'webhook_dispatch_shutdown',
ctx,
- ).then(() => this.outbox.close(webhook.id, idempotencyKey, 'failed'));
+ );
this.pendingBookkeeping.add(record);
try {
await record;