Skip to content

Free, Open Source, Self-Hosted WhatsApp API Gateway

fix(webhook): skip outbox rows still being dispatched on this node

Yudhi Armyndharisd736c957b33c
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 980b2a1b6..c9bb7d15d 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -20,6 +20,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- A session whose automatic reconnect fails to relaunch the engine (a network, DNS or browser launch error) keeps retrying with backoff instead of stopping in `failed` until restarted by hand; an authentication failure or a stale browser profile still ends in `failed` ([#1580](https://github.com/rmyndharis/OpenWA/issues/1580)).
- A session that runs out of reconnect attempts fires the `session:error` plugin hook when it lands in `failed`.
- An explicit `maxReconnectAttempts` is honoured and the reconnect delay is capped at 5 minutes; the budget used to restart once the backoff passed 5 minutes, so the limit was never reached and the documented 1-hour cap never applied.
+- The webhook delivery reconciler no longer replays a delivery that is still waiting for a dispatch slot or retrying on the node that dispatched it, which sent a duplicate outside `WEBHOOK_DISPATCH_CONCURRENCY` and could close a slow delivery as failed while it was still running.
- A webhook delivery shed at `WEBHOOK_DISPATCH_MAX_QUEUED` or refused during shutdown is no longer replayed by the delivery reconciler, so its delivery-failure row no longer reports an event that was delivered after all.
- `session.reconnect_loop` webhooks carry an idempotency key salted per occurrence, so an alert from a later outage that reaches the same attempt count is no longer deduplicated onto the earlier one or left without a delivery record.
diff --git a/docs/06-api-specification.md b/docs/06-api-specification.md
index 2b3e9712a..75408e673 100644
--- a/docs/06-api-specification.md
+++ b/docs/06-api-specification.md
@@ -6859,7 +6859,7 @@ Webhook delivery is **at-least-once**. A consumer can legitimately receive the s
- The underlying WhatsApp engine can re-fire an event for a single message.
- A failed delivery (non-2xx response, timeout, or network error) is retried.
-**Crash boundary.** Every delivery is recorded before it is attempted, and the record is retired once something durable owns it: the queue job in queued mode, the completed send in direct mode. A hard crash (SIGKILL, OOM) therefore leaves the record behind, and a bounded sweep (`WEBHOOK_RECONCILE_INTERVAL_MS`, default 60s) replays whatever is still stranded, reusing the stored `X-OpenWA-Idempotency-Key` so the retry stays deduplicable at your receiver. A delivery that keeps failing exhausts `WEBHOOK_RECONCILE_MAX_ATTEMPTS` and goes terminal rather than replaying forever. One window remains open: a crash between persisting the message and writing that record loses the delivery, because the two are not yet one transaction. The failure table records exhausted retries, plus over-budget, dispatch-capacity-exceeded, and shutdown-rejected deliveries with attempts 0. A delivery the dispatcher shed for capacity or refused during the drain is not replayed: its failure row is the only record, so recover it from there. Enabling the queue (`QUEUE_ENABLED=true`, needs Redis) makes the dispatch durable from the enqueue onward. In both modes, a graceful shutdown drains in-flight deliveries first: the queued path waits for each worker's current job, and the direct path waits up to `WEBHOOK_SHUTDOWN_DRAIN_MS` (default 5s; raise it to at least `WEBHOOK_TIMEOUT`, default 10s, if a slow receiver must finish).
+**Crash boundary.** Every delivery is recorded before it is attempted, and the record is retired once something durable owns it: the queue job in queued mode, the completed send in direct mode. A hard crash (SIGKILL, OOM) therefore leaves the record behind, and a bounded sweep (`WEBHOOK_RECONCILE_INTERVAL_MS`, default 60s) replays whatever is still stranded, reusing the stored `X-OpenWA-Idempotency-Key` so the retry stays deduplicable at your receiver. A delivery that keeps failing exhausts `WEBHOOK_RECONCILE_MAX_ATTEMPTS` and goes terminal rather than replaying forever. One window remains open: a crash between persisting the message and writing that record loses the delivery, because the two are not yet one transaction. The failure table records exhausted retries, plus over-budget, dispatch-capacity-exceeded, and shutdown-rejected deliveries with attempts 0. A delivery the dispatcher shed for capacity or refused during the drain is not replayed: its failure row is the only record, so recover it from there. A node's sweep also leaves alone a delivery that the same node is still waiting to dispatch or retrying, however long that takes; with several nodes on one database, another node can still replay it, and the stored idempotency key keeps that duplicate deduplicable. Enabling the queue (`QUEUE_ENABLED=true`, needs Redis) makes the dispatch durable from the enqueue onward. In both modes, a graceful shutdown drains in-flight deliveries first: the queued path waits for each worker's current job, and the direct path waits up to `WEBHOOK_SHUTDOWN_DRAIN_MS` (default 5s; raise it to at least `WEBHOOK_TIMEOUT`, default 10s, if a slow receiver must finish).
**Design your handler to be idempotent**, keyed on the `X-OpenWA-Idempotency-Key` header (see below). As a server-side safety net, OpenWA de-duplicates inbound `message.received` before dispatch (a re-fired event for an already-persisted message is dropped), so one webhook normally sees each inbound message once — but this is best-effort defense-in-depth and does not remove the need for consumer-side idempotency.
diff --git a/src/modules/webhook/webhook-delivery.service.spec.ts b/src/modules/webhook/webhook-delivery.service.spec.ts
index a81867f57..badecce4c 100644
--- a/src/modules/webhook/webhook-delivery.service.spec.ts
+++ b/src/modules/webhook/webhook-delivery.service.spec.ts
@@ -353,6 +353,36 @@ describe('WebhookDeliveryService', () => {
// purpose and leave a failure row behind for an event that was delivered after all.
expect(outboxService.close).toHaveBeenCalledWith('wh-b', expect.stringMatching(/_wh-b$/), 'failed');
expect(outboxService.close).toHaveBeenCalledWith('wh-a', expect.stringMatching(/_wh-a$/), 'dispatched');
+ expect(service.isLocallyPending((outboxService.close.mock.calls as string[][])[0][1])).toBe(false);
+ });
+
+ it('reports a delivery as locally pending while it is parked or in flight, and releases it once settled', async () => {
+ const wA = createMockWebhook({ id: 'wh-a', url: 'https://a.example/hook', events: ['message.received'] });
+ const wB = createMockWebhook({ id: 'wh-b', url: 'https://b.example/hook', events: ['message.received'] });
+ (repository.find as jest.Mock).mockResolvedValue([wA, wB]);
+ (repository.update as jest.Mock).mockResolvedValue({ affected: 1 });
+ (hookManager.execute as jest.Mock).mockImplementation((_event: string, data: unknown) =>
+ Promise.resolve({ continue: true, data }),
+ );
+ // One slot: A holds it, B parks behind it.
+ (service as unknown as { dispatchLimiter: ConcurrencyLimiter }).dispatchLimiter = new ConcurrencyLimiter(1, 10);
+
+ let release: (value: unknown) => void = () => undefined;
+ mockFetch.mockImplementation(() => new Promise(resolve => (release = resolve)));
+
+ const pending = service.dispatch('sess-1', 'message.received', { from: '[email protected]' });
+ for (let i = 0; i < 20 && mockFetch.mock.calls.length === 0; i++) await new Promise(r => setImmediate(r));
+
+ const keys = (outboxService.open.mock.calls as Array<[{ idempotencyKey: string }]>).map(c => c[0].idempotencyKey);
+ expect(keys).toHaveLength(2);
+ for (const key of keys) expect(service.isLocallyPending(key)).toBe(true);
+
+ release({ ok: true, status: 200 });
+ for (let i = 0; i < 20 && mockFetch.mock.calls.length < 2; i++) await new Promise(r => setImmediate(r));
+ release({ ok: true, status: 200 });
+ await pending;
+
+ for (const key of keys) expect(service.isLocallyPending(key)).toBe(false);
});
it('salts each sibling webhook with a distinct idempotency key so one receiver cannot dedupe out another', async () => {
diff --git a/src/modules/webhook/webhook-delivery.service.ts b/src/modules/webhook/webhook-delivery.service.ts
index 0bf930fc9..5b92f40d8 100644
--- a/src/modules/webhook/webhook-delivery.service.ts
+++ b/src/modules/webhook/webhook-delivery.service.ts
@@ -97,6 +97,13 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {
>();
/** Late bookkeeping (dead-letter rows) written by tasks the limiter already released — awaited on shutdown. */
private readonly pendingBookkeeping = new Set<Promise<void>>();
+ /**
+ * Outbox rows this node still owns, by idempotency key and counted (the same key can be dispatched
+ * twice), from the moment the row is opened until its dispatch settles: parked in the limiter,
+ * holding a slot, or inside a direct retry loop. The reconciler skips these so a slow delivery is
+ * not replayed alongside itself.
+ */
+ private readonly locallyPending = new Map<string, number>();
constructor(
@InjectRepository(Webhook, 'data')
@@ -616,6 +623,28 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {
deliveryId,
payload: ctx.baseData,
});
+ this.locallyPending.set(idempotencyKey, (this.locallyPending.get(idempotencyKey) ?? 0) + 1);
+ try {
+ await this.runLimited(webhook, deliveryId, idempotencyKey, ctx);
+ } finally {
+ const left = (this.locallyPending.get(idempotencyKey) ?? 1) - 1;
+ if (left > 0) this.locallyPending.set(idempotencyKey, left);
+ else this.locallyPending.delete(idempotencyKey);
+ }
+ }
+
+ /** True while a dispatch on this node still owns the outbox row for this key. */
+ isLocallyPending(idempotencyKey: string): boolean {
+ return this.locallyPending.has(idempotencyKey);
+ }
+
+ private async runLimited(
+ webhook: Webhook,
+ deliveryId: string,
+ idempotencyKey: string,
+ ctx: DispatchEventContext,
+ ): Promise<void> {
+ const { sessionId, event } = ctx;
await this.dispatchLimiter
.run(async () => {
this.inFlightDeliveries.set(deliveryId, {
diff --git a/src/modules/webhook/webhook-outbox.service.spec.ts b/src/modules/webhook/webhook-outbox.service.spec.ts
index 01f0fb78b..f8a2158d1 100644
--- a/src/modules/webhook/webhook-outbox.service.spec.ts
+++ b/src/modules/webhook/webhook-outbox.service.spec.ts
@@ -45,7 +45,8 @@ describe('WebhookOutboxService', () => {
repo.update.mockRejectedValue(new Error('disk full'));
await expect(service.close('wh-1', 'key_wh-1', 'dispatched')).resolves.toBeUndefined();
- await expect(service.countAttempt('row-1', 0)).resolves.toBeUndefined();
+ // A failed count must not block the replay: the write is best-effort like the rest.
+ await expect(service.countAttempt('row-1', 0)).resolves.toBe(true);
});
it('retires the payload when recording an outcome, so a settled row stops carrying a message body', async () => {
@@ -75,9 +76,20 @@ describe('WebhookOutboxService', () => {
});
it('counts an attempt by incrementing the stored number, not by recomputing it', async () => {
- await service.countAttempt('row-1', 2);
+ repo.update.mockResolvedValue({ affected: 1 });
- expect(repo.update).toHaveBeenCalledWith({ id: 'row-1' }, expect.objectContaining({ attempts: 3 }));
+ await expect(service.countAttempt('row-1', 2)).resolves.toBe(true);
+
+ expect(repo.update).toHaveBeenCalledWith(
+ { id: 'row-1', state: 'pending' },
+ expect.objectContaining({ attempts: 3 }),
+ );
+ });
+
+ it('reports a row that is no longer pending, so a stale copy is not replayed', async () => {
+ repo.update.mockResolvedValue({ affected: 0 });
+
+ await expect(service.countAttempt('row-1', 2)).resolves.toBe(false);
});
});
diff --git a/src/modules/webhook/webhook-outbox.service.ts b/src/modules/webhook/webhook-outbox.service.ts
index f1baf88d8..4a84a4556 100644
--- a/src/modules/webhook/webhook-outbox.service.ts
+++ b/src/modules/webhook/webhook-outbox.service.ts
@@ -148,12 +148,23 @@ export class WebhookOutboxService implements OnModuleInit, OnModuleDestroy {
}));
}
- /** Count one replay attempt against the row's budget. */
- async countAttempt(id: string, attempts: number): Promise<void> {
+ /**
+ * Count one replay attempt against the row's budget.
+ *
+ * Returns false when the row is no longer pending: the sweep reads its batch once, so a delivery
+ * that settled after that read must not be replayed from the stale copy. A failed write returns
+ * true, keeping the replay best-effort like every other write here.
+ */
+ async countAttempt(id: string, attempts: number): Promise<boolean> {
try {
- await this.outbox.update({ id }, { attempts: attempts + 1, lastAttemptAt: new Date() });
+ const result = await this.outbox.update(
+ { id, state: 'pending' },
+ { attempts: attempts + 1, lastAttemptAt: new Date() },
+ );
+ return (result.affected ?? 0) > 0;
} catch (error) {
this.logger.warn(`Could not count a replay attempt: ${String(error)}`);
+ return true;
}
}
}
diff --git a/src/modules/webhook/webhook-reconciler.service.spec.ts b/src/modules/webhook/webhook-reconciler.service.spec.ts
index 23b6cf0ee..5a8637675 100644
--- a/src/modules/webhook/webhook-reconciler.service.spec.ts
+++ b/src/modules/webhook/webhook-reconciler.service.spec.ts
@@ -37,13 +37,20 @@ describe('resolveWebhookReconcilerOptions', () => {
describe('WebhookReconcilerService', () => {
let outbox: { findStale: jest.Mock; close: jest.Mock; countAttempt: jest.Mock };
- let delivery: { redeliver: jest.Mock };
+ let delivery: { redeliver: jest.Mock; isLocallyPending: jest.Mock };
let webhooks: { findOne: jest.Mock };
let service: WebhookReconcilerService;
beforeEach(() => {
- outbox = { findStale: jest.fn().mockResolvedValue([]), close: jest.fn(), countAttempt: jest.fn() };
- delivery = { redeliver: jest.fn().mockResolvedValue('delivered') };
+ outbox = {
+ findStale: jest.fn().mockResolvedValue([]),
+ close: jest.fn(),
+ countAttempt: jest.fn().mockResolvedValue(true),
+ };
+ delivery = {
+ redeliver: jest.fn().mockResolvedValue('delivered'),
+ isLocallyPending: jest.fn().mockReturnValue(false),
+ };
webhooks = { findOne: jest.fn().mockResolvedValue({ id: 'wh-1', active: true }) };
service = new WebhookReconcilerService(webhooks as never, outbox as never, delivery as never);
});
@@ -140,6 +147,33 @@ describe('WebhookReconcilerService', () => {
expect(stats).toMatchObject({ skipped: 1 });
});
+ it('leaves a row alone while this node is still dispatching it', async () => {
+ // A direct delivery with retries, or one parked behind slow receivers, can stay pending past the
+ // grace window. Replaying it would POST alongside the original and spend its budget mid-flight.
+ outbox.findStale.mockResolvedValue([row({ attempts: 3 })]);
+ delivery.isLocallyPending.mockImplementation((key: string) => key === 'stored-key_wh-1');
+
+ const stats = await service.sweep(OPTS);
+
+ expect(delivery.redeliver).not.toHaveBeenCalled();
+ expect(outbox.countAttempt).not.toHaveBeenCalled();
+ expect(outbox.close).not.toHaveBeenCalled();
+ expect(stats).toMatchObject({ scanned: 1, skipped: 1, replayed: 0, failed: 0 });
+ });
+
+ it('does not replay a row that settled after the batch was read', async () => {
+ // The original dispatch finished while an earlier row in the pass was replaying: no longer
+ // locally pending, but no longer pending in the database either.
+ outbox.findStale.mockResolvedValue([row({ attempts: 1 })]);
+ outbox.countAttempt.mockResolvedValue(false);
+
+ const stats = await service.sweep(OPTS);
+
+ expect(delivery.redeliver).not.toHaveBeenCalled();
+ expect(outbox.close).not.toHaveBeenCalled();
+ expect(stats).toMatchObject({ scanned: 1, skipped: 1, replayed: 0, failed: 0 });
+ });
+
it('never stacks a second pass on top of a slow one', async () => {
let release: () => void = () => {};
outbox.findStale.mockImplementation(
diff --git a/src/modules/webhook/webhook-reconciler.service.ts b/src/modules/webhook/webhook-reconciler.service.ts
index b1360f184..8eb9b47ee 100644
--- a/src/modules/webhook/webhook-reconciler.service.ts
+++ b/src/modules/webhook/webhook-reconciler.service.ts
@@ -94,6 +94,13 @@ export class WebhookReconcilerService implements OnModuleInit, OnModuleDestroy {
const rows = await this.outbox.findStale(new Date(now.getTime() - opts.graceMs), opts.batchSize);
stats.scanned = rows.length;
for (const row of rows) {
+ if (this.delivery.isLocallyPending(row.idempotencyKey)) {
+ // Still owned by a dispatch on this node (parked in the limiter or mid retry loop), so it
+ // is slow rather than stranded. Replaying it would POST alongside the original and outside
+ // the dispatch concurrency bound, and would spend its budget while it is still running.
+ stats.skipped++;
+ continue;
+ }
if (row.attempts >= opts.maxAttempts) {
// Budget spent: stop replaying and leave the failure row as the recovery path.
await this.outbox.close(row.webhookId, row.idempotencyKey, 'failed');
@@ -108,7 +115,12 @@ export class WebhookReconcilerService implements OnModuleInit, OnModuleDestroy {
stats.skipped++;
continue;
}
- await this.outbox.countAttempt(row.id, row.attempts);
+ if (!(await this.outbox.countAttempt(row.id, row.attempts))) {
+ // Settled since the batch was read, typically a local dispatch that finished while an
+ // earlier row in this pass was replaying. The copy in hand is stale; replaying it duplicates.
+ stats.skipped++;
+ continue;
+ }
try {
// The outcome is a RETURN VALUE, not an exception. Every delivery failure is handled in
// place (dead-letter row, hook, log), so redeliver resolves either way and a catch here