Skip to content

Free, Open Source, Self-Hosted WhatsApp API Gateway

fix: stop background sweeps at shutdown (#1798)

Yudhi Armyndharis8f78149b9308
diff --git a/CHANGELOG.md b/CHANGELOG.md
index bdb84cb73..7da85bd39 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -24,6 +24,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
+- Stop in-flight webhook replay, ingress replay and pending-message sweeps at shutdown.
- Close queue workers before plugins shut down, so jobs taken during shutdown no longer spend attempts or get dead-lettered.
- Answer new HTTP requests with `503` once shutdown teardown begins.
- Refuse session starts with `503` once shutdown begins, and leave a start that shutdown interrupts for the next boot instead of marking it failed.
diff --git a/docs/06-api-specification.md b/docs/06-api-specification.md
index 2399d0a9e..7b6b15fed 100644
--- a/docs/06-api-specification.md
+++ b/docs/06-api-specification.md
@@ -7246,7 +7246,7 @@ Webhook delivery is **at-least-once**. A consumer can legitimately receive the s
**Crash boundary.** A successfully written outbox record retains the pre-hook event data until delivery succeeds, the subscription becomes ineligible, or the terminal failure is stored. Queued records keep this copy even after enqueue; the worker retires it only after one of those outcomes. If failure recording fails, the copy remains eligible for recovery. The bounded sweep (`WEBHOOK_RECONCILE_INTERVAL_MS`, default 60s) checks the stored job before replaying, leaves active jobs alone, and reuses the stored `X-OpenWA-Idempotency-Key`. If Redis cannot confirm the job state, replay waits. A missing or failed job can be replayed within `WEBHOOK_RECONCILE_MAX_ATTEMPTS`; exhausting that budget retires the outbox only after terminal failure recording succeeds. Receiver deduplication remains necessary because a crash after receiver processing can cause another POST.
-Outbox creation is best effort, and message persistence and outbox insertion are separate transactions. A write failure or crash between them can still lose the webhook delivery. Failure records with zero attempts are excluded from operator redrive. Capacity and shutdown rejections can still be recovered from an unsettled outbox record; shutdown during retry backoff may follow earlier POSTs. The direct dispatcher leaves its local in-flight work alone during reconciliation. Graceful shutdown drains current queue jobs; direct sends wait up to `WEBHOOK_SHUTDOWN_DRAIN_MS` (default 5s).
+Outbox creation is best effort, and message persistence and outbox insertion are separate transactions. A write failure or crash between them can still lose the webhook delivery. Failure records with zero attempts are excluded from operator redrive. Capacity and shutdown rejections can still be recovered from an unsettled outbox record; shutdown during retry backoff may follow earlier POSTs. The direct dispatcher leaves its local in-flight work alone during reconciliation. Graceful shutdown drains current queue jobs; direct sends wait up to `WEBHOOK_SHUTDOWN_DRAIN_MS` (default 5s). An outbox replay in flight at shutdown sends at most one more POST, bounded by `WEBHOOK_TIMEOUT`, and replays no further row; the next start replays what it left pending.
Before downgrading to a version that does not recognize the `queued` outbox state, drain or recover those records and take a backup. Its cleanup may otherwise treat an unsettled queued record as settled. Reverting the terminal-identity migration removes its index but does not restore historical duplicates.
diff --git a/src/modules/integration/ingress-reconciler.service.spec.ts b/src/modules/integration/ingress-reconciler.service.spec.ts
index 636a90690..cd147f067 100644
--- a/src/modules/integration/ingress-reconciler.service.spec.ts
+++ b/src/modules/integration/ingress-reconciler.service.spec.ts
@@ -52,7 +52,7 @@ describe('IngressReconcilerService.sweep', () => {
});
afterEach(async () => {
- service.onModuleDestroy();
+ await service.onModuleDestroy();
if (ds.isInitialized) await ds.destroy();
});
@@ -639,6 +639,41 @@ describe('IngressReconcilerService.sweep', () => {
expect(enqueue).toHaveBeenCalledTimes(2);
updateSpy.mockRestore();
});
+
+ // Plugin sandboxes stop right after this module is torn down: a replay landing later fails,
+ // spends an attempt, and dead-letters a row on its last one.
+ it('stops a running sweep on destroy and finishes the replay in hand before destroy resolves', async () => {
+ const first = await insertEvent({ createdAt: minutesAgo(6) });
+ const rest = [await insertEvent(), await insertEvent({ dispatchAttempts: 4 })];
+ let dispatching!: () => void;
+ const started = new Promise<void>(resolve => (dispatching = resolve));
+ let release!: () => void;
+ const held = new Promise<void>(resolve => (release = resolve));
+ enqueue.mockImplementationOnce(async () => {
+ dispatching();
+ await held;
+ return { outcome: 'queued' };
+ });
+ enqueue.mockResolvedValue({ outcome: 'failed', error: 'no live sandbox host' });
+
+ const sweep = service.sweep(OPTS);
+ await started;
+ const destroyed = service.onModuleDestroy();
+ release();
+ await destroyed;
+ const firstAtDestroy = (await stored(first)).dispatchState;
+ const stats = await sweep;
+
+ expect(firstAtDestroy).toBe('dispatched');
+ expect(stats).toMatchObject({ scanned: 1, replayed: 1, failed: 0 });
+ expect(enqueue).toHaveBeenCalledTimes(1);
+ for (const [i, id] of rest.entries()) {
+ const event = await stored(id);
+ expect(event.dispatchState).toBe('pending');
+ expect(event.dispatchAttempts).toBe([0, 4][i]);
+ }
+ expect(await failures.count()).toBe(0);
+ });
});
describe('IngressReconcilerService.onModuleInit (scheduling)', () => {
@@ -671,7 +706,7 @@ describe('IngressReconcilerService.onModuleInit (scheduling)', () => {
svc.onModuleInit();
jest.advanceTimersByTime(10 * 60_000);
expect(sweepSpy).not.toHaveBeenCalled();
- svc.onModuleDestroy();
+ void svc.onModuleDestroy();
} finally {
jest.useRealTimers();
}
@@ -694,7 +729,7 @@ describe('IngressReconcilerService.onModuleInit (scheduling)', () => {
svc.onModuleInit();
jest.advanceTimersByTime(60_000);
expect(sweepSpy).toHaveBeenCalledTimes(1);
- svc.onModuleDestroy();
+ void svc.onModuleDestroy();
} finally {
jest.useRealTimers();
}
@@ -717,7 +752,7 @@ describe('IngressReconcilerService.onModuleInit (scheduling)', () => {
svc.onModuleInit();
jest.advanceTimersByTime(60_000);
expect(sweepSpy).toHaveBeenCalledTimes(1);
- svc.onModuleDestroy();
+ void svc.onModuleDestroy();
sweepSpy.mockClear();
jest.advanceTimersByTime(60_000);
expect(sweepSpy).not.toHaveBeenCalled();
diff --git a/src/modules/integration/ingress-reconciler.service.ts b/src/modules/integration/ingress-reconciler.service.ts
index 25dd4bc54..343424eed 100644
--- a/src/modules/integration/ingress-reconciler.service.ts
+++ b/src/modules/integration/ingress-reconciler.service.ts
@@ -69,13 +69,16 @@ export interface IngressReconcileStats {
* (RedriveService), a dispatched event is the dispatch tier's concern.
*
* Mirrors IntegrationRetentionService's lifecycle: a raw unref'd setInterval started on module init
- * (first sweep after one interval, so plugin sandboxes have time to boot), cleared on destroy.
+ * (first sweep after one interval, so plugin sandboxes have time to boot), cleared on destroy, which
+ * also stops a pass in flight and waits for it.
*/
@Injectable()
export class IngressReconcilerService implements OnModuleInit, OnModuleDestroy {
private readonly logger = createLogger('IngressReconcilerService');
private timer?: ReturnType<typeof setInterval>;
- private sweeping = false;
+ // The pass in flight, settled when it ends; doubles as the overlap guard.
+ private inFlight?: Promise<void>;
+ private stopping = false;
constructor(
@InjectRepository(IngressEvent, 'data') private readonly events: Repository<IngressEvent>,
@@ -100,8 +103,16 @@ export class IngressReconcilerService implements OnModuleInit, OnModuleDestroy {
this.timer.unref?.();
}
- onModuleDestroy(): void {
+ /**
+ * Clearing the interval only stops the NEXT pass. A pass already running would go on replaying
+ * rows after PluginLoaderService has stopped the plugin sandboxes, so each remaining row fails,
+ * spends an attempt and, at its last one, is dead-lettered. Stop it at the next row and wait for
+ * the row in hand, while its plugin is still running.
+ */
+ async onModuleDestroy(): Promise<void> {
+ this.stopping = true;
if (this.timer) clearInterval(this.timer);
+ await this.inFlight;
}
/**
@@ -110,8 +121,9 @@ export class IngressReconcilerService implements OnModuleInit, OnModuleDestroy {
*/
async sweep(opts: IngressReconcilerOptions, now: Date = new Date()): Promise<IngressReconcileStats> {
const stats: IngressReconcileStats = { scanned: 0, replayed: 0, failed: 0, skipped: 0 };
- if (this.sweeping) return stats;
- this.sweeping = true;
+ if (this.inFlight || this.stopping) return stats;
+ let settle!: () => void;
+ this.inFlight = new Promise(resolve => (settle = resolve));
try {
const cutoff = new Date(now.getTime() - opts.graceMs);
// Only rows the sweep can act on may take a batch slot: a row of a disabled or deleted instance,
@@ -133,6 +145,7 @@ export class IngressReconcilerService implements OnModuleInit, OnModuleDestroy {
.limit(opts.batchSize)
.getMany();
for (const row of rows) {
+ if (this.stopping) break;
// A row whose latest attempt is still inside the grace window cools down between replays;
// it keeps its batch slot (bounded by maxAttempts, so the leak is capped) but is not hit again.
if (row.lastDispatchAt && row.lastDispatchAt > cutoff) {
@@ -187,7 +200,8 @@ export class IngressReconcilerService implements OnModuleInit, OnModuleDestroy {
}
return stats;
} finally {
- this.sweeping = false;
+ this.inFlight = undefined;
+ settle();
}
}
@@ -197,13 +211,15 @@ export class IngressReconcilerService implements OnModuleInit, OnModuleDestroy {
* delivery that was never dispatched: one stranded across downtime longer than the window, one of an
* instance disabled for that long, or any stranded row while the sweep is disabled. Same
* DLQ-before-retire order as the replay budget path. A row whose writes fail stays 'pending' with its
- * payload, which the prune keeps, and is retried on the next run. Skipped while a sweep is running.
+ * payload, which the prune keeps, and is retried on the next run. Skipped while a sweep is running,
+ * and stopped at the next row once shutdown begins, like a sweep.
* A row whose instance or session was deleted, or whose queue job still owns the delivery, gets no
* dead letter. Returns the number of rows dead-lettered.
*/
async deadLetterAgedPending(cutoff: Date, batchSize = 100): Promise<number> {
- if (this.sweeping) return 0;
- this.sweeping = true;
+ if (this.inFlight || this.stopping) return 0;
+ let settle!: () => void;
+ this.inFlight = new Promise(resolve => (settle = resolve));
let retired = 0;
try {
for (;;) {
@@ -214,6 +230,7 @@ export class IngressReconcilerService implements OnModuleInit, OnModuleDestroy {
});
let progressed = 0;
for (const row of rows) {
+ if (this.stopping) return retired;
if (!hasPayload(row)) continue;
const meta = { pluginId: row.pluginId, instanceId: row.instanceId, deliveryId: row.providerDeliveryId };
try {
@@ -287,7 +304,8 @@ export class IngressReconcilerService implements OnModuleInit, OnModuleDestroy {
if (rows.length < batchSize || progressed === 0) return retired;
}
} finally {
- this.sweeping = false;
+ this.inFlight = undefined;
+ settle();
}
}
diff --git a/src/modules/integration/integration-retention.service.spec.ts b/src/modules/integration/integration-retention.service.spec.ts
index 919cddd99..8e0085c67 100644
--- a/src/modules/integration/integration-retention.service.spec.ts
+++ b/src/modules/integration/integration-retention.service.spec.ts
@@ -645,7 +645,7 @@ describe('IntegrationRetentionService.pruneOlderThan with undispatched events',
it('keeps aged pending rows while a reconcile sweep holds the guard', async () => {
await insertEvent('stranded', {});
- (reconciler as unknown as { sweeping: boolean }).sweeping = true;
+ (reconciler as unknown as { inFlight?: Promise<void> }).inFlight = new Promise(() => undefined);
const result = await service.pruneOlderThan(7, 90);
@@ -653,4 +653,25 @@ describe('IntegrationRetentionService.pruneOlderThan with undispatched events',
expect((await events.findOneByOrFail({ id: 'stranded' })).payload).not.toBeNull();
expect(await failures.count()).toBe(0);
});
+
+ it('stops the hand-off at the next row once shutdown begins, and destroy waits for the row in hand', async () => {
+ await insertEvent('first', { createdAt: daysAgo(9) });
+ await insertEvent('second', {});
+ const realSave = failures.save.bind(failures) as (
+ row: IntegrationDeliveryFailure,
+ ) => Promise<IntegrationDeliveryFailure>;
+ let destroyed: Promise<void> | undefined;
+ jest.spyOn(failures, 'save').mockImplementationOnce(async row => {
+ destroyed = reconciler.onModuleDestroy();
+ return realSave(row as IntegrationDeliveryFailure);
+ });
+
+ await service.pruneOlderThan(7, 90);
+ await destroyed;
+
+ expect(await failures.count()).toBe(1);
+ const rows = await events.find({ order: { createdAt: 'ASC' } });
+ expect(rows.map(r => r.dispatchState)).toEqual(['failed', 'pending']);
+ expect(rows[1].payload).not.toBeNull();
+ });
});
diff --git a/src/modules/message/pending-message-reaper.service.spec.ts b/src/modules/message/pending-message-reaper.service.spec.ts
index d4071bf19..b8d1dc473 100644
--- a/src/modules/message/pending-message-reaper.service.spec.ts
+++ b/src/modules/message/pending-message-reaper.service.spec.ts
@@ -34,7 +34,7 @@ describe('PendingMessageReaperService.sweep', () => {
});
afterEach(async () => {
- service.onModuleDestroy();
+ await service.onModuleDestroy();
if (ds.isInitialized) await ds.destroy();
});
@@ -191,6 +191,42 @@ describe('PendingMessageReaperService.sweep', () => {
expect(row.waMessageId).toBe('wamid.race');
expect(persistedCalls()).toHaveLength(0);
});
+
+ // Plugins unregister their hooks and the DataSource closes right after this module is torn down,
+ // so a reap landing later would leave a provider's copy PENDING for good.
+ it('stops a running pass on destroy and finishes the row in hand before destroy resolves', async () => {
+ const ids: string[] = [];
+ for (const age of [4, 3, 2]) ids.push(await insertMessage({ createdAt: hoursAgo(age) }));
+ const realUpdate = messages.update.bind(messages);
+ let reaping!: () => void;
+ const started = new Promise<void>(resolve => (reaping = resolve));
+ let release!: () => void;
+ const held = new Promise<void>(resolve => (release = resolve));
+ const updateSpy = jest.spyOn(messages, 'update').mockImplementationOnce(async (criteria, partial) => {
+ reaping();
+ await held;
+ return realUpdate(criteria, partial);
+ });
+
+ const sweep = service.sweep(OPTS);
+ await started;
+ const destroyed = service.onModuleDestroy();
+ release();
+ await destroyed;
+ const writesAtDestroy = updateSpy.mock.calls.length;
+ const emissionsAtDestroy = persistedCalls().length;
+ const stats = await sweep;
+
+ expect(stats).toEqual({ scanned: 1, reaped: 1, failed: 0 });
+ expect(writesAtDestroy).toBe(1);
+ expect(emissionsAtDestroy).toBe(1);
+ expect(updateSpy).toHaveBeenCalledTimes(1);
+ expect(persistedCalls()).toHaveLength(1);
+ expect((await stored(ids[0])).status).toBe(MessageStatus.FAILED);
+ for (const id of ids.slice(1)) expect((await stored(id)).status).toBe(MessageStatus.PENDING);
+ // A stopped reaper starts no further pass.
+ expect(await service.sweep(OPTS)).toEqual({ scanned: 0, reaped: 0, failed: 0 });
+ });
});
describe('resolvePendingMessageReaperOptions', () => {
@@ -268,7 +304,7 @@ describe('PendingMessageReaperService.onModuleInit (scheduling)', () => {
svc.onModuleInit();
jest.advanceTimersByTime(10 * 600_000);
expect(sweepSpy).not.toHaveBeenCalled();
- svc.onModuleDestroy();
+ void svc.onModuleDestroy();
} finally {
jest.useRealTimers();
}
@@ -284,7 +320,7 @@ describe('PendingMessageReaperService.onModuleInit (scheduling)', () => {
svc.onModuleInit();
jest.advanceTimersByTime(60_000);
expect(sweepSpy).toHaveBeenCalledTimes(1);
- svc.onModuleDestroy();
+ void svc.onModuleDestroy();
sweepSpy.mockClear();
jest.advanceTimersByTime(60_000);
expect(sweepSpy).not.toHaveBeenCalled();
diff --git a/src/modules/message/pending-message-reaper.service.ts b/src/modules/message/pending-message-reaper.service.ts
index e4ab33c06..09fe3ea86 100644
--- a/src/modules/message/pending-message-reaper.service.ts
+++ b/src/modules/message/pending-message-reaper.service.ts
@@ -46,14 +46,16 @@ export interface PendingMessageReaperStats {
* state instead of staying stuck at the initial PENDING write.
*
* Mirrors IngressReconcilerService's lifecycle: a raw unref'd setInterval started on module init
- * (first sweep after one interval), cleared on destroy. Young PENDING rows (a send still in flight)
- * and non-outgoing rows are never touched.
+ * (first sweep after one interval), cleared on destroy, which also stops a pass in flight and waits
+ * for it. Young PENDING rows (a send still in flight) and non-outgoing rows are never touched.
*/
@Injectable()
export class PendingMessageReaperService implements OnModuleInit, OnModuleDestroy {
private readonly logger = createLogger('PendingMessageReaperService');
private timer?: ReturnType<typeof setInterval>;
- private sweeping = false;
+ // The pass in flight, settled when it ends; doubles as the overlap guard.
+ private inFlight?: Promise<void>;
+ private stopping = false;
constructor(
@InjectRepository(Message, 'data') private readonly messages: Repository<Message>,
@@ -74,8 +76,16 @@ export class PendingMessageReaperService implements OnModuleInit, OnModuleDestro
this.timer.unref?.();
}
- onModuleDestroy(): void {
+ /**
+ * Clearing the interval only stops the NEXT pass. A pass already running would go on writing
+ * FAILED rows while plugins unregister their hooks and the DataSource closes, so its re-emitted
+ * `message:persisted` reaches no handler and the provider's copy stays PENDING for good (the
+ * reaper never revisits a FAILED row). Stop it at the next row and wait for the row in hand.
+ */
+ async onModuleDestroy(): Promise<void> {
+ this.stopping = true;
if (this.timer) clearInterval(this.timer);
+ await this.inFlight;
}
/**
@@ -85,8 +95,9 @@ export class PendingMessageReaperService implements OnModuleInit, OnModuleDestro
*/
async sweep(opts: PendingMessageReaperOptions, now: Date = new Date()): Promise<PendingMessageReaperStats> {
const stats: PendingMessageReaperStats = { scanned: 0, reaped: 0, failed: 0 };
- if (this.sweeping) return stats;
- this.sweeping = true;
+ if (this.inFlight || this.stopping) return stats;
+ let settle!: () => void;
+ this.inFlight = new Promise(resolve => (settle = resolve));
try {
const cutoff = new Date(now.getTime() - opts.graceMs);
const rows = await this.messages.find({
@@ -99,6 +110,7 @@ export class PendingMessageReaperService implements OnModuleInit, OnModuleDestro
take: opts.batchSize,
});
for (const row of rows) {
+ if (this.stopping) break;
stats.scanned++;
try {
if (await this.reapRow(row, now)) {
@@ -122,7 +134,8 @@ export class PendingMessageReaperService implements OnModuleInit, OnModuleDestro
}
return stats;
} finally {
- this.sweeping = false;
+ this.inFlight = undefined;
+ settle();
}
}
diff --git a/src/modules/webhook/webhook-delivery.service.ts b/src/modules/webhook/webhook-delivery.service.ts
index 7cafdb162..fb21ef827 100644
--- a/src/modules/webhook/webhook-delivery.service.ts
+++ b/src/modules/webhook/webhook-delivery.service.ts
@@ -93,6 +93,11 @@ interface DispatchEventContext {
* which runs inside an HTTP request: the in-process backoff schedule would hold it for minutes.
*/
singleAttempt?: boolean;
+ /**
+ * Ends a replay's retry backoff early and gives no further attempt. A replay holds no dispatch
+ * slot, so the shutdown drain's limiter close cannot stop it; its owner aborts this instead.
+ */
+ signal?: AbortSignal;
}
/** Per-delivery options of the direct path (deliverWebhook), derived from the dispatch context. */
@@ -101,6 +106,7 @@ interface DirectDeliveryOptions {
/** Kept on the terminal failure row for a redrive; set only while payload retention is on. */
replayData?: Record<string, unknown>;
singleAttempt?: boolean;
+ signal?: AbortSignal;
}
class UnrecordedWebhookFailure extends Error {}
@@ -109,6 +115,15 @@ class UnrecordedWebhookFailure extends Error {}
const isLimiterClosed = (error: unknown): boolean =>
error instanceof Error && error.message === 'ConcurrencyLimiter closed';
+/**
+ * The replay's retries were stopped by its aborted signal, in a backoff or just before a retry POST.
+ * A POST that fails after the abort is not this: on the last attempt it is a terminal failure and is
+ * reported as one.
+ */
+export const isAbortedBackoff = (error: unknown, signal: AbortSignal | undefined): boolean =>
+ // Matched by name: the AbortError comes from node:timers, so instanceof Error is realm-dependent.
+ signal?.aborted === true && (error as Error | undefined)?.name === 'AbortError';
+
const isPlainObject = (value: unknown): boolean => typeof value === 'object' && value !== null && !Array.isArray(value);
/**
@@ -516,7 +531,8 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {
*
* Every failure path here is already handled in place (a dead-letter row, a hook, a log), so none
* of them reach the caller as an exception, except 'ConcurrencyLimiter closed' from a retry backoff
- * woken after shutdown, which runLimited handles (redeliver passes no yieldSlot and never sees it).
+ * woken after shutdown, which runLimited handles (redeliver passes no yieldSlot and never sees it),
+ * and the AbortError of a replay whose signal stopped its retries, which reaches its caller.
* The reconciler has to tell a delivered event from a
* dead-lettered one to know whether the outbox row may be retired, and a caught throw cannot tell
* it: there is none. This mirrors the inbound twin, where `ingressEnqueue.enqueue` returns an
@@ -638,9 +654,9 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {
{ sessionId, source: 'WebhookService' },
);
} catch (fallbackError) {
- // Shutdown closed the limiter during a retry backoff: not a delivery failure. runLimited
- // records it and keeps the outbox row pending for the next start.
- if (isLimiterClosed(fallbackError)) throw fallbackError;
+ // Shutdown closed the limiter or aborted the replay during a retry backoff: not a delivery
+ // failure. runLimited or the replay's caller keeps the outbox row pending for the next start.
+ if (isLimiterClosed(fallbackError) || isAbortedBackoff(fallbackError, ctx.signal)) throw fallbackError;
await this.hookManager.execute(
'webhook:error',
{
@@ -692,9 +708,9 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {
{ sessionId, source: 'WebhookService' },
);
} catch (error) {
- // Shutdown closed the limiter during a retry backoff: not a delivery failure. runLimited
- // records it and keeps the outbox row pending for the next start.
- if (isLimiterClosed(error)) throw error;
+ // Shutdown closed the limiter or aborted the replay during a retry backoff: not a delivery
+ // failure. runLimited or the replay's caller keeps the outbox row pending for the next start.
+ if (isLimiterClosed(error) || isAbortedBackoff(error, ctx.signal)) throw error;
// Execute hook on error
await this.hookManager.execute(
'webhook:error',
@@ -922,7 +938,7 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {
event: string,
idempotencyKey: string,
data: Record<string, unknown>,
- options: { singleAttempt?: boolean } = {},
+ options: { singleAttempt?: boolean; signal?: AbortSignal } = {},
): Promise<WebhookDeliveryOutcome> {
const deliveryId = generateDeliveryId();
const outcome = await this.deliverOne(webhook, deliveryId, idempotencyKey, {
@@ -930,6 +946,7 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {
event,
baseData: data,
singleAttempt: options.singleAttempt,
+ signal: options.signal,
});
// The receiver has the event. An operator redrive of a failure row can find the same delivery
// still pending in the outbox, which the next sweep would otherwise POST again.
@@ -984,6 +1001,7 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {
yieldSlot: ctx.yieldSlot,
replayData: this.keepsFailurePayload() ? ctx.baseData : undefined,
singleAttempt: ctx.singleAttempt,
+ signal: ctx.signal,
};
}
@@ -1031,6 +1049,8 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {
body,
attempt - 1,
);
+ // A retry gives way once its replay's owner has stopped, even when the backoff ended first.
+ if (attempt > 1) options.signal?.throwIfAborted();
await postWebhookPayload(current.url, body, headers, this.configService.get<number>('webhook.timeout', 10000));
this.failingWebhooks.delete(webhook.id);
@@ -1060,6 +1080,7 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {
});
return true;
} catch (error) {
+ if (isAbortedBackoff(error, options.signal)) throw error;
this.failingWebhooks.add(webhook.id);
this.logger.error(`Webhook delivery failed for ${webhook.id}`, String(error), {
webhookId: webhook.id,
@@ -1072,7 +1093,8 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {
// Without the dispatch slot: a backoff sends nothing, and holding the slot through it let
// a few failing receivers stall every other delivery. Taking it back throws once shutdown
// closed the limiter, which skips the terminal record below: the event was not given up.
- await yieldSlot(() => setTimeout(delay * 2 ** (attempt - 1)));
+ // An aborted replay signal ends the backoff the same way.
+ await yieldSlot(() => setTimeout(delay * 2 ** (attempt - 1), undefined, { signal: options.signal }));
continue;
}
// All direct-path retries exhausted — persist a durable failure record before giving up, mirroring
diff --git a/src/modules/webhook/webhook-reconciler.service.spec.ts b/src/modules/webhook/webhook-reconciler.service.spec.ts
index d671ed63f..bb42aa3d5 100644
--- a/src/modules/webhook/webhook-reconciler.service.spec.ts
+++ b/src/modules/webhook/webhook-reconciler.service.spec.ts
@@ -77,6 +77,7 @@ describe('WebhookReconcilerService', () => {
'message.received',
'stored-key_wh-1',
{ from: '[email protected]' },
+ { signal: expect.any(AbortSignal) as AbortSignal },
);
expect(outbox.close).toHaveBeenCalledWith('wh-1', 'stored-key_wh-1', 'dispatched');
expect(stats).toMatchObject({ scanned: 1, replayed: 1 });
diff --git a/src/modules/webhook/webhook-reconciler.service.ts b/src/modules/webhook/webhook-reconciler.service.ts
index f4a9729dc..5b2d0f323 100644
--- a/src/modules/webhook/webhook-reconciler.service.ts
+++ b/src/modules/webhook/webhook-reconciler.service.ts
@@ -3,7 +3,7 @@ import { InjectRepository } from '@nestjs/typeorm';
import { Repository } from 'typeorm';
import { Webhook } from './entities/webhook.entity';
import { WebhookOutboxService } from './webhook-outbox.service';
-import { WebhookDeliveryService } from './webhook-delivery.service';
+import { isAbortedBackoff, WebhookDeliveryService } from './webhook-delivery.service';
import { isDeliverableWebhook } from './utils/deliver-once';
import { createLogger } from '../../common/services/logger.service';
import { resolveNonNegativeIntEnv } from '../../config/configuration';
@@ -60,7 +60,11 @@ export interface WebhookReconcileStats {
export class WebhookReconcilerService implements OnModuleInit, OnModuleDestroy {
private readonly logger = createLogger('WebhookReconcilerService');
private timer?: ReturnType<typeof setInterval>;
- private sweeping = false;
+ // The pass in flight, settled when it ends; doubles as the overlap guard.
+ private inFlight?: Promise<void>;
+ // Aborted on destroy: stops the pass at the next row and the replay in hand before its next retry.
+ // A row already past its last check, or a retry POST already started, still sends that one POST.
+ private readonly stop = new AbortController();
private cursor?: string;
constructor(
@@ -83,15 +87,24 @@ export class WebhookReconcilerService implements OnModuleInit, OnModuleDestroy {
this.timer.unref?.();
}
- onModuleDestroy(): void {
+ /**
+ * Clearing the interval only stops the NEXT pass. A replay holds no dispatch slot, so the delivery
+ * drain neither waits for nor stops a pass already running: it would go on POSTing after the drain,
+ * and after PluginLoaderService has unregistered the `webhook:before` hooks. Stop it at the next row,
+ * stop the replay in hand without a further retry, and wait for it.
+ */
+ async onModuleDestroy(): Promise<void> {
+ this.stop.abort();
if (this.timer) clearInterval(this.timer);
+ await this.inFlight;
}
/** One bounded pass over the stranded backlog. Overlap-guarded. */
async sweep(opts: WebhookReconcilerOptions, now: Date = new Date()): Promise<WebhookReconcileStats> {
const stats: WebhookReconcileStats = { scanned: 0, replayed: 0, failed: 0, skipped: 0 };
- if (this.sweeping) return stats;
- this.sweeping = true;
+ if (this.inFlight || this.stop.signal.aborted) return stats;
+ let settle!: () => void;
+ this.inFlight = new Promise(resolve => (settle = resolve));
try {
const cutoff = new Date(now.getTime() - opts.graceMs);
let rows = await this.outbox.findStale(cutoff, opts.batchSize, this.cursor);
@@ -103,6 +116,7 @@ export class WebhookReconcilerService implements OnModuleInit, OnModuleDestroy {
this.cursor = rows.at(-1)?.id;
stats.scanned = rows.length;
for (const row of rows) {
+ if (this.stop.signal.aborted) break;
if (this.delivery.isLocallyPending(row.idempotencyKey)) {
// Still owned by a dispatch on this node (parked in the limiter or mid retry loop), so it
// is slow rather than stranded. Replaying it would POST alongside the original and outside
@@ -132,6 +146,8 @@ export class WebhookReconcilerService implements OnModuleInit, OnModuleDestroy {
stats.failed++;
continue;
}
+ // Shutdown may have begun during the lookups above; stop before spending budget or POSTing.
+ if (this.stop.signal.aborted) break;
if (!(await this.outbox.countAttempt(row.id, row.attempts))) {
// Settled since the batch was read, typically a local dispatch that finished while an
// earlier row in this pass was replaying. The copy in hand is stale; replaying it duplicates.
@@ -149,6 +165,7 @@ export class WebhookReconcilerService implements OnModuleInit, OnModuleDestroy {
row.event,
row.idempotencyKey,
row.payload,
+ { signal: this.stop.signal },
);
if (outcome === 'failed' || outcome === 'unrecorded') {
// Left 'pending' on purpose: the next sweep retries it until the budget is spent.
@@ -161,14 +178,20 @@ export class WebhookReconcilerService implements OnModuleInit, OnModuleDestroy {
if (outcome !== 'enqueued') await this.outbox.close(row.webhookId, row.idempotencyKey, 'dispatched');
stats.replayed++;
} catch (error) {
- // An exception is an unexpected fault rather than a delivery failure; the row stays
- // pending either way.
+ // A replay cut short in its retry backoff by shutdown was interrupted, not failed. Any other
+ // exception is an unexpected fault rather than a delivery failure. The row stays pending.
+ if (isAbortedBackoff(error, this.stop.signal)) {
+ this.logger.log(`Replay of ${row.event} to webhook ${row.webhookId} interrupted by shutdown`);
+ stats.skipped++;
+ continue;
+ }
this.logger.warn(`Replay of ${row.event} to webhook ${row.webhookId} failed: ${String(error)}`);
stats.failed++;
}
}
} finally {
- this.sweeping = false;
+ this.inFlight = undefined;
+ settle();
}
return stats;
}
diff --git a/src/modules/webhook/webhook-recovery-boundaries.spec.ts b/src/modules/webhook/webhook-recovery-boundaries.spec.ts
index c9ed09035..ee53c7874 100644
--- a/src/modules/webhook/webhook-recovery-boundaries.spec.ts
+++ b/src/modules/webhook/webhook-recovery-boundaries.spec.ts
@@ -214,4 +214,192 @@ describe('webhook recovery across live ownership and configuration changes', ()
expect(post).not.toHaveBeenCalled();
expect(await failures.count()).toBe(0);
});
+
+ // A replay holds no dispatch slot, so the delivery drain neither waits for nor stops it. Destroying
+ // the reconciler must, or the pass goes on POSTing after the drain and after plugin teardown.
+ it('stops a running sweep on destroy without POSTing a retry or another row after it', async () => {
+ await webhooks.update(primary.id, { retryCount: 3 });
+ const repository = ds.getRepository(WebhookOutboxEvent);
+ const outbox = new WebhookOutboxService(repository);
+ for (const [i, key] of ['key-a', 'key-b'].entries()) {
+ await outbox.open({
+ webhookId: primary.id,
+ sessionId: primary.sessionId,
+ event: 'message.received',
+ idempotencyKey: key,
+ deliveryId: randomUUID(),
+ payload: { id: i },
+ });
+ await repository.update({ idempotencyKey: key }, { createdAt: new Date(Date.now() - 600_000 + i) });
+ }
+ // key-b is over budget: a pass that went on past the stop would write its terminal failure row.
+ await repository.update({ idempotencyKey: 'key-b' }, { attempts: 5 });
+ // A backoff far beyond the test timeout: destroy only resolves if the stop cuts the running backoff short.
+ const config = new ConfigService({ webhook: { retryDelay: 600_000, timeout: 1000 } });
+ const hooks = new HookManager();
+ const execute = jest.spyOn(hooks, 'execute');
+ const delivery = new WebhookDeliveryService(webhooks, failures, config, hooks, outbox);
+ const reconciler = new WebhookReconcilerService(webhooks, outbox, delivery);
+ let posted!: () => void;
+ const firstPost = new Promise<void>(resolve => (posted = resolve));
+ const post = jest.spyOn(ssrf, 'withSafeFetch').mockImplementation((_url, _init, use) => {
+ posted();
+ return Promise.resolve(use(new Response(null, { status: 503 })));
+ });
+
+ const sweep = reconciler.sweep({ intervalMs: 1000, graceMs: 0, batchSize: 10, maxAttempts: 5 });
+ await firstPost;
+ // Let the 503 settle so the retry backoff timer is already running when the stop lands.
+ await new Promise(resolve => setImmediate(resolve));
+ await reconciler.onModuleDestroy();
+ const postsAtDestroy = post.mock.calls.length;
+ const stats = await sweep;
+
+ expect(postsAtDestroy).toBe(1);
+ expect(post).toHaveBeenCalledTimes(1);
+ // An interruption, not a delivery fault: counted as skipped, not failed.
+ expect(stats).toMatchObject({ replayed: 0, failed: 0, skipped: 1 });
+ const rows = await repository.find({ order: { idempotencyKey: 'ASC' } });
+ expect(rows.map(row => [row.idempotencyKey, row.state, row.attempts])).toEqual([
+ ['key-a', 'pending', 1],
+ ['key-b', 'pending', 5],
+ ]);
+ // Interrupted, not given up: no terminal failure row or error hook, and both stay pending for the next start.
+ expect(await failures.count()).toBe(0);
+ expect(execute.mock.calls.filter(([name]) => name === 'webhook:error')).toHaveLength(0);
+ });
+
+ it('does not replay a row whose pre-checks were under way when destroy began', async () => {
+ const repository = ds.getRepository(WebhookOutboxEvent);
+ const outbox = new WebhookOutboxService(repository);
+ for (const [i, key] of ['key-a', 'key-b'].entries()) {
+ await outbox.open({
+ webhookId: primary.id,
+ sessionId: primary.sessionId,
+ event: 'message.received',
+ idempotencyKey: key,
+ deliveryId: randomUUID(),
+ payload: { id: i },
+ });
+ await repository.update({ idempotencyKey: key }, { createdAt: new Date(Date.now() - 600_000 + i) });
+ }
+ const config = new ConfigService({ webhook: { retryDelay: 300, timeout: 1000 } });
+ const delivery = new WebhookDeliveryService(webhooks, failures, config, new HookManager(), outbox);
+ const reconciler = new WebhookReconcilerService(webhooks, outbox, delivery);
+ const post = jest
+ .spyOn(ssrf, 'withSafeFetch')
+ .mockImplementation((_url, _init, use) => Promise.resolve(use(new Response(null, { status: 200 }))));
+ let reached!: () => void;
+ const secondCheck = new Promise<void>(resolve => (reached = resolve));
+ let release!: () => void;
+ const held = new Promise<void>(resolve => (release = resolve));
+ jest
+ .spyOn(delivery, 'isQueueJobPending')
+ .mockImplementationOnce(() => Promise.resolve(false))
+ .mockImplementationOnce(async () => {
+ reached();
+ await held;
+ return false;
+ });
+
+ const sweep = reconciler.sweep({ intervalMs: 1000, graceMs: 0, batchSize: 10, maxAttempts: 5 });
+ await secondCheck;
+ const destroyed = reconciler.onModuleDestroy();
+ release();
+ await destroyed;
+ await sweep;
+
+ expect(post).toHaveBeenCalledTimes(1);
+ expect(await repository.findOneByOrFail({ idempotencyKey: 'key-b' })).toMatchObject({
+ state: 'pending',
+ attempts: 0,
+ });
+ });
+
+ it('does not POST a retry whose backoff ended before the stop', async () => {
+ await webhooks.update(primary.id, { retryCount: 2 });
+ const repository = ds.getRepository(WebhookOutboxEvent);
+ const outbox = new WebhookOutboxService(repository);
+ await outbox.open({
+ webhookId: primary.id,
+ sessionId: primary.sessionId,
+ event: 'message.received',
+ idempotencyKey: 'key-a',
+ deliveryId: randomUUID(),
+ payload: { id: 1 },
+ });
+ await repository.update({ idempotencyKey: 'key-a' }, { createdAt: new Date(Date.now() - 600_000) });
+ const hooks = new HookManager();
+ const execute = jest.spyOn(hooks, 'execute');
+ const config = new ConfigService({ webhook: { retryDelay: 1, timeout: 1000 } });
+ const delivery = new WebhookDeliveryService(webhooks, failures, config, hooks, outbox);
+ const reconciler = new WebhookReconcilerService(webhooks, outbox, delivery);
+ const post = jest
+ .spyOn(ssrf, 'withSafeFetch')
+ .mockImplementation((_url, _init, use) => Promise.resolve(use(new Response(null, { status: 503 }))));
+ // The stop lands while the retry re-reads its webhook, after the backoff has already ended.
+ const findOne = webhooks.findOne.bind(webhooks);
+ let destroyed: Promise<void> | undefined;
+ jest.spyOn(webhooks, 'findOne').mockImplementation(options => {
+ if (post.mock.calls.length === 1) destroyed ??= reconciler.onModuleDestroy();
+ return findOne(options);
+ });
+
+ const stats = await reconciler.sweep({ intervalMs: 1000, graceMs: 0, batchSize: 10, maxAttempts: 5 });
+ await destroyed;
+
+ expect(destroyed).toBeDefined();
+ expect(post).toHaveBeenCalledTimes(1);
+ expect(stats).toMatchObject({ replayed: 0, failed: 0, skipped: 1 });
+ expect(await failures.count()).toBe(0);
+ expect(execute.mock.calls.filter(([name]) => name === 'webhook:error')).toHaveLength(0);
+ expect(await repository.findOneByOrFail({ idempotencyKey: 'key-a' })).toMatchObject({
+ state: 'pending',
+ attempts: 1,
+ });
+ });
+
+ it('reports a last-attempt replay failure that lands after the stop as a terminal failure', async () => {
+ const repository = ds.getRepository(WebhookOutboxEvent);
+ const outbox = new WebhookOutboxService(repository);
+ await outbox.open({
+ webhookId: primary.id,
+ sessionId: primary.sessionId,
+ event: 'message.received',
+ idempotencyKey: 'key-a',
+ deliveryId: randomUUID(),
+ payload: { id: 1 },
+ });
+ await repository.update({ idempotencyKey: 'key-a' }, { createdAt: new Date(Date.now() - 600_000) });
+ const hooks = new HookManager();
+ const execute = jest.spyOn(hooks, 'execute');
+ const config = new ConfigService({ webhook: { retryDelay: 300, timeout: 1000 } });
+ const delivery = new WebhookDeliveryService(webhooks, failures, config, hooks, outbox);
+ const reconciler = new WebhookReconcilerService(webhooks, outbox, delivery);
+ let posted!: () => void;
+ const firstPost = new Promise<void>(resolve => (posted = resolve));
+ let fail!: () => void;
+ const held = new Promise<void>(resolve => (fail = resolve));
+ jest.spyOn(ssrf, 'withSafeFetch').mockImplementation(async (_url, _init, use) => {
+ posted();
+ await held;
+ return use(new Response(null, { status: 503 }));
+ });
+
+ const sweep = reconciler.sweep({ intervalMs: 1000, graceMs: 0, batchSize: 10, maxAttempts: 5 });
+ await firstPost;
+ const destroyed = reconciler.onModuleDestroy();
+ fail();
+ await destroyed;
+
+ // retryCount 1: the POST in flight at the stop was the only attempt, so its failure is final, and
+ // destroy waits for it to be recorded rather than leaving it to land after teardown.
+ expect(await failures.count()).toBe(1);
+ expect(execute.mock.calls.filter(([name]) => name === 'webhook:error')).toHaveLength(1);
+ expect(await sweep).toMatchObject({ replayed: 0, failed: 1 });
+ expect(await repository.findOneByOrFail({ idempotencyKey: 'key-a' })).toMatchObject({
+ state: 'pending',
+ attempts: 1,
+ });
+ });
});