Skip to content

Free, Open Source, Self-Hosted WhatsApp API Gateway

fix(shutdown): bound the reconciler wait for the pass in flight (#1835)

Yudhi Armyndharisabe088f15846
diff --git a/.env.example b/.env.example
index 9004d9fc5..5c4b2d68e 100644
--- a/.env.example
+++ b/.env.example
@@ -455,7 +455,8 @@ WEBHOOK_RETRY_DELAY=5000 # Base delay between retries in ms (doubles
# 4/3 x WEBHOOK_MEDIA_INLINE_MAX_BYTES as base64), up to about
# 1.3 GiB per process at the defaults; QUEUE_ENABLED=true keeps
# it in Redis instead.
-# WEBHOOK_SHUTDOWN_DRAIN_MS=5000 # on shutdown, in-flight deliveries get this long to finish
+# WEBHOOK_SHUTDOWN_DRAIN_MS=5000 # on shutdown, in-flight deliveries and an outbox replay in
+ # flight get this long to finish
# before they are logged abandoned (parked ones are
# dead-lettered). Startup warns when this is shorter than
# WEBHOOK_TIMEOUT — that cross silently truncates deliveries.
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 41d1a4341..c49616290 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -41,9 +41,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Stop the dashboard Chats page from marking the open chat read on a newly selected session.
- Refuse bulk batches once shutdown begins, and fail a batch saved during shutdown instead of sending it.
- Disable plugins whose enable is still running at shutdown, and refuse new plugin enables during teardown.
-- Stop in-flight webhook replay, ingress replay and pending-message sweeps at shutdown.
+- Stop in-flight webhook replay, ingress replay and pending-message sweeps at shutdown, waiting a bounded time for a replay in hand.
- Close queue workers, waiting up to 15 s for running jobs, before plugins shut down, so no new job is taken against stopped plugins.
-- Answer new HTTP requests with `503` once teardown begins after SIGTERM, SIGINT or `POST /api/infra/restart`.
+- Answer new HTTP requests other than health probes with `503` once teardown begins after SIGTERM, SIGINT or `POST /api/infra/restart`.
- Refuse session starts with `503` once shutdown begins, and leave a start that shutdown interrupts for the next boot instead of marking it failed.
- Stop dashboard multi-page loads and a pending chat mark-as-read from sending requests after logout.
- Keep a new dashboard login signed in when a request sent with the previous API key fails afterwards.
diff --git a/docs/06-api-specification.md b/docs/06-api-specification.md
index 9a6ec2d0d..01a570808 100644
--- a/docs/06-api-specification.md
+++ b/docs/06-api-specification.md
@@ -7246,7 +7246,7 @@ Webhook delivery is **at-least-once**. A consumer can legitimately receive the s
**Crash boundary.** A successfully written outbox record retains the pre-hook event data until delivery succeeds, the subscription becomes ineligible, or the terminal failure is stored. Queued records keep this copy even after enqueue; the worker retires it only after one of those outcomes. If failure recording fails, the copy remains eligible for recovery. The bounded sweep (`WEBHOOK_RECONCILE_INTERVAL_MS`, default 60s) checks the stored job before replaying, leaves active jobs alone, and reuses the stored `X-OpenWA-Idempotency-Key`. If Redis cannot confirm the job state, replay waits. A missing or failed job can be replayed within `WEBHOOK_RECONCILE_MAX_ATTEMPTS`; exhausting that budget retires the outbox only after terminal failure recording succeeds. Receiver deduplication remains necessary because a crash after receiver processing can cause another POST.
-Outbox creation is best effort, and message persistence and outbox insertion are separate transactions. A write failure or crash between them can still lose the webhook delivery. Failure records with zero attempts are excluded from operator redrive. Capacity and shutdown rejections can still be recovered from an unsettled outbox record; shutdown during retry backoff may follow earlier POSTs. The direct dispatcher leaves its local in-flight work alone during reconciliation. Graceful shutdown drains current queue jobs; direct sends wait up to `WEBHOOK_SHUTDOWN_DRAIN_MS` (default 5s). An outbox replay in flight at shutdown sends at most one more POST, bounded by `WEBHOOK_TIMEOUT`, and replays no further row; the next start replays what it left pending.
+Outbox creation is best effort, and message persistence and outbox insertion are separate transactions. A write failure or crash between them can still lose the webhook delivery. Failure records with zero attempts are excluded from operator redrive. Capacity and shutdown rejections can still be recovered from an unsettled outbox record; shutdown during retry backoff may follow earlier POSTs. The direct dispatcher leaves its local in-flight work alone during reconciliation. Graceful shutdown drains current queue jobs; direct sends, and an outbox replay in flight with its plugin hooks, get up to `WEBHOOK_SHUTDOWN_DRAIN_MS` (default 5s). That replay sends at most one more POST, bounded by `WEBHOOK_TIMEOUT`, and no further row is replayed; the next start replays what it left pending.
Before downgrading to a version that does not recognize the `queued` outbox state, drain or recover those records and take a backup. Its cleanup may otherwise treat an unsettled queued record as settled. Reverting the terminal-identity migration removes its index but does not restore historical duplicates.
diff --git a/src/modules/integration/ingress-reconciler.service.spec.ts b/src/modules/integration/ingress-reconciler.service.spec.ts
index cd147f067..b4144b157 100644
--- a/src/modules/integration/ingress-reconciler.service.spec.ts
+++ b/src/modules/integration/ingress-reconciler.service.spec.ts
@@ -7,6 +7,7 @@ import { IngressEnqueueService, sanitizeIngressJobId } from './ingress-enqueue.s
import { PluginInstanceService } from './plugin-instance.service';
import { PluginLoaderService } from '../../core/plugins/plugin-loader.service';
import { RedriveService } from './redrive.service';
+import { INGRESS_DISPATCH_TIMEOUT_MS } from './integration.constants';
import { IngressJobData } from '../queue/processors/ingress.processor';
import { ConfigService } from '@nestjs/config';
@@ -674,6 +675,87 @@ describe('IngressReconcilerService.sweep', () => {
}
expect(await failures.count()).toBe(0);
});
+
+ // The queue add in hand can be refused once teardown closes its connection, and the inline fallback
+ // then runs against plugins being stopped: neither failure may spend the row's last attempt.
+ it('spends no attempt on a replay that fails after destroy began', async () => {
+ const id = await insertEvent({ dispatchAttempts: 4 });
+ let dispatching!: () => void;
+ const started = new Promise<void>(resolve => (dispatching = resolve));
+ let release!: () => void;
+ const held = new Promise<void>(resolve => (release = resolve));
+ enqueue.mockImplementationOnce(async () => {
+ dispatching();
+ await held;
+ return { outcome: 'failed', error: 'no live sandbox host' };
+ });
+
+ const sweep = service.sweep(OPTS);
+ await started;
+ const destroyed = service.onModuleDestroy();
+ release();
+ await destroyed;
+ const stats = await sweep;
+
+ expect(stats).toMatchObject({ scanned: 1, replayed: 0, failed: 0, skipped: 1 });
+ expect(await stored(id)).toMatchObject({ dispatchState: 'pending', dispatchAttempts: 4 });
+ expect((await stored(id)).payload).not.toBeNull();
+ expect(await failures.count()).toBe(0);
+ });
+
+ // A job-state lookup made while Redis has never been reachable only returns once the queue connection
+ // is closed later in shutdown. Waiting for it kept every later destroy hook from running; dispatching
+ // once it returns would land after plugin teardown and dead-letter a row on its last attempt.
+ it('stops waiting for a job-state lookup that does not return, and leaves its row pending', async () => {
+ const id = await insertEvent({ dispatchAttempts: 4 });
+ let lookedUp!: () => void;
+ const reached = new Promise<void>(resolve => (lookedUp = resolve));
+ let closeConnection!: (error: Error) => void;
+ const getJobState = jest.fn(() => {
+ lookedUp();
+ return new Promise<never>((_resolve, reject) => (closeConnection = reject));
+ });
+ const dispatchWebhookForInstance = jest.fn().mockRejectedValue(new Error('no live sandbox host'));
+ service = new IngressReconcilerService(
+ events,
+ failures,
+ new IngressEnqueueService(
+ { dispatchWebhookForInstance } as unknown as PluginLoaderService,
+ { get: jest.fn().mockReturnValue(true) } as unknown as ConfigService,
+ { getJobState, add: jest.fn().mockRejectedValue(new Error('Connection is closed')) } as never,
+ ),
+ { getPlugin } as unknown as PluginLoaderService,
+ { resolve: resolveInstance } as unknown as PluginInstanceService,
+ );
+
+ const sweep = service.sweep(OPTS);
+ await reached;
+ let destroyed = false;
+ let destroyedBeforeDeadline: boolean;
+ let destroyedAtDeadline: boolean;
+ jest.useFakeTimers({ doNotFake: ['nextTick', 'setImmediate', 'queueMicrotask'] });
+ try {
+ void service.onModuleDestroy().then(() => (destroyed = true));
+ await jest.advanceTimersByTimeAsync(INGRESS_DISPATCH_TIMEOUT_MS - 1);
+ await new Promise(setImmediate);
+ destroyedBeforeDeadline = destroyed;
+ await jest.advanceTimersByTimeAsync(1);
+ await new Promise(setImmediate);
+ destroyedAtDeadline = destroyed;
+ } finally {
+ jest.useRealTimers();
+ }
+ closeConnection(new Error('Connection is closed'));
+ const stats = await sweep;
+
+ expect(destroyedBeforeDeadline).toBe(false);
+ expect(destroyedAtDeadline).toBe(true);
+ expect(stats).toMatchObject({ replayed: 0, failed: 0, skipped: 1 });
+ expect(dispatchWebhookForInstance).not.toHaveBeenCalled();
+ expect(await stored(id)).toMatchObject({ dispatchState: 'pending', dispatchAttempts: 4 });
+ expect((await stored(id)).payload).not.toBeNull();
+ expect(await failures.count()).toBe(0);
+ });
});
describe('IngressReconcilerService.onModuleInit (scheduling)', () => {
diff --git a/src/modules/integration/ingress-reconciler.service.ts b/src/modules/integration/ingress-reconciler.service.ts
index 343424eed..4d7f2e767 100644
--- a/src/modules/integration/ingress-reconciler.service.ts
+++ b/src/modules/integration/ingress-reconciler.service.ts
@@ -13,6 +13,7 @@ import {
resolveIngressJobOptions,
} from './ingress-enqueue.service';
import { extractConversationId } from './ingress.service';
+import { INGRESS_DISPATCH_TIMEOUT_MS } from './integration.constants';
import { PluginInstanceService } from './plugin-instance.service';
import { PluginLoaderService } from '../../core/plugins/plugin-loader.service';
import { IngressJobData } from '../queue/processors/ingress.processor';
@@ -68,9 +69,9 @@ export interface IngressReconcileStats {
* 'failed'/'dispatched' rows are never re-queued: a terminal failure lives in the DLQ
* (RedriveService), a dispatched event is the dispatch tier's concern.
*
- * Mirrors IntegrationRetentionService's lifecycle: a raw unref'd setInterval started on module init
- * (first sweep after one interval, so plugin sandboxes have time to boot), cleared on destroy, which
- * also stops a pass in flight and waits for it.
+ * Same timer lifecycle as IntegrationRetentionService: a raw unref'd setInterval started on module
+ * init (first sweep after one interval, so plugin sandboxes have time to boot) and cleared on destroy.
+ * Unlike it, destroy also stops a pass in flight and waits a bounded time for it.
*/
@Injectable()
export class IngressReconcilerService implements OnModuleInit, OnModuleDestroy {
@@ -107,12 +108,27 @@ export class IngressReconcilerService implements OnModuleInit, OnModuleDestroy {
* Clearing the interval only stops the NEXT pass. A pass already running would go on replaying
* rows after PluginLoaderService has stopped the plugin sandboxes, so each remaining row fails,
* spends an attempt and, at its last one, is dead-lettered. Stop it at the next row and wait for
- * the row in hand, while its plugin is still running.
+ * the row in hand, while its plugin is still running. The wait is bounded by the inline dispatch
+ * timeout: a queue lookup made while Redis has never been reachable does not return, and waiting on it
+ * would keep every later destroy hook (engines, session leases, plugins) from running.
+ * A row given up on stays 'pending': once its lookup returns, the row sees the stop and writes nothing.
*/
async onModuleDestroy(): Promise<void> {
this.stopping = true;
if (this.timer) clearInterval(this.timer);
- await this.inFlight;
+ if (!this.inFlight) return;
+ let timer: NodeJS.Timeout | undefined;
+ const settled = await Promise.race([
+ this.inFlight.then(() => true),
+ new Promise<false>(resolve => (timer = setTimeout(resolve, INGRESS_DISPATCH_TIMEOUT_MS, false))),
+ ]);
+ clearTimeout(timer);
+ if (!settled) {
+ this.logger.warn('Ingress reconcile pass still running at shutdown; continuing without it', {
+ waitMs: INGRESS_DISPATCH_TIMEOUT_MS,
+ action: 'ingress_reconcile_shutdown_timeout',
+ });
+ }
}
/**
@@ -185,7 +201,8 @@ export class IngressReconcilerService implements OnModuleInit, OnModuleDestroy {
stats.scanned++;
const outcome = await this.reconcileRow(row, opts.maxAttempts, now);
if (outcome === 'replayed') stats.replayed++;
- else stats.failed++;
+ else if (outcome === 'failed') stats.failed++;
+ else stats.skipped++;
} catch (err) {
// A bookkeeping failure (repo update/DLQ write) must not abort the batch; the row stays
// 'pending' and is retried next sweep.
@@ -252,6 +269,8 @@ export class IngressReconcilerService implements OnModuleInit, OnModuleDestroy {
// A job still in the queue (or completed) owns the delivery, as in reconcileRow: no dead
// letter while a copy can still deliver. A failed one falls through to its existing DLQ row.
const existing = await this.ingressEnqueue.existingJobState(jobData, row.providerDeliveryId);
+ // Shutdown began while the lookup was out (destroy may have stopped waiting for it).
+ if (this.stopping) return retired;
if (existing && existing !== 'failed') {
const settled = await this.events.update(
{ id: row.id, dispatchState: 'pending' },
@@ -348,12 +367,15 @@ export class IngressReconcilerService implements OnModuleInit, OnModuleDestroy {
row: IngressEvent & { payload: NonNullable<IngressEvent['payload']> },
maxAttempts: number,
now: Date,
- ): Promise<'replayed' | 'failed'> {
+ ): Promise<'replayed' | 'failed' | 'interrupted'> {
const jobData = this.jobDataFor(row);
// jobId = the ORIGINAL deliveryId, so the replay lands on any job the live path did enqueue before
// its outcome mark was lost. BullMQ resolves a duplicate add() whatever that job's state, so look
// first: a live job already owns the delivery, and a failed one would swallow the replay.
const existing = await this.ingressEnqueue.existingJobState(jobData, row.providerDeliveryId);
+ // Shutdown began while the lookup was out, and destroy may have stopped waiting for it: a dispatch
+ // now could land after plugin teardown and spend an attempt. Leave the row for the next start.
+ if (this.stopping) return 'interrupted';
if (existing === 'failed') {
// Every queue attempt already ran and IngressProcessor dead-lettered the delivery, or re-queued it
// and every copy failed too: a live or completed copy answers for the job (see existingJobState).
@@ -408,6 +430,10 @@ export class IngressReconcilerService implements OnModuleInit, OnModuleDestroy {
});
return 'replayed';
}
+ // A failure once shutdown began says nothing about the event: the add may have been refused by a
+ // queue connection closing under it, or the inline fallback met a plugin being torn down. Spend no
+ // attempt and leave the row for the next start.
+ if (this.stopping) return 'interrupted';
const attempts = (row.dispatchAttempts ?? 0) + 1;
const terminal = attempts >= maxAttempts;
diff --git a/src/modules/integration/integration-retention.service.spec.ts b/src/modules/integration/integration-retention.service.spec.ts
index 8e0085c67..159be1471 100644
--- a/src/modules/integration/integration-retention.service.spec.ts
+++ b/src/modules/integration/integration-retention.service.spec.ts
@@ -4,6 +4,7 @@ import { IntegrationDeliveryFailure } from './entities/integration-delivery-fail
import { IntegrationRetentionService } from './integration-retention.service';
import { IngressReconcilerService } from './ingress-reconciler.service';
import { IngressEnqueueService } from './ingress-enqueue.service';
+import { INGRESS_DISPATCH_TIMEOUT_MS } from './integration.constants';
import { PluginInstanceService } from './plugin-instance.service';
import { PluginLoaderService } from '../../core/plugins/plugin-loader.service';
import { Session } from '../session/entities/session.entity';
@@ -660,18 +661,71 @@ describe('IntegrationRetentionService.pruneOlderThan with undispatched events',
const realSave = failures.save.bind(failures) as (
row: IntegrationDeliveryFailure,
) => Promise<IntegrationDeliveryFailure>;
- let destroyed: Promise<void> | undefined;
+ let saving!: () => void;
+ const entered = new Promise<void>(resolve => (saving = resolve));
+ let release!: () => void;
+ const held = new Promise<void>(resolve => (release = resolve));
jest.spyOn(failures, 'save').mockImplementationOnce(async row => {
- destroyed = reconciler.onModuleDestroy();
+ saving();
+ await held;
return realSave(row as IntegrationDeliveryFailure);
});
- await service.pruneOlderThan(7, 90);
- await destroyed;
-
+ const pruned = service.pruneOlderThan(7, 90);
+ await entered;
+ let destroyed = false;
+ const destroy = reconciler.onModuleDestroy().then(() => (destroyed = true));
+ await new Promise(setImmediate);
+ const destroyedWhileHeld = destroyed;
+ release();
+ await destroy;
+ const firstAtDestroy = (await events.findOneByOrFail({ id: 'first' })).dispatchState;
+ await pruned;
+
+ expect(destroyedWhileHeld).toBe(false);
+ expect(firstAtDestroy).toBe('failed');
expect(await failures.count()).toBe(1);
const rows = await events.find({ order: { createdAt: 'ASC' } });
expect(rows.map(r => r.dispatchState)).toEqual(['failed', 'pending']);
expect(rows[1].payload).not.toBeNull();
});
+
+ it('stops waiting on destroy for a hand-off lookup that does not return, and keeps its row', async () => {
+ await insertEvent('stranded', {});
+ let answer!: (state: undefined) => void;
+ const lookedUp = new Promise<void>(resolve =>
+ existingJobState.mockImplementation(() => {
+ resolve();
+ return new Promise(settle => (answer = settle));
+ }),
+ );
+
+ const pruned = service.pruneOlderThan(7, 90);
+ await lookedUp;
+ let destroyed = false;
+ let destroyedBeforeDeadline: boolean;
+ let destroyedAtDeadline: boolean;
+ jest.useFakeTimers({ doNotFake: ['nextTick', 'setImmediate', 'queueMicrotask'] });
+ try {
+ void reconciler.onModuleDestroy().then(() => (destroyed = true));
+ await jest.advanceTimersByTimeAsync(INGRESS_DISPATCH_TIMEOUT_MS - 1);
+ await new Promise(setImmediate);
+ destroyedBeforeDeadline = destroyed;
+ await jest.advanceTimersByTimeAsync(1);
+ await new Promise(setImmediate);
+ destroyedAtDeadline = destroyed;
+ } finally {
+ jest.useRealTimers();
+ }
+ // The lookup returns once the queue connection closes later in shutdown; the row is left as it was.
+ answer(undefined);
+ await pruned;
+
+ expect(destroyedBeforeDeadline).toBe(false);
+ expect(destroyedAtDeadline).toBe(true);
+ expect(await failures.count()).toBe(0);
+ const row = await events.findOneByOrFail({ id: 'stranded' });
+ expect(row.dispatchState).toBe('pending');
+ expect(row.payload).not.toBeNull();
+ });
});
diff --git a/src/modules/message/pending-message-reaper.service.ts b/src/modules/message/pending-message-reaper.service.ts
index 09fe3ea86..8ed9d06f2 100644
--- a/src/modules/message/pending-message-reaper.service.ts
+++ b/src/modules/message/pending-message-reaper.service.ts
@@ -45,9 +45,9 @@ export interface PendingMessageReaperStats {
* performs for every persisted outbound state — so a provider's copy reconciles to the terminal
* state instead of staying stuck at the initial PENDING write.
*
- * Mirrors IngressReconcilerService's lifecycle: a raw unref'd setInterval started on module init
- * (first sweep after one interval), cleared on destroy, which also stops a pass in flight and waits
- * for it. Young PENDING rows (a send still in flight) and non-outgoing rows are never touched.
+ * Same timer lifecycle as IngressReconcilerService: a raw unref'd setInterval started on module init
+ * (first sweep after one interval) and cleared on destroy. Destroy also stops a pass in flight and
+ * waits for it. Young PENDING rows (a send still in flight) and non-outgoing rows are never touched.
*/
@Injectable()
export class PendingMessageReaperService implements OnModuleInit, OnModuleDestroy {
diff --git a/src/modules/queue/processors/close-worker.ts b/src/modules/queue/processors/close-worker.ts
index d51a74e2d..0afe60859 100644
--- a/src/modules/queue/processors/close-worker.ts
+++ b/src/modules/queue/processors/close-worker.ts
@@ -6,9 +6,10 @@ import { createLogger } from '../../../common/services/logger.service';
* Upper bound on a processor's early close. Worker.close() can stay pending while Redis is unreachable,
* and the global destroy hooks (API-key usage flush, plugin onDisable) wait behind it. The shipped kill
* deadline is 45s (docker-compose stop_grace_period, Helm terminationGracePeriodSeconds), and before
- * QueueModule is destroyed shutdown has already spent SHUTDOWN_DELAY_MS (3s), the engine teardown
- * (10s per-engine deadline, engines in parallel) and WEBHOOK_SHUTDOWN_DRAIN_MS (5s). 15s here leaves
- * 12s for the 5s-bounded usage flush, plugin onDisable and the rest of the teardown.
+ * QueueModule is destroyed shutdown has already spent SHUTDOWN_DELAY_MS (3s), the ingress reconciler
+ * wait (INGRESS_DISPATCH_TIMEOUT_MS, 5s), the engine teardown (10s per-engine deadline, engines in
+ * parallel) and WEBHOOK_SHUTDOWN_DRAIN_MS (5s). 15s here leaves 7s for the 5s-bounded usage flush,
+ * plugin onDisable and the rest of the teardown.
*/
export const MAX_WORKER_CLOSE_WAIT_MS = 15_000;
diff --git a/src/modules/queue/worker-shutdown.spec.ts b/src/modules/queue/worker-shutdown.spec.ts
index 708c3f7f7..6283cd2fe 100644
--- a/src/modules/queue/worker-shutdown.spec.ts
+++ b/src/modules/queue/worker-shutdown.spec.ts
@@ -20,6 +20,7 @@ import configuration from '../../config/configuration';
import { ApiKeyUsageTracker } from '../auth/api-key-usage-tracker.service';
import { IntegrationDeliveryFailure } from '../integration/entities/integration-delivery-failure.entity';
import { IngressEvent } from '../integration/entities/ingress-event.entity';
+import { INGRESS_DISPATCH_TIMEOUT_MS } from '../integration/integration.constants';
import { Webhook } from '../webhook/entities/webhook.entity';
import { WebhookDeliveryFailure } from '../webhook/entities/webhook-delivery-failure.entity';
import { WebhookOutboxService } from '../webhook/webhook-outbox.service';
@@ -156,8 +157,9 @@ describe('queue workers at shutdown', () => {
// The global destroy hooks run after the capped wait, so the cap must leave them room inside the
// shipped kill deadline after what shutdown spends before QueueModule is destroyed: SHUTDOWN_DELAY_MS,
- // the per-engine destroy deadline (engines torn down in parallel) and WEBHOOK_SHUTDOWN_DRAIN_MS; then
- // the bounded API-key usage flush. The defaults come from their sources, so raising one fails here.
+ // the ingress reconciler wait (INGRESS_DISPATCH_TIMEOUT_MS), the per-engine destroy deadline (engines
+ // torn down in parallel) and WEBHOOK_SHUTDOWN_DRAIN_MS; then the bounded API-key usage flush. The
+ // defaults come from their sources, so raising one fails here.
it('leaves the global destroy hooks room inside the shipped kill deadline', () => {
const root = join(__dirname, '../../..');
const compose = readFileSync(join(root, 'docker-compose.yml'), 'utf8');
@@ -169,7 +171,11 @@ describe('queue workers at shutdown', () => {
// Hard-coded: an inline literal in SessionLifecycleFences.teardownEngineSafely, not exported.
const engineTeardown = 10_000;
- const beforeQueueClose = DEFAULT_SHUTDOWN_DELAY_MS + engineTeardown + configuration().webhook.shutdownDrainMs;
+ const beforeQueueClose =
+ DEFAULT_SHUTDOWN_DELAY_MS +
+ INGRESS_DISPATCH_TIMEOUT_MS +
+ engineTeardown +
+ configuration().webhook.shutdownDrainMs;
const usageFlush = ApiKeyUsageTracker.SHUTDOWN_FLUSH_TIMEOUT_MS;
expect(beforeQueueClose + MAX_WORKER_CLOSE_WAIT_MS + usageFlush).toBeLessThan(Math.min(...graceMs));
});
diff --git a/src/modules/webhook/webhook-delivery.service.ts b/src/modules/webhook/webhook-delivery.service.ts
index fb21ef827..40347f774 100644
--- a/src/modules/webhook/webhook-delivery.service.ts
+++ b/src/modules/webhook/webhook-delivery.service.ts
@@ -66,15 +66,18 @@ const DEFAULT_WEBHOOK_MAX_PAYLOAD_BYTES = 1024 * 1024;
/**
* How long shutdown waits for in-flight direct deliveries (and their dead-letter bookkeeping) to
- * finish before abandoning them. Default 5s; override with WEBHOOK_SHUTDOWN_DRAIN_MS.
+ * finish before abandoning them, and for an outbox replay in flight. Default 5s; override with
+ * WEBHOOK_SHUTDOWN_DRAIN_MS.
*/
-const DEFAULT_WEBHOOK_SHUTDOWN_DRAIN_MS = 5000;
+export const DEFAULT_WEBHOOK_SHUTDOWN_DRAIN_MS = 5000;
/**
* The result of one delivery attempt. Reported, not thrown: every delivery failure below is already
- * handled in place, so a try/catch cannot tell a delivered event from a dead-lettered one. The one
- * exception that escapes is 'ConcurrencyLimiter closed' from a retry backoff woken after shutdown;
- * only runLimited can see it, and it records the shutdown and keeps the outbox row pending.
+ * handled in place, so a try/catch cannot tell a delivered event from a dead-lettered one. Two
+ * exceptions escape, both from shutdown, and both keep the outbox row pending: 'ConcurrencyLimiter
+ * closed' from a retry backoff woken after it, which only runLimited sees and records, and the
+ * AbortError of a replay whose signal stopped its retries, which reaches WebhookReconcilerService
+ * through redeliver.
* 'cancelled' is terminal like 'delivered' and must never be replayed: either a plugin suppressed the
* dispatch (nothing left the process), or a direct retry found the webhook removed, disabled or
* unsubscribed (an earlier attempt may already have been POSTed).
diff --git a/src/modules/webhook/webhook-reconciler.service.spec.ts b/src/modules/webhook/webhook-reconciler.service.spec.ts
index bb42aa3d5..42dc8ceb7 100644
--- a/src/modules/webhook/webhook-reconciler.service.spec.ts
+++ b/src/modules/webhook/webhook-reconciler.service.spec.ts
@@ -1,5 +1,7 @@
+import { ConfigService } from '@nestjs/config';
import { WebhookReconcilerService, resolveWebhookReconcilerOptions } from './webhook-reconciler.service';
import { ReplayableDelivery } from './webhook-outbox.service';
+import { MAX_TIMER_MS } from '../../config/configuration';
const OPTS = { intervalMs: 60_000, graceMs: 60_000, batchSize: 50, maxAttempts: 3 };
@@ -61,7 +63,7 @@ describe('WebhookReconcilerService', () => {
webhooks = {
findOne: jest.fn().mockResolvedValue({ id: 'wh-1', sessionId: 'sess-1', active: true, events: ['*'] }),
};
- service = new WebhookReconcilerService(webhooks as never, outbox as never, delivery as never);
+ service = new WebhookReconcilerService(webhooks as never, outbox as never, delivery as never, new ConfigService());
});
it('replays a stranded delivery with the STORED idempotency key', async () => {
@@ -227,6 +229,89 @@ describe('WebhookReconcilerService', () => {
expect(outbox.findStale).toHaveBeenCalledTimes(1);
});
+ // A queue lookup made while Redis has never been reachable does not return, and the destroy hooks
+ // after this one (engines, session leases, plugins) must not wait on it for ever.
+ it('stops waiting on destroy after WEBHOOK_SHUTDOWN_DRAIN_MS and replays nothing once the lookup returns', async () => {
+ outbox.findStale.mockResolvedValue([row({ deliveryId: 'job-1' })]);
+ let reached!: () => void;
+ const lookedUp = new Promise<void>(resolve => (reached = resolve));
+ let answer!: (pending: boolean) => void;
+ delivery.isQueueJobPending.mockImplementation(() => {
+ reached();
+ return new Promise<boolean>(resolve => (answer = resolve));
+ });
+ service = new WebhookReconcilerService(
+ webhooks as never,
+ outbox as never,
+ delivery as never,
+ new ConfigService({ webhook: { shutdownDrainMs: 2000 } }),
+ );
+
+ const sweep = service.sweep(OPTS);
+ await lookedUp;
+ let destroyed = false;
+ let destroyedBeforeDeadline: boolean;
+ let destroyedAtDeadline: boolean;
+ jest.useFakeTimers({ doNotFake: ['nextTick', 'setImmediate', 'queueMicrotask'] });
+ try {
+ void service.onModuleDestroy().then(() => (destroyed = true));
+ await jest.advanceTimersByTimeAsync(1999);
+ await new Promise(setImmediate);
+ destroyedBeforeDeadline = destroyed;
+ await jest.advanceTimersByTimeAsync(1);
+ await new Promise(setImmediate);
+ destroyedAtDeadline = destroyed;
+ } finally {
+ jest.useRealTimers();
+ }
+ answer(false);
+ const stats = await sweep;
+
+ expect(destroyedBeforeDeadline).toBe(false);
+ expect(destroyedAtDeadline).toBe(true);
+ expect(stats).toMatchObject({ replayed: 0, failed: 0 });
+ expect(outbox.countAttempt).not.toHaveBeenCalled();
+ expect(delivery.redeliver).not.toHaveBeenCalled();
+ expect(outbox.close).not.toHaveBeenCalled();
+ });
+
+ it('keeps waiting on destroy when WEBHOOK_SHUTDOWN_DRAIN_MS is above the Node timer ceiling', async () => {
+ outbox.findStale.mockResolvedValue([row({ deliveryId: 'job-1' })]);
+ let reached!: () => void;
+ const lookedUp = new Promise<void>(resolve => (reached = resolve));
+ let answer!: (pending: boolean) => void;
+ delivery.isQueueJobPending.mockImplementation(() => {
+ reached();
+ return new Promise<boolean>(resolve => (answer = resolve));
+ });
+ service = new WebhookReconcilerService(
+ webhooks as never,
+ outbox as never,
+ delivery as never,
+ new ConfigService({ webhook: { shutdownDrainMs: MAX_TIMER_MS + 1 } }),
+ );
+
+ const sweep = service.sweep(OPTS);
+ await lookedUp;
+ let destroyed = false;
+ let destroyedEarly: boolean;
+ jest.useFakeTimers({ doNotFake: ['nextTick', 'setImmediate', 'queueMicrotask'] });
+ try {
+ void service.onModuleDestroy().then(() => (destroyed = true));
+ await jest.advanceTimersByTimeAsync(60_000);
+ await new Promise(setImmediate);
+ destroyedEarly = destroyed;
+ } finally {
+ jest.useRealTimers();
+ }
+ answer(false);
+ await sweep;
+ await new Promise(setImmediate);
+
+ expect(destroyedEarly).toBe(false);
+ expect(destroyed).toBe(true);
+ });
+
it('does not start a timer when the interval disables it', () => {
const prev = process.env.WEBHOOK_RECONCILE_INTERVAL_MS;
process.env.WEBHOOK_RECONCILE_INTERVAL_MS = '0';
diff --git a/src/modules/webhook/webhook-reconciler.service.ts b/src/modules/webhook/webhook-reconciler.service.ts
index 5b2d0f323..2d93aab82 100644
--- a/src/modules/webhook/webhook-reconciler.service.ts
+++ b/src/modules/webhook/webhook-reconciler.service.ts
@@ -1,12 +1,17 @@
import { Injectable, OnModuleDestroy, OnModuleInit } from '@nestjs/common';
+import { ConfigService } from '@nestjs/config';
import { InjectRepository } from '@nestjs/typeorm';
import { Repository } from 'typeorm';
import { Webhook } from './entities/webhook.entity';
import { WebhookOutboxService } from './webhook-outbox.service';
-import { isAbortedBackoff, WebhookDeliveryService } from './webhook-delivery.service';
+import {
+ DEFAULT_WEBHOOK_SHUTDOWN_DRAIN_MS,
+ isAbortedBackoff,
+ WebhookDeliveryService,
+} from './webhook-delivery.service';
import { isDeliverableWebhook } from './utils/deliver-once';
import { createLogger } from '../../common/services/logger.service';
-import { resolveNonNegativeIntEnv } from '../../config/configuration';
+import { MAX_TIMER_MS, resolveNonNegativeIntEnv } from '../../config/configuration';
export interface WebhookReconcilerOptions {
/** 0 disables the sweep entirely. */
@@ -71,6 +76,7 @@ export class WebhookReconcilerService implements OnModuleInit, OnModuleDestroy {
@InjectRepository(Webhook, 'data') private readonly webhooks: Repository<Webhook>,
private readonly outbox: WebhookOutboxService,
private readonly delivery: WebhookDeliveryService,
+ private readonly configService: ConfigService,
) {}
onModuleInit(): void {
@@ -91,12 +97,32 @@ export class WebhookReconcilerService implements OnModuleInit, OnModuleDestroy {
* Clearing the interval only stops the NEXT pass. A replay holds no dispatch slot, so the delivery
* drain neither waits for nor stops a pass already running: it would go on POSTing after the drain,
* and after PluginLoaderService has unregistered the `webhook:before` hooks. Stop it at the next row,
- * stop the replay in hand without a further retry, and wait for it.
+ * stop the replay in hand without a further retry, and wait for it, at most WEBHOOK_SHUTDOWN_DRAIN_MS
+ * like the delivery drain beside it: the replay's hooks are not capped, and a queue lookup or add made
+ * while Redis has never been reachable does not return. A row given up on stays 'pending' unless the
+ * replay in hand completes before the process exits.
*/
async onModuleDestroy(): Promise<void> {
this.stop.abort();
if (this.timer) clearInterval(this.timer);
- await this.inFlight;
+ if (!this.inFlight) return;
+ // Above MAX_TIMER_MS a Node timer fires after 1 ms; the delivery drain honours such a value in full.
+ const waitMs = Math.min(
+ this.configService.get<number>('webhook.shutdownDrainMs', DEFAULT_WEBHOOK_SHUTDOWN_DRAIN_MS),
+ MAX_TIMER_MS,
+ );
+ let timer: NodeJS.Timeout | undefined;
+ const settled = await Promise.race([
+ this.inFlight.then(() => true),
+ new Promise<false>(resolve => (timer = setTimeout(resolve, waitMs, false))),
+ ]);
+ clearTimeout(timer);
+ if (!settled) {
+ this.logger.warn('Webhook reconcile pass still running at shutdown; continuing without it', {
+ waitMs,
+ action: 'webhook_reconcile_shutdown_timeout',
+ });
+ }
}
/** One bounded pass over the stranded backlog. Overlap-guarded. */
diff --git a/src/modules/webhook/webhook-recovery-boundaries.spec.ts b/src/modules/webhook/webhook-recovery-boundaries.spec.ts
index ee53c7874..68e4414a9 100644
--- a/src/modules/webhook/webhook-recovery-boundaries.spec.ts
+++ b/src/modules/webhook/webhook-recovery-boundaries.spec.ts
@@ -62,7 +62,7 @@ describe('webhook recovery across live ownership and configuration changes', ()
redeliver: jest.fn().mockResolvedValue('delivered'),
recordReplayExhaustion: jest.fn().mockResolvedValue(true),
};
- const reconciler = new WebhookReconcilerService(webhooks, outbox, delivery as never);
+ const reconciler = new WebhookReconcilerService(webhooks, outbox, delivery as never, new ConfigService());
const options = { intervalMs: 1000, graceMs: 0, batchSize: 2, maxAttempts: 5 };
for (let i = 0; i < 3; i++) expect((await reconciler.sweep(options)).scanned).toBeLessThanOrEqual(2);
expect(delivery.redeliver.mock.calls.map((call: unknown[]) => call[3])).toEqual(['key3', 'key4']);
@@ -148,7 +148,7 @@ describe('webhook recovery across live ownership and configuration changes', ()
const repository = ds.getRepository(WebhookOutboxEvent);
const outbox = new WebhookOutboxService(repository);
const delivery = new WebhookDeliveryService(webhooks, failures, config, new HookManager(), outbox);
- const reconciler = new WebhookReconcilerService(webhooks, outbox, delivery);
+ const reconciler = new WebhookReconcilerService(webhooks, outbox, delivery, config);
const redrive = new WebhookRedriveService(webhooks, failures, delivery, config);
const options = { intervalMs: 1000, graceMs: 0, batchSize: 50, maxAttempts: 5 };
// A dispatch interrupted before it settled leaves its row pending for the sweep.
@@ -239,7 +239,7 @@ describe('webhook recovery across live ownership and configuration changes', ()
const hooks = new HookManager();
const execute = jest.spyOn(hooks, 'execute');
const delivery = new WebhookDeliveryService(webhooks, failures, config, hooks, outbox);
- const reconciler = new WebhookReconcilerService(webhooks, outbox, delivery);
+ const reconciler = new WebhookReconcilerService(webhooks, outbox, delivery, config);
let posted!: () => void;
const firstPost = new Promise<void>(resolve => (posted = resolve));
const post = jest.spyOn(ssrf, 'withSafeFetch').mockImplementation((_url, _init, use) => {
@@ -269,6 +269,55 @@ describe('webhook recovery across live ownership and configuration changes', ()
expect(execute.mock.calls.filter(([name]) => name === 'webhook:error')).toHaveLength(0);
});
+ it('stops the direct fallback of a replay the queue refused in its backoff on destroy', async () => {
+ await webhooks.update(primary.id, { retryCount: 3 });
+ const repository = ds.getRepository(WebhookOutboxEvent);
+ const outbox = new WebhookOutboxService(repository);
+ await outbox.open({
+ webhookId: primary.id,
+ sessionId: primary.sessionId,
+ event: 'message.received',
+ idempotencyKey: 'key-a',
+ deliveryId: randomUUID(),
+ payload: { id: 1 },
+ });
+ await repository.update({ idempotencyKey: 'key-a' }, { createdAt: new Date(Date.now() - 600_000) });
+ const config = new ConfigService({ queue: { enabled: true }, webhook: { retryDelay: 600_000, timeout: 1000 } });
+ const hooks = new HookManager();
+ const execute = jest.spyOn(hooks, 'execute');
+ const queue = {
+ add: jest.fn().mockRejectedValue(new Error('Connection is closed')),
+ getJob: jest.fn().mockResolvedValue(undefined),
+ };
+ const delivery = new WebhookDeliveryService(webhooks, failures, config, hooks, outbox, undefined, queue as never);
+ const reconciler = new WebhookReconcilerService(webhooks, outbox, delivery, config);
+ let posted!: () => void;
+ const firstPost = new Promise<void>(resolve => (posted = resolve));
+ const post = jest.spyOn(ssrf, 'withSafeFetch').mockImplementation((_url, _init, use) => {
+ posted();
+ return Promise.resolve(use(new Response(null, { status: 503 })));
+ });
+
+ const sweep = reconciler.sweep({ intervalMs: 1000, graceMs: 0, batchSize: 10, maxAttempts: 5 });
+ await firstPost;
+ // Let the 503 settle so the retry backoff timer is already running when the stop lands.
+ await new Promise(resolve => setImmediate(resolve));
+ await reconciler.onModuleDestroy();
+ const stats = await sweep;
+
+ expect(queue.add).toHaveBeenCalledTimes(1);
+ expect(post).toHaveBeenCalledTimes(1);
+ expect(stats).toMatchObject({ replayed: 0, failed: 0, skipped: 1 });
+ expect(await failures.count()).toBe(0);
+ const errors = execute.mock.calls.filter(([name]) => name === 'webhook:error');
+ expect(errors).toHaveLength(1);
+ expect(errors[0][1]).toMatchObject({ error: expect.stringContaining('Queue failed') as string });
+ expect(await repository.findOneByOrFail({ idempotencyKey: 'key-a' })).toMatchObject({
+ state: 'pending',
+ attempts: 1,
+ });
+ });
+
it('does not replay a row whose pre-checks were under way when destroy began', async () => {
const repository = ds.getRepository(WebhookOutboxEvent);
const outbox = new WebhookOutboxService(repository);
@@ -285,7 +334,7 @@ describe('webhook recovery across live ownership and configuration changes', ()
}
const config = new ConfigService({ webhook: { retryDelay: 300, timeout: 1000 } });
const delivery = new WebhookDeliveryService(webhooks, failures, config, new HookManager(), outbox);
- const reconciler = new WebhookReconcilerService(webhooks, outbox, delivery);
+ const reconciler = new WebhookReconcilerService(webhooks, outbox, delivery, config);
const post = jest
.spyOn(ssrf, 'withSafeFetch')
.mockImplementation((_url, _init, use) => Promise.resolve(use(new Response(null, { status: 200 }))));
@@ -333,7 +382,7 @@ describe('webhook recovery across live ownership and configuration changes', ()
const execute = jest.spyOn(hooks, 'execute');
const config = new ConfigService({ webhook: { retryDelay: 1, timeout: 1000 } });
const delivery = new WebhookDeliveryService(webhooks, failures, config, hooks, outbox);
- const reconciler = new WebhookReconcilerService(webhooks, outbox, delivery);
+ const reconciler = new WebhookReconcilerService(webhooks, outbox, delivery, config);
const post = jest
.spyOn(ssrf, 'withSafeFetch')
.mockImplementation((_url, _init, use) => Promise.resolve(use(new Response(null, { status: 503 }))));
@@ -375,7 +424,7 @@ describe('webhook recovery across live ownership and configuration changes', ()
const execute = jest.spyOn(hooks, 'execute');
const config = new ConfigService({ webhook: { retryDelay: 300, timeout: 1000 } });
const delivery = new WebhookDeliveryService(webhooks, failures, config, hooks, outbox);
- const reconciler = new WebhookReconcilerService(webhooks, outbox, delivery);
+ const reconciler = new WebhookReconcilerService(webhooks, outbox, delivery, config);
let posted!: () => void;
const firstPost = new Promise<void>(resolve => (posted = resolve));
let fail!: () => void;