Skip to content

Free, Open Source, Self-Hosted WhatsApp API Gateway

fix(queue): fail fast on producer calls before Redis has connected (#1836)

Yudhi Armyndharisff34251921c0
diff --git a/.env.example b/.env.example
index 5c4b2d68e..b42388dc3 100644
--- a/.env.example
+++ b/.env.example
@@ -389,7 +389,10 @@ DATABASE_SYNCHRONIZE=false # WARNING: Set false in production! (data DB
# Redis settings (auto-configured if REDIS_BUILTIN=true)
# REDIS_HOST=localhost
# REDIS_PORT=6379
-# Fail Redis queue/cache connection attempts after this many ms.
+# Fail Redis queue/cache connection attempts after this many ms. Queue calls made before Redis has
+# ever connected wait at most this long from the first one, then take the same fallback as an
+# unreachable Redis until it connects. 0 disables the socket connection timeout; that queue wait then
+# still uses the default 5000.
# REDIS_CONNECT_TIMEOUT_MS=5000
# REDIS_USERNAME=
# REDIS_PASSWORD=
diff --git a/CHANGELOG.md b/CHANGELOG.md
index c49616290..261006ff8 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -28,6 +28,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
+- Fall back instead of hanging when an ingress, webhook or infra status queue call runs before Redis has ever connected.
- Drop whatsapp-web.js inbound media over `MEDIA_DOWNLOAD_MAX_BYTES` in the page when the sender understates its size.
- Trust `NODE_EXTRA_CA_CERTS` in `backup.sh`'s PostgreSQL TLS check on Node older than 22.15.
- Set `PGSSLMODE` in the `psql` import command `restore.sh` prints when `DATABASE_SSL=true`.
diff --git a/docs/25-integration-fabric.md b/docs/25-integration-fabric.md
index 189bd15e1..863e03401 100644
--- a/docs/25-integration-fabric.md
+++ b/docs/25-integration-fabric.md
@@ -231,8 +231,11 @@ job id; per-instance fairness via a token bucket; and a dead-letter record with
queued worker path, an **in-process** per-conversation lock prevents concurrent starts for the same lane;
strict FIFO is not preserved across retry/redrive, and the lock is single-node state rather than Redis or
PostgreSQL state. When the queue is disabled, ingress dispatches inline after persisting and does not
-serialize concurrent same-conversation deliveries. Providers already deliver over unordered,
-at-least-once HTTP, so plugin handlers must be idempotent and treat ingress as a reconciliation trigger.
+serialize concurrent same-conversation deliveries. A queued delivery falls back to the same inline
+dispatch when Redis is unreachable. Before Redis has ever connected, queue calls wait for it at most
+`REDIS_CONNECT_TIMEOUT_MS` (5s when unset or 0) from the first one, then fall back at once until it
+connects. Providers already deliver over unordered, at-least-once HTTP, so plugin handlers must be
+idempotent and treat ingress as a reconciliation trigger.
A job waiting on that lock still holds one of the `INGRESS_WORKER_CONCURRENCY` worker slots (default
10). A burst on one lane larger than that fills every slot with same-lane waiters, and events for other
@@ -275,12 +278,13 @@ whose dead-letter write fails is kept for the next run. Two cases get no dead-le
queue job that still owns the delivery takes the `dispatched` mark (closing any open dead-letter
row), and a row whose instance or bound session was deleted is dropped, since redrive refuses a
deleted instance and a session delete purges its dead letters. A delivery whose completed job was
-already removed from the queue, or whose job lookup failed while Redis was down, is dead-lettered
-anyway and may already have been delivered: check before redriving it. The startup prune runs before
-plugins load, so a dead-letter row it writes carries no conversation id and its redrive takes the
-per-instance lane. There is deliberately no per-instance row-count cap: eviction under a flood of
-forged delivery ids would silently drop legit dedup rows and re-admit their replays, which is worse
-than the bounded growth it would prevent.
+already removed from the queue, or whose job lookup failed while Redis was down (or, for a Redis
+slow to start at boot, not yet connected within that first-connect wait), is dead-lettered anyway and
+may already have been delivered: check before redriving it. A dead-letter row the startup prune writes
+before plugins load carries no conversation id, and its redrive takes the per-instance lane. There is
+deliberately no per-instance row-count cap: eviction under a flood of forged delivery ids would
+silently drop legit dedup rows and re-admit their replays, which is worse than the bounded growth it
+would prevent.
## 25.8 The Integration SDK (v1)
diff --git a/src/config/redis-options.ts b/src/config/redis-options.ts
index 3757ad48d..793d0d159 100644
--- a/src/config/redis-options.ts
+++ b/src/config/redis-options.ts
@@ -13,6 +13,9 @@ export interface RedisConnectionOptions {
tls?: TlsOptions;
}
+/** Default REDIS_CONNECT_TIMEOUT_MS, also the producer's first-connect wait when that is set to 0. */
+export const DEFAULT_REDIS_CONNECT_TIMEOUT_MS = 5000;
+
/**
* The connection fields every Redis client shares: the cache, the throttler store, the BullMQ
* producer and workers, and the WebSocket fan-out pair. Each caller spreads this and adds only its
@@ -30,7 +33,7 @@ export function redisConnectionOptions(): RedisConnectionOptions {
port: parseInt(process.env.REDIS_PORT || '6379', 10),
username: process.env.REDIS_USERNAME,
password: process.env.REDIS_PASSWORD,
- connectTimeout: parseInt(process.env.REDIS_CONNECT_TIMEOUT_MS || '5000', 10),
+ connectTimeout: parseInt(process.env.REDIS_CONNECT_TIMEOUT_MS || String(DEFAULT_REDIS_CONNECT_TIMEOUT_MS), 10),
...(process.env.REDIS_TLS === 'true' ? { tls: {} } : {}),
};
}
diff --git a/src/modules/infra/infra-status.controller.spec.ts b/src/modules/infra/infra-status.controller.spec.ts
index de859efe4..40fd3f60f 100644
--- a/src/modules/infra/infra-status.controller.spec.ts
+++ b/src/modules/infra/infra-status.controller.spec.ts
@@ -22,6 +22,7 @@ jest.mock('fs', () => {
import { InfraStatusController } from './infra-status.controller';
import { recordPinnedEnvKeys } from '../../config/env-precedence';
+import { producerConnectWaitMs } from '../queue/redis-connection';
describe('InfraStatusController.getStatus DB health (active SELECT 1 probe, not just isInitialized)', () => {
const build = (query: jest.Mock) => {
@@ -53,7 +54,10 @@ describe('InfraStatusController.getStatus DB health (active SELECT 1 probe, not
});
describe('InfraStatusController.getStatus queue job counts', () => {
- function buildStatusController(opts: { queueEnabled: boolean; queue?: { getJobCounts: jest.Mock } }) {
+ function buildStatusController(opts: {
+ queueEnabled: boolean;
+ queue?: { getJobCounts: jest.Mock; waitUntilReady?: jest.Mock };
+ }) {
const configService = {
// engine.type=baileys skips the wa-web-version registry fetch (no network in unit tests).
get: (key: string, def?: unknown) =>
@@ -72,7 +76,7 @@ describe('InfraStatusController.getStatus queue job counts', () => {
dockerService as never,
cacheService as never,
storageService as never,
- opts.queue as never,
+ (opts.queue && { waitUntilReady: jest.fn().mockResolvedValue(undefined), ...opts.queue }) as never,
);
}
@@ -86,6 +90,24 @@ describe('InfraStatusController.getStatus queue job counts', () => {
expect(status.queue).toEqual({ enabled: true, webhooks: { pending: 6, completed: 10, failed: 1 } });
});
+ // Until the first connect BullMQ holds a queue call for as long as Redis stays unreachable.
+ it('reports zeros once the connect timeout passes when Redis has never connected', async () => {
+ const getJobCounts = jest.fn();
+ const waitUntilReady = jest.fn(() => new Promise<never>(() => {}));
+ const controller = buildStatusController({ queueEnabled: true, queue: { getJobCounts, waitUntilReady } });
+ let status: Awaited<ReturnType<InfraStatusController['getStatus']>> | undefined;
+ jest.useFakeTimers({ doNotFake: ['nextTick', 'setImmediate', 'queueMicrotask'] });
+ try {
+ void controller.getStatus().then(s => (status = s));
+ await jest.advanceTimersByTimeAsync(producerConnectWaitMs());
+ } finally {
+ jest.useRealTimers();
+ }
+
+ expect(getJobCounts).not.toHaveBeenCalled();
+ expect(status?.queue).toEqual({ enabled: true, webhooks: { pending: 0, completed: 0, failed: 0 } });
+ });
+
it('reports zeros (and does not touch the queue) when the queue is disabled', async () => {
const getJobCounts = jest.fn();
const controller = buildStatusController({ queueEnabled: false, queue: { getJobCounts } });
diff --git a/src/modules/infra/infra-status.controller.ts b/src/modules/infra/infra-status.controller.ts
index 22c11f308..06c923fcf 100644
--- a/src/modules/infra/infra-status.controller.ts
+++ b/src/modules/infra/infra-status.controller.ts
@@ -10,6 +10,7 @@ import {
import { InjectQueue } from '@nestjs/bullmq';
import { Queue } from 'bullmq';
import { QUEUE_NAMES } from '../queue/queue-names';
+import { producerReady } from '../queue/redis-connection';
import { ConfigService } from '@nestjs/config';
import { DataSource } from 'typeorm';
import { InjectDataSource } from '@nestjs/typeorm';
@@ -198,6 +199,7 @@ export class InfraStatusController {
let webhooks = { pending: 0, completed: 0, failed: 0 };
if (queueEnabled && this.webhookQueue) {
try {
+ await producerReady(this.webhookQueue);
const counts = await this.webhookQueue.getJobCounts('wait', 'active', 'delayed', 'completed', 'failed');
webhooks = {
pending: (counts.wait ?? 0) + (counts.active ?? 0) + (counts.delayed ?? 0),
diff --git a/src/modules/integration/ingress-enqueue.service.spec.ts b/src/modules/integration/ingress-enqueue.service.spec.ts
index 07f36e2af..fa292915c 100644
--- a/src/modules/integration/ingress-enqueue.service.spec.ts
+++ b/src/modules/integration/ingress-enqueue.service.spec.ts
@@ -6,6 +6,7 @@ import {
} from './ingress-enqueue.service';
import { PluginLoaderService } from '../../core/plugins/plugin-loader.service';
import { ConfigService } from '@nestjs/config';
+import { producerConnectWaitMs } from '../queue/redis-connection';
describe('IngressEnqueueService', () => {
const data = {
@@ -19,12 +20,12 @@ describe('IngressEnqueueService', () => {
let loader: jest.Mocked<Partial<PluginLoaderService>>;
let config: jest.Mocked<Partial<ConfigService>>;
- let queue: { add: jest.Mock };
+ let queue: { add: jest.Mock; waitUntilReady: jest.Mock };
beforeEach(() => {
loader = { dispatchWebhookForInstance: jest.fn().mockResolvedValue(undefined) };
config = { get: jest.fn() };
- queue = { add: jest.fn().mockResolvedValue(undefined) };
+ queue = { add: jest.fn().mockResolvedValue(undefined), waitUntilReady: jest.fn().mockResolvedValue(undefined) };
});
it('adds a job to the ingress queue keyed by the namespaced jobId when queueing is enabled and a queue is present', async () => {
@@ -249,6 +250,37 @@ describe('IngressEnqueueService', () => {
expect(loader.dispatchWebhookForInstance).toHaveBeenCalledWith(data);
});
+ // Until the first connect, BullMQ holds every queue call without sending a command and retries the
+ // connection for ever, so enableOfflineQueue:false never rejects: with Redis unreachable since boot,
+ // the ingress request, a redrive and the reconciler's row in hand would wait indefinitely.
+ it('falls back to inline dispatch and reports no job when Redis has not connected within the connect timeout', async () => {
+ (config.get as jest.Mock).mockReturnValue(true);
+ const pending = () => new Promise<never>(() => {});
+ queue.waitUntilReady.mockImplementation(pending);
+ queue.add.mockImplementation(pending);
+ const getJobState = jest.fn(pending);
+ const svc = new IngressEnqueueService(
+ loader as PluginLoaderService,
+ config as ConfigService,
+ { ...queue, getJobState } as never,
+ );
+ let results: unknown;
+ jest.useFakeTimers();
+ try {
+ void Promise.all([svc.existingJobState(data, 'd1'), svc.enqueue(data, 'd1')]).then(r => (results = r));
+ await jest.advanceTimersByTimeAsync(producerConnectWaitMs() - 1);
+ expect(results).toBeUndefined();
+ await jest.advanceTimersByTimeAsync(1);
+ } finally {
+ jest.useRealTimers();
+ }
+
+ expect(results).toEqual([undefined, { outcome: 'dispatched' }]);
+ expect(getJobState).not.toHaveBeenCalled();
+ expect(queue.add).not.toHaveBeenCalled();
+ expect(loader.dispatchWebhookForInstance).toHaveBeenCalledWith(data);
+ });
+
describe('resolveIngressJobOptions', () => {
it('defaults to 3 attempts with exponential backoff', () => {
expect(resolveIngressJobOptions()).toEqual({ attempts: 3, backoff: { type: 'exponential', delay: 5000 } });
diff --git a/src/modules/integration/ingress-enqueue.service.ts b/src/modules/integration/ingress-enqueue.service.ts
index c5dee0b7b..e38b91799 100644
--- a/src/modules/integration/ingress-enqueue.service.ts
+++ b/src/modules/integration/ingress-enqueue.service.ts
@@ -7,16 +7,18 @@ import { PluginLoaderService } from '../../core/plugins/plugin-loader.service';
import { IngressJobData } from '../queue/processors/ingress.processor';
import { IntegrationDeliveryFailure } from './entities/integration-delivery-failure.entity';
import { QUEUE_NAMES } from '../queue/queue-names';
+import { producerReady } from '../queue/redis-connection';
import { createLogger } from '../../common/services/logger.service';
import { resolveNonNegativeIntEnv } from '../../config/configuration';
/**
* Outcome of an enqueue attempt. 'queued' = handed to BullMQ; 'dispatched' = delivered inline; 'failed'
- * = inline dispatch threw and was swallowed (`error` carries the message). enqueue() never throws, so
- * callers use the outcome (not exceptions) to decide durability follow-up: a LIVE ingress delivery must
- * persist a dead-letter row on 'failed' (see buildIngressDeadLetterRow, wired at the IngressService
- * factory) so RedriveService can replay it; RedriveService itself calls enqueue() directly, because a
- * failed replay must keep its EXISTING dead-letter row redrivable rather than write a second one.
+ * = inline dispatch threw and was swallowed (`error` carries the message). enqueue() does not throw
+ * (bar an aborted `signal`, see enqueue), so callers use the outcome (not exceptions) to decide
+ * durability follow-up: a LIVE ingress delivery must persist a dead-letter row on 'failed' (see
+ * buildIngressDeadLetterRow, wired at the IngressService factory) so RedriveService can replay it;
+ * RedriveService itself calls enqueue() directly, because a failed replay must keep its EXISTING
+ * dead-letter row redrivable rather than write a second one.
*/
export type EnqueueOutcome = { outcome: 'queued' | 'dispatched' | 'failed'; error?: string };
@@ -147,6 +149,7 @@ export class IngressEnqueueService implements OnApplicationBootstrap {
async existingJobState(data: IngressJobData, jobId: string): Promise<JobState | undefined> {
if (!this.config.get<boolean>('queue.enabled', false) || !this.ingressQueue) return undefined;
try {
+ await producerReady(this.ingressQueue);
const id = queueJobId(data, jobId);
const state = await this.ingressQueue.getJobState(id);
if (isLiveOrCompleted(state)) return state;
@@ -156,7 +159,8 @@ export class IngressEnqueueService implements OnApplicationBootstrap {
}
return state === 'unknown' ? undefined : state;
} catch (err) {
- // Redis unreachable: enqueue() then takes its own inline fallback, as it would without the probe.
+ // Redis unreachable or not yet connected: enqueue() then takes its own inline fallback, as it would
+ // without the probe.
this.logger.warn('Ingress job state lookup failed', {
pluginId: data.pluginId,
instanceId: data.instanceId,
@@ -168,7 +172,12 @@ export class IngressEnqueueService implements OnApplicationBootstrap {
}
}
- async enqueue(data: IngressJobData, jobId: string): Promise<EnqueueOutcome> {
+ /**
+ * `signal` lets a caller that stops at shutdown (the reconciler) refuse the inline fallback once it
+ * has aborted: a queue add that fails after the stop rethrows the abort reason instead of
+ * dispatching, since the plugin may already be torn down. That is the only way enqueue() throws.
+ */
+ async enqueue(data: IngressJobData, jobId: string, signal?: AbortSignal): Promise<EnqueueOutcome> {
const queueEnabled = this.config.get<boolean>('queue.enabled', false);
const useQueue = queueEnabled && !!this.ingressQueue;
@@ -178,6 +187,7 @@ export class IngressEnqueueService implements OnApplicationBootstrap {
// exponential-backoff attempts so a transient failure retries before landing in the DLQ. The id
// is sanitized because BullMQ refuses several id shapes at add() (see sanitizeIngressJobId),
// which would otherwise read as a Redis failure here and fall through to inline dispatch.
+ await producerReady(this.ingressQueue);
await this.ingressQueue.add('ingress', data, {
jobId: queueJobId(data, jobId),
...resolveIngressJobOptions(),
@@ -185,9 +195,11 @@ export class IngressEnqueueService implements OnApplicationBootstrap {
return { outcome: 'queued' };
} catch (err) {
// Redis unreachable (enableOfflineQueue:false makes add() reject) — fall through to inline
- // dispatch. Without this, the already-persisted event would be lost forever: the throw would
- // 500 the ingress request, the provider retries, dedup returns "duplicate", and no job was
- // ever enqueued (no DLQ row either). Mirrors WebhookService's queue-add fallback.
+ // dispatch (before the first connect, producerReady rejects instead). Without this, the
+ // already-persisted event would be lost forever: the throw would 500 the ingress request, the
+ // provider retries, dedup returns "duplicate", and no job was ever enqueued (no DLQ row either).
+ // Mirrors WebhookService's queue-add fallback.
+ signal?.throwIfAborted();
this.logger.error(
'Ingress queue add failed; dispatching inline',
err instanceof Error ? err.message : String(err),
diff --git a/src/modules/integration/ingress-reconciler.service.spec.ts b/src/modules/integration/ingress-reconciler.service.spec.ts
index b4144b157..3499e6dce 100644
--- a/src/modules/integration/ingress-reconciler.service.spec.ts
+++ b/src/modules/integration/ingress-reconciler.service.spec.ts
@@ -381,10 +381,14 @@ describe('IngressReconcilerService.sweep', () => {
// attempt: IngressProcessor already dead-lettered it. A replay under the same jobId is swallowed by
// BullMQ as a duplicate (add() still resolves), so it must not count as delivered.
describe('when the live path already queued a job for the delivery', () => {
- let queue: { add: jest.Mock; getJobState: jest.Mock };
+ let queue: { add: jest.Mock; getJobState: jest.Mock; waitUntilReady: jest.Mock };
beforeEach(() => {
- queue = { add: jest.fn().mockResolvedValue(undefined), getJobState: jest.fn() };
+ queue = {
+ add: jest.fn().mockResolvedValue(undefined),
+ getJobState: jest.fn(),
+ waitUntilReady: jest.fn().mockResolvedValue(undefined),
+ };
const realEnqueue = new IngressEnqueueService(
{ dispatchWebhookForInstance: jest.fn() } as unknown as PluginLoaderService,
{ get: jest.fn().mockReturnValue(true) } as unknown as ConfigService,
@@ -703,8 +707,8 @@ describe('IngressReconcilerService.sweep', () => {
expect(await failures.count()).toBe(0);
});
- // A job-state lookup made while Redis has never been reachable only returns once the queue connection
- // is closed later in shutdown. Waiting for it kept every later destroy hook from running; dispatching
+ // A job-state lookup against a stalled Redis may only return once the queue connection is closed
+ // later in shutdown. Waiting for it kept every later destroy hook from running; dispatching
// once it returns would land after plugin teardown and dead-letter a row on its last attempt.
it('stops waiting for a job-state lookup that does not return, and leaves its row pending', async () => {
const id = await insertEvent({ dispatchAttempts: 4 });
@@ -722,7 +726,11 @@ describe('IngressReconcilerService.sweep', () => {
new IngressEnqueueService(
{ dispatchWebhookForInstance } as unknown as PluginLoaderService,
{ get: jest.fn().mockReturnValue(true) } as unknown as ConfigService,
- { getJobState, add: jest.fn().mockRejectedValue(new Error('Connection is closed')) } as never,
+ {
+ getJobState,
+ add: jest.fn().mockRejectedValue(new Error('Connection is closed')),
+ waitUntilReady: jest.fn().mockResolvedValue(undefined),
+ } as never,
),
{ getPlugin } as unknown as PluginLoaderService,
{ resolve: resolveInstance } as unknown as PluginInstanceService,
@@ -756,6 +764,48 @@ describe('IngressReconcilerService.sweep', () => {
expect((await stored(id)).payload).not.toBeNull();
expect(await failures.count()).toBe(0);
});
+
+ // A queue add held by a stalled Redis can fail after destroy has begun. Its inline fallback would then
+ // race plugin teardown and, on the row's last attempt, dead-letter it.
+ it('does not dispatch inline when the queue add fails after shutdown began', async () => {
+ const id = await insertEvent({ dispatchAttempts: 4 });
+ let failAdd!: (error: Error) => void;
+ let onAdd!: () => void;
+ const adding = new Promise<void>(resolve => (onAdd = resolve));
+ const queue = {
+ getJobState: jest.fn().mockResolvedValue('unknown'),
+ add: jest.fn(() => {
+ onAdd();
+ return new Promise<never>((_resolve, reject) => (failAdd = reject));
+ }),
+ waitUntilReady: jest.fn().mockResolvedValue(undefined),
+ };
+ const dispatchWebhookForInstance = jest.fn().mockRejectedValue(new Error('no live sandbox host'));
+ service = new IngressReconcilerService(
+ events,
+ failures,
+ new IngressEnqueueService(
+ { dispatchWebhookForInstance } as unknown as PluginLoaderService,
+ { get: jest.fn().mockReturnValue(true) } as unknown as ConfigService,
+ queue as never,
+ ),
+ { getPlugin } as unknown as PluginLoaderService,
+ { resolve: resolveInstance } as unknown as PluginInstanceService,
+ );
+
+ const sweep = service.sweep(OPTS);
+ await adding;
+ const destroyed = service.onModuleDestroy();
+ failAdd(new Error('Connection is closed.'));
+ const stats = await sweep;
+ await destroyed;
+
+ expect(stats).toMatchObject({ replayed: 0, failed: 0, skipped: 1 });
+ expect(dispatchWebhookForInstance).not.toHaveBeenCalled();
+ expect(await stored(id)).toMatchObject({ dispatchState: 'pending', dispatchAttempts: 4 });
+ expect((await stored(id)).payload).not.toBeNull();
+ expect(await failures.count()).toBe(0);
+ });
});
describe('IngressReconcilerService.onModuleInit (scheduling)', () => {
diff --git a/src/modules/integration/ingress-reconciler.service.ts b/src/modules/integration/ingress-reconciler.service.ts
index 4d7f2e767..994e5098b 100644
--- a/src/modules/integration/ingress-reconciler.service.ts
+++ b/src/modules/integration/ingress-reconciler.service.ts
@@ -79,7 +79,7 @@ export class IngressReconcilerService implements OnModuleInit, OnModuleDestroy {
private timer?: ReturnType<typeof setInterval>;
// The pass in flight, settled when it ends; doubles as the overlap guard.
private inFlight?: Promise<void>;
- private stopping = false;
+ private readonly stop = new AbortController();
constructor(
@InjectRepository(IngressEvent, 'data') private readonly events: Repository<IngressEvent>,
@@ -109,12 +109,14 @@ export class IngressReconcilerService implements OnModuleInit, OnModuleDestroy {
* rows after PluginLoaderService has stopped the plugin sandboxes, so each remaining row fails,
* spends an attempt and, at its last one, is dead-lettered. Stop it at the next row and wait for
* the row in hand, while its plugin is still running. The wait is bounded by the inline dispatch
- * timeout: a queue lookup made while Redis has never been reachable does not return, and waiting on it
- * would keep every later destroy hook (engines, session leases, plugins) from running.
- * A row given up on stays 'pending': once its lookup returns, the row sees the stop and writes nothing.
+ * timeout: before Redis has ever connected, the row's queue lookup can wait up to
+ * REDIS_CONNECT_TIMEOUT_MS ahead of that dispatch, and a stalled Redis holds a lookup or add longer
+ * still; waiting it out would delay every later destroy hook (engines, session leases, plugins).
+ * A row given up on is never dispatched inline after the stop: it stays 'pending', unless its queue
+ * add still succeeds and the queued job takes the delivery.
*/
async onModuleDestroy(): Promise<void> {
- this.stopping = true;
+ this.stop.abort();
if (this.timer) clearInterval(this.timer);
if (!this.inFlight) return;
let timer: NodeJS.Timeout | undefined;
@@ -137,7 +139,7 @@ export class IngressReconcilerService implements OnModuleInit, OnModuleDestroy {
*/
async sweep(opts: IngressReconcilerOptions, now: Date = new Date()): Promise<IngressReconcileStats> {
const stats: IngressReconcileStats = { scanned: 0, replayed: 0, failed: 0, skipped: 0 };
- if (this.inFlight || this.stopping) return stats;
+ if (this.inFlight || this.stop.signal.aborted) return stats;
let settle!: () => void;
this.inFlight = new Promise(resolve => (settle = resolve));
try {
@@ -161,7 +163,7 @@ export class IngressReconcilerService implements OnModuleInit, OnModuleDestroy {
.limit(opts.batchSize)
.getMany();
for (const row of rows) {
- if (this.stopping) break;
+ if (this.stop.signal.aborted) break;
// A row whose latest attempt is still inside the grace window cools down between replays;
// it keeps its batch slot (bounded by maxAttempts, so the leak is capped) but is not hit again.
if (row.lastDispatchAt && row.lastDispatchAt > cutoff) {
@@ -234,7 +236,7 @@ export class IngressReconcilerService implements OnModuleInit, OnModuleDestroy {
* dead letter. Returns the number of rows dead-lettered.
*/
async deadLetterAgedPending(cutoff: Date, batchSize = 100): Promise<number> {
- if (this.inFlight || this.stopping) return 0;
+ if (this.inFlight || this.stop.signal.aborted) return 0;
let settle!: () => void;
this.inFlight = new Promise(resolve => (settle = resolve));
let retired = 0;
@@ -247,7 +249,7 @@ export class IngressReconcilerService implements OnModuleInit, OnModuleDestroy {
});
let progressed = 0;
for (const row of rows) {
- if (this.stopping) return retired;
+ if (this.stop.signal.aborted) return retired;
if (!hasPayload(row)) continue;
const meta = { pluginId: row.pluginId, instanceId: row.instanceId, deliveryId: row.providerDeliveryId };
try {
@@ -270,7 +272,7 @@ export class IngressReconcilerService implements OnModuleInit, OnModuleDestroy {
// letter while a copy can still deliver. A failed one falls through to its existing DLQ row.
const existing = await this.ingressEnqueue.existingJobState(jobData, row.providerDeliveryId);
// Shutdown began while the lookup was out (destroy may have stopped waiting for it).
- if (this.stopping) return retired;
+ if (this.stop.signal.aborted) return retired;
if (existing && existing !== 'failed') {
const settled = await this.events.update(
{ id: row.id, dispatchState: 'pending' },
@@ -375,7 +377,7 @@ export class IngressReconcilerService implements OnModuleInit, OnModuleDestroy {
const existing = await this.ingressEnqueue.existingJobState(jobData, row.providerDeliveryId);
// Shutdown began while the lookup was out, and destroy may have stopped waiting for it: a dispatch
// now could land after plugin teardown and spend an attempt. Leave the row for the next start.
- if (this.stopping) return 'interrupted';
+ if (this.stop.signal.aborted) return 'interrupted';
if (existing === 'failed') {
// Every queue attempt already ran and IngressProcessor dead-lettered the delivery, or re-queued it
// and every copy failed too: a live or completed copy answers for the job (see existingJobState).
@@ -410,9 +412,16 @@ export class IngressReconcilerService implements OnModuleInit, OnModuleDestroy {
});
return 'failed';
}
- const { outcome, error }: EnqueueOutcome = existing
+ // The add can also outlast destroy's wait (see onModuleDestroy). Given the stop signal, enqueue()
+ // throws instead of dispatching inline once it has aborted, and the row is left as above.
+ const result: EnqueueOutcome | undefined = existing
? { outcome: 'queued' }
- : await this.ingressEnqueue.enqueue(jobData, row.providerDeliveryId);
+ : await this.ingressEnqueue.enqueue(jobData, row.providerDeliveryId, this.stop.signal).catch(err => {
+ if (this.stop.signal.aborted) return undefined;
+ throw err;
+ });
+ if (!result) return 'interrupted';
+ const { outcome, error } = result;
if (outcome !== 'failed') {
// Retire the payload with the outcome: the dispatch tier owns the delivery from here (the
// BullMQ job data, or a DLQ row on an in-tier failure), so the dedup row slims to its marker.
@@ -433,7 +442,7 @@ export class IngressReconcilerService implements OnModuleInit, OnModuleDestroy {
// A failure once shutdown began says nothing about the event: the add may have been refused by a
// queue connection closing under it, or the inline fallback met a plugin being torn down. Spend no
// attempt and leave the row for the next start.
- if (this.stopping) return 'interrupted';
+ if (this.stop.signal.aborted) return 'interrupted';
const attempts = (row.dispatchAttempts ?? 0) + 1;
const terminal = attempts >= maxAttempts;
diff --git a/src/modules/integration/ingress.service.ts b/src/modules/integration/ingress.service.ts
index 607fd97a8..238cdbe8c 100644
--- a/src/modules/integration/ingress.service.ts
+++ b/src/modules/integration/ingress.service.ts
@@ -269,8 +269,8 @@ export class IngressService {
// Sync-response route: the ack is host-side and final; enqueue (queued or inline) must NOT block
// it — a queue-disabled deployment otherwise holds the HTTP response for up to the inline dispatch
// timeout. enqueue() is not awaited; the dedup row already persisted is the durability handle. The
- // .catch() is a defensive guard: enqueue() never rejects today (it swallows inline failures and the
- // factory wrapper writes a DLQ row on 'failed'), but a future regression must not become an unhandled
+ // .catch() is a defensive guard: enqueue() never rejects without a signal (it swallows inline failures
+ // and the factory wrapper writes a DLQ row on 'failed'), but a future regression must not become an unhandled
// rejection that crashes the process on the ingress hot path.
void this.deps.enqueue(jobData, deliveryId).catch(err => {
this.deps.log?.('ingress_enqueue_unhandled', {
diff --git a/src/modules/integration/integration-queue-wiring.spec.ts b/src/modules/integration/integration-queue-wiring.spec.ts
index 469307e02..95125431e 100644
--- a/src/modules/integration/integration-queue-wiring.spec.ts
+++ b/src/modules/integration/integration-queue-wiring.spec.ts
@@ -29,6 +29,7 @@ jest.mock('bullmq', () => {
this.name = name;
}
add = jest.fn().mockResolvedValue(undefined);
+ waitUntilReady = jest.fn().mockResolvedValue(undefined);
close = jest.fn().mockResolvedValue(undefined);
disconnect = jest.fn().mockResolvedValue(undefined);
}
diff --git a/src/modules/integration/integration-retention.service.spec.ts b/src/modules/integration/integration-retention.service.spec.ts
index 159be1471..ba08d2714 100644
--- a/src/modules/integration/integration-retention.service.spec.ts
+++ b/src/modules/integration/integration-retention.service.spec.ts
@@ -614,7 +614,7 @@ describe('IntegrationRetentionService.pruneOlderThan with undispatched events',
it('prunes aged settled markers while the hand-off is still waiting', async () => {
await insertEvent('dispatched', { dispatchState: 'dispatched', payload: null });
await insertEvent('stranded', {});
- // A job-state lookup against a queue that never became reachable does not return.
+ // A job-state lookup against a stalled Redis that has not returned yet.
const lookedUp = new Promise<void>(resolve =>
existingJobState.mockImplementation(() => {
resolve();
diff --git a/src/modules/metrics/metrics.service.spec.ts b/src/modules/metrics/metrics.service.spec.ts
index b4af95f1c..2e963a22f 100644
--- a/src/modules/metrics/metrics.service.spec.ts
+++ b/src/modules/metrics/metrics.service.spec.ts
@@ -3,6 +3,7 @@ import { ConfigService } from '@nestjs/config';
import { Queue } from 'bullmq';
import { MetricsService, METRICS_RENDER_TTL_MS, QUEUE_READ_TIMEOUT_MS } from './metrics.service';
import { StatsService, OverviewStats } from '../stats/stats.service';
+import { producerConnectWaitMs } from '../queue/redis-connection';
import { getWebhookDeliveryFailuresTotal } from '../../common/metrics/webhook-delivery-metrics';
import {
getSessionReconnectAttemptsTotal,
@@ -200,7 +201,8 @@ describe('MetricsService runtime series', () => {
};
const config = { get: () => undefined } as unknown as ConfigService;
const stats = { getOverview: jest.fn().mockResolvedValue(overview) } as unknown as StatsService;
- const queue = (getJobCounts: () => Promise<Record<string, number>>): Queue => ({ getJobCounts }) as unknown as Queue;
+ const queue = (getJobCounts: () => Promise<Record<string, number>>): Queue =>
+ ({ waitUntilReady: () => Promise.resolve(), getJobCounts }) as unknown as Queue;
const services: MetricsService[] = [];
const make = (webhook?: Queue, ingress?: Queue): MetricsService => {
const svc = new MetricsService(config, stats, webhook, ingress);
@@ -304,6 +306,22 @@ describe('MetricsService runtime series', () => {
expect(text).not.toContain('openwa_queue_jobs');
});
+ it('leaves no job-count read pending on a queue whose Redis has never connected', async () => {
+ jest.useFakeTimers();
+ const getJobCounts = jest.fn(() => Promise.resolve({ wait: 1, active: 0, delayed: 0, failed: 0 }));
+ const svc = make({ waitUntilReady: () => new Promise(() => undefined), getJobCounts } as unknown as Queue);
+
+ const rendering = svc.render();
+ await jest.advanceTimersByTimeAsync(QUEUE_READ_TIMEOUT_MS);
+ expect(await rendering).not.toContain('openwa_queue_jobs');
+
+ // Once the first-connect wait has passed, a scrape skips the queue without waiting.
+ await jest.advanceTimersByTimeAsync(producerConnectWaitMs());
+ (svc as unknown as { cachedRender: unknown }).cachedRender = null;
+ expect(await svc.render()).not.toContain('openwa_queue_jobs');
+ expect(getJobCounts).not.toHaveBeenCalled();
+ });
+
it('keeps the healthy queue when only the other one fails', async () => {
const text = await make(
queue(() => Promise.resolve({ wait: 1, active: 0, delayed: 0, failed: 0 })),
diff --git a/src/modules/metrics/metrics.service.ts b/src/modules/metrics/metrics.service.ts
index f910330b3..148dde049 100644
--- a/src/modules/metrics/metrics.service.ts
+++ b/src/modules/metrics/metrics.service.ts
@@ -12,6 +12,7 @@ import { ConfigService } from '@nestjs/config';
import { InjectQueue } from '@nestjs/bullmq';
import type { Queue } from 'bullmq';
import { QUEUE_NAMES } from '../queue/queue-names';
+import { producerReady } from '../queue/redis-connection';
import { constantTimeEqual } from '../../common/security/constantTimeEqual';
import { limiterKeyForIp, resolveClientIp, type RequestLike } from '../../common/utils/ip';
import { SlidingWindowLimiter } from '../events/ws-rate-limit';
@@ -264,7 +265,10 @@ export class MetricsService implements OnModuleDestroy {
return text;
}
- /** Job counts for each registered queue that answered within QUEUE_READ_TIMEOUT_MS. */
+ /**
+ * Job counts for each registered queue that answered within QUEUE_READ_TIMEOUT_MS. producerReady keeps a
+ * scrape from leaving a read pending on a Redis that has never connected.
+ */
private async readQueueCounts(): Promise<Array<[string, Record<string, number>]>> {
const queues = [
[QUEUE_NAMES.WEBHOOK, this.webhookQueue],
@@ -276,7 +280,7 @@ export class MetricsService implements OnModuleDestroy {
let timer: NodeJS.Timeout | undefined;
try {
const counts = await Promise.race([
- queue.getJobCounts(...QUEUE_JOB_STATES),
+ producerReady(queue).then(() => queue.getJobCounts(...QUEUE_JOB_STATES)),
new Promise<never>((_, reject) => {
timer = setTimeout(() => reject(new Error('timed out')), QUEUE_READ_TIMEOUT_MS);
}),
diff --git a/src/modules/queue/redis-connection.spec.ts b/src/modules/queue/redis-connection.spec.ts
index f1ae0b642..7d2856e38 100644
--- a/src/modules/queue/redis-connection.spec.ts
+++ b/src/modules/queue/redis-connection.spec.ts
@@ -1,5 +1,7 @@
import {
ingressWorkerConcurrency,
+ producerConnectWaitMs,
+ producerReady,
queueConnectionOptions,
workerConnectionOptions,
webhookWorkerConcurrency,
@@ -87,3 +89,86 @@ describe('webhookWorkerConcurrency', () => {
expect(ingressWorkerConcurrency()).toBe(10);
});
});
+
+describe('producerReady', () => {
+ const ORIGINAL_ENV = process.env;
+
+ beforeEach(() => jest.useFakeTimers());
+ afterEach(() => {
+ jest.useRealTimers();
+ process.env = ORIGINAL_ENV;
+ });
+
+ /** A queue whose first connect completes after `connectMs`, or never when undefined. */
+ const queueReadyAfter = (connectMs?: number) => ({
+ waitUntilReady: () =>
+ new Promise<never>(resolve => {
+ if (connectMs !== undefined) setTimeout(resolve, connectMs);
+ }),
+ });
+
+ /** Settle state of `promise` once the fake clock has advanced `ms`. */
+ async function settledAfter(promise: Promise<void>, ms: number): Promise<string> {
+ const state = promise.then(
+ () => 'resolved',
+ (error: Error) => `rejected: ${error.message}`,
+ );
+ let result = 'pending';
+ void state.then(s => (result = s));
+ await jest.advanceTimersByTimeAsync(ms);
+ return result;
+ }
+
+ it('waits for a first connect that lands within REDIS_CONNECT_TIMEOUT_MS', async () => {
+ process.env = { ...ORIGINAL_ENV, REDIS_CONNECT_TIMEOUT_MS: '1000' };
+ await expect(settledAfter(producerReady(queueReadyAfter(900)), 900)).resolves.toBe('resolved');
+ });
+
+ it('rejects once REDIS_CONNECT_TIMEOUT_MS passes without a connect', async () => {
+ process.env = { ...ORIGINAL_ENV, REDIS_CONNECT_TIMEOUT_MS: '1000' };
+ const ready = producerReady(queueReadyAfter());
+ await expect(settledAfter(ready, 999)).resolves.toBe('pending');
+ await expect(settledAfter(ready, 1)).resolves.toBe('rejected: Redis has not connected within 1000ms');
+ });
+
+ it('still waits the default for a first connect when REDIS_CONNECT_TIMEOUT_MS is 0', async () => {
+ // 0 turns off the socket connect timeout; it must not skip a healthy queue that is still connecting.
+ process.env = { ...ORIGINAL_ENV, REDIS_CONNECT_TIMEOUT_MS: '0' };
+ await expect(settledAfter(producerReady(queueReadyAfter(20)), 20)).resolves.toBe('resolved');
+
+ const never = producerReady(queueReadyAfter());
+ await expect(settledAfter(never, 4999)).resolves.toBe('pending');
+ await expect(settledAfter(never, 1)).resolves.toBe('rejected: Redis has not connected within 5000ms');
+ });
+
+ it('caps the wait at the Node timer ceiling instead of overflowing to 1 ms', async () => {
+ // Above the ceiling a Node timer fires after 1 ms and would skip a healthy queue that is connecting.
+ process.env = { ...ORIGINAL_ENV, REDIS_CONNECT_TIMEOUT_MS: '3000000000' };
+ expect(producerConnectWaitMs()).toBe(2147483647);
+ await expect(settledAfter(producerReady(queueReadyAfter()), 1000)).resolves.toBe('pending');
+ });
+
+ // Each wait on BullMQ's pending connect holds memory until Redis connects, which may be never.
+ it('shares one wait per queue: calls after an expired window reject at once until Redis connects', async () => {
+ process.env = { ...ORIGINAL_ENV, REDIS_CONNECT_TIMEOUT_MS: '1000' };
+ let connect!: () => void;
+ const queue = { waitUntilReady: jest.fn(() => new Promise<void>(resolve => (connect = resolve))) };
+
+ const first = producerReady(queue);
+ await expect(settledAfter(first, 600)).resolves.toBe('pending');
+ const inWindow = producerReady(queue);
+ await expect(settledAfter(inWindow, 400)).resolves.toBe('rejected: Redis has not connected within 1000ms');
+ await expect(settledAfter(first, 0)).resolves.toBe('rejected: Redis has not connected within 1000ms');
+
+ await expect(settledAfter(producerReady(queue), 0)).resolves.toBe(
+ 'rejected: Redis has not connected within 1000ms',
+ );
+ expect(queue.waitUntilReady).toHaveBeenCalledTimes(1);
+ expect(jest.getTimerCount()).toBe(0);
+
+ connect();
+ await jest.advanceTimersByTimeAsync(0);
+ await expect(settledAfter(producerReady(queue), 0)).resolves.toBe('resolved');
+ expect(queue.waitUntilReady).toHaveBeenCalledTimes(1);
+ });
+});
diff --git a/src/modules/queue/redis-connection.ts b/src/modules/queue/redis-connection.ts
index 3a41fedfb..13dc1ceda 100644
--- a/src/modules/queue/redis-connection.ts
+++ b/src/modules/queue/redis-connection.ts
@@ -1,15 +1,69 @@
-import { resolveNonNegativeIntEnv } from '../../config/configuration';
-import { redisConnectionOptions, type RedisConnectionOptions } from '../../config/redis-options';
+import type { Queue } from 'bullmq';
+import { MAX_TIMER_MS, resolveNonNegativeIntEnv } from '../../config/configuration';
+import {
+ DEFAULT_REDIS_CONNECT_TIMEOUT_MS,
+ redisConnectionOptions,
+ type RedisConnectionOptions,
+} from '../../config/redis-options';
/**
* BullMQ connection for the shared producer (`BullModule.forRootAsync` in queue.module.ts): the common
- * Redis fields plus `enableOfflineQueue: false`, so `queue.add()` fails fast when Redis is unreachable
- * and the dispatch path can fall back to direct delivery instead of buffering forever.
+ * Redis fields plus `enableOfflineQueue: false`, so once connected, `queue.add()` fails fast when Redis
+ * drops and the dispatch path can fall back to direct delivery instead of buffering forever. Before the
+ * first connect that option never applies: see producerReady.
*/
export function queueConnectionOptions(): RedisConnectionOptions & { enableOfflineQueue: false } {
return { ...redisConnectionOptions(), enableOfflineQueue: false };
}
+/**
+ * How long producerReady waits for a producer queue's first connect: REDIS_CONNECT_TIMEOUT_MS, or the
+ * default when it is 0 (see producerReady), capped at MAX_TIMER_MS. Above that a Node timer fires after
+ * 1 ms, which would bypass a healthy Redis at boot. Node truncates ioredis's socket connect timeout to
+ * the same ceiling.
+ */
+export function producerConnectWaitMs(): number {
+ return Math.min(redisConnectionOptions().connectTimeout || DEFAULT_REDIS_CONNECT_TIMEOUT_MS, MAX_TIMER_MS);
+}
+
+/** Per producer queue: whether its first connect has landed, and the shared wait for it. */
+const firstConnects = new WeakMap<object, { connected: boolean; wait: Promise<void> }>();
+
+/**
+ * Resolve once a producer queue has connected; reject if its first connect has not landed within
+ * REDIS_CONNECT_TIMEOUT_MS of the first call. Until that connect, BullMQ holds every queue call without
+ * sending a command, and its default retry strategy never gives up, so with Redis unreachable since
+ * boot a call never settles. Await this before each producer call so the caller's fallback runs
+ * instead. The wait is opened once per queue and shared: calls inside that window wait for its end,
+ * later calls reject at once until Redis connects (each call subscribing to BullMQ's pending connect
+ * would hold memory until it lands), and any call after a connect resolves at once. A setting of 0
+ * (no socket connect timeout in ioredis) still opens a window of the default: rejecting at once would
+ * bypass a healthy queue during startup, and waiting forever would bring the hang back. The window's
+ * timer is unref'd, so a wait still open at shutdown never holds the process alive.
+ */
+export async function producerReady(queue: Pick<Queue, 'waitUntilReady'>): Promise<void> {
+ let first = firstConnects.get(queue);
+ if (!first) {
+ const timeoutMs = producerConnectWaitMs();
+ const ready = queue.waitUntilReady();
+ let timer: NodeJS.Timeout | undefined;
+ const wait = Promise.race([
+ ready,
+ new Promise<never>((_resolve, reject) => {
+ timer = setTimeout(() => reject(new Error(`Redis has not connected within ${timeoutMs}ms`)), timeoutMs);
+ timer.unref();
+ }),
+ ]).finally(() => clearTimeout(timer));
+ const entry = { connected: false, wait };
+ ready.then(
+ () => (entry.connected = true),
+ () => undefined,
+ );
+ firstConnects.set(queue, (first = entry));
+ }
+ if (!first.connected) await first.wait;
+}
+
/**
* BullMQ connection for the webhook and ingress Workers. It must NOT inherit the producer-only
* fast-fail above: the Worker's blocking/internal commands (moveToActive, lock renewal) need to
diff --git a/src/modules/webhook/webhook-delivery.service.spec.ts b/src/modules/webhook/webhook-delivery.service.spec.ts
index 04a0f039c..ad3d8b519 100644
--- a/src/modules/webhook/webhook-delivery.service.spec.ts
+++ b/src/modules/webhook/webhook-delivery.service.spec.ts
@@ -35,6 +35,7 @@ import { LidMappingStoreService } from '../../engine/identity/lid-mapping-store.
import { userPart } from '../../engine/identity/wa-id';
import { HookManager } from '../../core/hooks';
import { QUEUE_NAMES } from '../queue/queue-names';
+import { producerConnectWaitMs } from '../queue/redis-connection';
import { getWebhookDeliveryFailuresTotal } from '../../common/metrics/webhook-delivery-metrics';
import { ConcurrencyLimiter } from '../../common/utils/concurrency-limiter';
@@ -169,6 +170,7 @@ describe('WebhookDeliveryService', () => {
webhookQueue = {
add: jest.fn().mockResolvedValue(undefined),
+ waitUntilReady: jest.fn().mockResolvedValue(undefined),
};
lidStore = {
@@ -826,6 +828,8 @@ describe('WebhookDeliveryService', () => {
await expect(
service.redeliver(webhook, 'sess-1', 'message.received', 'stored-key-2', { from: '[email protected]' }),
).resolves.toBe('delivered');
+ // The reconciler leaves a delivered replay's outbox row to this close.
+ expect(outboxService.close).toHaveBeenCalledWith(webhook.id, 'stored-key-2', 'dispatched');
});
it('reports a plugin-cancelled dispatch as cancelled, recording no failure and sending nothing', async () => {
@@ -1956,6 +1960,44 @@ describe('WebhookDeliveryService', () => {
expect.anything(),
);
});
+
+ // Until the first connect BullMQ holds every queue call without sending a command and retries the
+ // connection for ever, so enableOfflineQueue:false never rejects: with Redis unreachable since boot,
+ // each dispatch held its slot indefinitely and the reconciler's job lookup never answered.
+ it('delivers directly, and reads the job as pending, when Redis has not connected within the connect timeout', async () => {
+ const queueService = await buildQueueService(<T>(key: string, def?: T): T | boolean | number => {
+ if (key === 'queue.enabled') return true;
+ return def as T;
+ });
+ const webhook = createMockWebhook({ events: ['message.received'], retryCount: 1 });
+ const mockFetch = undiciFetch as jest.Mock;
+ (repository.find as jest.Mock).mockResolvedValue([webhook]);
+ (repository.update as jest.Mock).mockResolvedValue({ affected: 1 });
+ const pending = () => new Promise<never>(() => {});
+ webhookQueue.waitUntilReady.mockImplementation(pending);
+ webhookQueue.add.mockImplementation(pending);
+ webhookQueue.getJob = jest.fn(pending);
+ mockFetch.mockResolvedValue({ ok: true, status: 200 });
+ let settled: unknown;
+ jest.useFakeTimers({ doNotFake: ['nextTick', 'setImmediate', 'queueMicrotask'] });
+ try {
+ void Promise.all([
+ queueService.dispatch('sess-1', 'message.received', {}),
+ queueService.isQueueJobPending('job-1'),
+ ]).then(r => (settled = r));
+ await jest.advanceTimersByTimeAsync(producerConnectWaitMs());
+ } finally {
+ jest.useRealTimers();
+ }
+
+ expect(settled).toEqual([undefined, true]);
+ expect(webhookQueue.add).not.toHaveBeenCalled();
+ expect(webhookQueue.getJob).not.toHaveBeenCalled();
+ expect(mockFetch).toHaveBeenCalledWith(
+ 'https://example.com/webhook',
+ expect.objectContaining({ method: 'POST' }),
+ );
+ });
});
describe('direct retries', () => {
diff --git a/src/modules/webhook/webhook-delivery.service.ts b/src/modules/webhook/webhook-delivery.service.ts
index 40347f774..404dab91c 100644
--- a/src/modules/webhook/webhook-delivery.service.ts
+++ b/src/modules/webhook/webhook-delivery.service.ts
@@ -19,6 +19,7 @@ import { createLogger } from '../../common/services/logger.service';
import { DEFAULT_WEBHOOK_MEDIA_INLINE_MAX_BYTES, shedInlineMedia } from '../../common/utils/inline-media';
import { incrementWebhookDeliveryFailures } from '../../common/metrics/webhook-delivery-metrics';
import { QUEUE_NAMES } from '../queue/queue-names';
+import { producerReady } from '../queue/redis-connection';
import { generateIdempotencyKey, generateDeliveryId } from './utils/idempotency.util';
import { evaluateFilters } from './filters/filter-evaluator';
import { LidMappingStoreService } from '../../engine/identity/lid-mapping-store.service';
@@ -76,8 +77,8 @@ export const DEFAULT_WEBHOOK_SHUTDOWN_DRAIN_MS = 5000;
* handled in place, so a try/catch cannot tell a delivered event from a dead-lettered one. Two
* exceptions escape, both from shutdown, and both keep the outbox row pending: 'ConcurrencyLimiter
* closed' from a retry backoff woken after it, which only runLimited sees and records, and the
- * AbortError of a replay whose signal stopped its retries, which reaches WebhookReconcilerService
- * through redeliver.
+ * AbortError of a replay whose signal stopped its retries or its queue fallback, which reaches
+ * WebhookReconcilerService through redeliver.
* 'cancelled' is terminal like 'delivered' and must never be replayed: either a plugin suppressed the
* dispatch (nothing left the process), or a direct retry found the webhook removed, disabled or
* unsubscribed (an earlier attempt may already have been POSTed).
@@ -535,7 +536,8 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {
* Every failure path here is already handled in place (a dead-letter row, a hook, a log), so none
* of them reach the caller as an exception, except 'ConcurrencyLimiter closed' from a retry backoff
* woken after shutdown, which runLimited handles (redeliver passes no yieldSlot and never sees it),
- * and the AbortError of a replay whose signal stopped its retries, which reaches its caller.
+ * and the AbortError of a replay whose signal stopped its retries or its queue fallback, which
+ * reaches its caller.
* The reconciler has to tell a delivered event from a
* dead-lettered one to know whether the outbox row may be retired, and a caught throw cannot tell
* it: there is none. This mirrors the inbound twin, where `ingressEnqueue.enqueue` returns an
@@ -593,6 +595,7 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {
...(this.keepsFailurePayload() ? { replayData: ctx.baseData } : {}),
};
+ await producerReady(this.webhookQueue!);
await this.webhookQueue!.add(`webhook-${webhook.id}`, jobData, {
// jobId = deliveryId makes this add() idempotent within BullMQ (same precedent as the ingress
// producer). It does not dedup a crash replay: if the process dies before the outbox row is
@@ -625,6 +628,10 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {
action: 'webhook_queued',
});
} catch (error) {
+ // A replay whose owner stopped while the add was out (it can wait for a first connect, or on a
+ // stalled Redis) must not POST or run hooks after the drain: the reconciler reads the abort as
+ // interrupted.
+ ctx.signal?.throwIfAborted();
// Execute hook on queue error (not delivery error - that happens in processor)
await this.hookManager.execute(
'webhook:error',
@@ -637,7 +644,7 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {
action: 'webhook_queue_failed',
});
- // Fallback: deliver directly when the queue add failed (e.g. Redis unreachable with the
+ // Fallback: deliver directly when the queue add failed (Redis not connected yet, or down with the
// producer's enableOfflineQueue:false). This is at-least-once — if add() actually reached
// Redis before rejecting, the queued job AND this fallback may both POST. Both paths carry the
// same X-OpenWA-Idempotency-Key / X-OpenWA-Delivery-Id, so a conformant receiver dedupes.
@@ -961,6 +968,7 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {
async isQueueJobPending(deliveryId: string): Promise<boolean> {
if (!this.webhookQueue) return false;
try {
+ await producerReady(this.webhookQueue);
const job = await this.webhookQueue.getJob(deliveryId);
if (!job) return false;
return !['completed', 'failed', 'unknown'].includes(await job.getState());
diff --git a/src/modules/webhook/webhook-reconciler.service.spec.ts b/src/modules/webhook/webhook-reconciler.service.spec.ts
index 42dc8ceb7..1b169047d 100644
--- a/src/modules/webhook/webhook-reconciler.service.spec.ts
+++ b/src/modules/webhook/webhook-reconciler.service.spec.ts
@@ -81,7 +81,8 @@ describe('WebhookReconcilerService', () => {
{ from: '[email protected]' },
{ signal: expect.any(AbortSignal) as AbortSignal },
);
- expect(outbox.close).toHaveBeenCalledWith('wh-1', 'stored-key_wh-1', 'dispatched');
+ // redeliver retires a delivered event's row itself; a second close here only repeats the write.
+ expect(outbox.close).not.toHaveBeenCalled();
expect(stats).toMatchObject({ scanned: 1, replayed: 1 });
});
@@ -229,8 +230,8 @@ describe('WebhookReconcilerService', () => {
expect(outbox.findStale).toHaveBeenCalledTimes(1);
});
- // A queue lookup made while Redis has never been reachable does not return, and the destroy hooks
- // after this one (engines, session leases, plugins) must not wait on it for ever.
+ // A queue lookup against a stalled Redis may not return, and the destroy hooks after this one
+ // (engines, session leases, plugins) must not wait on it for ever.
it('stops waiting on destroy after WEBHOOK_SHUTDOWN_DRAIN_MS and replays nothing once the lookup returns', async () => {
outbox.findStale.mockResolvedValue([row({ deliveryId: 'job-1' })]);
let reached!: () => void;
diff --git a/src/modules/webhook/webhook-reconciler.service.ts b/src/modules/webhook/webhook-reconciler.service.ts
index 2d93aab82..4caa87545 100644
--- a/src/modules/webhook/webhook-reconciler.service.ts
+++ b/src/modules/webhook/webhook-reconciler.service.ts
@@ -99,8 +99,8 @@ export class WebhookReconcilerService implements OnModuleInit, OnModuleDestroy {
* and after PluginLoaderService has unregistered the `webhook:before` hooks. Stop it at the next row,
* stop the replay in hand without a further retry, and wait for it, at most WEBHOOK_SHUTDOWN_DRAIN_MS
* like the delivery drain beside it: the replay's hooks are not capped, and a queue lookup or add made
- * while Redis has never been reachable does not return. A row given up on stays 'pending' unless the
- * replay in hand completes before the process exits.
+ * before Redis has ever connected waits up to REDIS_CONNECT_TIMEOUT_MS. A row given up on stays
+ * 'pending' unless the replay in hand completes before the process exits.
*/
async onModuleDestroy(): Promise<void> {
this.stop.abort();
@@ -199,9 +199,9 @@ export class WebhookReconcilerService implements OnModuleInit, OnModuleDestroy {
stats.failed++;
continue;
}
- // Enqueued rows retain their copy until the worker settles them. Success and deliberate
- // cancellation can retire immediately.
- if (outcome !== 'enqueued') await this.outbox.close(row.webhookId, row.idempotencyKey, 'dispatched');
+ // Enqueued rows retain their copy until the worker settles them, and redeliver has already
+ // retired a delivered one. A deliberate cancellation retires here.
+ if (outcome === 'cancelled') await this.outbox.close(row.webhookId, row.idempotencyKey, 'dispatched');
stats.replayed++;
} catch (error) {
// A replay cut short in its retry backoff by shutdown was interrupted, not failed. Any other
diff --git a/src/modules/webhook/webhook-recovery-boundaries.spec.ts b/src/modules/webhook/webhook-recovery-boundaries.spec.ts
index 68e4414a9..c1324ebb1 100644
--- a/src/modules/webhook/webhook-recovery-boundaries.spec.ts
+++ b/src/modules/webhook/webhook-recovery-boundaries.spec.ts
@@ -12,6 +12,7 @@ import { WebhookOutboxService } from './webhook-outbox.service';
import { WebhookReconcilerService } from './webhook-reconciler.service';
import { WebhookRedriveService } from './webhook-redrive.service';
import { WebhookDeliveryService } from './webhook-delivery.service';
+import { producerConnectWaitMs } from '../queue/redis-connection';
describe('webhook recovery across live ownership and configuration changes', () => {
let ds: DataSource;
@@ -59,7 +60,11 @@ describe('webhook recovery across live ownership and configuration changes', ()
const delivery = {
isLocallyPending: jest.fn((key: string) => key === 'key0'),
isQueueJobPending: jest.fn((id: string) => Promise.resolve(id === 'job1' || id === 'job2')),
- redeliver: jest.fn().mockResolvedValue('delivered'),
+ // Like the real redeliver, a delivered replay retires its own outbox row.
+ redeliver: jest.fn(async (webhook: Webhook, _sessionId: string, _event: string, key: string) => {
+ await outbox.close(webhook.id, key, 'dispatched');
+ return 'delivered';
+ }),
recordReplayExhaustion: jest.fn().mockResolvedValue(true),
};
const reconciler = new WebhookReconcilerService(webhooks, outbox, delivery as never, new ConfigService());
@@ -288,6 +293,7 @@ describe('webhook recovery across live ownership and configuration changes', ()
const queue = {
add: jest.fn().mockRejectedValue(new Error('Connection is closed')),
getJob: jest.fn().mockResolvedValue(undefined),
+ waitUntilReady: jest.fn().mockResolvedValue(undefined),
};
const delivery = new WebhookDeliveryService(webhooks, failures, config, hooks, outbox, undefined, queue as never);
const reconciler = new WebhookReconcilerService(webhooks, outbox, delivery, config);
@@ -318,6 +324,53 @@ describe('webhook recovery across live ownership and configuration changes', ()
});
});
+ // Before Redis has ever connected, the queue add waits up to the connect timeout and then falls back
+ // to a direct POST. A replay whose owner stopped during that wait must send nothing and run no hooks.
+ it('sends nothing for a replay stopped while its queue add waited for a first connect', async () => {
+ const config = new ConfigService({ queue: { enabled: true }, webhook: { timeout: 1000 } });
+ const hooks = new HookManager();
+ const execute = jest.spyOn(hooks, 'execute');
+ let onWait!: () => void;
+ const waiting = new Promise<void>(resolve => (onWait = resolve));
+ const never = () => new Promise<never>(() => {});
+ const queue = {
+ add: jest.fn(never),
+ waitUntilReady: jest.fn(() => {
+ onWait();
+ return never();
+ }),
+ };
+ const outbox = new WebhookOutboxService(ds.getRepository(WebhookOutboxEvent));
+ const delivery = new WebhookDeliveryService(webhooks, failures, config, hooks, outbox, undefined, queue as never);
+ const post = jest
+ .spyOn(ssrf, 'withSafeFetch')
+ .mockImplementation((_url, _init, use) => Promise.resolve(use(new Response(null, { status: 200 }))));
+ const stop = new AbortController();
+
+ let result: unknown;
+ jest.useFakeTimers({ doNotFake: ['nextTick', 'setImmediate', 'queueMicrotask'] });
+ try {
+ const running = delivery
+ .redeliver(primary, primary.sessionId, 'message.received', 'key-a', {}, { signal: stop.signal })
+ .then(
+ outcome => outcome,
+ (error: Error) => error.name,
+ );
+ await waiting;
+ stop.abort();
+ await jest.advanceTimersByTimeAsync(producerConnectWaitMs());
+ result = await running;
+ } finally {
+ jest.useRealTimers();
+ }
+
+ expect(result).toBe('AbortError');
+ expect(queue.add).not.toHaveBeenCalled();
+ expect(post).not.toHaveBeenCalled();
+ expect(execute.mock.calls.map(([name]) => name).filter(name => name !== 'webhook:before')).toEqual([]);
+ expect(await failures.count()).toBe(0);
+ });
+
it('does not replay a row whose pre-checks were under way when destroy began', async () => {
const repository = ds.getRepository(WebhookOutboxEvent);
const outbox = new WebhookOutboxService(repository);