Free, Open Source, Self-Hosted WhatsApp API Gateway
TypeScript apibotgatewayself-hostedwhatsappwhatsapp-apiwhatsapp-automationwhatsapp-botwhatsapp-gatewaywhatsapp-gateway-apiwhatsapp-webwhatsapp-web-api
fix(queue): fail fast on producer calls before Redis has connected (#1836)
Yudhi Armyndharisff34251921c0
diff --git a/.env.example b/.env.exampleindex 5c4b2d68e..b42388dc3 100644--- a/.env.example+++ b/.env.example@@ -389,7 +389,10 @@ DATABASE_SYNCHRONIZE=false # WARNING: Set false in production! (data DB# Redis settings (auto-configured if REDIS_BUILTIN=true)# REDIS_HOST=localhost# REDIS_PORT=6379-# Fail Redis queue/cache connection attempts after this many ms.+# Fail Redis queue/cache connection attempts after this many ms. Queue calls made before Redis has+# ever connected wait at most this long from the first one, then take the same fallback as an+# unreachable Redis until it connects. 0 disables the socket connection timeout; that queue wait then+# still uses the default 5000.# REDIS_CONNECT_TIMEOUT_MS=5000# REDIS_USERNAME=# REDIS_PASSWORD=diff --git a/CHANGELOG.md b/CHANGELOG.mdindex c49616290..261006ff8 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -28,6 +28,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Fixed+- Fall back instead of hanging when an ingress, webhook or infra status queue call runs before Redis has ever connected.- Drop whatsapp-web.js inbound media over `MEDIA_DOWNLOAD_MAX_BYTES` in the page when the sender understates its size.- Trust `NODE_EXTRA_CA_CERTS` in `backup.sh`'s PostgreSQL TLS check on Node older than 22.15.- Set `PGSSLMODE` in the `psql` import command `restore.sh` prints when `DATABASE_SSL=true`.diff --git a/docs/25-integration-fabric.md b/docs/25-integration-fabric.mdindex 189bd15e1..863e03401 100644--- a/docs/25-integration-fabric.md+++ b/docs/25-integration-fabric.md@@ -231,8 +231,11 @@ job id; per-instance fairness via a token bucket; and a dead-letter record withqueued worker path, an **in-process** per-conversation lock prevents concurrent starts for the same lane;strict FIFO is not preserved across retry/redrive, and the lock is single-node state rather than Redis orPostgreSQL state. When the queue is disabled, ingress dispatches inline after persisting and does not-serialize concurrent same-conversation deliveries. Providers already deliver over unordered,-at-least-once HTTP, so plugin handlers must be idempotent and treat ingress as a reconciliation trigger.+serialize concurrent same-conversation deliveries. A queued delivery falls back to the same inline+dispatch when Redis is unreachable. Before Redis has ever connected, queue calls wait for it at most+`REDIS_CONNECT_TIMEOUT_MS` (5s when unset or 0) from the first one, then fall back at once until it+connects. Providers already deliver over unordered, at-least-once HTTP, so plugin handlers must be+idempotent and treat ingress as a reconciliation trigger.A job waiting on that lock still holds one of the `INGRESS_WORKER_CONCURRENCY` worker slots (default10). A burst on one lane larger than that fills every slot with same-lane waiters, and events for other@@ -275,12 +278,13 @@ whose dead-letter write fails is kept for the next run. Two cases get no dead-lequeue job that still owns the delivery takes the `dispatched` mark (closing any open dead-letterrow), and a row whose instance or bound session was deleted is dropped, since redrive refuses adeleted instance and a session delete purges its dead letters. A delivery whose completed job was-already removed from the queue, or whose job lookup failed while Redis was down, is dead-lettered-anyway and may already have been delivered: check before redriving it. The startup prune runs before-plugins load, so a dead-letter row it writes carries no conversation id and its redrive takes the-per-instance lane. There is deliberately no per-instance row-count cap: eviction under a flood of-forged delivery ids would silently drop legit dedup rows and re-admit their replays, which is worse-than the bounded growth it would prevent.+already removed from the queue, or whose job lookup failed while Redis was down (or, for a Redis+slow to start at boot, not yet connected within that first-connect wait), is dead-lettered anyway and+may already have been delivered: check before redriving it. A dead-letter row the startup prune writes+before plugins load carries no conversation id, and its redrive takes the per-instance lane. There is+deliberately no per-instance row-count cap: eviction under a flood of forged delivery ids would+silently drop legit dedup rows and re-admit their replays, which is worse than the bounded growth it+would prevent.## 25.8 The Integration SDK (v1)diff --git a/src/config/redis-options.ts b/src/config/redis-options.tsindex 3757ad48d..793d0d159 100644--- a/src/config/redis-options.ts+++ b/src/config/redis-options.ts@@ -13,6 +13,9 @@ export interface RedisConnectionOptions {tls?: TlsOptions;}+/** Default REDIS_CONNECT_TIMEOUT_MS, also the producer's first-connect wait when that is set to 0. */+export const DEFAULT_REDIS_CONNECT_TIMEOUT_MS = 5000;+/*** The connection fields every Redis client shares: the cache, the throttler store, the BullMQ* producer and workers, and the WebSocket fan-out pair. Each caller spreads this and adds only its@@ -30,7 +33,7 @@ export function redisConnectionOptions(): RedisConnectionOptions {port: parseInt(process.env.REDIS_PORT || '6379', 10),username: process.env.REDIS_USERNAME,password: process.env.REDIS_PASSWORD,- connectTimeout: parseInt(process.env.REDIS_CONNECT_TIMEOUT_MS || '5000', 10),+ connectTimeout: parseInt(process.env.REDIS_CONNECT_TIMEOUT_MS || String(DEFAULT_REDIS_CONNECT_TIMEOUT_MS), 10),...(process.env.REDIS_TLS === 'true' ? { tls: {} } : {}),};}diff --git a/src/modules/infra/infra-status.controller.spec.ts b/src/modules/infra/infra-status.controller.spec.tsindex de859efe4..40fd3f60f 100644--- a/src/modules/infra/infra-status.controller.spec.ts+++ b/src/modules/infra/infra-status.controller.spec.ts@@ -22,6 +22,7 @@ jest.mock('fs', () => {import { InfraStatusController } from './infra-status.controller';import { recordPinnedEnvKeys } from '../../config/env-precedence';+import { producerConnectWaitMs } from '../queue/redis-connection';describe('InfraStatusController.getStatus DB health (active SELECT 1 probe, not just isInitialized)', () => {const build = (query: jest.Mock) => {@@ -53,7 +54,10 @@ describe('InfraStatusController.getStatus DB health (active SELECT 1 probe, not});describe('InfraStatusController.getStatus queue job counts', () => {- function buildStatusController(opts: { queueEnabled: boolean; queue?: { getJobCounts: jest.Mock } }) {+ function buildStatusController(opts: {+ queueEnabled: boolean;+ queue?: { getJobCounts: jest.Mock; waitUntilReady?: jest.Mock };+ }) {const configService = {// engine.type=baileys skips the wa-web-version registry fetch (no network in unit tests).get: (key: string, def?: unknown) =>@@ -72,7 +76,7 @@ describe('InfraStatusController.getStatus queue job counts', () => {dockerService as never,cacheService as never,storageService as never,- opts.queue as never,+ (opts.queue && { waitUntilReady: jest.fn().mockResolvedValue(undefined), ...opts.queue }) as never,);}@@ -86,6 +90,24 @@ describe('InfraStatusController.getStatus queue job counts', () => {expect(status.queue).toEqual({ enabled: true, webhooks: { pending: 6, completed: 10, failed: 1 } });});+ // Until the first connect BullMQ holds a queue call for as long as Redis stays unreachable.+ it('reports zeros once the connect timeout passes when Redis has never connected', async () => {+ const getJobCounts = jest.fn();+ const waitUntilReady = jest.fn(() => new Promise<never>(() => {}));+ const controller = buildStatusController({ queueEnabled: true, queue: { getJobCounts, waitUntilReady } });+ let status: Awaited<ReturnType<InfraStatusController['getStatus']>> | undefined;+ jest.useFakeTimers({ doNotFake: ['nextTick', 'setImmediate', 'queueMicrotask'] });+ try {+ void controller.getStatus().then(s => (status = s));+ await jest.advanceTimersByTimeAsync(producerConnectWaitMs());+ } finally {+ jest.useRealTimers();+ }++ expect(getJobCounts).not.toHaveBeenCalled();+ expect(status?.queue).toEqual({ enabled: true, webhooks: { pending: 0, completed: 0, failed: 0 } });+ });+it('reports zeros (and does not touch the queue) when the queue is disabled', async () => {const getJobCounts = jest.fn();const controller = buildStatusController({ queueEnabled: false, queue: { getJobCounts } });diff --git a/src/modules/infra/infra-status.controller.ts b/src/modules/infra/infra-status.controller.tsindex 22c11f308..06c923fcf 100644--- a/src/modules/infra/infra-status.controller.ts+++ b/src/modules/infra/infra-status.controller.ts@@ -10,6 +10,7 @@ import {import { InjectQueue } from '@nestjs/bullmq';import { Queue } from 'bullmq';import { QUEUE_NAMES } from '../queue/queue-names';+import { producerReady } from '../queue/redis-connection';import { ConfigService } from '@nestjs/config';import { DataSource } from 'typeorm';import { InjectDataSource } from '@nestjs/typeorm';@@ -198,6 +199,7 @@ export class InfraStatusController {let webhooks = { pending: 0, completed: 0, failed: 0 };if (queueEnabled && this.webhookQueue) {try {+ await producerReady(this.webhookQueue);const counts = await this.webhookQueue.getJobCounts('wait', 'active', 'delayed', 'completed', 'failed');webhooks = {pending: (counts.wait ?? 0) + (counts.active ?? 0) + (counts.delayed ?? 0),diff --git a/src/modules/integration/ingress-enqueue.service.spec.ts b/src/modules/integration/ingress-enqueue.service.spec.tsindex 07f36e2af..fa292915c 100644--- a/src/modules/integration/ingress-enqueue.service.spec.ts+++ b/src/modules/integration/ingress-enqueue.service.spec.ts@@ -6,6 +6,7 @@ import {} from './ingress-enqueue.service';import { PluginLoaderService } from '../../core/plugins/plugin-loader.service';import { ConfigService } from '@nestjs/config';+import { producerConnectWaitMs } from '../queue/redis-connection';describe('IngressEnqueueService', () => {const data = {@@ -19,12 +20,12 @@ describe('IngressEnqueueService', () => {let loader: jest.Mocked<Partial<PluginLoaderService>>;let config: jest.Mocked<Partial<ConfigService>>;- let queue: { add: jest.Mock };+ let queue: { add: jest.Mock; waitUntilReady: jest.Mock };beforeEach(() => {loader = { dispatchWebhookForInstance: jest.fn().mockResolvedValue(undefined) };config = { get: jest.fn() };- queue = { add: jest.fn().mockResolvedValue(undefined) };+ queue = { add: jest.fn().mockResolvedValue(undefined), waitUntilReady: jest.fn().mockResolvedValue(undefined) };});it('adds a job to the ingress queue keyed by the namespaced jobId when queueing is enabled and a queue is present', async () => {@@ -249,6 +250,37 @@ describe('IngressEnqueueService', () => {expect(loader.dispatchWebhookForInstance).toHaveBeenCalledWith(data);});+ // Until the first connect, BullMQ holds every queue call without sending a command and retries the+ // connection for ever, so enableOfflineQueue:false never rejects: with Redis unreachable since boot,+ // the ingress request, a redrive and the reconciler's row in hand would wait indefinitely.+ it('falls back to inline dispatch and reports no job when Redis has not connected within the connect timeout', async () => {+ (config.get as jest.Mock).mockReturnValue(true);+ const pending = () => new Promise<never>(() => {});+ queue.waitUntilReady.mockImplementation(pending);+ queue.add.mockImplementation(pending);+ const getJobState = jest.fn(pending);+ const svc = new IngressEnqueueService(+ loader as PluginLoaderService,+ config as ConfigService,+ { ...queue, getJobState } as never,+ );+ let results: unknown;+ jest.useFakeTimers();+ try {+ void Promise.all([svc.existingJobState(data, 'd1'), svc.enqueue(data, 'd1')]).then(r => (results = r));+ await jest.advanceTimersByTimeAsync(producerConnectWaitMs() - 1);+ expect(results).toBeUndefined();+ await jest.advanceTimersByTimeAsync(1);+ } finally {+ jest.useRealTimers();+ }++ expect(results).toEqual([undefined, { outcome: 'dispatched' }]);+ expect(getJobState).not.toHaveBeenCalled();+ expect(queue.add).not.toHaveBeenCalled();+ expect(loader.dispatchWebhookForInstance).toHaveBeenCalledWith(data);+ });+describe('resolveIngressJobOptions', () => {it('defaults to 3 attempts with exponential backoff', () => {expect(resolveIngressJobOptions()).toEqual({ attempts: 3, backoff: { type: 'exponential', delay: 5000 } });diff --git a/src/modules/integration/ingress-enqueue.service.ts b/src/modules/integration/ingress-enqueue.service.tsindex c5dee0b7b..e38b91799 100644--- a/src/modules/integration/ingress-enqueue.service.ts+++ b/src/modules/integration/ingress-enqueue.service.ts@@ -7,16 +7,18 @@ import { PluginLoaderService } from '../../core/plugins/plugin-loader.service';import { IngressJobData } from '../queue/processors/ingress.processor';import { IntegrationDeliveryFailure } from './entities/integration-delivery-failure.entity';import { QUEUE_NAMES } from '../queue/queue-names';+import { producerReady } from '../queue/redis-connection';import { createLogger } from '../../common/services/logger.service';import { resolveNonNegativeIntEnv } from '../../config/configuration';/*** Outcome of an enqueue attempt. 'queued' = handed to BullMQ; 'dispatched' = delivered inline; 'failed'- * = inline dispatch threw and was swallowed (`error` carries the message). enqueue() never throws, so- * callers use the outcome (not exceptions) to decide durability follow-up: a LIVE ingress delivery must- * persist a dead-letter row on 'failed' (see buildIngressDeadLetterRow, wired at the IngressService- * factory) so RedriveService can replay it; RedriveService itself calls enqueue() directly, because a- * failed replay must keep its EXISTING dead-letter row redrivable rather than write a second one.+ * = inline dispatch threw and was swallowed (`error` carries the message). enqueue() does not throw+ * (bar an aborted `signal`, see enqueue), so callers use the outcome (not exceptions) to decide+ * durability follow-up: a LIVE ingress delivery must persist a dead-letter row on 'failed' (see+ * buildIngressDeadLetterRow, wired at the IngressService factory) so RedriveService can replay it;+ * RedriveService itself calls enqueue() directly, because a failed replay must keep its EXISTING+ * dead-letter row redrivable rather than write a second one.*/export type EnqueueOutcome = { outcome: 'queued' | 'dispatched' | 'failed'; error?: string };@@ -147,6 +149,7 @@ export class IngressEnqueueService implements OnApplicationBootstrap {async existingJobState(data: IngressJobData, jobId: string): Promise<JobState | undefined> {if (!this.config.get<boolean>('queue.enabled', false) || !this.ingressQueue) return undefined;try {+ await producerReady(this.ingressQueue);const id = queueJobId(data, jobId);const state = await this.ingressQueue.getJobState(id);if (isLiveOrCompleted(state)) return state;@@ -156,7 +159,8 @@ export class IngressEnqueueService implements OnApplicationBootstrap {}return state === 'unknown' ? undefined : state;} catch (err) {- // Redis unreachable: enqueue() then takes its own inline fallback, as it would without the probe.+ // Redis unreachable or not yet connected: enqueue() then takes its own inline fallback, as it would+ // without the probe.this.logger.warn('Ingress job state lookup failed', {pluginId: data.pluginId,instanceId: data.instanceId,@@ -168,7 +172,12 @@ export class IngressEnqueueService implements OnApplicationBootstrap {}}- async enqueue(data: IngressJobData, jobId: string): Promise<EnqueueOutcome> {+ /**+ * `signal` lets a caller that stops at shutdown (the reconciler) refuse the inline fallback once it+ * has aborted: a queue add that fails after the stop rethrows the abort reason instead of+ * dispatching, since the plugin may already be torn down. That is the only way enqueue() throws.+ */+ async enqueue(data: IngressJobData, jobId: string, signal?: AbortSignal): Promise<EnqueueOutcome> {const queueEnabled = this.config.get<boolean>('queue.enabled', false);const useQueue = queueEnabled && !!this.ingressQueue;@@ -178,6 +187,7 @@ export class IngressEnqueueService implements OnApplicationBootstrap {// exponential-backoff attempts so a transient failure retries before landing in the DLQ. The id// is sanitized because BullMQ refuses several id shapes at add() (see sanitizeIngressJobId),// which would otherwise read as a Redis failure here and fall through to inline dispatch.+ await producerReady(this.ingressQueue);await this.ingressQueue.add('ingress', data, {jobId: queueJobId(data, jobId),...resolveIngressJobOptions(),@@ -185,9 +195,11 @@ export class IngressEnqueueService implements OnApplicationBootstrap {return { outcome: 'queued' };} catch (err) {// Redis unreachable (enableOfflineQueue:false makes add() reject) — fall through to inline- // dispatch. Without this, the already-persisted event would be lost forever: the throw would- // 500 the ingress request, the provider retries, dedup returns "duplicate", and no job was- // ever enqueued (no DLQ row either). Mirrors WebhookService's queue-add fallback.+ // dispatch (before the first connect, producerReady rejects instead). Without this, the+ // already-persisted event would be lost forever: the throw would 500 the ingress request, the+ // provider retries, dedup returns "duplicate", and no job was ever enqueued (no DLQ row either).+ // Mirrors WebhookService's queue-add fallback.+ signal?.throwIfAborted();this.logger.error('Ingress queue add failed; dispatching inline',err instanceof Error ? err.message : String(err),diff --git a/src/modules/integration/ingress-reconciler.service.spec.ts b/src/modules/integration/ingress-reconciler.service.spec.tsindex b4144b157..3499e6dce 100644--- a/src/modules/integration/ingress-reconciler.service.spec.ts+++ b/src/modules/integration/ingress-reconciler.service.spec.ts@@ -381,10 +381,14 @@ describe('IngressReconcilerService.sweep', () => {// attempt: IngressProcessor already dead-lettered it. A replay under the same jobId is swallowed by// BullMQ as a duplicate (add() still resolves), so it must not count as delivered.describe('when the live path already queued a job for the delivery', () => {- let queue: { add: jest.Mock; getJobState: jest.Mock };+ let queue: { add: jest.Mock; getJobState: jest.Mock; waitUntilReady: jest.Mock };beforeEach(() => {- queue = { add: jest.fn().mockResolvedValue(undefined), getJobState: jest.fn() };+ queue = {+ add: jest.fn().mockResolvedValue(undefined),+ getJobState: jest.fn(),+ waitUntilReady: jest.fn().mockResolvedValue(undefined),+ };const realEnqueue = new IngressEnqueueService({ dispatchWebhookForInstance: jest.fn() } as unknown as PluginLoaderService,{ get: jest.fn().mockReturnValue(true) } as unknown as ConfigService,@@ -703,8 +707,8 @@ describe('IngressReconcilerService.sweep', () => {expect(await failures.count()).toBe(0);});- // A job-state lookup made while Redis has never been reachable only returns once the queue connection- // is closed later in shutdown. Waiting for it kept every later destroy hook from running; dispatching+ // A job-state lookup against a stalled Redis may only return once the queue connection is closed+ // later in shutdown. Waiting for it kept every later destroy hook from running; dispatching// once it returns would land after plugin teardown and dead-letter a row on its last attempt.it('stops waiting for a job-state lookup that does not return, and leaves its row pending', async () => {const id = await insertEvent({ dispatchAttempts: 4 });@@ -722,7 +726,11 @@ describe('IngressReconcilerService.sweep', () => {new IngressEnqueueService({ dispatchWebhookForInstance } as unknown as PluginLoaderService,{ get: jest.fn().mockReturnValue(true) } as unknown as ConfigService,- { getJobState, add: jest.fn().mockRejectedValue(new Error('Connection is closed')) } as never,+ {+ getJobState,+ add: jest.fn().mockRejectedValue(new Error('Connection is closed')),+ waitUntilReady: jest.fn().mockResolvedValue(undefined),+ } as never,),{ getPlugin } as unknown as PluginLoaderService,{ resolve: resolveInstance } as unknown as PluginInstanceService,@@ -756,6 +764,48 @@ describe('IngressReconcilerService.sweep', () => {expect((await stored(id)).payload).not.toBeNull();expect(await failures.count()).toBe(0);});++ // A queue add held by a stalled Redis can fail after destroy has begun. Its inline fallback would then+ // race plugin teardown and, on the row's last attempt, dead-letter it.+ it('does not dispatch inline when the queue add fails after shutdown began', async () => {+ const id = await insertEvent({ dispatchAttempts: 4 });+ let failAdd!: (error: Error) => void;+ let onAdd!: () => void;+ const adding = new Promise<void>(resolve => (onAdd = resolve));+ const queue = {+ getJobState: jest.fn().mockResolvedValue('unknown'),+ add: jest.fn(() => {+ onAdd();+ return new Promise<never>((_resolve, reject) => (failAdd = reject));+ }),+ waitUntilReady: jest.fn().mockResolvedValue(undefined),+ };+ const dispatchWebhookForInstance = jest.fn().mockRejectedValue(new Error('no live sandbox host'));+ service = new IngressReconcilerService(+ events,+ failures,+ new IngressEnqueueService(+ { dispatchWebhookForInstance } as unknown as PluginLoaderService,+ { get: jest.fn().mockReturnValue(true) } as unknown as ConfigService,+ queue as never,+ ),+ { getPlugin } as unknown as PluginLoaderService,+ { resolve: resolveInstance } as unknown as PluginInstanceService,+ );++ const sweep = service.sweep(OPTS);+ await adding;+ const destroyed = service.onModuleDestroy();+ failAdd(new Error('Connection is closed.'));+ const stats = await sweep;+ await destroyed;++ expect(stats).toMatchObject({ replayed: 0, failed: 0, skipped: 1 });+ expect(dispatchWebhookForInstance).not.toHaveBeenCalled();+ expect(await stored(id)).toMatchObject({ dispatchState: 'pending', dispatchAttempts: 4 });+ expect((await stored(id)).payload).not.toBeNull();+ expect(await failures.count()).toBe(0);+ });});describe('IngressReconcilerService.onModuleInit (scheduling)', () => {diff --git a/src/modules/integration/ingress-reconciler.service.ts b/src/modules/integration/ingress-reconciler.service.tsindex 4d7f2e767..994e5098b 100644--- a/src/modules/integration/ingress-reconciler.service.ts+++ b/src/modules/integration/ingress-reconciler.service.ts@@ -79,7 +79,7 @@ export class IngressReconcilerService implements OnModuleInit, OnModuleDestroy {private timer?: ReturnType<typeof setInterval>;// The pass in flight, settled when it ends; doubles as the overlap guard.private inFlight?: Promise<void>;- private stopping = false;+ private readonly stop = new AbortController();constructor(@InjectRepository(IngressEvent, 'data') private readonly events: Repository<IngressEvent>,@@ -109,12 +109,14 @@ export class IngressReconcilerService implements OnModuleInit, OnModuleDestroy {* rows after PluginLoaderService has stopped the plugin sandboxes, so each remaining row fails,* spends an attempt and, at its last one, is dead-lettered. Stop it at the next row and wait for* the row in hand, while its plugin is still running. The wait is bounded by the inline dispatch- * timeout: a queue lookup made while Redis has never been reachable does not return, and waiting on it- * would keep every later destroy hook (engines, session leases, plugins) from running.- * A row given up on stays 'pending': once its lookup returns, the row sees the stop and writes nothing.+ * timeout: before Redis has ever connected, the row's queue lookup can wait up to+ * REDIS_CONNECT_TIMEOUT_MS ahead of that dispatch, and a stalled Redis holds a lookup or add longer+ * still; waiting it out would delay every later destroy hook (engines, session leases, plugins).+ * A row given up on is never dispatched inline after the stop: it stays 'pending', unless its queue+ * add still succeeds and the queued job takes the delivery.*/async onModuleDestroy(): Promise<void> {- this.stopping = true;+ this.stop.abort();if (this.timer) clearInterval(this.timer);if (!this.inFlight) return;let timer: NodeJS.Timeout | undefined;@@ -137,7 +139,7 @@ export class IngressReconcilerService implements OnModuleInit, OnModuleDestroy {*/async sweep(opts: IngressReconcilerOptions, now: Date = new Date()): Promise<IngressReconcileStats> {const stats: IngressReconcileStats = { scanned: 0, replayed: 0, failed: 0, skipped: 0 };- if (this.inFlight || this.stopping) return stats;+ if (this.inFlight || this.stop.signal.aborted) return stats;let settle!: () => void;this.inFlight = new Promise(resolve => (settle = resolve));try {@@ -161,7 +163,7 @@ export class IngressReconcilerService implements OnModuleInit, OnModuleDestroy {.limit(opts.batchSize).getMany();for (const row of rows) {- if (this.stopping) break;+ if (this.stop.signal.aborted) break;// A row whose latest attempt is still inside the grace window cools down between replays;// it keeps its batch slot (bounded by maxAttempts, so the leak is capped) but is not hit again.if (row.lastDispatchAt && row.lastDispatchAt > cutoff) {@@ -234,7 +236,7 @@ export class IngressReconcilerService implements OnModuleInit, OnModuleDestroy {* dead letter. Returns the number of rows dead-lettered.*/async deadLetterAgedPending(cutoff: Date, batchSize = 100): Promise<number> {- if (this.inFlight || this.stopping) return 0;+ if (this.inFlight || this.stop.signal.aborted) return 0;let settle!: () => void;this.inFlight = new Promise(resolve => (settle = resolve));let retired = 0;@@ -247,7 +249,7 @@ export class IngressReconcilerService implements OnModuleInit, OnModuleDestroy {});let progressed = 0;for (const row of rows) {- if (this.stopping) return retired;+ if (this.stop.signal.aborted) return retired;if (!hasPayload(row)) continue;const meta = { pluginId: row.pluginId, instanceId: row.instanceId, deliveryId: row.providerDeliveryId };try {@@ -270,7 +272,7 @@ export class IngressReconcilerService implements OnModuleInit, OnModuleDestroy {// letter while a copy can still deliver. A failed one falls through to its existing DLQ row.const existing = await this.ingressEnqueue.existingJobState(jobData, row.providerDeliveryId);// Shutdown began while the lookup was out (destroy may have stopped waiting for it).- if (this.stopping) return retired;+ if (this.stop.signal.aborted) return retired;if (existing && existing !== 'failed') {const settled = await this.events.update({ id: row.id, dispatchState: 'pending' },@@ -375,7 +377,7 @@ export class IngressReconcilerService implements OnModuleInit, OnModuleDestroy {const existing = await this.ingressEnqueue.existingJobState(jobData, row.providerDeliveryId);// Shutdown began while the lookup was out, and destroy may have stopped waiting for it: a dispatch// now could land after plugin teardown and spend an attempt. Leave the row for the next start.- if (this.stopping) return 'interrupted';+ if (this.stop.signal.aborted) return 'interrupted';if (existing === 'failed') {// Every queue attempt already ran and IngressProcessor dead-lettered the delivery, or re-queued it// and every copy failed too: a live or completed copy answers for the job (see existingJobState).@@ -410,9 +412,16 @@ export class IngressReconcilerService implements OnModuleInit, OnModuleDestroy {});return 'failed';}- const { outcome, error }: EnqueueOutcome = existing+ // The add can also outlast destroy's wait (see onModuleDestroy). Given the stop signal, enqueue()+ // throws instead of dispatching inline once it has aborted, and the row is left as above.+ const result: EnqueueOutcome | undefined = existing? { outcome: 'queued' }- : await this.ingressEnqueue.enqueue(jobData, row.providerDeliveryId);+ : await this.ingressEnqueue.enqueue(jobData, row.providerDeliveryId, this.stop.signal).catch(err => {+ if (this.stop.signal.aborted) return undefined;+ throw err;+ });+ if (!result) return 'interrupted';+ const { outcome, error } = result;if (outcome !== 'failed') {// Retire the payload with the outcome: the dispatch tier owns the delivery from here (the// BullMQ job data, or a DLQ row on an in-tier failure), so the dedup row slims to its marker.@@ -433,7 +442,7 @@ export class IngressReconcilerService implements OnModuleInit, OnModuleDestroy {// A failure once shutdown began says nothing about the event: the add may have been refused by a// queue connection closing under it, or the inline fallback met a plugin being torn down. Spend no// attempt and leave the row for the next start.- if (this.stopping) return 'interrupted';+ if (this.stop.signal.aborted) return 'interrupted';const attempts = (row.dispatchAttempts ?? 0) + 1;const terminal = attempts >= maxAttempts;diff --git a/src/modules/integration/ingress.service.ts b/src/modules/integration/ingress.service.tsindex 607fd97a8..238cdbe8c 100644--- a/src/modules/integration/ingress.service.ts+++ b/src/modules/integration/ingress.service.ts@@ -269,8 +269,8 @@ export class IngressService {// Sync-response route: the ack is host-side and final; enqueue (queued or inline) must NOT block// it — a queue-disabled deployment otherwise holds the HTTP response for up to the inline dispatch// timeout. enqueue() is not awaited; the dedup row already persisted is the durability handle. The- // .catch() is a defensive guard: enqueue() never rejects today (it swallows inline failures and the- // factory wrapper writes a DLQ row on 'failed'), but a future regression must not become an unhandled+ // .catch() is a defensive guard: enqueue() never rejects without a signal (it swallows inline failures+ // and the factory wrapper writes a DLQ row on 'failed'), but a future regression must not become an unhandled// rejection that crashes the process on the ingress hot path.void this.deps.enqueue(jobData, deliveryId).catch(err => {this.deps.log?.('ingress_enqueue_unhandled', {diff --git a/src/modules/integration/integration-queue-wiring.spec.ts b/src/modules/integration/integration-queue-wiring.spec.tsindex 469307e02..95125431e 100644--- a/src/modules/integration/integration-queue-wiring.spec.ts+++ b/src/modules/integration/integration-queue-wiring.spec.ts@@ -29,6 +29,7 @@ jest.mock('bullmq', () => {this.name = name;}add = jest.fn().mockResolvedValue(undefined);+ waitUntilReady = jest.fn().mockResolvedValue(undefined);close = jest.fn().mockResolvedValue(undefined);disconnect = jest.fn().mockResolvedValue(undefined);}diff --git a/src/modules/integration/integration-retention.service.spec.ts b/src/modules/integration/integration-retention.service.spec.tsindex 159be1471..ba08d2714 100644--- a/src/modules/integration/integration-retention.service.spec.ts+++ b/src/modules/integration/integration-retention.service.spec.ts@@ -614,7 +614,7 @@ describe('IntegrationRetentionService.pruneOlderThan with undispatched events',it('prunes aged settled markers while the hand-off is still waiting', async () => {await insertEvent('dispatched', { dispatchState: 'dispatched', payload: null });await insertEvent('stranded', {});- // A job-state lookup against a queue that never became reachable does not return.+ // A job-state lookup against a stalled Redis that has not returned yet.const lookedUp = new Promise<void>(resolve =>existingJobState.mockImplementation(() => {resolve();diff --git a/src/modules/metrics/metrics.service.spec.ts b/src/modules/metrics/metrics.service.spec.tsindex b4af95f1c..2e963a22f 100644--- a/src/modules/metrics/metrics.service.spec.ts+++ b/src/modules/metrics/metrics.service.spec.ts@@ -3,6 +3,7 @@ import { ConfigService } from '@nestjs/config';import { Queue } from 'bullmq';import { MetricsService, METRICS_RENDER_TTL_MS, QUEUE_READ_TIMEOUT_MS } from './metrics.service';import { StatsService, OverviewStats } from '../stats/stats.service';+import { producerConnectWaitMs } from '../queue/redis-connection';import { getWebhookDeliveryFailuresTotal } from '../../common/metrics/webhook-delivery-metrics';import {getSessionReconnectAttemptsTotal,@@ -200,7 +201,8 @@ describe('MetricsService runtime series', () => {};const config = { get: () => undefined } as unknown as ConfigService;const stats = { getOverview: jest.fn().mockResolvedValue(overview) } as unknown as StatsService;- const queue = (getJobCounts: () => Promise<Record<string, number>>): Queue => ({ getJobCounts }) as unknown as Queue;+ const queue = (getJobCounts: () => Promise<Record<string, number>>): Queue =>+ ({ waitUntilReady: () => Promise.resolve(), getJobCounts }) as unknown as Queue;const services: MetricsService[] = [];const make = (webhook?: Queue, ingress?: Queue): MetricsService => {const svc = new MetricsService(config, stats, webhook, ingress);@@ -304,6 +306,22 @@ describe('MetricsService runtime series', () => {expect(text).not.toContain('openwa_queue_jobs');});+ it('leaves no job-count read pending on a queue whose Redis has never connected', async () => {+ jest.useFakeTimers();+ const getJobCounts = jest.fn(() => Promise.resolve({ wait: 1, active: 0, delayed: 0, failed: 0 }));+ const svc = make({ waitUntilReady: () => new Promise(() => undefined), getJobCounts } as unknown as Queue);++ const rendering = svc.render();+ await jest.advanceTimersByTimeAsync(QUEUE_READ_TIMEOUT_MS);+ expect(await rendering).not.toContain('openwa_queue_jobs');++ // Once the first-connect wait has passed, a scrape skips the queue without waiting.+ await jest.advanceTimersByTimeAsync(producerConnectWaitMs());+ (svc as unknown as { cachedRender: unknown }).cachedRender = null;+ expect(await svc.render()).not.toContain('openwa_queue_jobs');+ expect(getJobCounts).not.toHaveBeenCalled();+ });+it('keeps the healthy queue when only the other one fails', async () => {const text = await make(queue(() => Promise.resolve({ wait: 1, active: 0, delayed: 0, failed: 0 })),diff --git a/src/modules/metrics/metrics.service.ts b/src/modules/metrics/metrics.service.tsindex f910330b3..148dde049 100644--- a/src/modules/metrics/metrics.service.ts+++ b/src/modules/metrics/metrics.service.ts@@ -12,6 +12,7 @@ import { ConfigService } from '@nestjs/config';import { InjectQueue } from '@nestjs/bullmq';import type { Queue } from 'bullmq';import { QUEUE_NAMES } from '../queue/queue-names';+import { producerReady } from '../queue/redis-connection';import { constantTimeEqual } from '../../common/security/constantTimeEqual';import { limiterKeyForIp, resolveClientIp, type RequestLike } from '../../common/utils/ip';import { SlidingWindowLimiter } from '../events/ws-rate-limit';@@ -264,7 +265,10 @@ export class MetricsService implements OnModuleDestroy {return text;}- /** Job counts for each registered queue that answered within QUEUE_READ_TIMEOUT_MS. */+ /**+ * Job counts for each registered queue that answered within QUEUE_READ_TIMEOUT_MS. producerReady keeps a+ * scrape from leaving a read pending on a Redis that has never connected.+ */private async readQueueCounts(): Promise<Array<[string, Record<string, number>]>> {const queues = [[QUEUE_NAMES.WEBHOOK, this.webhookQueue],@@ -276,7 +280,7 @@ export class MetricsService implements OnModuleDestroy {let timer: NodeJS.Timeout | undefined;try {const counts = await Promise.race([- queue.getJobCounts(...QUEUE_JOB_STATES),+ producerReady(queue).then(() => queue.getJobCounts(...QUEUE_JOB_STATES)),new Promise<never>((_, reject) => {timer = setTimeout(() => reject(new Error('timed out')), QUEUE_READ_TIMEOUT_MS);}),diff --git a/src/modules/queue/redis-connection.spec.ts b/src/modules/queue/redis-connection.spec.tsindex f1ae0b642..7d2856e38 100644--- a/src/modules/queue/redis-connection.spec.ts+++ b/src/modules/queue/redis-connection.spec.ts@@ -1,5 +1,7 @@import {ingressWorkerConcurrency,+ producerConnectWaitMs,+ producerReady,queueConnectionOptions,workerConnectionOptions,webhookWorkerConcurrency,@@ -87,3 +89,86 @@ describe('webhookWorkerConcurrency', () => {expect(ingressWorkerConcurrency()).toBe(10);});});++describe('producerReady', () => {+ const ORIGINAL_ENV = process.env;++ beforeEach(() => jest.useFakeTimers());+ afterEach(() => {+ jest.useRealTimers();+ process.env = ORIGINAL_ENV;+ });++ /** A queue whose first connect completes after `connectMs`, or never when undefined. */+ const queueReadyAfter = (connectMs?: number) => ({+ waitUntilReady: () =>+ new Promise<never>(resolve => {+ if (connectMs !== undefined) setTimeout(resolve, connectMs);+ }),+ });++ /** Settle state of `promise` once the fake clock has advanced `ms`. */+ async function settledAfter(promise: Promise<void>, ms: number): Promise<string> {+ const state = promise.then(+ () => 'resolved',+ (error: Error) => `rejected: ${error.message}`,+ );+ let result = 'pending';+ void state.then(s => (result = s));+ await jest.advanceTimersByTimeAsync(ms);+ return result;+ }++ it('waits for a first connect that lands within REDIS_CONNECT_TIMEOUT_MS', async () => {+ process.env = { ...ORIGINAL_ENV, REDIS_CONNECT_TIMEOUT_MS: '1000' };+ await expect(settledAfter(producerReady(queueReadyAfter(900)), 900)).resolves.toBe('resolved');+ });++ it('rejects once REDIS_CONNECT_TIMEOUT_MS passes without a connect', async () => {+ process.env = { ...ORIGINAL_ENV, REDIS_CONNECT_TIMEOUT_MS: '1000' };+ const ready = producerReady(queueReadyAfter());+ await expect(settledAfter(ready, 999)).resolves.toBe('pending');+ await expect(settledAfter(ready, 1)).resolves.toBe('rejected: Redis has not connected within 1000ms');+ });++ it('still waits the default for a first connect when REDIS_CONNECT_TIMEOUT_MS is 0', async () => {+ // 0 turns off the socket connect timeout; it must not skip a healthy queue that is still connecting.+ process.env = { ...ORIGINAL_ENV, REDIS_CONNECT_TIMEOUT_MS: '0' };+ await expect(settledAfter(producerReady(queueReadyAfter(20)), 20)).resolves.toBe('resolved');++ const never = producerReady(queueReadyAfter());+ await expect(settledAfter(never, 4999)).resolves.toBe('pending');+ await expect(settledAfter(never, 1)).resolves.toBe('rejected: Redis has not connected within 5000ms');+ });++ it('caps the wait at the Node timer ceiling instead of overflowing to 1 ms', async () => {+ // Above the ceiling a Node timer fires after 1 ms and would skip a healthy queue that is connecting.+ process.env = { ...ORIGINAL_ENV, REDIS_CONNECT_TIMEOUT_MS: '3000000000' };+ expect(producerConnectWaitMs()).toBe(2147483647);+ await expect(settledAfter(producerReady(queueReadyAfter()), 1000)).resolves.toBe('pending');+ });++ // Each wait on BullMQ's pending connect holds memory until Redis connects, which may be never.+ it('shares one wait per queue: calls after an expired window reject at once until Redis connects', async () => {+ process.env = { ...ORIGINAL_ENV, REDIS_CONNECT_TIMEOUT_MS: '1000' };+ let connect!: () => void;+ const queue = { waitUntilReady: jest.fn(() => new Promise<void>(resolve => (connect = resolve))) };++ const first = producerReady(queue);+ await expect(settledAfter(first, 600)).resolves.toBe('pending');+ const inWindow = producerReady(queue);+ await expect(settledAfter(inWindow, 400)).resolves.toBe('rejected: Redis has not connected within 1000ms');+ await expect(settledAfter(first, 0)).resolves.toBe('rejected: Redis has not connected within 1000ms');++ await expect(settledAfter(producerReady(queue), 0)).resolves.toBe(+ 'rejected: Redis has not connected within 1000ms',+ );+ expect(queue.waitUntilReady).toHaveBeenCalledTimes(1);+ expect(jest.getTimerCount()).toBe(0);++ connect();+ await jest.advanceTimersByTimeAsync(0);+ await expect(settledAfter(producerReady(queue), 0)).resolves.toBe('resolved');+ expect(queue.waitUntilReady).toHaveBeenCalledTimes(1);+ });+});diff --git a/src/modules/queue/redis-connection.ts b/src/modules/queue/redis-connection.tsindex 3a41fedfb..13dc1ceda 100644--- a/src/modules/queue/redis-connection.ts+++ b/src/modules/queue/redis-connection.ts@@ -1,15 +1,69 @@-import { resolveNonNegativeIntEnv } from '../../config/configuration';-import { redisConnectionOptions, type RedisConnectionOptions } from '../../config/redis-options';+import type { Queue } from 'bullmq';+import { MAX_TIMER_MS, resolveNonNegativeIntEnv } from '../../config/configuration';+import {+ DEFAULT_REDIS_CONNECT_TIMEOUT_MS,+ redisConnectionOptions,+ type RedisConnectionOptions,+} from '../../config/redis-options';/*** BullMQ connection for the shared producer (`BullModule.forRootAsync` in queue.module.ts): the common- * Redis fields plus `enableOfflineQueue: false`, so `queue.add()` fails fast when Redis is unreachable- * and the dispatch path can fall back to direct delivery instead of buffering forever.+ * Redis fields plus `enableOfflineQueue: false`, so once connected, `queue.add()` fails fast when Redis+ * drops and the dispatch path can fall back to direct delivery instead of buffering forever. Before the+ * first connect that option never applies: see producerReady.*/export function queueConnectionOptions(): RedisConnectionOptions & { enableOfflineQueue: false } {return { ...redisConnectionOptions(), enableOfflineQueue: false };}+/**+ * How long producerReady waits for a producer queue's first connect: REDIS_CONNECT_TIMEOUT_MS, or the+ * default when it is 0 (see producerReady), capped at MAX_TIMER_MS. Above that a Node timer fires after+ * 1 ms, which would bypass a healthy Redis at boot. Node truncates ioredis's socket connect timeout to+ * the same ceiling.+ */+export function producerConnectWaitMs(): number {+ return Math.min(redisConnectionOptions().connectTimeout || DEFAULT_REDIS_CONNECT_TIMEOUT_MS, MAX_TIMER_MS);+}++/** Per producer queue: whether its first connect has landed, and the shared wait for it. */+const firstConnects = new WeakMap<object, { connected: boolean; wait: Promise<void> }>();++/**+ * Resolve once a producer queue has connected; reject if its first connect has not landed within+ * REDIS_CONNECT_TIMEOUT_MS of the first call. Until that connect, BullMQ holds every queue call without+ * sending a command, and its default retry strategy never gives up, so with Redis unreachable since+ * boot a call never settles. Await this before each producer call so the caller's fallback runs+ * instead. The wait is opened once per queue and shared: calls inside that window wait for its end,+ * later calls reject at once until Redis connects (each call subscribing to BullMQ's pending connect+ * would hold memory until it lands), and any call after a connect resolves at once. A setting of 0+ * (no socket connect timeout in ioredis) still opens a window of the default: rejecting at once would+ * bypass a healthy queue during startup, and waiting forever would bring the hang back. The window's+ * timer is unref'd, so a wait still open at shutdown never holds the process alive.+ */+export async function producerReady(queue: Pick<Queue, 'waitUntilReady'>): Promise<void> {+ let first = firstConnects.get(queue);+ if (!first) {+ const timeoutMs = producerConnectWaitMs();+ const ready = queue.waitUntilReady();+ let timer: NodeJS.Timeout | undefined;+ const wait = Promise.race([+ ready,+ new Promise<never>((_resolve, reject) => {+ timer = setTimeout(() => reject(new Error(`Redis has not connected within ${timeoutMs}ms`)), timeoutMs);+ timer.unref();+ }),+ ]).finally(() => clearTimeout(timer));+ const entry = { connected: false, wait };+ ready.then(+ () => (entry.connected = true),+ () => undefined,+ );+ firstConnects.set(queue, (first = entry));+ }+ if (!first.connected) await first.wait;+}+/*** BullMQ connection for the webhook and ingress Workers. It must NOT inherit the producer-only* fast-fail above: the Worker's blocking/internal commands (moveToActive, lock renewal) need todiff --git a/src/modules/webhook/webhook-delivery.service.spec.ts b/src/modules/webhook/webhook-delivery.service.spec.tsindex 04a0f039c..ad3d8b519 100644--- a/src/modules/webhook/webhook-delivery.service.spec.ts+++ b/src/modules/webhook/webhook-delivery.service.spec.ts@@ -35,6 +35,7 @@ import { LidMappingStoreService } from '../../engine/identity/lid-mapping-store.import { userPart } from '../../engine/identity/wa-id';import { HookManager } from '../../core/hooks';import { QUEUE_NAMES } from '../queue/queue-names';+import { producerConnectWaitMs } from '../queue/redis-connection';import { getWebhookDeliveryFailuresTotal } from '../../common/metrics/webhook-delivery-metrics';import { ConcurrencyLimiter } from '../../common/utils/concurrency-limiter';@@ -169,6 +170,7 @@ describe('WebhookDeliveryService', () => {webhookQueue = {add: jest.fn().mockResolvedValue(undefined),+ waitUntilReady: jest.fn().mockResolvedValue(undefined),};lidStore = {@@ -826,6 +828,8 @@ describe('WebhookDeliveryService', () => {await expect(service.redeliver(webhook, 'sess-1', 'message.received', 'stored-key-2', { from: '[email protected]' }),).resolves.toBe('delivered');+ // The reconciler leaves a delivered replay's outbox row to this close.+ expect(outboxService.close).toHaveBeenCalledWith(webhook.id, 'stored-key-2', 'dispatched');});it('reports a plugin-cancelled dispatch as cancelled, recording no failure and sending nothing', async () => {@@ -1956,6 +1960,44 @@ describe('WebhookDeliveryService', () => {expect.anything(),);});++ // Until the first connect BullMQ holds every queue call without sending a command and retries the+ // connection for ever, so enableOfflineQueue:false never rejects: with Redis unreachable since boot,+ // each dispatch held its slot indefinitely and the reconciler's job lookup never answered.+ it('delivers directly, and reads the job as pending, when Redis has not connected within the connect timeout', async () => {+ const queueService = await buildQueueService(<T>(key: string, def?: T): T | boolean | number => {+ if (key === 'queue.enabled') return true;+ return def as T;+ });+ const webhook = createMockWebhook({ events: ['message.received'], retryCount: 1 });+ const mockFetch = undiciFetch as jest.Mock;+ (repository.find as jest.Mock).mockResolvedValue([webhook]);+ (repository.update as jest.Mock).mockResolvedValue({ affected: 1 });+ const pending = () => new Promise<never>(() => {});+ webhookQueue.waitUntilReady.mockImplementation(pending);+ webhookQueue.add.mockImplementation(pending);+ webhookQueue.getJob = jest.fn(pending);+ mockFetch.mockResolvedValue({ ok: true, status: 200 });+ let settled: unknown;+ jest.useFakeTimers({ doNotFake: ['nextTick', 'setImmediate', 'queueMicrotask'] });+ try {+ void Promise.all([+ queueService.dispatch('sess-1', 'message.received', {}),+ queueService.isQueueJobPending('job-1'),+ ]).then(r => (settled = r));+ await jest.advanceTimersByTimeAsync(producerConnectWaitMs());+ } finally {+ jest.useRealTimers();+ }++ expect(settled).toEqual([undefined, true]);+ expect(webhookQueue.add).not.toHaveBeenCalled();+ expect(webhookQueue.getJob).not.toHaveBeenCalled();+ expect(mockFetch).toHaveBeenCalledWith(+ 'https://example.com/webhook',+ expect.objectContaining({ method: 'POST' }),+ );+ });});describe('direct retries', () => {diff --git a/src/modules/webhook/webhook-delivery.service.ts b/src/modules/webhook/webhook-delivery.service.tsindex 40347f774..404dab91c 100644--- a/src/modules/webhook/webhook-delivery.service.ts+++ b/src/modules/webhook/webhook-delivery.service.ts@@ -19,6 +19,7 @@ import { createLogger } from '../../common/services/logger.service';import { DEFAULT_WEBHOOK_MEDIA_INLINE_MAX_BYTES, shedInlineMedia } from '../../common/utils/inline-media';import { incrementWebhookDeliveryFailures } from '../../common/metrics/webhook-delivery-metrics';import { QUEUE_NAMES } from '../queue/queue-names';+import { producerReady } from '../queue/redis-connection';import { generateIdempotencyKey, generateDeliveryId } from './utils/idempotency.util';import { evaluateFilters } from './filters/filter-evaluator';import { LidMappingStoreService } from '../../engine/identity/lid-mapping-store.service';@@ -76,8 +77,8 @@ export const DEFAULT_WEBHOOK_SHUTDOWN_DRAIN_MS = 5000;* handled in place, so a try/catch cannot tell a delivered event from a dead-lettered one. Two* exceptions escape, both from shutdown, and both keep the outbox row pending: 'ConcurrencyLimiter* closed' from a retry backoff woken after it, which only runLimited sees and records, and the- * AbortError of a replay whose signal stopped its retries, which reaches WebhookReconcilerService- * through redeliver.+ * AbortError of a replay whose signal stopped its retries or its queue fallback, which reaches+ * WebhookReconcilerService through redeliver.* 'cancelled' is terminal like 'delivered' and must never be replayed: either a plugin suppressed the* dispatch (nothing left the process), or a direct retry found the webhook removed, disabled or* unsubscribed (an earlier attempt may already have been POSTed).@@ -535,7 +536,8 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {* Every failure path here is already handled in place (a dead-letter row, a hook, a log), so none* of them reach the caller as an exception, except 'ConcurrencyLimiter closed' from a retry backoff* woken after shutdown, which runLimited handles (redeliver passes no yieldSlot and never sees it),- * and the AbortError of a replay whose signal stopped its retries, which reaches its caller.+ * and the AbortError of a replay whose signal stopped its retries or its queue fallback, which+ * reaches its caller.* The reconciler has to tell a delivered event from a* dead-lettered one to know whether the outbox row may be retired, and a caught throw cannot tell* it: there is none. This mirrors the inbound twin, where `ingressEnqueue.enqueue` returns an@@ -593,6 +595,7 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {...(this.keepsFailurePayload() ? { replayData: ctx.baseData } : {}),};+ await producerReady(this.webhookQueue!);await this.webhookQueue!.add(`webhook-${webhook.id}`, jobData, {// jobId = deliveryId makes this add() idempotent within BullMQ (same precedent as the ingress// producer). It does not dedup a crash replay: if the process dies before the outbox row is@@ -625,6 +628,10 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {action: 'webhook_queued',});} catch (error) {+ // A replay whose owner stopped while the add was out (it can wait for a first connect, or on a+ // stalled Redis) must not POST or run hooks after the drain: the reconciler reads the abort as+ // interrupted.+ ctx.signal?.throwIfAborted();// Execute hook on queue error (not delivery error - that happens in processor)await this.hookManager.execute('webhook:error',@@ -637,7 +644,7 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {action: 'webhook_queue_failed',});- // Fallback: deliver directly when the queue add failed (e.g. Redis unreachable with the+ // Fallback: deliver directly when the queue add failed (Redis not connected yet, or down with the// producer's enableOfflineQueue:false). This is at-least-once — if add() actually reached// Redis before rejecting, the queued job AND this fallback may both POST. Both paths carry the// same X-OpenWA-Idempotency-Key / X-OpenWA-Delivery-Id, so a conformant receiver dedupes.@@ -961,6 +968,7 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {async isQueueJobPending(deliveryId: string): Promise<boolean> {if (!this.webhookQueue) return false;try {+ await producerReady(this.webhookQueue);const job = await this.webhookQueue.getJob(deliveryId);if (!job) return false;return !['completed', 'failed', 'unknown'].includes(await job.getState());diff --git a/src/modules/webhook/webhook-reconciler.service.spec.ts b/src/modules/webhook/webhook-reconciler.service.spec.tsindex 42dc8ceb7..1b169047d 100644--- a/src/modules/webhook/webhook-reconciler.service.spec.ts+++ b/src/modules/webhook/webhook-reconciler.service.spec.ts@@ -81,7 +81,8 @@ describe('WebhookReconcilerService', () => {{ from: '[email protected]' },{ signal: expect.any(AbortSignal) as AbortSignal },);- expect(outbox.close).toHaveBeenCalledWith('wh-1', 'stored-key_wh-1', 'dispatched');+ // redeliver retires a delivered event's row itself; a second close here only repeats the write.+ expect(outbox.close).not.toHaveBeenCalled();expect(stats).toMatchObject({ scanned: 1, replayed: 1 });});@@ -229,8 +230,8 @@ describe('WebhookReconcilerService', () => {expect(outbox.findStale).toHaveBeenCalledTimes(1);});- // A queue lookup made while Redis has never been reachable does not return, and the destroy hooks- // after this one (engines, session leases, plugins) must not wait on it for ever.+ // A queue lookup against a stalled Redis may not return, and the destroy hooks after this one+ // (engines, session leases, plugins) must not wait on it for ever.it('stops waiting on destroy after WEBHOOK_SHUTDOWN_DRAIN_MS and replays nothing once the lookup returns', async () => {outbox.findStale.mockResolvedValue([row({ deliveryId: 'job-1' })]);let reached!: () => void;diff --git a/src/modules/webhook/webhook-reconciler.service.ts b/src/modules/webhook/webhook-reconciler.service.tsindex 2d93aab82..4caa87545 100644--- a/src/modules/webhook/webhook-reconciler.service.ts+++ b/src/modules/webhook/webhook-reconciler.service.ts@@ -99,8 +99,8 @@ export class WebhookReconcilerService implements OnModuleInit, OnModuleDestroy {* and after PluginLoaderService has unregistered the `webhook:before` hooks. Stop it at the next row,* stop the replay in hand without a further retry, and wait for it, at most WEBHOOK_SHUTDOWN_DRAIN_MS* like the delivery drain beside it: the replay's hooks are not capped, and a queue lookup or add made- * while Redis has never been reachable does not return. A row given up on stays 'pending' unless the- * replay in hand completes before the process exits.+ * before Redis has ever connected waits up to REDIS_CONNECT_TIMEOUT_MS. A row given up on stays+ * 'pending' unless the replay in hand completes before the process exits.*/async onModuleDestroy(): Promise<void> {this.stop.abort();@@ -199,9 +199,9 @@ export class WebhookReconcilerService implements OnModuleInit, OnModuleDestroy {stats.failed++;continue;}- // Enqueued rows retain their copy until the worker settles them. Success and deliberate- // cancellation can retire immediately.- if (outcome !== 'enqueued') await this.outbox.close(row.webhookId, row.idempotencyKey, 'dispatched');+ // Enqueued rows retain their copy until the worker settles them, and redeliver has already+ // retired a delivered one. A deliberate cancellation retires here.+ if (outcome === 'cancelled') await this.outbox.close(row.webhookId, row.idempotencyKey, 'dispatched');stats.replayed++;} catch (error) {// A replay cut short in its retry backoff by shutdown was interrupted, not failed. Any otherdiff --git a/src/modules/webhook/webhook-recovery-boundaries.spec.ts b/src/modules/webhook/webhook-recovery-boundaries.spec.tsindex 68e4414a9..c1324ebb1 100644--- a/src/modules/webhook/webhook-recovery-boundaries.spec.ts+++ b/src/modules/webhook/webhook-recovery-boundaries.spec.ts@@ -12,6 +12,7 @@ import { WebhookOutboxService } from './webhook-outbox.service';import { WebhookReconcilerService } from './webhook-reconciler.service';import { WebhookRedriveService } from './webhook-redrive.service';import { WebhookDeliveryService } from './webhook-delivery.service';+import { producerConnectWaitMs } from '../queue/redis-connection';describe('webhook recovery across live ownership and configuration changes', () => {let ds: DataSource;@@ -59,7 +60,11 @@ describe('webhook recovery across live ownership and configuration changes', ()const delivery = {isLocallyPending: jest.fn((key: string) => key === 'key0'),isQueueJobPending: jest.fn((id: string) => Promise.resolve(id === 'job1' || id === 'job2')),- redeliver: jest.fn().mockResolvedValue('delivered'),+ // Like the real redeliver, a delivered replay retires its own outbox row.+ redeliver: jest.fn(async (webhook: Webhook, _sessionId: string, _event: string, key: string) => {+ await outbox.close(webhook.id, key, 'dispatched');+ return 'delivered';+ }),recordReplayExhaustion: jest.fn().mockResolvedValue(true),};const reconciler = new WebhookReconcilerService(webhooks, outbox, delivery as never, new ConfigService());@@ -288,6 +293,7 @@ describe('webhook recovery across live ownership and configuration changes', ()const queue = {add: jest.fn().mockRejectedValue(new Error('Connection is closed')),getJob: jest.fn().mockResolvedValue(undefined),+ waitUntilReady: jest.fn().mockResolvedValue(undefined),};const delivery = new WebhookDeliveryService(webhooks, failures, config, hooks, outbox, undefined, queue as never);const reconciler = new WebhookReconcilerService(webhooks, outbox, delivery, config);@@ -318,6 +324,53 @@ describe('webhook recovery across live ownership and configuration changes', ()});});+ // Before Redis has ever connected, the queue add waits up to the connect timeout and then falls back+ // to a direct POST. A replay whose owner stopped during that wait must send nothing and run no hooks.+ it('sends nothing for a replay stopped while its queue add waited for a first connect', async () => {+ const config = new ConfigService({ queue: { enabled: true }, webhook: { timeout: 1000 } });+ const hooks = new HookManager();+ const execute = jest.spyOn(hooks, 'execute');+ let onWait!: () => void;+ const waiting = new Promise<void>(resolve => (onWait = resolve));+ const never = () => new Promise<never>(() => {});+ const queue = {+ add: jest.fn(never),+ waitUntilReady: jest.fn(() => {+ onWait();+ return never();+ }),+ };+ const outbox = new WebhookOutboxService(ds.getRepository(WebhookOutboxEvent));+ const delivery = new WebhookDeliveryService(webhooks, failures, config, hooks, outbox, undefined, queue as never);+ const post = jest+ .spyOn(ssrf, 'withSafeFetch')+ .mockImplementation((_url, _init, use) => Promise.resolve(use(new Response(null, { status: 200 }))));+ const stop = new AbortController();++ let result: unknown;+ jest.useFakeTimers({ doNotFake: ['nextTick', 'setImmediate', 'queueMicrotask'] });+ try {+ const running = delivery+ .redeliver(primary, primary.sessionId, 'message.received', 'key-a', {}, { signal: stop.signal })+ .then(+ outcome => outcome,+ (error: Error) => error.name,+ );+ await waiting;+ stop.abort();+ await jest.advanceTimersByTimeAsync(producerConnectWaitMs());+ result = await running;+ } finally {+ jest.useRealTimers();+ }++ expect(result).toBe('AbortError');+ expect(queue.add).not.toHaveBeenCalled();+ expect(post).not.toHaveBeenCalled();+ expect(execute.mock.calls.map(([name]) => name).filter(name => name !== 'webhook:before')).toEqual([]);+ expect(await failures.count()).toBe(0);+ });+it('does not replay a row whose pre-checks were under way when destroy began', async () => {const repository = ds.getRepository(WebhookOutboxEvent);const outbox = new WebhookOutboxService(repository);