Free, Open Source, Self-Hosted WhatsApp API Gateway
TypeScript apibotgatewayself-hostedwhatsappwhatsapp-apiwhatsapp-automationwhatsapp-botwhatsapp-gatewaywhatsapp-gateway-apiwhatsapp-webwhatsapp-web-api
fix(shutdown): fit the teardown inside the kill deadline (#1855)
Yudhi Armyndharis8a45366a2877
diff --git a/.env.example b/.env.exampleindex 1be46afbc..0027a833d 100644--- a/.env.example+++ b/.env.example@@ -72,8 +72,10 @@ STATUS_SEED_ON_READY=false# Graceful-shutdown drain: on SIGTERM/SIGINT the app flips readiness to 503 (so a load balancer stops# routing) and keeps serving in-flight requests for this many ms before teardown. Default 3000 unless# NODE_ENV is development or test (0 there, so a dev hot-reload/Ctrl+C is not slowed); an unset-# NODE_ENV also gets 3000. Capped at 30000. A second signal forces an immediate exit. In Docker, keep-# `stop_grace_period` >= this + your worst-case teardown.+# NODE_ENV also gets 3000. Capped at 30000. A second signal forces an immediate exit. At the default 3 s+# delay, the shipped 75 s `stop_grace_period` is this delay plus a 67 s teardown limit plus a 5 s+# margin; raising this moves the in-process force-exit later by the same amount, so raise+# `stop_grace_period` by the same amount.# SHUTDOWN_DELAY_MS=3000# HTTP server timeouts. Raise these when a reverse proxy in front holds connections longer.@@ -459,11 +461,13 @@ WEBHOOK_RETRY_DELAY=5000 # Base delay between retries in ms (doubles# 4/3 x WEBHOOK_MEDIA_INLINE_MAX_BYTES as base64), up to about# 1.3 GiB per process at the defaults; QUEUE_ENABLED=true keeps# it in Redis instead.-# WEBHOOK_SHUTDOWN_DRAIN_MS=5000 # on shutdown, in-flight deliveries and an outbox replay in- # flight get this long to finish+# WEBHOOK_SHUTDOWN_DRAIN_MS=5000 # on shutdown, in-flight deliveries, an outbox replay and an+ # operator redrive in flight get this long to finish# before they are logged abandoned (parked ones are# dead-lettered). Startup warns when this is shorter than# WEBHOOK_TIMEOUT — that cross silently truncates deliveries.+ # Raising it moves the in-process force-exit and the needed+ # `stop_grace_period` later by the same amount.# WEBHOOK_MAX_PER_SESSION=16 # max webhooks registered per session; new registrations at/over# the cap get 400 (existing ones are grandfathered); 0 = unlimited# AUTOMATION_MAX_PER_SESSION=32 # max autoreply rules per session; new rules at/over the cap getdiff --git a/CHANGELOG.md b/CHANGELOG.mdindex 82fa704ff..ac49af93b 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -69,8 +69,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0- Stop the dashboard Chats page from marking the open chat read on a newly selected session.- Refuse bulk batches once shutdown begins, and answer `503` for a batch saved during shutdown instead of sending it.- Stop in-flight webhook replay, ingress replay and pending-message sweeps at shutdown, waiting a bounded time for a replay in hand.-- Close queue workers, waiting up to 15 s for running jobs, before plugins shut down, so no new job is taken against stopped plugins.+- Close queue workers, waiting up to 10 s for running jobs, before plugins shut down, so no new job is taken against stopped plugins.- Answer new HTTP requests other than health probes with `503` once teardown begins after SIGTERM, SIGINT or `POST /api/infra/restart`.+- Disable plugins in parallel at shutdown, capping the whole plugin phase, including waits for enables in progress, at 10 s.+- Bound the webhook redrive wait at shutdown by `WEBHOOK_SHUTDOWN_DRAIN_MS`, like the delivery drain and reconciler.+- Exit with status 1 when teardown runs 67 s past the shutdown delay, so a stalled close no longer runs into the kill deadline or hangs `POST /api/infra/restart`.+- Destroy open HTTP connections and WebSocket clients when the server closes at shutdown, so an open request cannot hold up the database close.+- Stop waiting for a queue worker whose close outlasts its 10 s cap, so it cannot hold up the database close.+- Settle a failed or hung queue producer close within 2 s per queue at shutdown, so a Redis outage or network partition no longer skips the database close.+- Raise `stop_grace_period` and `terminationGracePeriodSeconds` from 45 to 75 s to cover the worst-case teardown.- Stop dashboard multi-page loads and a pending chat mark-as-read from sending requests after logout.- Keep a new dashboard login signed in when a request sent with the previous API key fails afterwards.- Enforce the plugin package size limit before reading `manifest.json`, and reject archive entries whose path starts with `/`.@@ -130,6 +137,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Upgrade notes (behavior changes)+- Set `stop_grace_period` (Docker) or `terminationGracePeriodSeconds` (Kubernetes) to at least 75 s if you pinned your own 45 s value, plus the excess of `SHUTDOWN_DELAY_MS` over 3 s and of `WEBHOOK_SHUTDOWN_DRAIN_MS` over 5 s.+- A stop that runs 67 s past the shutdown delay now exits with status 1 instead of being killed (137), which matters if you alert on exit codes.+- A sandboxed plugin's `onDisable` now shares one 10 s cap with all other plugins at shutdown, and a worker still running then dies with the process.+- `docker-compose.dev.yml` now waits up to 75 s on a stuck stop instead of Docker's 10 s default.+- The queue worker close cap fell from 15 s to 10 s, so a queued webhook or plugin ingress job still running then is re-run after the next start; receivers must deduplicate on `X-OpenWA-Idempotency-Key`.- Boot fails on a `PORT` with surrounding whitespace or a whitespace-only `PORT`; remove the padding.- Aged undispatched ingress events are dead-lettered instead of deleted; some may already have been delivered, so check before redriving them.- Java SDK (next SDK release after 0.5.1): the request timeout (default 30 s) also bounds the response body; raise it for large media downloads.diff --git a/charts/openwa/values.yaml b/charts/openwa/values.yamlindex 41bd573fe..d55d0e462 100644--- a/charts/openwa/values.yaml+++ b/charts/openwa/values.yaml@@ -77,9 +77,14 @@ resources:memory: 2Gicpu: 1000m-# Matches docker-compose stop_grace_period: graceful drain + per-engine Chromium-# teardown can exceed the 30s k8s default.-terminationGracePeriodSeconds: 45+# Matches docker-compose stop_grace_period. Sized for the worst case (QUEUE_ENABLED, Redis, a plugin+# enabled), where the bounded teardown stages add up to 63s: drain 3 + ingress reconciler 5 + engines+# 10 (in parallel, whatever the session count) + auto-start wait 5 + webhook drain 5 + worker close 10+# + producer queue close 2 for each of two queues + usage flush 5 + plugin phase 10 + three Redis quits+# 2 each. The process force-exits 67s after the drain and 5s before this deadline. Raising+# SHUTDOWN_DELAY_MS or WEBHOOK_SHUTDOWN_DRAIN_MS moves that exit later by the same amount, so raise+# this by the same amount too.+terminationGracePeriodSeconds: 75# Pod-level securityContext, empty by default. The default image start is as# root: the entrypoint re-owns /app/data and the XDG dirs, then drops to thediff --git a/docker-compose.dev.yml b/docker-compose.dev.ymlindex a42fd6c06..c87ea90e8 100644--- a/docker-compose.dev.yml+++ b/docker-compose.dev.yml@@ -296,6 +296,9 @@ services:volumes:- ./data:/app/datarestart: unless-stopped+ # Same teardown budget as docker-compose.yml, which has the arithmetic: Docker's 10s default would+ # SIGKILL the container while an engine is still stopping.+ stop_grace_period: 75shealthcheck:test: ['CMD', 'curl', '-f', 'http://localhost:2785/api/health/ready']interval: 30sdiff --git a/docker-compose.yml b/docker-compose.ymlindex dbfbbb73e..8423841c1 100644--- a/docker-compose.yml+++ b/docker-compose.yml@@ -49,11 +49,16 @@ services:dockerfile: Dockerfilecontainer_name: openwa-apirestart: unless-stopped- # Give the graceful drain room to complete before Docker SIGKILLs the container: the shutdown- # grace (SHUTDOWN_DELAY_MS, default 3s) plus the worst-case per-engine teardown bound (~10s) can- # exceed Docker's 10s default, which would otherwise kill Chromium mid-teardown and orphan a- # session profile. Tune down if you raise neither SHUTDOWN_DELAY_MS nor run many sessions.- stop_grace_period: 45s+ # Give the ordered teardown room to finish before Docker SIGKILLs the container, which would cut+ # Chromium, the usage flush and the database close short. Bounded stages in order: drain 3s+ # (SHUTDOWN_DELAY_MS) + ingress reconciler 5s + engines 10s + auto-start wait 5s + webhook drain 5s+ # (WEBHOOK_SHUTDOWN_DRAIN_MS) + usage flush 5s = 33s. An enabled plugin adds 10s. QUEUE_ENABLED and+ # REDIS_ENABLED add 20s (worker close 10s, producer queue close 2s for each of two queues, three+ # Redis quits 2s each): 63s worst case. The process force-exits 67s after the drain and 5s before+ # this deadline. Raising SHUTDOWN_DELAY_MS or WEBHOOK_SHUTDOWN_DRAIN_MS moves that exit later by the+ # same amount, so raise this by the same amount too. The session count does not add: engines stop+ # in parallel.+ stop_grace_period: 75s# App network for datastores + the isolated network to reach# docker-proxy (DOCKER_HOST). createService() in docker.service.ts attaches# orchestrated containers to the literal `openwa-network`, so its name is fixed.diff --git a/docs/06-api-specification.md b/docs/06-api-specification.mdindex be21b76ba..0553d3752 100644--- a/docs/06-api-specification.md+++ b/docs/06-api-specification.md@@ -4745,7 +4745,7 @@ Queued deliveries retain their pre-hook outbox copy until the worker completes dReplay recorded webhook deliveries that still hold their event data, in one bounded batch, with the lowest attempt counts first, then oldest first. A row is replayable when it is terminal (`attempts` > 0; an `attempts: 0` row is the outbox's to replay) and was recorded while `WEBHOOK_FAILURE_PAYLOAD_RETENTION_HOURS` > 0, until that window passes. With the default `0` nothing is kept and this call replays nothing.-Each replay reuses the row's stored `idempotencyKey` (sent as `X-OpenWA-Idempotency-Key`), so receivers can deduplicate a POST that timed out after processing. `webhook:before` hooks run again on the stored pre-hook data. Each row gets **one direct POST** inside the request, including when ordinary dispatch uses the queue. Success removes its row and closes any outbox copy the sweep still holds for that event, so the sweep does not send it again; another failure keeps it and raises its attempt count. A sweep replay of the same event already under way can still POST it again under the same key. Calls on one node run serially. Graceful shutdown stops a batch at its next row and waits for the replays still running (at most four), including their `webhook:before` hooks; each POST is bounded by `WEBHOOK_TIMEOUT`. Rows it did not reach stay for a later call, and a call queued behind it replays nothing. The audit log records counts without payload content.+Each replay reuses the row's stored `idempotencyKey` (sent as `X-OpenWA-Idempotency-Key`), so receivers can deduplicate a POST that timed out after processing. `webhook:before` hooks run again on the stored pre-hook data. Each row gets **one direct POST** inside the request, including when ordinary dispatch uses the queue. Success removes its row and closes any outbox copy the sweep still holds for that event, so the sweep does not send it again; another failure keeps it and raises its attempt count. A sweep replay of the same event already under way can still POST it again under the same key. Calls on one node run serially. Graceful shutdown stops a batch at its next row and waits for the replays still running (at most four), including their `webhook:before` hooks, for at most `WEBHOOK_SHUTDOWN_DRAIN_MS`; each POST is bounded by `WEBHOOK_TIMEOUT`. Rows it did not reach stay for a later call, and a call queued behind it replays nothing. The audit log records counts without payload content.Choose a batch limit that fits the client's timeout, especially for unavailable receivers. A clienttimeout does not cancel the batch; inspect the failure log before retrying it.@@ -5753,7 +5753,7 @@ Request a graceful server restart, optionally orchestrating Docker profiles (add}```-`estimatedTime` (seconds) = base 15 + 20/postgres + 13/redis + 15/minio + 5/removal. The echoed `profilesToRemove`, and the removal count, already omit any bundled service the environment pins the app to (for example `DATABASE_HOST=postgres` set in the host environment or the project `.env`); such a service is never stopped. `orchestration` is present only when Docker is available and at least one managed profile (`postgres`, `redis`, `minio`) remains in `profiles`; `removal` only when Docker is available and at least one managed profile remains in `profilesToRemove` after dropping pinned services and profiles that are also being started. Without Docker, a `data/.orchestration-request.json` signal file is written instead. After responding, `shutdownService.shutdown()` runs (default ~3s grace); readiness returns `503` during drain.+`estimatedTime` (seconds, a typical healthy value rather than a bound; teardown is force-exited 67 s after the grace at the default `WEBHOOK_SHUTDOWN_DRAIN_MS`) = base 15 + 20/postgres + 13/redis + 15/minio + 5/removal. The echoed `profilesToRemove`, and the removal count, already omit any bundled service the environment pins the app to (for example `DATABASE_HOST=postgres` set in the host environment or the project `.env`); such a service is never stopped. `orchestration` is present only when Docker is available and at least one managed profile (`postgres`, `redis`, `minio`) remains in `profiles`; `removal` only when Docker is available and at least one managed profile remains in `profilesToRemove` after dropping pinned services and profiles that are also being started. Without Docker, a `data/.orchestration-request.json` signal file is written instead. After responding, `shutdownService.shutdown()` runs (default ~3s grace); readiness returns `503` during drain.**Errors:** `401` · `403`@@ -7246,7 +7246,7 @@ Webhook delivery is **at-least-once**. A consumer can legitimately receive the s**Crash boundary.** For an ordinary dispatch, a successfully written outbox record retains the pre-hook event data until delivery succeeds, the subscription becomes ineligible, or the terminal failure is stored. Queued records keep this copy even after enqueue; the worker retires it only after one of those outcomes. If failure recording fails, the copy remains eligible for recovery. The bounded sweep (`WEBHOOK_RECONCILE_INTERVAL_MS`, default 60s) checks the stored job before replaying, leaves active jobs alone, and reuses the stored `X-OpenWA-Idempotency-Key`. If Redis cannot confirm the job state, replay waits. A missing or failed job, or a stranded direct delivery, can be replayed within `WEBHOOK_RECONCILE_MAX_ATTEMPTS`. A replay sent directly (the queue off, or its enqueue failed) that exhausts its retries stores its terminal failure row but keeps the outbox copy pending, so later sweeps replay it until that budget is spent; exhausting it retires the outbox only after terminal failure recording succeeds. Receiver deduplication remains necessary because a crash after receiver processing can cause another POST.-Outbox creation is best effort, and message persistence and outbox insertion are separate transactions. A write failure or crash between them can still lose the webhook delivery. Failure records with zero attempts are excluded from operator redrive. Capacity and shutdown rejections can still be recovered from an unsettled outbox record; shutdown during retry backoff may follow earlier POSTs. The direct dispatcher leaves its local in-flight work alone during reconciliation. Graceful shutdown drains current queue jobs; direct sends, and an outbox replay in flight with its plugin hooks, get up to `WEBHOOK_SHUTDOWN_DRAIN_MS` (default 5s). That replay sends at most one more POST, bounded by `WEBHOOK_TIMEOUT`, and no further row is replayed; the next start replays what it left pending.+Outbox creation is best effort, and message persistence and outbox insertion are separate transactions. A write failure or crash between them can still lose the webhook delivery. Failure records with zero attempts are excluded from operator redrive. Capacity and shutdown rejections can still be recovered from an unsettled outbox record; shutdown during retry backoff may follow earlier POSTs. The direct dispatcher leaves its local in-flight work alone during reconciliation. Graceful shutdown drains current queue jobs for at most 10 s; direct sends, an operator redrive replay in flight, and an outbox replay in flight with its plugin hooks, get up to `WEBHOOK_SHUTDOWN_DRAIN_MS` (default 5s). That replay sends at most one more POST, bounded by `WEBHOOK_TIMEOUT`, and no further row is replayed; the next start replays what it left pending.Before downgrading to a version that does not recognize the `queued` outbox state, drain or recover those records and take a backup. Its cleanup may otherwise treat an unsettled queued record as settled. Reverting the terminal-identity migration removes its index but does not restore historical duplicates.diff --git a/docs/19-plugin-architecture.md b/docs/19-plugin-architecture.mdindex cd0a309d7..76f5325ab 100644--- a/docs/19-plugin-architecture.md+++ b/docs/19-plugin-architecture.md@@ -583,16 +583,18 @@ double-enable, and engines must match the configured active engine):**Disable / unload / uninstall.** `disablePlugin` runs `onDisable` (force-terminating the worker for asandboxed plugin, even if `onDisable` hangs or throws) and unregisters the plugin's hooks. `onModuleDestroy`-disables every enabled plugin on graceful shutdown so stateful plugins can flush. Running plugins are-disabled first; then it waits for enables already in flight (a sandboxed enable is three 30 s-bounded-calls) and disables those plugins too. New enables are refused once teardown starts. A disable that-arrives while the same plugin is still being disabled (shutdown overlapping a REST disable, an-uninstall or an update unload) joins the running teardown, so `onDisable` runs once; an unload that-joins this way gets no worker-side `onUnload`. `uninstallPlugin` disables + unloads, drops the-registry entry, and deletes the plugin's directory and its `ctx.storage` data dir (built-ins are-protected and cannot be uninstalled). The unload path dispatches `onUnload`: for a sandboxed plugin-it runs in the worker between `onDisable` and terminate (a plain disable does-NOT fire `onUnload` — disable is reversible and its cleanup hook is `onDisable`).+disables every enabled plugin on graceful shutdown so stateful plugins can flush. Plugins are disabled+in parallel, so the order of `onDisable` across plugins is unspecified and a handler must not rely on+another plugin still being enabled. Running plugins go first; then it waits for enables already in+flight (a sandboxed enable is three 30 s-bounded calls) and disables those plugins too. The whole+phase, waits included, is capped at 10 s; a plugin still running at the cap is terminated with the+process. New enables are refused once teardown starts. A disable that arrives while the same plugin is+still being disabled (shutdown overlapping a REST disable, an uninstall or an update unload) joins the+running teardown, so `onDisable` runs once; an unload that joins this way gets no worker-side `onUnload`.+`uninstallPlugin` disables + unloads, drops the registry entry, and deletes the plugin's directory and+its `ctx.storage` data dir (built-ins are protected and cannot be uninstalled). The unload path+dispatches `onUnload`: for a sandboxed plugin it runs in the worker between `onDisable` and terminate+(a plain disable does NOT fire `onUnload`; disable is reversible and its cleanup hook is `onDisable`).**Context.** `createPluginContext` builds the `PluginContext` (§19.4): a per-plugin logger,`registerHook` (wrapped with the per-session activation gate), `registerWebhook`, the livediff --git a/docs/30-plugin-sandboxing.md b/docs/30-plugin-sandboxing.mdindex 3bf3de01a..16184ad22 100644--- a/docs/30-plugin-sandboxing.md+++ b/docs/30-plugin-sandboxing.md@@ -56,7 +56,8 @@ from disk is user-installed and sandboxed.still working through a backlog of dispatches reads the ping late, but every result it returns countsas an answer. A worker whose event loop is blocked answers nothing for 5 s, so the host terminates itand sets the plugin to `ERROR` (no automatic restart, the same as a crash); the operator re-enables it- once fixed.+ once fixed. At graceful shutdown every plugin's `onDisable` shares one 10 s cap, and a worker still+ running at the cap dies with the process.- **Memory-kind boundary.** The heap cap bounds the V8 heap only. `Buffer`/`ArrayBuffer` allocationsare native memory outside `maxOldGenerationSizeMb`, and worker threads share the host's addressspace, so a plugin that accumulates Buffers grows host RSS until the container's memory limitdiff --git a/src/common/services/shutdown-budget.spec.ts b/src/common/services/shutdown-budget.spec.tsnew file mode 100644index 000000000..1c27f6bee--- /dev/null+++ b/src/common/services/shutdown-budget.spec.ts@@ -0,0 +1,109 @@+import { readFileSync } from 'fs';+import { join } from 'path';+import { CACHE_QUIT_TIMEOUT_MS } from '../cache/cache.service';+import { THROTTLER_REDIS_QUIT_TIMEOUT_MS } from '../throttler/redis-throttler.storage';+import configuration from '../../config/configuration';+import { PLUGIN_SHUTDOWN_WAIT_MS } from '../../core/plugins/plugin-loader.service';+import { ApiKeyUsageTracker } from '../../modules/auth/api-key-usage-tracker.service';+import { WS_REDIS_QUIT_TIMEOUT_MS } from '../../modules/events/redis-io.adapter';+import { INGRESS_DISPATCH_TIMEOUT_MS } from '../../modules/integration/integration.constants';+import { MAX_WORKER_CLOSE_WAIT_MS } from '../../modules/queue/processors/close-worker';+import { QUEUE_NAMES } from '../../modules/queue/queue-names';+import { QUEUE_CLOSE_TIMEOUT_MS } from '../../modules/queue/shutdown-safe-queue';+import { ENGINE_TEARDOWN_TIMEOUT_MS } from '../../modules/session/session-lifecycle-fences';+import { AUTOSTART_SHUTDOWN_WAIT_MS } from '../../modules/session/session.service';+import {+ DEFAULT_SHUTDOWN_DELAY_MS,+ DEFAULT_WEBHOOK_SHUTDOWN_DRAIN_MS,+ SHUTDOWN_KILL_DEADLINE_MS,+ SHUTDOWN_TEARDOWN_LIMIT_MS,+ shutdownTeardownLimitMs,+} from './shutdown-budget';++/**+ * The bounded teardown stages, summed in Nest destroy order, must fit the shipped kill deadline. The+ * figures come from the constants the stages use, so raising one without raising the deadline (and the+ * table in shutdown-budget.ts) fails here. A new bounded stage has to be added to the sums below.+ */+describe('shutdown time budget', () => {+ const MIN_RESERVE_MS = 5_000;+ const root = join(__dirname, '../../..');+ // Each queue is its own BullModule.registerQueue module, and Nest closes modules one after the other.+ const QUEUE_COUNT = Object.keys(QUEUE_NAMES).length;+ const base =+ DEFAULT_SHUTDOWN_DELAY_MS ++ INGRESS_DISPATCH_TIMEOUT_MS ++ ENGINE_TEARDOWN_TIMEOUT_MS ++ AUTOSTART_SHUTDOWN_WAIT_MS ++ DEFAULT_WEBHOOK_SHUTDOWN_DRAIN_MS ++ ApiKeyUsageTracker.SHUTDOWN_FLUSH_TIMEOUT_MS;+ const withPlugins = base + PLUGIN_SHUTDOWN_WAIT_MS;+ const clustered =+ withPlugins ++ MAX_WORKER_CLOSE_WAIT_MS ++ QUEUE_CLOSE_TIMEOUT_MS * QUEUE_COUNT ++ THROTTLER_REDIS_QUIT_TIMEOUT_MS ++ CACHE_QUIT_TIMEOUT_MS ++ WS_REDIS_QUIT_TIMEOUT_MS;++ it('sums to the documented totals', () => {+ expect([base, withPlugins, clustered]).toEqual([33_000, 43_000, 63_000]);+ });++ it('registers one queue module per queue name, so the queue stage is counted per queue', () => {+ const source = readFileSync(join(root, 'src/modules/queue/queue.module.ts'), 'utf8');+ expect(source.match(/BullModule\.registerQueue(Async)?\(/g)).toHaveLength(QUEUE_COUNT);+ });++ it('uses the webhook drain default the budget is written for', () => {+ expect(configuration().webhook.shutdownDrainMs).toBe(DEFAULT_WEBHOOK_SHUTDOWN_DRAIN_MS);+ });++ // The reserve pays for the uncapped hooks after the HTTP close. A stage added or raised eats into it; the+ // message names the fix, so the failure does not read as a bare number mismatch.+ it('leaves a reserve for the hooks after the HTTP close, even when every cap is hit at once', () => {+ const reserveMs = SHUTDOWN_TEARDOWN_LIMIT_MS - (clustered - DEFAULT_SHUTDOWN_DELAY_MS);+ expect(+ reserveMs >= MIN_RESERVE_MS+ ? 'ok'+ : `reserve ${reserveMs} ms < ${MIN_RESERVE_MS} ms: raise SHUTDOWN_KILL_DEADLINE_MS, both compose files and the chart in step`,+ ).toBe('ok');+ });++ it('keeps the 67 s teardown limit and the 5 s kill margin the docs quote', () => {+ expect(SHUTDOWN_TEARDOWN_LIMIT_MS).toBe(67_000);+ expect(SHUTDOWN_KILL_DEADLINE_MS - DEFAULT_SHUTDOWN_DELAY_MS - SHUTDOWN_TEARDOWN_LIMIT_MS).toBe(5_000);+ });++ // The prose that quotes the limit and the totals has to follow the constants.+ it.each([+ ['CHANGELOG.md', [SHUTDOWN_TEARDOWN_LIMIT_MS, SHUTDOWN_KILL_DEADLINE_MS]],+ ['.env.example', [SHUTDOWN_TEARDOWN_LIMIT_MS, SHUTDOWN_KILL_DEADLINE_MS]],+ ['docs/06-api-specification.md', [SHUTDOWN_TEARDOWN_LIMIT_MS]],+ ['docker-compose.yml', [SHUTDOWN_TEARDOWN_LIMIT_MS, base, clustered]],+ ['charts/openwa/values.yaml', [SHUTDOWN_TEARDOWN_LIMIT_MS, clustered]],+ ])('quotes the current figures in %s', (file, figures) => {+ const text = readFileSync(join(root, file), 'utf8');+ for (const ms of figures) expect(text).toMatch(new RegExp(`\\b${ms / 1000} ?s\\b`));+ });++ it('moves the teardown limit 1:1 with WEBHOOK_SHUTDOWN_DRAIN_MS and keeps it a valid timer delay', () => {+ expect(shutdownTeardownLimitMs(DEFAULT_WEBHOOK_SHUTDOWN_DRAIN_MS)).toBe(SHUTDOWN_TEARDOWN_LIMIT_MS);+ expect(shutdownTeardownLimitMs(DEFAULT_WEBHOOK_SHUTDOWN_DRAIN_MS + 7_000)).toBe(SHUTDOWN_TEARDOWN_LIMIT_MS + 7_000);+ expect(shutdownTeardownLimitMs(0)).toBe(SHUTDOWN_TEARDOWN_LIMIT_MS - DEFAULT_WEBHOOK_SHUTDOWN_DRAIN_MS);+ expect(shutdownTeardownLimitMs(Number.MAX_SAFE_INTEGER)).toBe(2 ** 31 - 1);+ });++ it('matches the shipped kill deadlines in both compose files and the Helm chart', () => {+ const grace = (file: string) =>+ Number(/^\s+stop_grace_period: (\d+)s$/m.exec(readFileSync(join(root, file), 'utf8'))?.[1]) * 1000;+ const values = readFileSync(join(root, 'charts/openwa/values.yaml'), 'utf8');+ expect(grace('docker-compose.yml')).toBe(SHUTDOWN_KILL_DEADLINE_MS);+ expect(grace('docker-compose.dev.yml')).toBe(SHUTDOWN_KILL_DEADLINE_MS);+ expect(Number(/^terminationGracePeriodSeconds: (\d+)$/m.exec(values)?.[1]) * 1000).toBe(SHUTDOWN_KILL_DEADLINE_MS);+ });++ it('destroys open sockets at the HTTP close, which the table counts as 0 s', () => {+ expect(readFileSync(join(root, 'src/main.ts'), 'utf8')).toMatch(/^\s+forceCloseConnections: true,/m);+ });+});diff --git a/src/common/services/shutdown-budget.ts b/src/common/services/shutdown-budget.tsnew file mode 100644index 000000000..69f610081--- /dev/null+++ b/src/common/services/shutdown-budget.ts@@ -0,0 +1,87 @@+/**+ * Shutdown time budget. On SIGTERM/SIGINT (and POST /api/infra/restart) the ordered teardown has to+ * finish before the orchestrator's kill deadline, or the late destroy hooks (API-key usage flush,+ * plugin onDisable, DataSource close) are SIGKILLed half way. The bounded stages below run one after+ * another in Nest destroy order (modules in sequence, hooks inside one module in parallel); sessions+ * do not add up, because engines are torn down in parallel under one per-engine deadline.+ *+ * stage cap applies when+ * SHUTDOWN_DELAY_MS drain 3 s always (default; the grace before teardown)+ * ingress reconciler wait 5 s a pass in flight (worst case)+ * engine teardown 10 s always+ * auto-start wait 5 s shutdown during the boot window+ * throttler Redis quit 2 s REDIS_ENABLED+ * webhook drain 5 s a delivery in flight (worst case; WEBHOOK_SHUTDOWN_DRAIN_MS)+ * queue worker close 10 s QUEUE_ENABLED+ * producer queue close 4 s QUEUE_ENABLED (2 s per queue, one queue module each)+ * API-key usage flush 5 s pending counters+ * cache Redis quit 2 s REDIS_ENABLED+ * plugin phase (disable + pending enable) 10 s a plugin is enabled+ * WebSocket adapter Redis quit 2 s REDIS_ENABLED+ * HTTP close 0 s main.ts sets forceCloseConnections+ *+ * Base (single replica, no plugins): 3 + 5 + 10 + 5 + 5 + 5 = 33 s. With a plugin enabled: 43 s.+ * Clustered (QUEUE + REDIS + plugins): 63 s, of which 3 s is the grace.+ *+ * The teardown after the grace has SHUTDOWN_TEARDOWN_LIMIT_MS (67 s), against 60 s of stages in the+ * clustered case. The 7 s left over is the reserve for the waits that have no cap of their own, before+ * and after the HTTP close: the database and plugin hook waits listed at the end, the BullModule+ * re-await of the worker close (closeWorkerIfStarted settles it once its cap is hit), the DataSource+ * close and the ffmpeg kill. They take milliseconds when nothing is wedged.+ *+ * The producer queue close is a stage of its own: ShutdownSafeQueue settles it, rejected or hung (a+ * half-open Redis socket never answers QUIT), within QUEUE_CLOSE_TIMEOUT_MS. Each queue is registered+ * in its own module (queue.module.ts) and Nest closes modules one after the other, so the stage is+ * QUEUE_CLOSE_TIMEOUT_MS times the number of queues in QUEUE_NAMES. A close that rejects makes Nest+ * skip every hook after it, the DataSource close included, and one that hangs holds them until the+ * force-exit.+ *+ * ShutdownService force-exits with status 1 when the teardown overruns that limit. It arms the timer+ * when the grace ends, so a raised SHUTDOWN_DELAY_MS moves the exit and the kill deadline together,+ * and it adds the excess of WEBHOOK_SHUTDOWN_DRAIN_MS over its 5 s default, the one other+ * operator-set input to a stage. The kill deadline an operator needs is therefore+ *+ * 75 s + (SHUTDOWN_DELAY_MS - 3 s) + (WEBHOOK_SHUTDOWN_DRAIN_MS - 5 s)+ *+ * and the shipped 75 s holds for the defaults. A healthy stop exits on its own and never waits. The+ * force-exit covers only SIGTERM, SIGINT and the admin restart; the other signals Nest handles itself+ * (main.ts) call app.close() with no exit timer.+ *+ * Waits with no cap of their own, which only the force-exit bounds: the bulk-message destroy hook (a+ * lookup and an update per processing batch), the pending-message reaper (it awaits its running+ * sweep, whose in-hand row re-emits message:persisted to plugin hooks) and the session ownership+ * release. They wait on the database, and the reaper also on plugin hooks. A wedged database ends the+ * stop at the force-exit, with status 1, before the kill; the stages behind these waits (worker close,+ * usage flush, plugin onDisable) get only the time that is left.+ */++/** Grace before teardown, so a load balancer sees the 503 first. SHUTDOWN_DELAY_MS overrides it. */+export const DEFAULT_SHUTDOWN_DELAY_MS = 3_000;++/** Default for WEBHOOK_SHUTDOWN_DRAIN_MS, the one other operator-set input to a stage in the table. */+export const DEFAULT_WEBHOOK_SHUTDOWN_DRAIN_MS = 5_000;++/**+ * Must equal docker-compose.yml and docker-compose.dev.yml stop_grace_period and+ * charts/openwa/values.yaml terminationGracePeriodSeconds. It holds for the default delay and drain.+ */+export const SHUTDOWN_KILL_DEADLINE_MS = 75_000;++/** The force-exit fires this long before the kill deadline. */+export const SHUTDOWN_KILL_MARGIN_MS = 5_000;++/** How long the teardown may run after the grace, with the default drain: 67 s. */+export const SHUTDOWN_TEARDOWN_LIMIT_MS =+ SHUTDOWN_KILL_DEADLINE_MS - DEFAULT_SHUTDOWN_DELAY_MS - SHUTDOWN_KILL_MARGIN_MS;++/**+ * The largest delay a Node timer accepts; above it the timer fires after 1 ms. Defined here because+ * configuration.ts imports this file, and re-exported from there for the other callers.+ */+export const MAX_TIMER_MS = 2 ** 31 - 1;++/** The teardown limit for a given WEBHOOK_SHUTDOWN_DRAIN_MS: it counts 1:1 against the kill deadline. */+export function shutdownTeardownLimitMs(webhookDrainMs: number): number {+ const limit = SHUTDOWN_TEARDOWN_LIMIT_MS + webhookDrainMs - DEFAULT_WEBHOOK_SHUTDOWN_DRAIN_MS;+ return Math.min(limit, MAX_TIMER_MS);+}diff --git a/src/common/services/shutdown.service.spec.ts b/src/common/services/shutdown.service.spec.tsindex d1828e160..e686cbaf4 100644--- a/src/common/services/shutdown.service.spec.ts+++ b/src/common/services/shutdown.service.spec.ts@@ -1,3 +1,8 @@+import { ConfigModule, ConfigService } from '@nestjs/config';+import { Test } from '@nestjs/testing';+import configuration from '../../config/configuration';+import { LoggerModule } from './logger.module';+import { SHUTDOWN_TEARDOWN_LIMIT_MS } from './shutdown-budget';import { ShutdownService } from './shutdown.service';/**@@ -173,3 +178,143 @@ describe('ShutdownService exit status (teardown outcome)', () => {expect(exitSpy).toHaveBeenCalledWith(1);});});++describe('ShutdownService force-exit backstop', () => {+ let exitSpy: jest.SpyInstance;++ beforeEach(() => {+ jest.useFakeTimers();+ exitSpy = jest.spyOn(process, 'exit').mockImplementation((): never => undefined as never);+ });++ afterEach(() => {+ jest.clearAllTimers();+ jest.useRealTimers();+ exitSpy.mockRestore();+ delete process.env.SHUTDOWN_DELAY_MS;+ });++ const hung = (delay: string, webhookDrainMs?: number): ShutdownService => {+ process.env.SHUTDOWN_DELAY_MS = delay;+ const config = { get: () => webhookDrainMs } as unknown as ConfigService;+ const svc = new ShutdownService(webhookDrainMs === undefined ? undefined : config);+ svc.setShutdownCallback(() => new Promise<void>(() => undefined));+ jest.spyOn(svc['logger'], 'error').mockImplementation(() => undefined);+ return svc;+ };++ it('exits 1 exactly at SHUTDOWN_TEARDOWN_LIMIT_MS after the grace when teardown never finishes', async () => {+ hung('0').shutdown();+ await jest.advanceTimersByTimeAsync(SHUTDOWN_TEARDOWN_LIMIT_MS - 1);+ expect(exitSpy).not.toHaveBeenCalled();+ await jest.advanceTimersByTimeAsync(1);+ expect(exitSpy).toHaveBeenCalledTimes(1);+ expect(exitSpy).toHaveBeenCalledWith(1);+ });++ it('measures the limit from the end of the grace, so a raised SHUTDOWN_DELAY_MS moves the exit with it', async () => {+ hung('20000').shutdown();+ await jest.advanceTimersByTimeAsync(20_000 + SHUTDOWN_TEARDOWN_LIMIT_MS - 1);+ expect(exitSpy).not.toHaveBeenCalled();+ await jest.advanceTimersByTimeAsync(1);+ expect(exitSpy).toHaveBeenCalledTimes(1);+ });++ it('adds the excess of WEBHOOK_SHUTDOWN_DRAIN_MS over its 5 s default to the limit', async () => {+ hung('0', 15_000).shutdown();+ await jest.advanceTimersByTimeAsync(SHUTDOWN_TEARDOWN_LIMIT_MS + 10_000 - 1);+ expect(exitSpy).not.toHaveBeenCalled();+ await jest.advanceTimersByTimeAsync(1);+ expect(exitSpy).toHaveBeenCalledTimes(1);+ });++ it('lowers the limit for a drain shorter than the default', async () => {+ hung('0', 0).shutdown();+ await jest.advanceTimersByTimeAsync(SHUTDOWN_TEARDOWN_LIMIT_MS - 5_000);+ expect(exitSpy).toHaveBeenCalledTimes(1);+ });++ it('keeps the timer inside what Node accepts for an enormous drain', async () => {+ hung('0', Number.MAX_SAFE_INTEGER).shutdown();+ await jest.advanceTimersByTimeAsync(0);+ expect(exitSpy).not.toHaveBeenCalled();+ await jest.advanceTimersByTimeAsync(2 ** 31 - 2);+ expect(exitSpy).not.toHaveBeenCalled();+ await jest.advanceTimersByTimeAsync(1);+ expect(exitSpy).toHaveBeenCalledTimes(1);+ });++ it('is cleared when teardown completes, so no second exit follows', async () => {+ process.env.SHUTDOWN_DELAY_MS = '0';+ const svc = new ShutdownService();+ svc.setShutdownCallback(jest.fn().mockResolvedValue(undefined));+ svc.shutdown();+ await jest.advanceTimersByTimeAsync(0);+ expect(exitSpy).toHaveBeenCalledTimes(1);+ expect(exitSpy).toHaveBeenCalledWith(0);+ await jest.advanceTimersByTimeAsync(SHUTDOWN_TEARDOWN_LIMIT_MS);+ expect(exitSpy).toHaveBeenCalledTimes(1);+ });++ it('is cleared when teardown fails, leaving the single exit(1) of the failure', async () => {+ process.env.SHUTDOWN_DELAY_MS = '0';+ const svc = new ShutdownService();+ svc.setShutdownCallback(jest.fn().mockRejectedValue(new Error('boom')));+ jest.spyOn(svc['logger'], 'error').mockImplementation(() => undefined);+ svc.shutdown();+ await jest.advanceTimersByTimeAsync(SHUTDOWN_TEARDOWN_LIMIT_MS);+ expect(exitSpy).toHaveBeenCalledTimes(1);+ expect(exitSpy).toHaveBeenCalledWith(1);+ });++ // Fake timers make unref a no-op, so the handle of the budget timer is watched directly.+ it('keeps the timer referenced, so a teardown parked on a handle-less promise cannot exit 0', async () => {+ const real = global.setTimeout;+ const unrefs: jest.SpyInstance[] = [];+ const spy = jest.spyOn(global, 'setTimeout').mockImplementation((fn: () => void, ms?: number) => {+ const handle = real(fn, ms);+ if (ms === SHUTDOWN_TEARDOWN_LIMIT_MS) unrefs.push(jest.spyOn(handle, 'unref'));+ return handle;+ });+ try {+ hung('0').shutdown();+ await jest.advanceTimersByTimeAsync(0);+ expect(unrefs).toHaveLength(1);+ expect(unrefs[0]).not.toHaveBeenCalled();+ } finally {+ spy.mockRestore();+ }+ });++ it('arms one timer however often shutdown is requested', async () => {+ const svc = hung('0');+ svc.shutdown();+ svc.shutdown();+ await jest.advanceTimersByTimeAsync(0);+ expect(jest.getTimerCount()).toBe(1);+ await jest.advanceTimersByTimeAsync(SHUTDOWN_TEARDOWN_LIMIT_MS);+ expect(exitSpy).toHaveBeenCalledTimes(1);+ });++ // The constructor parameter is @Optional(), so a wiring break would silently fall back to the default+ // drain and the force-exit would stop following WEBHOOK_SHUTDOWN_DRAIN_MS.+ it('receives the real ConfigService, so the exit follows WEBHOOK_SHUTDOWN_DRAIN_MS', async () => {+ process.env.WEBHOOK_SHUTDOWN_DRAIN_MS = '12000';+ process.env.SHUTDOWN_DELAY_MS = '0';+ try {+ const moduleRef = await Test.createTestingModule({+ imports: [ConfigModule.forRoot({ isGlobal: true, ignoreEnvFile: true, load: [configuration] }), LoggerModule],+ }).compile();+ const svc = moduleRef.get(ShutdownService);+ svc.setShutdownCallback(() => new Promise<void>(() => undefined));+ jest.spyOn(svc['logger'], 'error').mockImplementation(() => undefined);+ svc.shutdown();+ await jest.advanceTimersByTimeAsync(SHUTDOWN_TEARDOWN_LIMIT_MS + 7_000 - 1);+ expect(exitSpy).not.toHaveBeenCalled();+ await jest.advanceTimersByTimeAsync(1);+ expect(exitSpy).toHaveBeenCalledTimes(1);+ } finally {+ delete process.env.WEBHOOK_SHUTDOWN_DRAIN_MS;+ }+ });+});diff --git a/src/common/services/shutdown.service.ts b/src/common/services/shutdown.service.tsindex 35727e891..fad11bd65 100644--- a/src/common/services/shutdown.service.ts+++ b/src/common/services/shutdown.service.ts@@ -1,8 +1,13 @@-import { Injectable } from '@nestjs/common';+import { Injectable, Optional } from '@nestjs/common';+import { ConfigService } from '@nestjs/config';import { createLogger } from './logger.service';+import {+ DEFAULT_SHUTDOWN_DELAY_MS,+ DEFAULT_WEBHOOK_SHUTDOWN_DRAIN_MS,+ shutdownTeardownLimitMs,+} from './shutdown-budget';-/** Default grace before teardown; capped so a misconfigured value can't exceed a typical SIGKILL window. */-export const DEFAULT_SHUTDOWN_DELAY_MS = 3000;+// The delay counts 1:1 against the kill deadline; see shutdown-budget.ts.const MAX_SHUTDOWN_DELAY_MS = 30_000;@Injectable()@@ -14,6 +19,8 @@ export class ShutdownService {private shutdownScheduled = false;private tearingDown = false;+ constructor(@Optional() private readonly config?: ConfigService) {}+/*** Set the shutdown callback (called from main.ts after app creation)*/@@ -73,12 +80,23 @@ export class ShutdownService {setTimeout(() => {this.logger.log('Initiating shutdown...');this.tearingDown = true;+ // The only bound on the waits that have none of their own (the database waits, queue and pool+ // close), and the only one on the admin-restart path, where no orchestrator kill follows. Armed+ // here, after the grace, so a raised SHUTDOWN_DELAY_MS moves it with the kill deadline; see+ // shutdown-budget.ts. Not unref'd: a teardown parked on a promise that holds no handle would let+ // the loop drain and the process exit 0 with the remaining hooks never run.+ const budgetMs = shutdownTeardownLimitMs(+ this.config?.get<number>('webhook.shutdownDrainMs') ?? DEFAULT_WEBHOOK_SHUTDOWN_DRAIN_MS,+ );+ const forceExit = setTimeout(() => {+ this.logger.error('Shutdown teardown exceeded its budget, forcing exit', undefined, { budgetMs });+ process.exit(1);+ }, budgetMs);const doShutdown = async () => {// The exit status mirrors the teardown outcome: 0 when teardown completed, 1 when it- // failed — an orchestrator (k8s, systemd, docker restart policies) must not read a- // resource-leaking shutdown as a clean one. (A teardown that HANGS never reaches this- // exit at all; that case is bounded externally by the second-signal force-exit in- // main.ts or the orchestrator's SIGKILL deadline.)+ // failed, so an orchestrator (k8s, systemd, docker restart policies) must not read a+ // resource-leaking shutdown as a clean one. A teardown that hangs is cut by the force-exit+ // timer above (or by a second signal in main.ts).let exitCode = 0;try {if (this.destroyCallback) {@@ -91,6 +109,7 @@ export class ShutdownService {error instanceof Error ? error.message : String(error),);} finally {+ clearTimeout(forceExit);process.exit(exitCode);}};diff --git a/src/config/configuration.ts b/src/config/configuration.tsindex ba849a4fd..ecffd543a 100644--- a/src/config/configuration.ts+++ b/src/config/configuration.ts@@ -4,6 +4,7 @@ import { computeSendPacingConfig } from '../modules/message/send-pacing.config';import { resolveInflightBodyBudgetBytes } from './inflight-body-budget';import { resolveRequestTimeoutMs } from './http-timeouts';import { readWsRateLimitConfig } from '../modules/events/ws-rate-limit';+import { DEFAULT_WEBHOOK_SHUTDOWN_DRAIN_MS, MAX_TIMER_MS } from '../common/services/shutdown-budget';/*** Root of the host's persistent state. Relative on purpose: the image sets WORKDIR /app and mounts@@ -60,7 +61,7 @@ export function resolveNonNegativeIntEnv(raw: string | undefined, fallback: numb* arms `protocolTimeout` with a plain `setTimeout` (`common/CallbackRegistry.js`), so the ceiling* applies to it directly and the browser never finishes launching.*/-export const MAX_TIMER_MS = 2147483647;+export { MAX_TIMER_MS };/*** The UI locale Chromium is pinned to. Without a pin the browser's language is whatever the launched@@ -309,7 +310,7 @@ export default () => ({// How long shutdown waits for in-flight direct deliveries to finish before abandoning them.// 0 = don't wait (explicit opt-out); blank/garbage falls back to the default — a NaN here would// silently remove the drain deadline downstream (Math.max(0, NaN) is NaN).- shutdownDrainMs: resolveNonNegativeIntEnv(process.env.WEBHOOK_SHUTDOWN_DRAIN_MS, 5000),+ shutdownDrainMs: resolveNonNegativeIntEnv(process.env.WEBHOOK_SHUTDOWN_DRAIN_MS, DEFAULT_WEBHOOK_SHUTDOWN_DRAIN_MS),// How long a terminal delivery-failure row keeps the event data it was built from, so// `POST /webhooks/delivery-failures/redrive` can replay it. 0 (the default) stores nothing, which// is the pre-redrive behaviour. The retention sweep nulls an expired payload and keeps the row.diff --git a/src/configure-app.ts b/src/configure-app.tsindex bf4e64e73..31f202d30 100644--- a/src/configure-app.ts+++ b/src/configure-app.ts@@ -224,7 +224,8 @@ export function configureApp(app: INestApplication, options: ConfigureAppOptions// Once the shutdown grace has elapsed, teardown runs every destroy hook before the listener// closes, and a request admitted then would run against services being torn down. Refuse new- // requests from that point; admitted ones run to completion. The health probes stay answered so+ // requests from that point; admitted ones run until teardown ends and are then cut by the HTTP close+ // (forceCloseConnections in main.ts). The health probes stay answered so// readiness keeps reporting its draining state. Ahead of the budget, so no body is read.const shutdown = optional(ShutdownService);app.use((req: Request, res: Response, next: NextFunction) => {diff --git a/src/core/plugins/plugin-loader.service.spec.ts b/src/core/plugins/plugin-loader.service.spec.tsindex 465c01fd9..a753d7443 100644--- a/src/core/plugins/plugin-loader.service.spec.ts+++ b/src/core/plugins/plugin-loader.service.spec.ts@@ -116,7 +116,7 @@ describe('dispatchConversationMedia', () => {import * as fs from 'fs';import * as os from 'os';-import { PluginLoaderService } from './plugin-loader.service';+import { PLUGIN_SHUTDOWN_WAIT_MS, PluginLoaderService } from './plugin-loader.service';import { ConfigService } from '@nestjs/config';import { ModuleRef } from '@nestjs/core';import { HookManager } from '../hooks';@@ -479,6 +479,91 @@ describe('PluginLoaderService — graceful shutdown (onModuleDestroy)', () => {expect(loader.getPlugin('gone-plg')).toBeUndefined();});+ describe('time cap', () => {+ beforeEach(() => jest.useFakeTimers());+ afterEach(() => jest.useRealTimers());++ const never = (): Promise<void> => new Promise<void>(() => undefined);++ it('disables plugins in parallel and returns at PLUGIN_SHUTDOWN_WAIT_MS when onDisable never settles', async () => {+ const a = jest.fn(never);+ const b = jest.fn(never);+ loader.registerBuiltInPlugin(ext('a-plg'), { onDisable: a });+ loader.registerBuiltInPlugin(ext('b-plg'), { onDisable: b });+ await loader.enablePlugin('a-plg');+ await loader.enablePlugin('b-plg');+ const warn = jest.spyOn(loader['logger'], 'warn').mockImplementation(() => undefined);++ let done = false;+ void loader.onModuleDestroy().then(() => (done = true));+ await jest.advanceTimersByTimeAsync(0);+ expect(a).toHaveBeenCalledTimes(1);+ expect(b).toHaveBeenCalledTimes(1);++ await jest.advanceTimersByTimeAsync(PLUGIN_SHUTDOWN_WAIT_MS - 1);+ expect(done).toBe(false);+ await jest.advanceTimersByTimeAsync(1);+ expect(done).toBe(true);+ expect(warn).toHaveBeenCalledWith(+ expect.stringContaining('time cap'),+ expect.objectContaining({ action: 'plugin_shutdown_timeout' }),+ );+ });++ it('stops waiting for an enable that never settles at the same cap', async () => {+ loader.registerBuiltInPlugin(ext('stuck-plg'), { onEnable: never });+ void loader.enablePlugin('stuck-plg').catch(() => undefined);+ jest.spyOn(loader['logger'], 'warn').mockImplementation(() => undefined);++ let done = false;+ void loader.onModuleDestroy().then(() => (done = true));+ await jest.advanceTimersByTimeAsync(PLUGIN_SHUTDOWN_WAIT_MS - 1);+ expect(done).toBe(false);+ await jest.advanceTimersByTimeAsync(1);+ expect(done).toBe(true);+ });++ it('still disables an enable that lands at 2 s and returns then, without the warning', async () => {+ const onDisable = jest.fn(() => Promise.resolve());+ loader.registerBuiltInPlugin(ext('late-plg'), {+ onEnable: () => new Promise<void>(resolve => setTimeout(resolve, 2000)),+ onDisable,+ });+ void loader.enablePlugin('late-plg');+ const warn = jest.spyOn(loader['logger'], 'warn').mockImplementation(() => undefined);++ let done = false;+ void loader.onModuleDestroy().then(() => (done = true));+ await jest.advanceTimersByTimeAsync(1999);+ expect(done).toBe(false);+ await jest.advanceTimersByTimeAsync(1);+ expect(done).toBe(true);+ expect(onDisable).toHaveBeenCalledTimes(1);+ expect(warn).not.toHaveBeenCalled();+ });++ it('leaves no timer behind after a teardown that finishes within the cap', async () => {+ loader.registerBuiltInPlugin(ext('ok-plg'), { onDisable: () => Promise.resolve() });+ await loader.enablePlugin('ok-plg');++ await loader.onModuleDestroy();++ expect(jest.getTimerCount()).toBe(0);+ });++ it('does not let one failing onDisable stop the others', async () => {+ const ok = jest.fn(() => Promise.resolve());+ loader.registerBuiltInPlugin(ext('bad-plg'), { onDisable: () => Promise.reject(new Error('flush failed')) });+ loader.registerBuiltInPlugin(ext('ok-plg'), { onDisable: ok });+ await loader.enablePlugin('bad-plg');+ await loader.enablePlugin('ok-plg');+ jest.spyOn(loader['logger'], 'error').mockImplementation(() => undefined);++ await expect(loader.onModuleDestroy()).resolves.toBeUndefined();+ expect(ok).toHaveBeenCalledTimes(1);+ });+ });+it('refuses an enable once teardown has begun', async () => {const onEnable = jest.fn(() => Promise.resolve());loader.registerBuiltInPlugin(ext('late-plg'), { onEnable });diff --git a/src/core/plugins/plugin-loader.service.ts b/src/core/plugins/plugin-loader.service.tsindex 3193ff149..dd00f134b 100644--- a/src/core/plugins/plugin-loader.service.ts+++ b/src/core/plugins/plugin-loader.service.ts@@ -53,6 +53,9 @@ const SANDBOX_MAX_INFLIGHT_CAPS = 32;*/const SANDBOX_CAP_TIMEOUT_MS = 30000;+/** Total time the plugin shutdown phase may take (disable pass, in-flight enables, second pass). See shutdown-budget.ts. */+export const PLUGIN_SHUTDOWN_WAIT_MS = 10_000;+/*** How long a worker has to answer the liveness ping the host sends after a dispatch times out. A worker* stuck in a synchronous loop is terminated about 5 s after its first dispatch timeout (about 10 s after@@ -249,7 +252,7 @@ export class PluginLoaderService implements OnModuleInit, OnApplicationBootstrap* code executes. Built-ins are skipped: an engine is enabled by EngineFactory against the configured* engine.type, and enabling a non-active engine here would be rejected anyway.*- * Best-effort and sequential, like the shutdown teardown: a plugin that cannot come back is logged+ * Best-effort and sequential: a plugin that cannot come back is logged* and left in ERROR, and never holds up the gateway.*/async onApplicationBootstrap(): Promise<void> {@@ -281,14 +284,36 @@ export class PluginLoaderService implements OnModuleInit, OnApplicationBootstrap* Graceful shutdown (SIGTERM → app.close()): run onDisable for every enabled plugin so it can flush* buffers, close connections, and persist state. Previously onDisable only ran via the REST disable* and uninstall paths, so a normal restart/deploy/scale-down skipped it and stateful plugins lost- * in-flight work. Best-effort and sequential: one plugin's failure must not block the others.+ * in-flight work. Best-effort: plugins are disabled in parallel and one plugin's failure must not+ * block the others.** Running plugins are disabled first, so a slow enable still in flight cannot hold back their onDisable.* Then the in-flight enables are awaited and a second pass disables the plugins that finished enabling* mid-teardown, instead of leaving them running with onDisable never called. Later enables are refused.+ *+ * The whole phase shares one cap, PLUGIN_SHUTDOWN_WAIT_MS, because a sandboxed plugin's lifecycle call+ * alone is bounded at 30 s and the destroy hooks after this one (database close, lease release) have+ * to finish inside the kill deadline. A sandbox worker still running at the cap dies with the process.*/async onModuleDestroy(): Promise<void> {this.shuttingDown = true;+ let timer: NodeJS.Timeout | undefined;+ const done = await Promise.race([+ this.disableForShutdown().then(() => true),+ new Promise<false>(resolve => {+ timer = setTimeout(resolve, PLUGIN_SHUTDOWN_WAIT_MS, false);+ }),+ ]);+ clearTimeout(timer);+ if (!done) {+ this.logger.warn('Plugin shutdown still running at its time cap; continuing without it', {+ waitMs: PLUGIN_SHUTDOWN_WAIT_MS,+ action: 'plugin_shutdown_timeout',+ });+ }+ }++ private async disableForShutdown(): Promise<void> {await this.disableEnabledPlugins();await Promise.allSettled(this.pendingEnables);await this.disableEnabledPlugins();@@ -296,17 +321,19 @@ export class PluginLoaderService implements OnModuleInit, OnApplicationBootstrapprivate async disableEnabledPlugins(): Promise<void> {const enabled = this.getAllPlugins().filter(p => p.status === PluginStatus.ENABLED);- for (const plugin of enabled) {- try {- await this.disablePlugin(plugin.manifest.id);- } catch (error) {- this.logger.error(- `Failed to disable plugin ${plugin.manifest.id} during shutdown`,- error instanceof Error ? error.message : String(error),- { pluginId: plugin.manifest.id, action: 'plugin_shutdown_disable_failed' },- );- }- }+ await Promise.allSettled(+ enabled.map(async plugin => {+ try {+ await this.disablePlugin(plugin.manifest.id);+ } catch (error) {+ this.logger.error(+ `Failed to disable plugin ${plugin.manifest.id} during shutdown`,+ error instanceof Error ? error.message : String(error),+ { pluginId: plugin.manifest.id, action: 'plugin_shutdown_disable_failed' },+ );+ }+ }),+ );}/**diff --git a/src/main.ts b/src/main.tsindex 873e4efac..8eba494df 100644--- a/src/main.ts+++ b/src/main.ts@@ -142,7 +142,15 @@ async function bootstrap() {// Disable Nest's default body parser so we can set an explicit size cap below. Framework lines// (route mapping, unhandled-exception stacks) go through the app logger so LOG_LEVEL, LOG_FORMAT and// the request id apply to them too.- const app = await NestFactory.create(AppModule, { bodyParser: false, logger: createLogger('Nest') });+ // forceCloseConnections: the HTTP close runs after every destroy hook, and a request still in flight+ // would hold it until the force-exit; destroying the open sockets there keeps it at 0 s in the+ // shutdown budget (shutdown-budget.ts). It also cuts in-flight requests and WebSocket clients at that+ // point (the socket.io server is not closed first), after the destroy hooks have run.+ const app = await NestFactory.create(AppModule, {+ bodyParser: false,+ logger: createLogger('Nest'),+ forceCloseConnections: true,+ });appInstance = app;// Cross-replica WebSocket fan-out: when Redis is enabled, broadcasts reach clients on everydiff --git a/src/modules/auth/api-key-usage-tracker.service.ts b/src/modules/auth/api-key-usage-tracker.service.tsindex 4960bae6c..8ba372fe6 100644--- a/src/modules/auth/api-key-usage-tracker.service.ts+++ b/src/modules/auth/api-key-usage-tracker.service.ts@@ -91,7 +91,7 @@ export class ApiKeyUsageTracker {* Best-effort flush of the coalesced counters on teardown. Nest runs the owning service's* onModuleDestroy before the TypeORM connection closes (the DataSource is destroyed in* onApplicationShutdown, the last lifecycle hook), so the DB is still writable here. Bounded so a- * wedged DB cannot stall shutdown past the grace window; whatever is still unflushed after the+ * wedged DB cannot stall the teardown (a line item in shutdown-budget.ts); whatever is unflushed after the* bound is dropped — the counters are advisory statistics, authentication never depends on them.*/async flushOnShutdown(): Promise<void> {diff --git a/src/modules/events/redis-io.adapter.ts b/src/modules/events/redis-io.adapter.tsindex cfb1cde8f..887278db0 100644--- a/src/modules/events/redis-io.adapter.ts+++ b/src/modules/events/redis-io.adapter.ts@@ -104,7 +104,9 @@ export class RedisIoAdapter extends IoAdapter {// Socket.IO's adapters unsubscribe from their Redis channels as the server closes. Tearing the// clients down FIRST left those unsubscribes to be issued on dead clients, and each rejection// surfaced as an unhandled rejection — a handful of ERROR lines on every graceful shutdown, for- // a shutdown that had in fact gone fine. Close the server first, then release the pair.+ // a shutdown that had in fact gone fine. Close the server first, then release the pair. Under+ // forceCloseConnections Nest skips the server close when the socket.io server shares the HTTP+ // server, so only the pub/sub pair is released here and the HTTP close tears the sockets down.await super.close(server);await Promise.allSettled([this.quitClient(this.pubClient), this.quitClient(this.subClient)]);this.pubClient = undefined;diff --git a/src/modules/infra/infra-config.controller.ts b/src/modules/infra/infra-config.controller.tsindex a83545681..f01c42e4e 100644--- a/src/modules/infra/infra-config.controller.ts+++ b/src/modules/infra/infra-config.controller.ts@@ -529,7 +529,8 @@ export class InfraConfigController {// default 3s) — readiness reports 503 during the window so traffic drains first.void this.shutdownService.shutdown();- // Calculate estimated time - base 15s + additional for each service (increased for reliability)+ // Calculate estimated time - base 15s + additional for each service (increased for reliability).+ // A typical healthy value, not a bound: ShutdownService force-exits teardown 67s after the grace.let estimatedTime = 15;if (profiles.includes('postgres')) estimatedTime += 20;if (profiles.includes('redis')) estimatedTime += 13;diff --git a/src/modules/queue/processors/close-worker.spec.ts b/src/modules/queue/processors/close-worker.spec.tsnew file mode 100644index 000000000..0a3ecc09f--- /dev/null+++ b/src/modules/queue/processors/close-worker.spec.ts@@ -0,0 +1,34 @@+import type { WorkerHost } from '@nestjs/bullmq';+import { closeWorkerIfStarted, MAX_WORKER_CLOSE_WAIT_MS } from './close-worker';++describe('closeWorkerIfStarted', () => {+ beforeEach(() => jest.useFakeTimers());+ afterEach(() => jest.useRealTimers());++ const hungHost = (): WorkerHost =>+ ({ worker: { name: 'q', close: () => new Promise<void>(() => undefined) } }) as unknown as WorkerHost;++ const settledAt = async (waitMs: number): Promise<number> => {+ let doneAt = -1;+ const started = Date.now();+ void closeWorkerIfStarted(hungHost(), waitMs).then(() => (doneAt = Date.now() - started));+ await jest.advanceTimersByTimeAsync(MAX_WORKER_CLOSE_WAIT_MS + 1);+ return doneAt;+ };++ it('releases the hook at the requested wait when it is shorter than the cap', async () => {+ expect(await settledAt(2_000)).toBe(2_000);+ });++ it('clamps a longer wait to MAX_WORKER_CLOSE_WAIT_MS', async () => {+ expect(await settledAt(MAX_WORKER_CLOSE_WAIT_MS * 3)).toBe(MAX_WORKER_CLOSE_WAIT_MS);+ });++ it('settles a later worker.close() once the wait is over, as BullModule awaits it again', async () => {+ const host = hungHost();+ const pending = closeWorkerIfStarted(host, 1_000);+ await jest.advanceTimersByTimeAsync(1_000);+ await pending;+ await expect(host.worker.close()).resolves.toBeUndefined();+ });+});diff --git a/src/modules/queue/processors/close-worker.ts b/src/modules/queue/processors/close-worker.tsindex 78968d186..a4f496edb 100644--- a/src/modules/queue/processors/close-worker.ts+++ b/src/modules/queue/processors/close-worker.ts@@ -4,14 +4,13 @@ import { createLogger } from '../../../common/services/logger.service';/*** Upper bound on a processor's early close. Worker.close() can stay pending while Redis is unreachable,- * and the global destroy hooks (API-key usage flush, plugin onDisable) wait behind it. The shipped kill- * deadline is 45s (docker-compose stop_grace_period, Helm terminationGracePeriodSeconds), and before- * QueueModule is destroyed shutdown has already spent SHUTDOWN_DELAY_MS (3s), the ingress reconciler- * wait (INGRESS_DISPATCH_TIMEOUT_MS, 5s), the engine teardown (10s per-engine deadline, engines in- * parallel) and WEBHOOK_SHUTDOWN_DRAIN_MS (5s). 15s here leaves 7s for the 5s-bounded usage flush,- * plugin onDisable and the rest of the teardown.+ * and the global destroy hooks (API-key usage flush, plugin onDisable) wait behind it. 10s equals the+ * default webhook delivery timeout, so a job whose POST uses the whole timeout has no time left for its+ * bookkeeping. A job still active at the cap is re-run after the next start, so receivers must+ * deduplicate on the stable idempotency key. The sum of every stage before and after this one, against+ * the kill deadline, is in shutdown-budget.ts.*/-export const MAX_WORKER_CLOSE_WAIT_MS = 15_000;+export const MAX_WORKER_CLOSE_WAIT_MS = 10_000;const logger = createLogger('QueueWorkerShutdown');@@ -31,10 +30,11 @@ export function startedWorker(host: WorkerHost): Worker | undefined {/*** Close a processor's Worker from its onModuleDestroy hook, waiting at most `waitMs` (capped at- * MAX_WORKER_CLOSE_WAIT_MS) for its running jobs. Past the wait the hook returns and the teardown goes on- * as it would without the early close; BullModule's onApplicationShutdown awaits the same pending close.- * A rejected close is left to that later await too, so it cannot skip the global destroy hooks. A no-op- * for a Worker that was never created.+ * MAX_WORKER_CLOSE_WAIT_MS) for its running jobs. Past the wait the hook returns and the teardown goes on.+ * A close that is still pending then is abandoned: Worker.close() hands every caller its first promise,+ * and BullModule awaits it again in onApplicationShutdown with no timeout, ahead of the DataSource+ * close, so it is replaced by one that resolves at once. A rejected close is left to that later await,+ * so it cannot skip the global destroy hooks. A no-op for a Worker that was never created.*/export async function closeWorkerIfStarted(host: WorkerHost, waitMs: number): Promise<void> {const worker = startedWorker(host);@@ -53,5 +53,6 @@ export async function closeWorkerIfStarted(host: WorkerHost, waitMs: number): PrclearTimeout(timer);if (!closed) {logger.warn('Queue worker did not close in time; continuing shutdown', { queue: worker.name, waitMs: limitMs });+ worker.close = () => Promise.resolve();}}diff --git a/src/modules/queue/processors/ingress.processor.spec.ts b/src/modules/queue/processors/ingress.processor.spec.tsindex e27742c10..a9569acea 100644--- a/src/modules/queue/processors/ingress.processor.spec.ts+++ b/src/modules/queue/processors/ingress.processor.spec.ts@@ -1,3 +1,4 @@+import { MAX_WORKER_CLOSE_WAIT_MS } from './close-worker';import { IngressProcessor } from './ingress.processor';function job(overrides = {}) {@@ -16,6 +17,27 @@ function job(overrides = {}) {}describe('IngressProcessor', () => {+ // Running jobs that share an ordering key run one after another, so the close takes the whole cap.+ it('waits MAX_WORKER_CLOSE_WAIT_MS for a worker that never closes, then returns', async () => {+ jest.useFakeTimers();+ try {+ const proc = new IngressProcessor({} as never, {} as never, {} as never, {} as never, {} as never);+ (proc as unknown as { _worker: unknown })._worker = {+ name: 'ingress',+ close: () => new Promise<void>(() => undefined),+ };+ let doneAt = -1;+ const started = Date.now();+ void proc.onModuleDestroy().then(() => (doneAt = Date.now() - started));+ await jest.advanceTimersByTimeAsync(MAX_WORKER_CLOSE_WAIT_MS - 1);+ expect(doneAt).toBe(-1);+ await jest.advanceTimersByTimeAsync(1);+ expect(doneAt).toBe(MAX_WORKER_CLOSE_WAIT_MS);+ } finally {+ jest.useRealTimers();+ }+ });+it('dispatches the event into the worker via dispatchWebhook', async () => {const dispatchWebhook = jest.fn().mockResolvedValue({ ok: true, status: 200 });const loader = { dispatchWebhookForInstance: dispatchWebhook };diff --git a/src/modules/queue/processors/ingress.processor.ts b/src/modules/queue/processors/ingress.processor.tsindex c9c578a19..262de02ef 100644--- a/src/modules/queue/processors/ingress.processor.ts+++ b/src/modules/queue/processors/ingress.processor.ts@@ -89,6 +89,8 @@ export class IngressProcessor extends WorkerHost {* Closing here, before the global modules, lets the jobs already running finish against live plugins* and leaves the rest in Redis for the next start. Running jobs that share an ordering key dispatch* one after another, so a full worker of them can outlast any wait; this one takes the whole cap.+ * A job still running at the cap is cut mid-dispatch and re-run after the next start; BullMQ allows one+ * stall per job, so a second one fails it.*/async onModuleDestroy(): Promise<void> {await closeWorkerIfStarted(this, MAX_WORKER_CLOSE_WAIT_MS);diff --git a/src/modules/queue/processors/webhook.processor.spec.ts b/src/modules/queue/processors/webhook.processor.spec.tsindex d8951e701..38b797bbe 100644--- a/src/modules/queue/processors/webhook.processor.spec.ts+++ b/src/modules/queue/processors/webhook.processor.spec.ts@@ -3,6 +3,7 @@ import { FindOperator, Repository } from 'typeorm';import { ConfigService } from '@nestjs/config';import { WebhookOutboxService } from '../../webhook/webhook-outbox.service';import { WebhookProcessor } from './webhook.processor';+import { MAX_WORKER_CLOSE_WAIT_MS } from './close-worker';import { Webhook } from '../../webhook/entities/webhook.entity';import { WebhookDeliveryFailure } from '../../webhook/entities/webhook-delivery-failure.entity';import { HookManager } from '../../../core/hooks';@@ -703,4 +704,35 @@ describe('WebhookProcessor', () => {await Promise.all(jobs);});});++ describe('onModuleDestroy', () => {+ // The wait for running jobs is the delivery timeout plus 5 s of bookkeeping, capped at the worker close cap.+ const settleAfter = async (timeoutMs: number): Promise<number> => {+ jest.useFakeTimers();+ try {+ configService.get.mockImplementation((key: string, def?: unknown) =>+ key === 'webhook.timeout' ? timeoutMs : def,+ );+ (processor as unknown as { _worker: unknown })._worker = {+ name: 'webhook',+ close: () => new Promise<void>(() => undefined),+ };+ let doneAt = -1;+ const started = Date.now();+ void processor.onModuleDestroy().then(() => (doneAt = Date.now() - started));+ await jest.advanceTimersByTimeAsync(MAX_WORKER_CLOSE_WAIT_MS + 1);+ return doneAt;+ } finally {+ jest.useRealTimers();+ }+ };++ it('waits for the delivery timeout plus 5 s when that is under the close cap', async () => {+ expect(await settleAfter(2_000)).toBe(7_000);+ });++ it('waits no longer than the close cap for a long delivery timeout', async () => {+ expect(await settleAfter(25_000)).toBe(MAX_WORKER_CLOSE_WAIT_MS);+ });+ });});diff --git a/src/modules/queue/processors/webhook.processor.ts b/src/modules/queue/processors/webhook.processor.tsindex 6168a59f5..40d788dc0 100644--- a/src/modules/queue/processors/webhook.processor.ts+++ b/src/modules/queue/processors/webhook.processor.ts@@ -83,7 +83,9 @@ export class WebhookProcessor extends WorkerHost {}// Stop taking jobs before the plugins are torn down, so a final attempt's webhook:error hook still- // reaches them; see IngressProcessor.onModuleDestroy. The wait covers one POST plus its bookkeeping.+ // reaches them; see IngressProcessor.onModuleDestroy. The wait is min(WEBHOOK_TIMEOUT + 5 s, 10 s), so it+ // covers one POST plus its bookkeeping only while WEBHOOK_TIMEOUT is 5 s or less; a job still active at the+ // cap is re-run after the next start.async onModuleDestroy(): Promise<void> {await closeWorkerIfStarted(this, this.configService.get<number>('webhook.timeout', 10000) + 5000);}diff --git a/src/modules/queue/queue.module.spec.ts b/src/modules/queue/queue.module.spec.tsindex 0b6c2d17d..ca6550049 100644--- a/src/modules/queue/queue.module.spec.ts+++ b/src/modules/queue/queue.module.spec.ts@@ -1,4 +1,32 @@-import { WEBHOOK_QUEUE_JOB_OPTIONS } from './queue.module';+import 'reflect-metadata';+import { FactoryProvider, Provider } from '@nestjs/common';+import { getQueueToken } from '@nestjs/bullmq';+import { QUEUE_NAMES } from './queue-names';+import { QueueModule, WEBHOOK_QUEUE_JOB_OPTIONS } from './queue.module';+import { ShutdownSafeQueue } from './shutdown-safe-queue';++// A stub base keeps the factory from opening a Redis connection; ShutdownSafeQueue still extends it.+jest.mock('bullmq', () => ({+ ...jest.requireActual<object>('bullmq'),+ Queue: class {+ constructor(readonly name: string) {}+ },+}));++// The producers must be built as ShutdownSafeQueue: without it a dropped Redis rejects or hangs their+// close and Nest skips the DataSource close behind it. registerQueue captures the class when the+// @Module decorator runs, so the check builds the registered producers instead of reading the static.+describe('QueueModule producers', () => {+ it.each([QUEUE_NAMES.WEBHOOK, QUEUE_NAMES.INGRESS])('%s is created as a ShutdownSafeQueue', async name => {+ const imports = Reflect.getMetadata('imports', QueueModule) as { providers?: Provider[] }[];+ const provider = imports+ .flatMap(m => m.providers ?? [])+ .find(p => (p as { provide?: unknown }).provide === getQueueToken(name)) as FactoryProvider | undefined;+ expect(provider).toBeDefined();++ expect(await provider!.useFactory({ name })).toBeInstanceOf(ShutdownSafeQueue);+ });+});/*** Failed webhook jobs retain their full payload in Redis until eviction, so the retention windowdiff --git a/src/modules/queue/queue.module.ts b/src/modules/queue/queue.module.tsindex 243429d9a..c96480f6f 100644--- a/src/modules/queue/queue.module.ts+++ b/src/modules/queue/queue.module.ts@@ -8,6 +8,7 @@ import { WebhookProcessor } from './processors/webhook.processor';import { IngressProcessor } from './processors/ingress.processor';import { QUEUE_NAMES } from './queue-names';import { queueConnectionOptions } from './redis-connection';+import { ShutdownSafeQueue } from './shutdown-safe-queue';import { Webhook } from '../webhook/entities/webhook.entity';import { WebhookOutboxService } from '../webhook/webhook-outbox.service';import { WebhookOutboxEvent } from '../webhook/entities/webhook-outbox-event.entity';@@ -31,6 +32,9 @@ export const WEBHOOK_QUEUE_JOB_OPTIONS = {removeOnFail: { age: 86400, count: 5000 },} as const;+// Read when registerQueue runs below, so it has to be set before the @Module decorator is evaluated.+BullModule.queueClass = ShutdownSafeQueue;+@Module({imports: [// Required for WebhookProcessor to inject Repository<Webhook> + Repository<WebhookDeliveryFailure>;diff --git a/src/modules/queue/shutdown-safe-queue.spec.ts b/src/modules/queue/shutdown-safe-queue.spec.tsnew file mode 100644index 000000000..330ddf6d3--- /dev/null+++ b/src/modules/queue/shutdown-safe-queue.spec.ts@@ -0,0 +1,118 @@+import { Global, Module } from '@nestjs/common';+import { NestFactory } from '@nestjs/core';+import { BullModule } from '@nestjs/bullmq';+import { Queue } from 'bullmq';+import { LoggerModule } from '../../common/services/logger.module';+import { LoggerService } from '../../common/services/logger.service';+import { QUEUE_CLOSE_TIMEOUT_MS, ShutdownSafeQueue } from './shutdown-safe-queue';++const REDIS_DOWN = "Stream isn't writeable and enableOfflineQueue options is false";++/** A ShutdownSafeQueue with no Redis behind it: Queue.close() rejects the way a dropped connection does. */+class DroppedQueue {+ constructor(readonly name: string) {+ return Object.assign(Object.create(ShutdownSafeQueue.prototype), { name }) as DroppedQueue;+ }+}++describe('ShutdownSafeQueue', () => {+ let superClose: jest.SpyInstance;+ let disconnect: jest.SpyInstance;+ let warn: jest.SpyInstance;++ beforeEach(() => {+ superClose = jest.spyOn(Queue.prototype, 'close').mockRejectedValue(new Error(REDIS_DOWN));+ disconnect = jest.spyOn(Object.getPrototypeOf(Queue.prototype), 'disconnect').mockResolvedValue(undefined);+ warn = jest.spyOn(LoggerService.prototype, 'warn').mockImplementation(() => undefined);+ });++ afterEach(() => jest.restoreAllMocks());++ afterAll(() => {+ BullModule.queueClass = Queue;+ });++ it('settles a close that rejects, logs it and drops the connection', async () => {+ const queue = new DroppedQueue('webhook') as unknown as ShutdownSafeQueue;++ await expect(queue.close()).resolves.toBeUndefined();++ expect(superClose).toHaveBeenCalledTimes(1);+ expect(disconnect).toHaveBeenCalledTimes(1);+ expect(JSON.stringify(warn.mock.calls)).toContain(REDIS_DOWN);+ });++ // A half-open socket swallows the QUIT: the close neither resolves nor rejects.+ it('settles a close that never settles at the deadline, logs it and drops the connection', async () => {+ jest.useFakeTimers();+ try {+ superClose.mockReturnValue(new Promise(() => undefined));+ const queue = new DroppedQueue('webhook') as unknown as ShutdownSafeQueue;+ let settled = false;+ const closing = queue.close().then(() => {+ settled = true;+ });++ await jest.advanceTimersByTimeAsync(QUEUE_CLOSE_TIMEOUT_MS - 1);+ expect(settled).toBe(false);+ await jest.advanceTimersByTimeAsync(1);+ await closing;++ expect(disconnect).toHaveBeenCalledTimes(1);+ expect(JSON.stringify(warn.mock.calls)).toContain(`${QUEUE_CLOSE_TIMEOUT_MS} ms`);+ } finally {+ jest.useRealTimers();+ }+ });++ it('leaves no timer behind after a close that resolves', async () => {+ jest.useFakeTimers();+ try {+ superClose.mockResolvedValue(undefined);+ const queue = new DroppedQueue('webhook') as unknown as ShutdownSafeQueue;+ await queue.close();++ expect(jest.getTimerCount()).toBe(0);+ expect(disconnect).not.toHaveBeenCalled();+ } finally {+ jest.useRealTimers();+ }+ });++ it('does not wait for a disconnect that never settles', async () => {+ disconnect.mockReturnValue(new Promise(() => undefined));+ const queue = new DroppedQueue('webhook') as unknown as ShutdownSafeQueue;++ await expect(queue.close()).resolves.toBeUndefined();+ });++ // Nest does not catch a rejected shutdown hook, so one queue that fails to close would skip the hooks+ // of every module destroyed after it. Global modules, the DataSource's among them, are destroyed last.+ it('lets the later shutdown hooks run when the producer close fails in a real app', async () => {+ BullModule.queueClass = DroppedQueue;+ let databaseClosed = false;++ @Global()+ @Module({})+ class DatabaseStubModule {+ onApplicationShutdown(): void {+ databaseClosed = true;+ }+ }+ @Module({+ imports: [+ LoggerModule,+ DatabaseStubModule,+ BullModule.forRoot({ connection: {} }),+ BullModule.registerQueue({ name: 'webhook' }),+ ],+ })+ class RootModule {}++ const app = await NestFactory.createApplicationContext(RootModule, { logger: false });+ await expect(app.close()).resolves.toBeUndefined();++ expect(superClose).toHaveBeenCalled();+ expect(databaseClosed).toBe(true);+ });+});diff --git a/src/modules/queue/shutdown-safe-queue.ts b/src/modules/queue/shutdown-safe-queue.tsnew file mode 100644index 000000000..7940cb409--- /dev/null+++ b/src/modules/queue/shutdown-safe-queue.ts@@ -0,0 +1,42 @@+import { Queue } from 'bullmq';+import { createLogger } from '../../common/services/logger.service';++const logger = createLogger('QueueShutdown');++/** How long a producer queue close may take. Same bound as the other Redis quits (cache, throttler, WebSocket). */+export const QUEUE_CLOSE_TIMEOUT_MS = 2000;++/**+ * A producer Queue whose close() neither rejects nor hangs. BullModule awaits that close in the+ * queue's onApplicationShutdown hook, Nest does not catch a rejected hook, and every later hook is+ * skipped: the global DataSource close and the ffmpeg kill. Two Redis failures reach it. Once Redis+ * has dropped, the producer connection has no offline queue and the QUIT inside Queue.close()+ * rejects at once. On a half-open socket (a network partition) the QUIT is written and no reply ever+ * arrives, so the close never settles and the later hooks wait for the force-exit. Either way the+ * failure is logged and the connection is dropped, within QUEUE_CLOSE_TIMEOUT_MS. The disconnect is+ * not awaited: on a socket that is already gone ioredis never emits the 'end' it would wait for.+ */+export class ShutdownSafeQueue extends Queue {+ override async close(): Promise<void> {+ let timer: NodeJS.Timeout | undefined;+ try {+ await Promise.race([+ super.close(),+ new Promise<never>((_, reject) => {+ timer = setTimeout(+ () => reject(new Error(`close did not settle within ${QUEUE_CLOSE_TIMEOUT_MS} ms`)),+ QUEUE_CLOSE_TIMEOUT_MS,+ );+ }),+ ]);+ } catch (err) {+ logger.warn('Queue close failed; dropping the connection and continuing shutdown', {+ queue: this.name,+ error: err instanceof Error ? err.message : String(err),+ });+ this.disconnect().catch(() => undefined);+ } finally {+ clearTimeout(timer);+ }+ }+}diff --git a/src/modules/queue/worker-shutdown.spec.ts b/src/modules/queue/worker-shutdown.spec.tsindex 9ea42d670..addbff059 100644--- a/src/modules/queue/worker-shutdown.spec.ts+++ b/src/modules/queue/worker-shutdown.spec.ts@@ -1,5 +1,3 @@-import { readFileSync } from 'fs';-import { join } from 'path';import { Global, Module } from '@nestjs/common';import { GLOBAL_MODULE_METADATA, MODULE_METADATA } from '@nestjs/common/constants';import { ConfigService } from '@nestjs/config';@@ -15,13 +13,10 @@ import { QueueModule } from './queue.module';import { MAX_WORKER_CLOSE_WAIT_MS } from './processors/close-worker';import { PluginLoaderService } from '../../core/plugins/plugin-loader.service';import { HookManager } from '../../core/hooks';-import { DEFAULT_SHUTDOWN_DELAY_MS, ShutdownService } from '../../common/services/shutdown.service';+import { ShutdownService } from '../../common/services/shutdown.service';import { LoggerModule } from '../../common/services/logger.module';-import configuration from '../../config/configuration';-import { ApiKeyUsageTracker } from '../auth/api-key-usage-tracker.service';import { IntegrationDeliveryFailure } from '../integration/entities/integration-delivery-failure.entity';import { IngressEvent } from '../integration/entities/ingress-event.entity';-import { INGRESS_DISPATCH_TIMEOUT_MS } from '../integration/integration.constants';import { Webhook } from '../webhook/entities/webhook.entity';import { WebhookDeliveryFailure } from '../webhook/entities/webhook-delivery-failure.entity';import { WebhookOutboxService } from '../webhook/webhook-outbox.service';@@ -170,12 +165,15 @@ describe('queue workers at shutdown', () => {const warn = jest.spyOn(console, 'warn').mockImplementation(() => undefined);jest.useFakeTimers();try {- void app.close();+ const closing = app.close();await jest.advanceTimersByTimeAsync(MAX_WORKER_CLOSE_WAIT_MS - 1);expect(pluginTeardownRan).toBe(false);await jest.advanceTimersByTimeAsync(1);expect(pluginTeardownRan).toBe(true);+ // BullModule awaits the same worker close again in onApplicationShutdown, ahead of the DataSource+ // close; it must not wait on the close that never settles.+ await expect(closing).resolves.toBeUndefined();const warned = warn.mock.calls.flat().join('\n');expect(warned).toContain(QUEUE_NAMES.INGRESS);expect(warned).toContain(QUEUE_NAMES.WEBHOOK);@@ -185,31 +183,6 @@ describe('queue workers at shutdown', () => {}});- // The global destroy hooks run after the capped wait, so the cap must leave them room inside the- // shipped kill deadline after what shutdown spends before QueueModule is destroyed: SHUTDOWN_DELAY_MS,- // the ingress reconciler wait (INGRESS_DISPATCH_TIMEOUT_MS), the per-engine destroy deadline (engines- // torn down in parallel) and WEBHOOK_SHUTDOWN_DRAIN_MS; then the bounded API-key usage flush. The- // defaults come from their sources, so raising one fails here.- it('leaves the global destroy hooks room inside the shipped kill deadline', () => {- const root = join(__dirname, '../../..');- const compose = readFileSync(join(root, 'docker-compose.yml'), 'utf8');- const values = readFileSync(join(root, 'charts/openwa/values.yaml'), 'utf8');- const graceMs = [- Number(/^\s+stop_grace_period: (\d+)s$/m.exec(compose)?.[1]) * 1000,- Number(/^terminationGracePeriodSeconds: (\d+)$/m.exec(values)?.[1]) * 1000,- ];-- // Hard-coded: an inline literal in SessionLifecycleFences.teardownEngineSafely, not exported.- const engineTeardown = 10_000;- const beforeQueueClose =- DEFAULT_SHUTDOWN_DELAY_MS +- INGRESS_DISPATCH_TIMEOUT_MS +- engineTeardown +- configuration().webhook.shutdownDrainMs;- const usageFlush = ApiKeyUsageTracker.SHUTDOWN_FLUSH_TIMEOUT_MS;- expect(beforeQueueClose + MAX_WORKER_CLOSE_WAIT_MS + usageFlush).toBeLessThan(Math.min(...graceMs));- });-// A bootstrap that fails before listen() closes an app whose workers were never created; the destroy// hooks must not reject, or the remaining teardown (plugins, database, Redis) is skipped.it('tears down an app closed before its workers were created', async () => {diff --git a/src/modules/session/session-lifecycle-fences.spec.ts b/src/modules/session/session-lifecycle-fences.spec.tsindex 20a0a9a20..2966f910b 100644--- a/src/modules/session/session-lifecycle-fences.spec.ts+++ b/src/modules/session/session-lifecycle-fences.spec.ts@@ -1,7 +1,7 @@import { ConflictException } from '@nestjs/common';import { EngineRegistry } from '../../engine/engine-registry.service';import { IWhatsAppEngine } from '../../engine/interfaces/whatsapp-engine.interface';-import { SessionLifecycleFences } from './session-lifecycle-fences';+import { ENGINE_TEARDOWN_TIMEOUT_MS, SessionLifecycleFences } from './session-lifecycle-fences';/*** Direct invariant coverage for the credential-teardown / initial-status fences. Until now these@@ -43,6 +43,23 @@ describe('SessionLifecycleFences', () => {);});+ it('gives up on N hung engines at one ENGINE_TEARDOWN_TIMEOUT_MS, not N of them (the shutdown budget counts it once)', async () => {+ jest.useFakeTimers();+ try {+ const { fences } = makeFences();+ const hung = () => engine({ destroy: jest.fn(() => new Promise<void>(() => undefined)) });+ const all = Promise.all([1, 2, 3, 4].map(n => fences.destroyEngineSafely(`s${n}`, hung())));+ let settled = false;+ void all.then(() => (settled = true));+ await jest.advanceTimersByTimeAsync(ENGINE_TEARDOWN_TIMEOUT_MS - 1);+ expect(settled).toBe(false);+ await jest.advanceTimersByTimeAsync(1);+ await expect(all).resolves.toEqual([false, false, false, false]);+ } finally {+ jest.useRealTimers();+ }+ });+it('tracks the RAW logout promise under the session NAME before racing the deadline', async () => {const { fences, pendingTeardowns } = makeFences();let settle!: () => void;diff --git a/src/modules/session/session-lifecycle-fences.ts b/src/modules/session/session-lifecycle-fences.tsindex 556cfc54e..cf66313ec 100644--- a/src/modules/session/session-lifecycle-fences.ts+++ b/src/modules/session/session-lifecycle-fences.ts@@ -3,6 +3,9 @@ import { EngineRegistry } from '../../engine/engine-registry.service';import { IWhatsAppEngine } from '../../engine/interfaces/whatsapp-engine.interface';import { type createLogger } from '../../common/services/logger.service';+/** Per-engine teardown deadline. Shutdown tears engines down in parallel, so it counts once. See shutdown-budget.ts. */+export const ENGINE_TEARDOWN_TIMEOUT_MS = 10_000;+/*** The credential-teardown / initial-status fences extracted from SessionEngineLifecycle. Plain* class (NOT a NestJS provider — the lifecycle's constructor signature is frozen by specs), built@@ -88,7 +91,7 @@ export class SessionLifecycleFences {await Promise.race([raw,new Promise<never>((_, reject) => {- timer = setTimeout(() => reject(new Error(`engine.${label}() timed out`)), 10_000);+ timer = setTimeout(() => reject(new Error(`engine.${label}() timed out`)), ENGINE_TEARDOWN_TIMEOUT_MS);}),]);return true;diff --git a/src/modules/session/session.service.ts b/src/modules/session/session.service.tsindex d79b0bb6d..e9a111b25 100644--- a/src/modules/session/session.service.ts+++ b/src/modules/session/session.service.ts@@ -88,8 +88,9 @@ function isTransientLaunchFailure(error: unknown): boolean {export const AUTOSTART_THROTTLE_MS = 2_000;/**- * How long shutdown waits for the auto-start run once the engines are destroyed. Short: the 45s kill- * deadline in compose and the Helm chart still has to cover releaseAll and the database close.+ * How long shutdown waits for the auto-start run once the engines are destroyed. Short: the webhook+ * drain, worker close, usage flush, plugin phase, releaseAll and the HTTP and database close still have+ * to fit the kill deadline (see shutdown-budget.ts).*/export const AUTOSTART_SHUTDOWN_WAIT_MS = 5_000;diff --git a/src/modules/webhook/webhook-delivery.service.ts b/src/modules/webhook/webhook-delivery.service.tsindex 404dab91c..425356dc2 100644--- a/src/modules/webhook/webhook-delivery.service.ts+++ b/src/modules/webhook/webhook-delivery.service.ts@@ -16,6 +16,7 @@ import {recordTerminalFailure,} from './utils/deliver-once';import { createLogger } from '../../common/services/logger.service';+import { DEFAULT_WEBHOOK_SHUTDOWN_DRAIN_MS } from '../../common/services/shutdown-budget';import { DEFAULT_WEBHOOK_MEDIA_INLINE_MAX_BYTES, shedInlineMedia } from '../../common/utils/inline-media';import { incrementWebhookDeliveryFailures } from '../../common/metrics/webhook-delivery-metrics';import { QUEUE_NAMES } from '../queue/queue-names';@@ -68,9 +69,9 @@ const DEFAULT_WEBHOOK_MAX_PAYLOAD_BYTES = 1024 * 1024;/*** How long shutdown waits for in-flight direct deliveries (and their dead-letter bookkeeping) to* finish before abandoning them, and for an outbox replay in flight. Default 5s; override with- * WEBHOOK_SHUTDOWN_DRAIN_MS.+ * WEBHOOK_SHUTDOWN_DRAIN_MS. Defined in shutdown-budget.ts, which sizes the kill deadline from it.*/-export const DEFAULT_WEBHOOK_SHUTDOWN_DRAIN_MS = 5000;+export { DEFAULT_WEBHOOK_SHUTDOWN_DRAIN_MS };/*** The result of one delivery attempt. Reported, not thrown: every delivery failure below is already@@ -233,7 +234,7 @@ export class WebhookDeliveryService implements OnModuleInit, OnModuleDestroy {* up to WEBHOOK_SHUTDOWN_DRAIN_MS to finish; anything still running after that is about to be* dropped by process exit, so it is logged per delivery — a dead-letter row would be wrong there,* since the receiver may already have gotten the event. Nest awaits this hook during app.close(),- * so the bound also keeps app.close() itself bounded.+ * so the bound limits this hook only; the whole teardown is the sum in shutdown-budget.ts.*/async onModuleDestroy(): Promise<void> {if (!this.queueEnabled) {diff --git a/src/modules/webhook/webhook-redrive.service.spec.ts b/src/modules/webhook/webhook-redrive.service.spec.tsindex 23db973f7..1611406af 100644--- a/src/modules/webhook/webhook-redrive.service.spec.ts+++ b/src/modules/webhook/webhook-redrive.service.spec.ts@@ -6,6 +6,7 @@ import { Webhook } from './entities/webhook.entity';import { WebhookDeliveryFailure } from './entities/webhook-delivery-failure.entity';import { WebhookDeliveryService } from './webhook-delivery.service';import { MAX_WEBHOOK_REDRIVE_LIMIT, WebhookRedriveService } from './webhook-redrive.service';+import { MAX_TIMER_MS } from '../../common/services/shutdown-budget';import { recordWebhookDeliveryFailure } from './utils/record-delivery-failure';describe('WebhookRedriveService', () => {@@ -15,6 +16,7 @@ describe('WebhookRedriveService', () => {let primary: Webhook;let otherSession: Webhook;let retentionHours: number;+ let drainMs: number;let delivery: { redeliver: jest.MockedFunction<WebhookDeliveryService['redeliver']> };let service: WebhookRedriveService;@@ -61,6 +63,7 @@ describe('WebhookRedriveService', () => {),);retentionHours = 24;+ drainMs = 5000;delivery = {redeliver: jest.fn<ReturnType<WebhookDeliveryService['redeliver']>,@@ -92,7 +95,9 @@ describe('WebhookRedriveService', () => {webhooks,failures,delivery as unknown as WebhookDeliveryService,- { get: () => retentionHours } as unknown as ConfigService,+ {+ get: (key: string) => (key === 'webhook.shutdownDrainMs' ? drainMs : retentionHours),+ } as unknown as ConfigService,);});@@ -364,4 +369,74 @@ describe('WebhookRedriveService', () => {expect(select).not.toHaveBeenCalled();expect(delivery.redeliver).toHaveBeenCalledTimes(4);});++ describe('shutdown wait bound', () => {+ const holdReplay = async (): Promise<() => void> => {+ await addFailure();+ let release!: () => void;+ const gate = new Promise<void>(resolve => (release = resolve));+ const deliver = delivery.redeliver.getMockImplementation()!;+ delivery.redeliver.mockImplementation(async (...args) => {+ await gate;+ return deliver(...args);+ });+ void service.redrive({});+ await new Promise(resolve => setImmediate(resolve));+ expect(delivery.redeliver).toHaveBeenCalledTimes(1);+ return release;+ };++ it('stops waiting for a replay that never settles after WEBHOOK_SHUTDOWN_DRAIN_MS and logs it', async () => {+ drainMs = 30;+ const release = await holdReplay();+ const warn = jest.spyOn(service['logger'], 'warn').mockImplementation(() => undefined);+ const startedAt = Date.now();++ await service.onModuleDestroy();++ expect(Date.now() - startedAt).toBeGreaterThanOrEqual(25);+ expect(warn).toHaveBeenCalledWith(+ expect.stringContaining('redrive still running'),+ expect.objectContaining({ action: 'webhook_redrive_shutdown_timeout', waitMs: 30 }),+ );+ release();+ });++ it('caps the wait at the largest timer delay when the drain exceeds it', async () => {+ drainMs = 2 ** 31 + 1000;+ const release = await holdReplay();+ const warn = jest.spyOn(service['logger'], 'warn').mockImplementation(() => undefined);+ const setTimeoutSpy = jest.spyOn(global, 'setTimeout');++ const destroy = service.onModuleDestroy();+ release();+ await destroy;++ expect(setTimeoutSpy).toHaveBeenCalledWith(expect.any(Function), MAX_TIMER_MS, false);+ expect(warn).not.toHaveBeenCalled();+ });++ it('returns without waiting when the drain is 0', async () => {+ drainMs = 0;+ const release = await holdReplay();+ const warn = jest.spyOn(service['logger'], 'warn').mockImplementation(() => undefined);++ await service.onModuleDestroy();++ expect(warn).toHaveBeenCalledTimes(1);+ release();+ });++ it('resolves as soon as the replay settles and logs nothing', async () => {+ drainMs = 60_000;+ const release = await holdReplay();+ const warn = jest.spyOn(service['logger'], 'warn').mockImplementation(() => undefined);++ const destroy = service.onModuleDestroy();+ release();+ await destroy;++ expect(warn).not.toHaveBeenCalled();+ });+ });});diff --git a/src/modules/webhook/webhook-redrive.service.ts b/src/modules/webhook/webhook-redrive.service.tsindex cb2e42b6e..219c53831 100644--- a/src/modules/webhook/webhook-redrive.service.ts+++ b/src/modules/webhook/webhook-redrive.service.ts@@ -4,8 +4,9 @@ import { InjectRepository } from '@nestjs/typeorm';import { In, IsNull, MoreThan, Not, Repository } from 'typeorm';import { Webhook } from './entities/webhook.entity';import { WebhookDeliveryFailure } from './entities/webhook-delivery-failure.entity';-import { WebhookDeliveryService } from './webhook-delivery.service';+import { DEFAULT_WEBHOOK_SHUTDOWN_DRAIN_MS, WebhookDeliveryService } from './webhook-delivery.service';import { isDeliverableWebhook } from './utils/deliver-once';+import { MAX_TIMER_MS } from '../../config/configuration';import { resolveSessionScope } from '../../common/security/session-scope';import { createLogger } from '../../common/services/logger.service';@@ -81,11 +82,29 @@ export class WebhookRedriveService implements OnModuleDestroy {* on POSTing after PluginLoaderService has unregistered the `webhook:before` hooks, sending the* stored pre-hook data. Stop at the next row and wait for the replays in hand: this module is* destroyed before the global plugin module, so those still run with their hooks. Rows not reached- * stay for a later call and are counted in `remaining`.+ * stay for a later call and are counted in `remaining`. The wait is at most WEBHOOK_SHUTDOWN_DRAIN_MS,+ * like the delivery drain and the reconciler beside it: a replay in hand is one POST plus hooks and+ * database writes that carry no cap of their own.*/async onModuleDestroy(): Promise<void> {this.stop.abort();- await this.running;+ // Above MAX_TIMER_MS a Node timer fires after 1 ms; the delivery drain honours such a value in full.+ const waitMs = Math.min(+ this.configService.get<number>('webhook.shutdownDrainMs', DEFAULT_WEBHOOK_SHUTDOWN_DRAIN_MS),+ MAX_TIMER_MS,+ );+ let timer: NodeJS.Timeout | undefined;+ const settled = await Promise.race([+ this.running.then(() => true),+ new Promise<false>(resolve => (timer = setTimeout(resolve, waitMs, false))),+ ]);+ clearTimeout(timer);+ if (!settled) {+ this.logger.warn('Webhook redrive still running at shutdown; continuing without it', {+ waitMs,+ action: 'webhook_redrive_shutdown_timeout',+ });+ }}redrive(request: WebhookRedriveRequest, allowedSessions?: string[] | null): Promise<WebhookRedriveResult> {