Free, Open Source, Self-Hosted WhatsApp API Gateway
TypeScript apibotgatewayself-hostedwhatsappwhatsapp-apiwhatsapp-automationwhatsapp-botwhatsapp-gatewaywhatsapp-gateway-apiwhatsapp-webwhatsapp-web-api
fix(core): refuse new requests once shutdown teardown begins (#1796)
Yudhi Armyndharis0b983d7d18b5
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex f3b73688f..3e81969c2 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -24,6 +24,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Fixed+- Answer new HTTP requests with `503` once shutdown teardown begins.- Refuse session starts with `503` once shutdown begins, and leave a start that shutdown interrupts for the next boot instead of marking it failed.- Retire the pending webhook outbox record when a redrive delivers, so the sweep does not send the event again.- Stop dashboard multi-page loads and a pending chat mark-as-read from sending requests after logout.diff --git a/docs/06-api-specification.md b/docs/06-api-specification.mdindex 982579e1a..2399d0a9e 100644--- a/docs/06-api-specification.md+++ b/docs/06-api-specification.md@@ -124,7 +124,7 @@ Validation failures (`statusCode: 400`) return `message` as an **array** of fiel| `500` | Internal Server Error | Send failed at the WhatsApp engine or an unexpected server error; when WhatsApp Web itself threw during a whatsapp-web.js send or status post, the body carries `code: 'ENGINE_PAGE_ERROR'`, `pageError` (`name`, `message`) and, when the page could read it, the WhatsApp Web `build` || `501` | Not Implemented | The operation is not supported by the active engine (see the capability matrix, docs/29) || `502` | Bad Gateway | An engine transport failure (e.g. a dead Baileys socket; retryable), an upstream component returned something unusable (not retryable; each route section carries the exact wording), or, on a multi-node deployment, forwarding a session-scoped request to its owner node failed after the request may already have been sent, so a non-idempotent call must not be replayed blindly (see docs/13) |-| `503` | Service Unavailable | A dependency or the session is not ready (boot draining, a datastore down, the engine reconnecting, a media `url` fetch through the session's egress proxy failing before any response, whether the proxy or the target is at fault), or WhatsApp did not answer an engine operation within its deadline; retryable, except that a write which timed out waiting for WhatsApp may still have been applied, so a non-idempotent call must not be replayed blindly (treat a `503` on a write as outcome-unknown unless the route section says repeating it is safe). On a multi-node deployment a forward that fails before reaching the owner node answers `503` and was not carried out (see docs/13); a `503` from the owner itself is relayed unchanged and keeps that route's meaning. Also answered, with `Retry-After`, when too many request bodies are in flight at once |+| `503` | Service Unavailable | A dependency or the session is not ready (boot draining or shutdown teardown, a datastore down, the engine reconnecting, a media `url` fetch through the session's egress proxy failing before any response, whether the proxy or the target is at fault), or WhatsApp did not answer an engine operation within its deadline; retryable, except that a write which timed out waiting for WhatsApp may still have been applied, so a non-idempotent call must not be replayed blindly (treat a `503` on a write as outcome-unknown unless the route section says repeating it is safe). On a multi-node deployment a forward that fails before reaching the owner node answers `503` and was not carried out (see docs/13); a `503` from the owner itself is relayed unchanged and keeps that route's meaning. Also answered, with `Retry-After`, when too many request bodies are in flight at once || `504` | Gateway Timeout | An upstream the gateway waits on did not answer in time: on a multi-node deployment, the owner node did not answer a forwarded session-scoped request within `SESSION_PROXY_TIMEOUT_MS`; on `POST /sessions/{sessionId}/start` (see §6.4.1), the engine did not finish starting within its timeout, either because initialization never finished at all (WhatsApp Web, the network or the session `proxyUrl` unreachable in a way that hangs the connection, or a browser stalled mid-startup, for example under a container memory or resource limit) or, on whatsapp-web.js, because its auth poll expired after the page loaded (typically an unreachable `proxyUrl`; on Baileys an unreachable proxy does not fail the start), and the message names which. On the forwarding path the request may still have been carried out on the owner node, so a non-idempotent call must not be replayed blindly (see docs/13) |### Timestamp Conventionsdiff --git a/src/common/services/shutdown.service.spec.ts b/src/common/services/shutdown.service.spec.tsindex db7a9f380..d1828e160 100644--- a/src/common/services/shutdown.service.spec.ts+++ b/src/common/services/shutdown.service.spec.ts@@ -52,6 +52,21 @@ describe('ShutdownService.shutdown (idempotent, bounded grace)', () => {expect(jest.getTimerCount()).toBe(1);});+ it('marks teardown only once the grace has elapsed, before the teardown callback runs', async () => {+ process.env.SHUTDOWN_DELAY_MS = '2000';+ const svc = new ShutdownService();+ let tearingDownInCallback: boolean | undefined;+ svc.setShutdownCallback(() => {+ tearingDownInCallback = svc.isTearingDown();+ return Promise.resolve();+ });+ svc.shutdown();+ await jest.advanceTimersByTimeAsync(1999);+ expect(svc.isTearingDown()).toBe(false); // still draining: requests are served+ await jest.advanceTimersByTimeAsync(1);+ expect(tearingDownInCallback).toBe(true);+ });+it('runs the teardown callback and exits exactly once even when called repeatedly', async () => {process.env.SHUTDOWN_DELAY_MS = '0';const { svc, cb } = svcWithCb();diff --git a/src/common/services/shutdown.service.ts b/src/common/services/shutdown.service.tsindex c8da8d1c9..88d52afcc 100644--- a/src/common/services/shutdown.service.ts+++ b/src/common/services/shutdown.service.ts@@ -11,6 +11,7 @@ export class ShutdownService {private destroyCallback: (() => Promise<void>) | null = null;private shuttingDown = false;private shutdownScheduled = false;+ private tearingDown = false;/*** Set the shutdown callback (called from main.ts after app creation)@@ -27,6 +28,15 @@ export class ShutdownService {return this.shuttingDown;}+ /**+ * True once the grace has elapsed and teardown has started. Nest keeps the HTTP listener open+ * until every destroy hook has finished, so the request gate in configure-app.ts refuses new+ * requests from this point on.+ */+ isTearingDown(): boolean {+ return this.tearingDown;+ }+/** Flip the draining flag (idempotent). Safe to call synchronously from a signal handler. */markShuttingDown(): void {if (!this.shuttingDown) {@@ -52,6 +62,7 @@ export class ShutdownService {setTimeout(() => {this.logger.log('Initiating shutdown...');+ this.tearingDown = true;const doShutdown = async () => {// The exit status mirrors the teardown outcome: 0 when teardown completed, 1 when it// failed — an orchestrator (k8s, systemd, docker restart policies) must not read adiff --git a/src/configure-app.ts b/src/configure-app.tsindex 4ede4f53a..bf4e64e73 100644--- a/src/configure-app.ts+++ b/src/configure-app.ts@@ -1,4 +1,4 @@-import { INestApplication } from '@nestjs/common';+import { INestApplication, Type } from '@nestjs/common';import { ModuleRef } from '@nestjs/core';import helmet from 'helmet';import { Request, Response, NextFunction, json, urlencoded } from 'express';@@ -14,6 +14,7 @@ import {import { ActiveKeyIndex } from './modules/auth/active-key-index';import { requestContextMiddleware } from './common/middleware/request-context.middleware';import { createLogger } from './common/services/logger.service';+import { ShutdownService } from './common/services/shutdown.service';import { injectDashboardCspNonce } from './config/dashboard-csp';import { resolveCorsPolicy, isUpgradeInsecureRequestsEnabled, resolveBodyLimit } from './config/bootstrap-security';import { resolveRequestTimeoutMs } from './config/http-timeouts';@@ -211,6 +212,32 @@ export function configureApp(app: INestApplication, options: ConfigureAppOptionsres.status(400).json({ statusCode: 400, message: 'URL must not contain an encoded NUL', error: 'Bad Request' });});+ // Providers some test modules lack. Looked up through ModuleRef: a failed app.get() aborts the+ // process instead of throwing.+ const optional = <T>(token: Type<T>): T | undefined => {+ try {+ return app.get(ModuleRef).get(token, { strict: false });+ } catch {+ return undefined;+ }+ };++ // Once the shutdown grace has elapsed, teardown runs every destroy hook before the listener+ // closes, and a request admitted then would run against services being torn down. Refuse new+ // requests from that point; admitted ones run to completion. The health probes stay answered so+ // readiness keeps reporting its draining state. Ahead of the budget, so no body is read.+ const shutdown = optional(ShutdownService);+ app.use((req: Request, res: Response, next: NextFunction) => {+ if (!shutdown?.isTearingDown()) return next();+ const path = req.path.toLowerCase();+ if (path === '/api/health' || path.startsWith('/api/health/')) return next();+ res.status(503).set('Connection', 'close').json({+ statusCode: 503,+ message: 'Server is shutting down',+ error: 'Service Unavailable',+ });+ });+// Aggregate in-flight body budget (DoS hardening): once too many body bytes are being buffered// across ALL connections, new requests get 503 + Retry-After without their body being read.// This is a deliberate PRE-GUARD: the throttler/auth guards run at the Nest routing layer —@@ -227,13 +254,7 @@ export function configureApp(app: INestApplication, options: ConfigureAppOptions// Requests without a recognised API key share a pool of a quarter of the budget, never less than// two BODY_SIZE_LIMIT bodies (half the default budget). With no AuthModule in the app (some test// modules) every request is unrecognised, which is the stricter side.- // Looked up through ModuleRef: a failed app.get() aborts the process instead of throwing.- let keyIndex: ActiveKeyIndex | undefined;- try {- keyIndex = app.get(ModuleRef).get(ActiveKeyIndex, { strict: false });- } catch {- keyIndex = undefined;- }+ const keyIndex = optional(ActiveKeyIndex);app.use(createInflightBodyBudget(inflightBudgetBytes, {trustedProxies: (process.env.TRUSTED_PROXIES || '')diff --git a/test/configure-app.e2e-spec.ts b/test/configure-app.e2e-spec.tsindex e7076f2dd..8d4439357 100644--- a/test/configure-app.e2e-spec.ts+++ b/test/configure-app.e2e-spec.ts@@ -10,6 +10,8 @@ import {Param,All,Req,+ Injectable,+ OnModuleDestroy,} from '@nestjs/common';import type { Request } from 'express';import { NestFactory } from '@nestjs/core';@@ -23,6 +25,7 @@ import { configureApp } from '../src/configure-app';import { applyGlobalValidation } from '../src/config/app-validation';import { DASHBOARD_CSP_NONCE_PLACEHOLDER } from '../src/config/dashboard-csp';import { ActiveKeyIndex } from '../src/modules/auth/active-key-index';+import { ShutdownService } from '../src/common/services/shutdown.service';import { documentErrorResponses, ERROR_RESPONSE_SCHEMA } from '../src/config/swagger.config';/**@@ -381,3 +384,102 @@ describe('in-flight body budget tiers (configureApp)', () => {}});});++describe('requests during shutdown teardown (configureApp)', () => {+ // Nest keeps the listener open until every destroy hook has finished, so the hook below can still+ // reach the server: it records what a client arriving mid-teardown gets back.+ let base = '';+ const handled: string[] = [];+ const duringTeardown: Record<string, request.Response> = {};+ let slowEntered!: () => void;+ const slowAdmitted = new Promise<void>(resolve => (slowEntered = resolve));+ let releaseSlow!: () => void;+ const slowReleased = new Promise<void>(resolve => (releaseSlow = resolve));++ @Controller()+ class TeardownProbeController {+ @Get('work')+ work() {+ handled.push('work');+ return { ok: true };+ }++ @Get('slow')+ async slow() {+ handled.push('slow');+ slowEntered();+ await slowReleased;+ return { ok: true };+ }++ @Get('health/live')+ live() {+ return { status: 'ok' };+ }+ }++ @Injectable()+ class TeardownProbe implements OnModuleDestroy {+ async onModuleDestroy() {+ for (const path of ['/api/work', '/api/health/live']) {+ duringTeardown[path] = await request(base).get(path).set('Connection', 'keep-alive');+ }+ releaseSlow();+ }+ }++ @Module({ controllers: [TeardownProbeController], providers: [ShutdownService, TeardownProbe] })+ class TeardownSurfaceModule {}++ it('refuses new requests once teardown begins, while admitted requests and the health probes complete', async () => {+ const app = await NestFactory.create<INestApplication<App>>(TeardownSurfaceModule, {+ bodyParser: false,+ logger: false,+ });+ configureApp(app);+ applyGlobalValidation(app);+ await app.listen(0, '127.0.0.1');+ base = await app.getUrl();++ let exited!: () => void;+ const exitCalled = new Promise<void>(resolve => (exited = resolve));+ const exitSpy = jest.spyOn(process, 'exit').mockImplementation((() => exited()) as never);+ let teardownStarted = false;+ try {+ const shutdown = app.get(ShutdownService);+ shutdown.setShutdownCallback(() => app.close());++ // The drain: readiness reports 503 so the balancer stops routing, but traffic is still served.+ shutdown.markShuttingDown();+ expect((await request(base).get('/api/work')).status).toBe(200);++ const slow = request(base)+ .get('/api/slow')+ .then(res => res);+ await slowAdmitted;+ teardownStarted = true;+ shutdown.shutdown(0);+ await exitCalled;++ const refused = duringTeardown['/api/work'];+ expect(refused.status).toBe(503);+ expect(refused.headers.connection).toBe('close');+ expect(refused.body).toEqual({+ statusCode: 503,+ message: 'Server is shutting down',+ error: 'Service Unavailable',+ });+ // The handler never ran for the refused request: only the drain-time call and the slow one did.+ expect(handled).toEqual(['work', 'slow']);+ expect(duringTeardown['/api/health/live'].status).toBe(200);+ expect((await slow).status).toBe(200);+ } finally {+ exitSpy.mockRestore();+ // A failure before teardown would otherwise leave the listener and the slow handler open.+ if (!teardownStarted) {+ releaseSlow();+ await app.close();+ }+ }+ });+});