Free, Open Source, Self-Hosted WhatsApp API Gateway
TypeScript apibotgatewayself-hostedwhatsappwhatsapp-apiwhatsapp-automationwhatsapp-botwhatsapp-gatewaywhatsapp-gateway-apiwhatsapp-webwhatsapp-web-api
fix(http): keep most of the body budget for requests with an API key
Yudhi Armyndharis2c62ed790dc8
diff --git a/.env.example b/.env.exampleindex 8fb23aef7..641f7fa49 100644--- a/.env.example+++ b/.env.example@@ -905,9 +905,15 @@ PLUGINS_DIR=./data/plugins # Plugin directory (default: ./data/plugins)# Aggregate cap on request-body bytes buffered across ALL connections. Once exceeded, new requests# get 503 + Retry-After without their body being read (protects against slow-body memory pinning).# Keep it >= BODY_SIZE_LIMIT. Default: 4 x BODY_SIZE_LIMIT (100 MiB with the default 25mb cap).-# One client IP may hold at most HALF of this budget in flight and gets the same 503 + Retry-After-# past that. Behind a reverse proxy set TRUSTED_PROXIES, or every caller resolves to the proxy-# address and shares that one half. A request holds its declared size (a chunked body 1 MiB, or-# what has arrived past that) until it finishes; a body that sends nothing for 15 s, or that falls-# behind the pace needed to deliver what it holds within REQUEST_TIMEOUT_MS, is dropped.+# Below 2 x BODY_SIZE_LIMIT one full-size unkeyed body can leave no room for ingress deliveries.+# Requests without an active API key (ingress webhooks, unauthenticated callers) share a quarter of+# it (at least two BODY_SIZE_LIMIT); the rest is kept for keyed requests. Each API key may hold at+# most HALF of the budget, and each unkeyed client IP half of the unkeyed pool (or one+# BODY_SIZE_LIMIT, if larger); past that the same 503 + Retry-After. Behind a reverse proxy set+# TRUSTED_PROXIES. Without it every unkeyed caller, ingress webhooks included, resolves to the+# proxy address and they all draw on one client share: at the defaults a single full-size unkeyed+# upload then refuses ingress deliveries with 503 until it finishes. A request holds its declared+# size (a chunked body 1 MiB, or what has arrived past that) until it finishes; a body that sends+# nothing for 15 s, or that falls behind the pace needed to deliver what it holds within+# REQUEST_TIMEOUT_MS, is dropped.# INFLIGHT_BODY_BUDGET_BYTES=104857600diff --git a/docs/04-security-design.md b/docs/04-security-design.mdindex 06541638f..09e1b78ff 100644--- a/docs/04-security-design.md+++ b/docs/04-security-design.md@@ -281,7 +281,7 @@ Each window is counted **per route handler per client IP** (the IP resolved throTTL values are in milliseconds. The `/api/metrics` and `/api/health*` routes are exempt (`@SkipThrottle`); `/api/health` still looks up at most 30 failed key presentations per client per minute, and past that answers without `version` and without a key lookup. To enforce tighter per-route limits, lower the global windows or add a limiter at your reverse proxy.-An IPv6 client is keyed on its /64, so rotating addresses inside one allocation does not mint fresh buckets. The same key is used by every other per-client limit: the MCP and Bull Board pre-auth throttles, the WebSocket limits below, the per-client share of the in-flight body budget, the health route's key-lookup budget and auth-failure audit bound, and the REST, queue-dashboard and MCP auth-failure audit bound. `allowedIps` matching and audit rows still see the full address.+An IPv6 client is keyed on its /64, so rotating addresses inside one allocation does not mint fresh buckets. The same key is used by every other per-client limit: the MCP and Bull Board pre-auth throttles, the WebSocket limits below, the per-client share of the in-flight body budget for requests without an API key, the health route's key-lookup budget and auth-failure audit bound, and the REST, queue-dashboard and MCP auth-failure audit bound. `allowedIps` matching and audit rows still see the full address.### Response on limit@@ -303,8 +303,10 @@ The API exposes the rate-limit headers via CORS (`exposedHeaders`) so browser clThe windows above run at the routing layer, after a request body has been buffered, so they cannot bound memory held by slow uploads. A pre-routing middleware does: it caps the request-body bytes in flight across all connections at `INFLIGHT_BODY_BUDGET_BYTES` (default 4 x `BODY_SIZE_LIMIT`) and answers a request that would cross it with `503` and `Retry-After`, before reading its body.-- One client (IPv6 on its /64) may hold at most half of the budget.-- A request holds its declared `Content-Length` until it finishes. After a 15-second grace it must keep up the pace that lands the whole body within `REQUEST_TIMEOUT_MS` (by default 25 MiB in 300 seconds, about 85 KiB/s); a request that falls behind is dropped and its reservation released. A chunked body holds a 1 MiB placeholder, or the bytes that have arrived once they pass it, and must deliver the placeholder at the same pace (about 3.4 KiB/s by default). If that growth crosses the budget or its client's half, that request is aborted.+- Requests without an active API key (public ingress webhooks, unauthenticated callers) share a pool of a quarter of the budget, never less than twice `BODY_SIZE_LIMIT` (so half of the default budget), and are charged at most `BODY_SIZE_LIMIT` each. The rest is kept for requests that carry an active key. With `INFLIGHT_BODY_BUDGET_BYTES` below three times `BODY_SIZE_LIMIT` that reserve is smaller than one full-size body, and at twice `BODY_SIZE_LIMIT` or less there is none.+- A request with an active key may hold at most half of the budget per key, wherever it connects from. An unkeyed client (IPv6 on its /64) may hold at most half of the unkeyed pool, or one `BODY_SIZE_LIMIT` if that is larger, so one full-size body from one address leaves room for other unkeyed senders once `INFLIGHT_BODY_BUDGET_BYTES` is at least twice `BODY_SIZE_LIMIT`; below that the room left is the budget minus one `BODY_SIZE_LIMIT`, and at a budget equal to `BODY_SIZE_LIMIT` a single full-size unkeyed body refuses every other body-carrying request, ingress deliveries included, with `503` until it finishes. Behind a reverse proxy without `TRUSTED_PROXIES`, every unkeyed request, ingress included, keys on the proxy address and draws on that one share, so at the defaults a single full-size unkeyed upload refuses ingress deliveries with `503` until it finishes.+- Keys are recognised from an in-memory index of active keys, refreshed at once when this node changes a key and every 30 seconds for a direct write to this node's main database. Keys are per node, so a change made on another node never reaches it. The index only picks the tier; the API-key guard still authorises every request.+- A request holds its declared `Content-Length` until it finishes. After a 15-second grace it must keep up the pace that lands the whole body within `REQUEST_TIMEOUT_MS` (by default 25 MiB in 300 seconds, about 85 KiB/s); a request that falls behind is dropped and its reservation released. A chunked body holds a 1 MiB placeholder, or the bytes that have arrived once they pass it, and must deliver the placeholder at the same pace (about 3.4 KiB/s by default). If that growth crosses the budget, its client's share, or (without a key) the unkeyed pool, that request is aborted.- A body that sends nothing for 15 seconds is dropped and its reservation released.- Requests without a body are never refused for lack of budget.diff --git a/docs/06-api-specification.md b/docs/06-api-specification.mdindex 7dc68a60e..93131d0ca 100644--- a/docs/06-api-specification.md+++ b/docs/06-api-specification.md@@ -109,7 +109,7 @@ Validation failures (`statusCode: 400`) return `message` as an **array** of fiel| `500` | Internal Server Error | Send failed at the WhatsApp engine or an unexpected server error; when WhatsApp Web itself threw during a whatsapp-web.js send or status post, the body carries `code: 'ENGINE_PAGE_ERROR'`, `pageError` (`name`, `message`) and, when the page could read it, the WhatsApp Web `build` || `501` | Not Implemented | The operation is not supported by the active engine (see the capability matrix, docs/29) || `502` | Bad Gateway | An engine transport failure (e.g. a dead Baileys socket; retryable), an upstream component returned something unusable (not retryable; each route section carries the exact wording), or, on a multi-node deployment, forwarding a session-scoped request to its owner node failed after the request may already have been sent, so a non-idempotent call must not be replayed blindly (see docs/13) |-| `503` | Service Unavailable | A dependency or the session is not ready (boot draining, a datastore down, the engine reconnecting, a media `url` fetch through the session's egress proxy failing before any response, whether the proxy or the target is at fault); retryable. On a multi-node deployment a forwarded request answers `503` only when the owner node was never reached, so the request was not carried out (see docs/13) |+| `503` | Service Unavailable | A dependency or the session is not ready (boot draining, a datastore down, the engine reconnecting, a media `url` fetch through the session's egress proxy failing before any response, whether the proxy or the target is at fault); retryable. On a multi-node deployment a forwarded request answers `503` only when the owner node was never reached, so the request was not carried out (see docs/13). Also answered, with `Retry-After`, when too many request bodies are in flight at once || `504` | Gateway Timeout | An upstream the gateway waits on did not answer in time: on a multi-node deployment, the owner node did not answer a forwarded session-scoped request within `SESSION_PROXY_TIMEOUT_MS`; on `POST /sessions/{sessionId}/start` (see §6.4.1), the engine did not finish starting within its timeout, either because initialization never finished at all (WhatsApp Web, the network or the session `proxyUrl` unreachable in a way that hangs the connection, or a browser stalled mid-startup, for example under a container memory or resource limit) or because the engine's own auth poll expired after the page loaded (typically an unreachable `proxyUrl`), and the message names which. On the forwarding path the request may still have been carried out on the owner node, so a non-idempotent call must not be replayed blindly (see docs/13) |### Timestamp Conventionsdiff --git a/src/config/inflight-body-budget.spec.ts b/src/config/inflight-body-budget.spec.tsindex b83511a89..25f6a54be 100644--- a/src/config/inflight-body-budget.spec.ts+++ b/src/config/inflight-body-budget.spec.ts@@ -307,6 +307,114 @@ describe('createInflightBodyBudget middleware', () => {});});+describe('anonymous tier', () => {+ beforeEach(() => jest.useFakeTimers());+ afterEach(() => jest.useRealTimers());++ // Budget 1000 with a 250-byte body cap: the anonymous pool is max(2 x 250, 25% of 1000) = 500,+ // each anonymous IP's share is 250 and each key's share is half the budget, 500.+ const tiered = () => {+ const classify = jest.fn((req: Request) => {+ const key = req.headers['x-api-key'];+ return typeof key === 'string' ? `hash-${key}` : undefined;+ });+ return { budget: createInflightBodyBudget(1000, { classify, bodyLimitBytes: 250 }), classify };+ };+ const from = (ip: string, headers: Record<string, string>): Request & EventEmitter => {+ const req = makeReq(headers);+ (req as unknown as { socket: { remoteAddress: string } }).socket.remoteAddress = ip;+ return req;+ };++ it('keeps unrecognised requests inside their pool while a keyed request still gets in', () => {+ const { budget } = tiered();+ expect(run(budget, from('198.51.100.1', { 'content-length': '250' })).next).toHaveBeenCalledTimes(1);+ expect(run(budget, from('198.51.100.2', { 'content-length': '250' })).next).toHaveBeenCalledTimes(1);+ expect(budget.anonymousBytes()).toBe(500);++ // The aggregate has 500 free, but none of it is open to a third anonymous client.+ const refused = run(budget, from('198.51.100.3', { 'content-length': '10' }));+ expect(refused.next).not.toHaveBeenCalled();+ expect(refused.state.code).toBe(503);++ const keyed = run(budget, from('198.51.100.3', { 'content-length': '400', 'x-api-key': 'k1' }));+ expect(keyed.next).toHaveBeenCalledTimes(1);+ expect(budget.currentBytes()).toBe(900);+ expect(budget.anonymousBytes()).toBe(500);+ });++ it('leaves room for other unkeyed senders while one client holds a full-size body (defaults)', () => {+ const budget = createInflightBodyBudget(100 * MB, { classify: () => undefined, bodyLimitBytes: 25 * MB });+ const big = run(budget, from('198.51.100.1', { 'content-length': String(25 * MB) }));+ expect(big.next).toHaveBeenCalledTimes(1);+ // The same client gets no more than one full-size body.+ expect(run(budget, from('198.51.100.1', { 'content-length': '2048' })).state.code).toBe(503);++ // A small delivery from another address (an ingress webhook, say) still gets in.+ const other = run(budget, from('203.0.113.9', { 'content-length': '2048' }));+ expect(other.next).toHaveBeenCalledTimes(1);+ expect(budget.anonymousBytes()).toBe(25 * MB + 2048);+ });++ it('charges an anonymous body no more than the body cap, so the parser can still answer 413', () => {+ const { budget } = tiered();+ const { next, res } = run(budget, from('198.51.100.1', { 'content-length': '900' }));+ expect(next).toHaveBeenCalledTimes(1);+ expect(budget.currentBytes()).toBe(250);+ res.emit('finish');+ expect(budget.currentBytes()).toBe(0);+ expect(budget.anonymousBytes()).toBe(0);+ });++ it('shares per key across addresses, and per address never across keys', () => {+ const { budget } = tiered();+ expect(run(budget, from('198.51.100.1', { 'content-length': '400', 'x-api-key': 'k1' })).next).toHaveBeenCalled();+ const sameKey = run(budget, from('198.51.100.2', { 'content-length': '200', 'x-api-key': 'k1' }));+ expect(sameKey.state.code).toBe(503);+ const otherKey = run(budget, from('198.51.100.1', { 'content-length': '400', 'x-api-key': 'k2' }));+ expect(otherKey.next).toHaveBeenCalledTimes(1);+ });++ it('aborts an anonymous body whose growth crosses the pool, releasing every counter once', () => {+ // Budget 8 MiB, body cap 2 MiB: pool 4 MiB, 2 MiB per anonymous IP, 1 MiB chunked placeholder.+ const budget = createInflightBodyBudget(8 * MB, { classify: () => undefined, bodyLimitBytes: 2 * MB });+ const slow = from('198.51.100.1', { 'transfer-encoding': 'chunked' });+ const slowRes = run(budget, slow).res;+ const second = run(budget, from('198.51.100.2', { 'content-length': String(2 * MB) }));+ const third = run(budget, from('198.51.100.3', { 'content-length': String(MB) }));+ expect(second.next).toHaveBeenCalledTimes(1);+ expect(third.next).toHaveBeenCalledTimes(1);+ expect(budget.anonymousBytes()).toBe(4 * MB);++ // Within its own share and the aggregate, but past the pool.+ (slow as unknown as { socket: { bytesRead: number } }).socket.bytesRead = 1.5 * MB;+ jest.advanceTimersByTime(5_000);+ expect((slow as unknown as { destroy: jest.Mock }).destroy).toHaveBeenCalledTimes(1);+ slow.emit('close');+ slowRes.emit('finish');+ expect(budget.anonymousBytes()).toBe(3 * MB);+ expect(budget.clientBytes(slow)).toBe(0);++ second.res.emit('finish');+ third.res.emit('finish');+ expect(budget.anonymousBytes()).toBe(0);+ expect(budget.currentBytes()).toBe(0);+ });++ it('does not classify a request without a body', () => {+ const { budget, classify } = tiered();+ run(budget, from('198.51.100.1', { 'x-api-key': 'k1' }));+ expect(classify).not.toHaveBeenCalled();+ });++ it('keeps one pool with per-IP shares when no classifier is given', () => {+ const budget = createInflightBodyBudget(1000, { bodyLimitBytes: 250 });+ expect(run(budget, from('198.51.100.1', { 'content-length': '400' })).next).toHaveBeenCalledTimes(1);+ expect(run(budget, from('198.51.100.2', { 'content-length': '400' })).next).toHaveBeenCalledTimes(1);+ expect(budget.anonymousBytes()).toBe(0);+ });+});+describe('stall reaper', () => {beforeEach(() => jest.useFakeTimers());afterEach(() => jest.useRealTimers());@@ -436,6 +544,41 @@ describe('stall reaper', () => {expect(budget.currentBytes()).toBeGreaterThan(MB);});+ it('paces an anonymous chunked body against the smaller amount it is charged', () => {+ const budget = createInflightBodyBudget(MB, { classify: () => undefined, bodyLimitBytes: 64 * 1024 });+ const req = makeReq({ 'transfer-encoding': 'chunked' });+ run(budget, req);+ expect(budget.anonymousBytes()).toBe(64 * 1024);++ // 64 KiB in 300 s is about 220 B/s; 1 KiB every 4 s keeps up, 1 MiB in 300 s would not.+ trickle([req], 1024, 60_000);+ expect(destroyMock(req)).not.toHaveBeenCalled();+ });++ it('does not let trickling unkeyed chunked bodies hold the unkeyed pool past the pace window', () => {+ const budget = createInflightBodyBudget(100 * MB, { classify: () => undefined, bodyLimitBytes: 25 * MB });+ const held: Request[] = [];+ for (const ip of ['198.51.100.1', '198.51.100.2']) {+ for (let i = 0; i < 25; i++) {+ const req = makeReq({ 'transfer-encoding': 'chunked' });+ socketOf(req).remoteAddress = ip;+ expect(run(budget, req).next).toHaveBeenCalledTimes(1);+ held.push(req);+ }+ }+ expect(budget.anonymousBytes()).toBe(50 * MB);+ const early = makeReq({ 'content-length': '2000' });+ socketOf(early).remoteAddress = '203.0.113.9';+ expect(run(budget, early).state.code).toBe(503);++ trickle(held, 1, 20_000);+ for (const r of held) expect(destroyMock(r)).toHaveBeenCalledTimes(1);+ expect(budget.anonymousBytes()).toBe(0);+ const late = makeReq({ 'content-length': '2000' });+ socketOf(late).remoteAddress = '203.0.113.9';+ expect(run(budget, late).next).toHaveBeenCalledTimes(1);+ });+it('keeps a client share held by trickling requests', () => {const budget = createInflightBodyBudget(1000); // share 0.5 -> cap 500const a = makeReq({ 'content-length': '250' });diff --git a/src/config/inflight-body-budget.ts b/src/config/inflight-body-budget.tsindex 218be8048..fee1c7c81 100644--- a/src/config/inflight-body-budget.ts+++ b/src/config/inflight-body-budget.ts@@ -124,6 +124,20 @@ export interface InflightBodyBudgetOptions {* a legitimate bulk uploader above the share gets 503 + Retry-After, not a hang.*/perClientShare?: number;+ /**+ * Turns on the anonymous tier. Returns a stable id (the stored key hash) when the request carries+ * an API key known to be active, undefined otherwise. Unrecognised requests share a pool of+ * ANONYMOUS_POOL_FRACTION of the budget (at least two body caps), so they can never take the rest+ * away from keyed traffic; a recognised key draws on the whole budget with its own per-key share, wherever it+ * connects from. Without this option every request draws on the whole budget with a per-IP share.+ */+ classify?: (req: Request, clientIp: string) => string | undefined;+ /**+ * The per-request body cap (BODY_SIZE_LIMIT) in bytes. An anonymous request is never charged+ * more: the parser refuses a longer declared body with 413 without buffering it. Default 25 MiB,+ * the parser's own default.+ */+ bodyLimitBytes?: number;/*** Node's request timeout (REQUEST_TIMEOUT_MS) in milliseconds. A body that falls behind the pace* needed to deliver its reservation within it (the declared size, or a chunked body's@@ -137,17 +151,34 @@ export interface InflightBodyBudget {middleware: (req: Request, res: Response, next: NextFunction) => void;/** Aggregate bytes currently attributed to in-flight request bodies (observability/tests). */currentBytes: () => number;- /** Bytes currently attributed to one client's in-flight bodies (observability/tests). */+ /** Bytes currently attributed to one client IP's in-flight bodies (observability/tests). */clientBytes: (req: Request) => number;+ /** Bytes currently charged to the anonymous tier (observability/tests). */+ anonymousBytes: () => number;}+/** Share of the budget open to requests without a recognised API key (see the classify option). */+const ANONYMOUS_POOL_FRACTION = 0.25;+export function createInflightBodyBudget(budgetBytes: number, options?: InflightBodyBudgetOptions): InflightBodyBudget {let inFlightBytes = 0;const retryAfter = String(options?.retryAfterSeconds ?? 1);const trustedProxies = options?.trustedProxies ?? [];const share = Math.min(1, Math.max(Number.EPSILON, options?.perClientShare ?? 0.5));const perClientCap = Math.max(1, Math.floor(budgetBytes * share));+ const classify = options?.classify;+ const bodyLimitBytes = options?.bodyLimitBytes ?? FALLBACK_LIMIT_BYTES;const requestTimeoutMs = options?.requestTimeoutMs ?? DEFAULT_REQUEST_TIMEOUT_MS;+ // The anonymous pool holds at least two full-size bodies, so with the default half share one+ // client never fills it and other unkeyed senders, ingress deliveries among them, keep room. A+ // budget below three body caps leaves keyed traffic less than a full-size body, which the docs+ // call out.+ const anonPool = Math.min(+ budgetBytes,+ Math.max(2 * bodyLimitBytes, Math.floor(budgetBytes * ANONYMOUS_POOL_FRACTION)),+ );+ const anonClientCap = Math.max(Math.min(bodyLimitBytes, anonPool), Math.floor(anonPool * share));+ let anonInFlight = 0;// Per-client in-flight bytes, keyed on the resolved client IP (an IPv6 client on its /64).// Entries are created lazily and deleted by the same exactly-once release that decrements the// aggregate, so the map cannot leak a client that finished. Only body-carrying requests are@@ -238,15 +269,32 @@ export function createInflightBodyBudget(budgetBytes: number, options?: Inflight// Admission control on the RESERVED size: a request that would push the aggregate OR ITS// CLIENT'S SHARE past the budget is refused before a single byte of its body is buffered.// The per-client share bounds what one source can hold.- const clientKey = limiterKeyForIp(resolveClientIp(req, trustedProxies));+ // A recognised key is shared per key (its entries are bounded by the key count, so they are+ // exempt from the map cap); anything else per client IP.+ const clientIp = resolveClientIp(req, trustedProxies);+ const keyId = classify?.(req, clientIp);+ const anonymous = classify !== undefined && keyId === undefined;+ const clientKey = keyId !== undefined ? `key:${keyId}` : limiterKeyForIp(clientIp);+ const shareCap = anonymous ? anonClientCap : perClientCap;+ const ceiling = anonymous ? bodyLimitBytes : Infinity;const clientBusy = clientInFlight.get(clientKey) ?? 0;- const mapAtCapacity = clientInFlight.size >= MAX_TRACKED_CLIENTS && !clientInFlight.has(clientKey);- if (inFlightBytes + reserved > budgetBytes || clientBusy + reserved > perClientCap || mapAtCapacity) {+ const mapAtCapacity =+ keyId === undefined && clientInFlight.size >= MAX_TRACKED_CLIENTS && !clientInFlight.has(clientKey);+ // The aggregate check uses the full declared size; the tier and share checks the charged size.+ const charge = Math.min(reserved, ceiling);+ if (+ inFlightBytes + reserved > budgetBytes ||+ clientBusy + charge > shareCap ||+ (anonymous && anonInFlight + charge > anonPool) ||+ mapAtCapacity+ ) {rejectBusy(req, res);return;}+ reserved = charge;inFlightBytes += reserved;+ if (anonymous) anonInFlight += reserved;clientInFlight.set(clientKey, clientBusy + reserved);let released = false;@@ -265,6 +313,7 @@ export function createInflightBodyBudget(budgetBytes: number, options?: Inflightreleased = true;disarmStallReaper();inFlightBytes -= reserved;+ if (anonymous) anonInFlight -= reserved;const busy = (clientInFlight.get(clientKey) ?? reserved) - reserved;if (busy > 0) clientInFlight.set(clientKey, busy);else clientInFlight.delete(clientKey);@@ -289,21 +338,23 @@ export function createInflightBodyBudget(budgetBytes: number, options?: Inflight// A declared body keeps its declared size until it is released. A chunked body is re-priced at// the bytes that have arrived, never below its opening placeholder until it is complete: budget// handed back mid-stream would be taken back, unchecked, by a body that then completes between- // polls. Growth that crosses the aggregate or the client share aborts the+ // polls. Growth that crosses the aggregate, the client share or the anonymous pool aborts the// request mid-stream, the same bound a declared length gets at admission. A complete body is// re-priced at its real size but never aborted: it is already buffered. reserved is updated// BEFORE release() so the exactly-once decrement subtracts the reconciled size.const reconcile = (complete: boolean): void => {if (released || declared !== undefined) return;const floor = complete ? 0 : undeclaredReservation;- const actual = Math.max(floor, socket.bytesRead - startBytes);+ const actual = Math.min(Math.max(floor, socket.bytesRead - startBytes), ceiling);const delta = actual - reserved;if (delta === 0) return;inFlightBytes += delta;+ if (anonymous) anonInFlight += delta;reserved = actual;const busy = (clientInFlight.get(clientKey) ?? 0) + delta;clientInFlight.set(clientKey, busy);- if (!complete && delta > 0 && (busy > perClientCap || inFlightBytes > budgetBytes)) {+ const overTier = anonymous && anonInFlight > anonPool;+ if (!complete && delta > 0 && (busy > shareCap || inFlightBytes > budgetBytes || overTier)) {release();req.destroy();}@@ -364,6 +415,7 @@ export function createInflightBodyBudget(budgetBytes: number, options?: Inflightmiddleware,currentBytes: () => inFlightBytes,clientBytes: req => clientInFlight.get(limiterKeyForIp(resolveClientIp(req, trustedProxies))) ?? 0,+ anonymousBytes: () => anonInFlight,};}diff --git a/src/configure-app.ts b/src/configure-app.tsindex e66642116..4b030f0b1 100644--- a/src/configure-app.ts+++ b/src/configure-app.ts@@ -1,11 +1,17 @@import { INestApplication } from '@nestjs/common';+import { ModuleRef } from '@nestjs/core';import helmet from 'helmet';import { Request, Response, NextFunction, json, urlencoded } from 'express';import { randomBytes } from 'crypto';import { existsSync, readFileSync } from 'fs';import { extname, join } from 'path';import { DASHBOARD_DIST, dashboardServingEnabled, dashboardBuildPresent } from './app.module';-import { createInflightBodyBudget, resolveInflightBodyBudgetBytes } from './config/inflight-body-budget';+import {+ createInflightBodyBudget,+ parseBodyLimitBytes,+ resolveInflightBodyBudgetBytes,+} from './config/inflight-body-budget';+import { ActiveKeyIndex } from './modules/auth/active-key-index';import { requestContextMiddleware } from './common/middleware/request-context.middleware';import { injectDashboardCspNonce } from './config/dashboard-csp';import { resolveCorsPolicy, isUpgradeInsecureRequestsEnabled, resolveBodyLimit } from './config/bootstrap-security';@@ -60,19 +66,30 @@ export function configureApp(app: INestApplication, options: ConfigureAppOptionsprocess.env.INFLIGHT_BODY_BUDGET_BYTES,process.env.BODY_SIZE_LIMIT,);+ // Cap request body size (DoS hardening). Media sends carry base64 in the JSON body,+ // so the default is generous; tune with BODY_SIZE_LIMIT.+ const bodyLimit = resolveBodyLimit(process.env.BODY_SIZE_LIMIT);+ // Requests without a recognised API key share a pool of a quarter of the budget, never less than+ // two BODY_SIZE_LIMIT bodies (half the default budget). With no AuthModule in the app (some test+ // modules) every request is unrecognised, which is the stricter side.+ // Looked up through ModuleRef: a failed app.get() aborts the process instead of throwing.+ let keyIndex: ActiveKeyIndex | undefined;+ try {+ keyIndex = app.get(ModuleRef).get(ActiveKeyIndex, { strict: false });+ } catch {+ keyIndex = undefined;+ }app.use(createInflightBodyBudget(inflightBudgetBytes, {trustedProxies: (process.env.TRUSTED_PROXIES || '').split(',').map(p => p.trim()).filter(Boolean),+ classify: (req, clientIp) => keyIndex?.recognise(req.headers, clientIp),+ bodyLimitBytes: parseBodyLimitBytes(bodyLimit),requestTimeoutMs: resolveRequestTimeoutMs(process.env.REQUEST_TIMEOUT_MS),}).middleware,);-- // Cap request body size (DoS hardening). Media sends carry base64 in the JSON body,- // so the default is generous; tune with BODY_SIZE_LIMIT.- const bodyLimit = resolveBodyLimit(process.env.BODY_SIZE_LIMIT);// The `verify` callback stashes the EXACT bytes json() received on req.rawBody, byte-identical to// what a provider signed, so the @Public ingress controller can HMAC-verify over the raw body// (JSON.stringify(req.body) is NOT byte-identical). Cheap for every route; non-ingress routes ignore it.diff --git a/src/modules/auth/active-key-index.spec.ts b/src/modules/auth/active-key-index.spec.tsnew file mode 100644index 000000000..6bbfa3f58--- /dev/null+++ b/src/modules/auth/active-key-index.spec.ts@@ -0,0 +1,115 @@+import { createHash, createHmac } from 'crypto';+import { Repository } from 'typeorm';+import { ActiveKeyIndex } from './active-key-index';+import { ApiKey } from './entities/api-key.entity';++const sha = (key: string) => createHash('sha256').update(key).digest('hex');++type Row = Pick<ApiKey, 'keyHash' | 'allowedIps' | 'expiresAt'>;++const indexOver = (rows: Row[] | (() => Promise<Row[]>)) => {+ const find = jest.fn(typeof rows === 'function' ? rows : () => Promise.resolve(rows));+ const index = new ActiveKeyIndex({ find } as unknown as Repository<ApiKey>);+ return { index, find };+};++const row = (key: string, extra: Partial<Row> = {}): Row => ({+ keyHash: sha(key),+ allowedIps: null,+ expiresAt: null,+ ...extra,+});++describe('ActiveKeyIndex', () => {+ const pepper = process.env.API_KEY_PEPPER;+ afterEach(() => {+ if (pepper === undefined) delete process.env.API_KEY_PEPPER;+ else process.env.API_KEY_PEPPER = pepper;+ jest.useRealTimers();+ });++ it('recognises an active key from X-API-Key or a Bearer token, trimmed', async () => {+ const { index, find } = indexOver([row('k1')]);+ await index.onApplicationBootstrap();+ index.onModuleDestroy();++ expect(find).toHaveBeenCalledWith(expect.objectContaining({ where: { isActive: true } }));+ expect(index.recognise({ 'x-api-key': ' k1 ' }, '192.0.2.1')).toBe(sha('k1'));+ expect(index.recognise({ authorization: 'Bearer k1' }, '192.0.2.1')).toBe(sha('k1'));+ expect(index.recognise({ 'x-api-key': 'nope' }, '192.0.2.1')).toBeUndefined();+ expect(index.recognise({}, '192.0.2.1')).toBeUndefined();+ });++ it('reads X-API-Key before the Bearer token, as the guard does', async () => {+ const { index } = indexOver([row('k1')]);+ await index.onApplicationBootstrap();+ index.onModuleDestroy();+ expect(index.recognise({ 'x-api-key': 'nope', authorization: 'Bearer k1' }, '192.0.2.1')).toBeUndefined();+ });++ it('hashes with the pepper when one is set', async () => {+ process.env.API_KEY_PEPPER = 'pepper';+ const peppered = createHmac('sha256', 'pepper').update('k1').digest('hex');+ const { index } = indexOver([{ keyHash: peppered, allowedIps: null, expiresAt: null }]);+ await index.onApplicationBootstrap();+ index.onModuleDestroy();+ expect(index.recognise({ 'x-api-key': 'k1' }, '192.0.2.1')).toBe(peppered);+ });++ it('ignores an expired key and one used from outside its allowed addresses', async () => {+ const { index } = indexOver([+ row('old', { expiresAt: new Date(Date.now() - 1000) }),+ row('fenced', { allowedIps: ['10.0.0.0/8'] }),+ ]);+ await index.onApplicationBootstrap();+ index.onModuleDestroy();+ expect(index.recognise({ 'x-api-key': 'old' }, '192.0.2.1')).toBeUndefined();+ expect(index.recognise({ 'x-api-key': 'fenced' }, '192.0.2.1')).toBeUndefined();+ expect(index.recognise({ 'x-api-key': 'fenced' }, '10.1.2.3')).toBe(sha('fenced'));+ });++ it('keeps the previous view when a refresh fails', async () => {+ let fail = false;+ const { index } = indexOver(() => (fail ? Promise.reject(new Error('db down')) : Promise.resolve([row('k1')])));+ await index.onApplicationBootstrap();+ fail = true;+ index.refreshSoon();+ await (index as unknown as { loading: Promise<void> }).loading;+ index.onModuleDestroy();+ expect(index.recognise({ 'x-api-key': 'k1' }, '192.0.2.1')).toBe(sha('k1'));+ });++ it('runs one more load when asked to refresh during a load, and none after destroy', async () => {+ let rows = [row('k1')];+ const { index, find } = indexOver(() => Promise.resolve(rows));+ await index.onApplicationBootstrap();+ expect(find).toHaveBeenCalledTimes(1);++ rows = [row('k2')];+ index.refreshSoon();+ index.refreshSoon();+ index.refreshSoon();+ await (index as unknown as { loading: Promise<void> }).loading;+ expect(find).toHaveBeenCalledTimes(3);+ expect(index.recognise({ 'x-api-key': 'k2' }, '192.0.2.1')).toBe(sha('k2'));+ expect(index.recognise({ 'x-api-key': 'k1' }, '192.0.2.1')).toBeUndefined();++ index.onModuleDestroy();+ index.refreshSoon();+ expect(find).toHaveBeenCalledTimes(3);+ });++ it('refreshes on an interval until destroyed', async () => {+ jest.useFakeTimers();+ const { index, find } = indexOver([row('k1')]);+ await index.onApplicationBootstrap();+ expect(find).toHaveBeenCalledTimes(1);++ await jest.advanceTimersByTimeAsync(30_000);+ expect(find).toHaveBeenCalledTimes(2);++ index.onModuleDestroy();+ await jest.advanceTimersByTimeAsync(60_000);+ expect(find).toHaveBeenCalledTimes(2);+ });+});diff --git a/src/modules/auth/active-key-index.ts b/src/modules/auth/active-key-index.tsnew file mode 100644index 000000000..842c7d65e--- /dev/null+++ b/src/modules/auth/active-key-index.ts@@ -0,0 +1,107 @@+import { Injectable, OnApplicationBootstrap, OnModuleDestroy } from '@nestjs/common';+import { InjectRepository } from '@nestjs/typeorm';+import { Repository } from 'typeorm';+import type { IncomingHttpHeaders } from 'http';+import { ApiKey } from './entities/api-key.entity';+import { hashApiKey } from './api-key-hash';+import { bearerToken } from '../../common/security/bearer-token';+import { ipMatches } from '../../common/utils/ip';+import { createLogger } from '../../common/services/logger.service';++const REFRESH_INTERVAL_MS = 30_000;++interface IndexedKey {+ allowedIps: string[] | null;+ expiresAt: Date | null;+}++/**+ * In-memory view of the active API keys, used BEFORE a request body is read to decide which tier+ * of the in-flight body budget the request draws on. It is advisory only: it never authorises+ * anything, the API-key guard still validates every request against the database. A key this+ * node creates, changes or revokes is picked up at once; a direct write to this node's main+ * database within REFRESH_INTERVAL_MS. Keys are per node, so a change made on another node never+ * reaches this index.+ */+@Injectable()+export class ActiveKeyIndex implements OnApplicationBootstrap, OnModuleDestroy {+ private readonly logger = createLogger('ActiveKeyIndex');+ private keys = new Map<string, IndexedKey>();+ private timer: ReturnType<typeof setInterval> | undefined;+ private loading: Promise<void> | undefined;+ private reloadAgain = false;+ private destroyed = false;++ constructor(+ @InjectRepository(ApiKey, 'main')+ private readonly apiKeyRepository: Repository<ApiKey>,+ ) {}++ async onApplicationBootstrap(): Promise<void> {+ this.refreshSoon();+ await this.loading;+ this.timer = setInterval(() => this.refreshSoon(), REFRESH_INTERVAL_MS);+ this.timer.unref();+ }++ onModuleDestroy(): void {+ this.destroyed = true;+ clearInterval(this.timer);+ this.timer = undefined;+ }++ /**+ * Reload without blocking the caller. A call that lands while a load is running queues exactly+ * one more load, so a write that committed after the running SELECT started is never missed.+ */+ refreshSoon(): void {+ if (this.destroyed) return;+ if (this.loading) {+ this.reloadAgain = true;+ return;+ }+ this.loading = (async () => {+ do {+ this.reloadAgain = false;+ await this.load();+ } while (this.reloadAgain && !this.destroyed);+ })().finally(() => {+ this.loading = undefined;+ });+ }++ /**+ * The stored hash of the request's key when it names an active, unexpired key that may be used+ * from clientIp; undefined otherwise. Reads the key the way the guard does: X-API-Key first, then+ * a Bearer token, trimmed.+ */+ recognise(headers: IncomingHttpHeaders, clientIp: string): string | undefined {+ const header = headers['x-api-key'];+ const raw = (typeof header === 'string' && header ? header : bearerToken(headers.authorization))?.trim();+ if (!raw) return undefined;+ const keyHash = hashApiKey(raw, process.env.API_KEY_PEPPER);+ const key = this.keys.get(keyHash);+ if (!key) return undefined;+ // Expiry is checked here rather than in the query: SQLite stores dates as strings, and a key+ // that expires between two refreshes must stop counting at once.+ if (key.expiresAt && key.expiresAt < new Date()) return undefined;+ if (key.allowedIps?.length && !key.allowedIps.some(entry => ipMatches(clientIp, entry))) return undefined;+ return keyHash;+ }++ private async load(): Promise<void> {+ try {+ const rows = await this.apiKeyRepository.find({+ select: { keyHash: true, allowedIps: true, expiresAt: true },+ where: { isActive: true },+ });+ this.keys = new Map(rows.map(row => [row.keyHash, { allowedIps: row.allowedIps, expiresAt: row.expiresAt }]));+ } catch (error) {+ // Keep the last good view: a transient database error must not move every keyed request into+ // the anonymous tier.+ this.logger.warn('Could not refresh the active API key index; keeping the previous one', {+ error: error instanceof Error ? error.message : String(error),+ });+ }+ }+}diff --git a/src/modules/auth/auth.module.ts b/src/modules/auth/auth.module.tsindex 2eb4a80c8..e26181ab9 100644--- a/src/modules/auth/auth.module.ts+++ b/src/modules/auth/auth.module.ts@@ -5,6 +5,7 @@ import { ApiKey } from './entities/api-key.entity';import { AuthService } from './auth.service';import { ApiKeyUsageTracker } from './api-key-usage-tracker.service';import { ChatScopeService } from './chat-scope.service';+import { ActiveKeyIndex } from './active-key-index';import { AuthController } from './auth.controller';import { AuthValidateController } from './auth-validate.controller';import { ApiKeyGuard } from './guards/api-key.guard';@@ -18,6 +19,7 @@ import { ProxyAwareThrottlerGuard } from '../../common/security/proxy-aware-throAuthService,ApiKeyUsageTracker,ChatScopeService,+ ActiveKeyIndex,{provide: APP_GUARD,useClass: ProxyAwareThrottlerGuard,@@ -27,6 +29,6 @@ import { ProxyAwareThrottlerGuard } from '../../common/security/proxy-aware-throuseClass: ApiKeyGuard,},],- exports: [AuthService, ChatScopeService],+ exports: [AuthService, ChatScopeService, ActiveKeyIndex],})export class AuthModule {}diff --git a/src/modules/auth/auth.service.spec.ts b/src/modules/auth/auth.service.spec.tsindex 19b0cc6a3..2dd32145f 100644--- a/src/modules/auth/auth.service.spec.ts+++ b/src/modules/auth/auth.service.spec.ts@@ -11,6 +11,7 @@ import * as fs from 'fs';import { AuthService, resolveSeedApiKey, bannerKeyLine, UnresolvedApiKeyException } from './auth.service';import { ApiKeyUsageTracker } from './api-key-usage-tracker.service';import { getRequestActor, runWithRequestId } from '../../common/services/request-context';+import { ActiveKeyIndex } from './active-key-index';import { ApiKey, ApiKeyRole } from './entities/api-key.entity';// Helpers@@ -112,8 +113,10 @@ describe('AuthService', () => {let committedWrites: Array<{ mode: 'update' | 'delete'; patch?: Record<string, unknown>; guarded: boolean }>;/** The raw AND-fragment the last-admin guard binds, so the SQL predicate itself is asserted. */let lastAdminFragments: string[];+ let keyIndex: { refreshSoon: jest.Mock };beforeEach(async () => {+ keyIndex = { refreshSoon: jest.fn() };repository = {count: jest.fn(),find: jest.fn(),@@ -135,6 +138,7 @@ describe('AuthService', () => {provide: getRepositoryToken(ApiKey, 'main'),useValue: repository,},+ { provide: ActiveKeyIndex, useValue: keyIndex },],}).compile();@@ -416,6 +420,22 @@ describe('AuthService', () => {});});+ describe('active key index', () => {+ it('is refreshed after every key write', async () => {+ (repository.create as jest.Mock).mockImplementation((dto: Partial<ApiKey>) => ({ ...dto, id: 'uuid-new' }));+ await service.createApiKey({ name: 'new' });+ expect(keyIndex.refreshSoon).toHaveBeenCalledTimes(1);++ setupKeys([createMockApiKey({ id: 'uuid-1' }), createMockApiKey({ id: 'uuid-2' })]);+ await service.update('uuid-1', { name: 'renamed' });+ expect(keyIndex.refreshSoon).toHaveBeenCalledTimes(2);+ await service.revoke('uuid-1');+ expect(keyIndex.refreshSoon).toHaveBeenCalledTimes(3);+ await service.delete('uuid-2');+ expect(keyIndex.refreshSoon).toHaveBeenCalledTimes(4);+ });+ });+describe('revoke', () => {it('should set isActive to false', async () => {setupKeys([createMockApiKey({ id: 'uuid-1', isActive: true })]);diff --git a/src/modules/auth/auth.service.ts b/src/modules/auth/auth.service.tsindex 901419b7f..2692cea00 100644--- a/src/modules/auth/auth.service.ts+++ b/src/modules/auth/auth.service.ts@@ -3,6 +3,7 @@ import {ForbiddenException,Injectable,NotFoundException,+ Optional,UnauthorizedException,OnModuleInit,OnModuleDestroy,@@ -19,6 +20,7 @@ import { createLogger } from '../../common/services/logger.service';import { setRequestActor } from '../../common/services/request-context';import { readBootstrapKey, removeBootstrapKey, writeBootstrapKey } from './bootstrap-key-file';import { ApiKeyUsageTracker } from './api-key-usage-tracker.service';+import { ActiveKeyIndex } from './active-key-index';import { apiKeyAuthorizationFingerprint, normalizeScopeList } from './api-key-authorization';import { normalizeChatAllowList } from '../../common/security/chat-scope';import { EventsGateway, type ApiKeyEvictionReason } from '../events/events.gateway';@@ -78,6 +80,7 @@ export class AuthService implements OnModuleInit, OnModuleDestroy {private readonly apiKeyRepository: Repository<ApiKey>,private readonly usageTracker: ApiKeyUsageTracker,private readonly moduleRef: ModuleRef,+ @Optional() private readonly keyIndex?: ActiveKeyIndex,) {}async onModuleInit(): Promise<void> {@@ -209,6 +212,7 @@ export class AuthService implements OnModuleInit, OnModuleDestroy {});const saved = await this.apiKeyRepository.save(apiKey);+ this.keyIndex?.refreshSoon();this.logger.log(`API key created: ${saved.name}`, {keyId: saved.id,role: saved.role,@@ -281,6 +285,7 @@ export class AuthService implements OnModuleInit, OnModuleDestroy {// The row's post-write state, for the eviction comparison below.saved = await this.findOne(id);}+ this.keyIndex?.refreshSoon();// One fingerprint definition, two callers: this immediate eviction and the gateway's periodic// re-validation sweep. Sharing it keeps the two from disagreeing about what an authorization@@ -307,6 +312,7 @@ export class AuthService implements OnModuleInit, OnModuleDestroy {this.usageTracker.forget(id);this.removeBootstrapKeyFileIfMatching(apiKey);this.evictActiveSockets(id, 'deleted');+ this.keyIndex?.refreshSoon();this.logger.log(`API key deleted: ${apiKey.name}`, {keyId: id,action: 'api_key_deleted',@@ -335,6 +341,7 @@ export class AuthService implements OnModuleInit, OnModuleDestroy {// Kick any WebSocket connections already authenticated with this key: without this, a revoked// key keeps receiving events on already-subscribed sockets until they happen to disconnect.this.evictActiveSockets(id, 'revoked');+ this.keyIndex?.refreshSoon();return saved;}diff --git a/test/configure-app.e2e-spec.ts b/test/configure-app.e2e-spec.tsindex fbdeb1597..227d6f38a 100644--- a/test/configure-app.e2e-spec.ts+++ b/test/configure-app.e2e-spec.ts@@ -3,11 +3,13 @@ import { NestFactory } from '@nestjs/core';import request from 'supertest';import { App } from 'supertest/types';import { mkdtempSync, mkdirSync, writeFileSync } from 'fs';+import { request as httpRequest } from 'http';import { tmpdir } from 'os';import { join } from 'path';import { configureApp } from '../src/configure-app';import { applyGlobalValidation } from '../src/config/app-validation';import { DASHBOARD_CSP_NONCE_PLACEHOLDER } from '../src/config/dashboard-csp';+import { ActiveKeyIndex } from '../src/modules/auth/active-key-index';/*** The production HTTP surface, run for real. Every other e2e builds a bare Nest app, so the stack@@ -161,3 +163,63 @@ describe('production HTTP surface (configureApp)', () => {expect(res.status).toBe(415);});});++describe('in-flight body budget tiers (configureApp)', () => {+ let app: INestApplication<App>;+ const previousBodyLimit = process.env.BODY_SIZE_LIMIT;++ // Stands in for the real index: only 'known-key' is an active key.+ const keyIndex = {+ recognise: (headers: { 'x-api-key'?: unknown }) => (headers['x-api-key'] === 'known-key' ? 'h' : undefined),+ };++ @Module({ controllers: [EchoController], providers: [{ provide: ActiveKeyIndex, useValue: keyIndex }] })+ class TieredSurfaceModule {}++ beforeAll(async () => {+ // 1mb cap: a 4mb budget whose anonymous pool is 2mb, 1mb per client address.+ process.env.BODY_SIZE_LIMIT = '1mb';+ app = await NestFactory.create<INestApplication<App>>(TieredSurfaceModule, { bodyParser: false, logger: false });+ configureApp(app);+ applyGlobalValidation(app);+ await app.listen(0, '127.0.0.1');+ });++ afterAll(async () => {+ await app?.close();+ if (previousBodyLimit === undefined) delete process.env.BODY_SIZE_LIMIT;+ else process.env.BODY_SIZE_LIMIT = previousBodyLimit;+ });++ it('keeps room for a recognised key while an unrecognised client holds its whole share', async () => {+ const { port } = new URL(await app.getUrl());+ // Declares a full-size body, this address's whole share, and never finishes sending it.+ const held = httpRequest({+ host: '127.0.0.1',+ port,+ path: '/api/echo',+ method: 'POST',+ headers: { 'Content-Type': 'application/json', 'Content-Length': String(1024 * 1024) },+ });+ held.on('error', () => undefined);+ held.write('{"a":"');+ await new Promise(resolve => setTimeout(resolve, 200));++ try {+ const anonymous = await request(app.getHttpServer())+ .post('/api/echo')+ .set('Content-Type', 'application/json')+ .send({ small: true });+ expect(anonymous.status).toBe(503);++ const keyed = await request(app.getHttpServer())+ .post('/api/echo')+ .set('Content-Type', 'application/json')+ .set('X-API-Key', 'known-key')+ .send({ small: true });+ expect(keyed.status).toBe(201);+ } finally {+ held.destroy();+ }+ });+});