Skip to content

Free, Open Source, Self-Hosted WhatsApp API Gateway

fix(http): keep most of the body budget for requests with an API key

Yudhi Armyndharis2c62ed790dc8
diff --git a/.env.example b/.env.example
index 8fb23aef7..641f7fa49 100644
--- a/.env.example
+++ b/.env.example
@@ -905,9 +905,15 @@ PLUGINS_DIR=./data/plugins # Plugin directory (default: ./data/plugins)
# Aggregate cap on request-body bytes buffered across ALL connections. Once exceeded, new requests
# get 503 + Retry-After without their body being read (protects against slow-body memory pinning).
# Keep it >= BODY_SIZE_LIMIT. Default: 4 x BODY_SIZE_LIMIT (100 MiB with the default 25mb cap).
-# One client IP may hold at most HALF of this budget in flight and gets the same 503 + Retry-After
-# past that. Behind a reverse proxy set TRUSTED_PROXIES, or every caller resolves to the proxy
-# address and shares that one half. A request holds its declared size (a chunked body 1 MiB, or
-# what has arrived past that) until it finishes; a body that sends nothing for 15 s, or that falls
-# behind the pace needed to deliver what it holds within REQUEST_TIMEOUT_MS, is dropped.
+# Below 2 x BODY_SIZE_LIMIT one full-size unkeyed body can leave no room for ingress deliveries.
+# Requests without an active API key (ingress webhooks, unauthenticated callers) share a quarter of
+# it (at least two BODY_SIZE_LIMIT); the rest is kept for keyed requests. Each API key may hold at
+# most HALF of the budget, and each unkeyed client IP half of the unkeyed pool (or one
+# BODY_SIZE_LIMIT, if larger); past that the same 503 + Retry-After. Behind a reverse proxy set
+# TRUSTED_PROXIES. Without it every unkeyed caller, ingress webhooks included, resolves to the
+# proxy address and they all draw on one client share: at the defaults a single full-size unkeyed
+# upload then refuses ingress deliveries with 503 until it finishes. A request holds its declared
+# size (a chunked body 1 MiB, or what has arrived past that) until it finishes; a body that sends
+# nothing for 15 s, or that falls behind the pace needed to deliver what it holds within
+# REQUEST_TIMEOUT_MS, is dropped.
# INFLIGHT_BODY_BUDGET_BYTES=104857600
diff --git a/docs/04-security-design.md b/docs/04-security-design.md
index 06541638f..09e1b78ff 100644
--- a/docs/04-security-design.md
+++ b/docs/04-security-design.md
@@ -281,7 +281,7 @@ Each window is counted **per route handler per client IP** (the IP resolved thro
TTL values are in milliseconds. The `/api/metrics` and `/api/health*` routes are exempt (`@SkipThrottle`); `/api/health` still looks up at most 30 failed key presentations per client per minute, and past that answers without `version` and without a key lookup. To enforce tighter per-route limits, lower the global windows or add a limiter at your reverse proxy.
-An IPv6 client is keyed on its /64, so rotating addresses inside one allocation does not mint fresh buckets. The same key is used by every other per-client limit: the MCP and Bull Board pre-auth throttles, the WebSocket limits below, the per-client share of the in-flight body budget, the health route's key-lookup budget and auth-failure audit bound, and the REST, queue-dashboard and MCP auth-failure audit bound. `allowedIps` matching and audit rows still see the full address.
+An IPv6 client is keyed on its /64, so rotating addresses inside one allocation does not mint fresh buckets. The same key is used by every other per-client limit: the MCP and Bull Board pre-auth throttles, the WebSocket limits below, the per-client share of the in-flight body budget for requests without an API key, the health route's key-lookup budget and auth-failure audit bound, and the REST, queue-dashboard and MCP auth-failure audit bound. `allowedIps` matching and audit rows still see the full address.
### Response on limit
@@ -303,8 +303,10 @@ The API exposes the rate-limit headers via CORS (`exposedHeaders`) so browser cl
The windows above run at the routing layer, after a request body has been buffered, so they cannot bound memory held by slow uploads. A pre-routing middleware does: it caps the request-body bytes in flight across all connections at `INFLIGHT_BODY_BUDGET_BYTES` (default 4 x `BODY_SIZE_LIMIT`) and answers a request that would cross it with `503` and `Retry-After`, before reading its body.
-- One client (IPv6 on its /64) may hold at most half of the budget.
-- A request holds its declared `Content-Length` until it finishes. After a 15-second grace it must keep up the pace that lands the whole body within `REQUEST_TIMEOUT_MS` (by default 25 MiB in 300 seconds, about 85 KiB/s); a request that falls behind is dropped and its reservation released. A chunked body holds a 1 MiB placeholder, or the bytes that have arrived once they pass it, and must deliver the placeholder at the same pace (about 3.4 KiB/s by default). If that growth crosses the budget or its client's half, that request is aborted.
+- Requests without an active API key (public ingress webhooks, unauthenticated callers) share a pool of a quarter of the budget, never less than twice `BODY_SIZE_LIMIT` (so half of the default budget), and are charged at most `BODY_SIZE_LIMIT` each. The rest is kept for requests that carry an active key. With `INFLIGHT_BODY_BUDGET_BYTES` below three times `BODY_SIZE_LIMIT` that reserve is smaller than one full-size body, and at twice `BODY_SIZE_LIMIT` or less there is none.
+- A request with an active key may hold at most half of the budget per key, wherever it connects from. An unkeyed client (IPv6 on its /64) may hold at most half of the unkeyed pool, or one `BODY_SIZE_LIMIT` if that is larger, so one full-size body from one address leaves room for other unkeyed senders once `INFLIGHT_BODY_BUDGET_BYTES` is at least twice `BODY_SIZE_LIMIT`; below that the room left is the budget minus one `BODY_SIZE_LIMIT`, and at a budget equal to `BODY_SIZE_LIMIT` a single full-size unkeyed body refuses every other body-carrying request, ingress deliveries included, with `503` until it finishes. Behind a reverse proxy without `TRUSTED_PROXIES`, every unkeyed request, ingress included, keys on the proxy address and draws on that one share, so at the defaults a single full-size unkeyed upload refuses ingress deliveries with `503` until it finishes.
+- Keys are recognised from an in-memory index of active keys, refreshed at once when this node changes a key and every 30 seconds for a direct write to this node's main database. Keys are per node, so a change made on another node never reaches it. The index only picks the tier; the API-key guard still authorises every request.
+- A request holds its declared `Content-Length` until it finishes. After a 15-second grace it must keep up the pace that lands the whole body within `REQUEST_TIMEOUT_MS` (by default 25 MiB in 300 seconds, about 85 KiB/s); a request that falls behind is dropped and its reservation released. A chunked body holds a 1 MiB placeholder, or the bytes that have arrived once they pass it, and must deliver the placeholder at the same pace (about 3.4 KiB/s by default). If that growth crosses the budget, its client's share, or (without a key) the unkeyed pool, that request is aborted.
- A body that sends nothing for 15 seconds is dropped and its reservation released.
- Requests without a body are never refused for lack of budget.
diff --git a/docs/06-api-specification.md b/docs/06-api-specification.md
index 7dc68a60e..93131d0ca 100644
--- a/docs/06-api-specification.md
+++ b/docs/06-api-specification.md
@@ -109,7 +109,7 @@ Validation failures (`statusCode: 400`) return `message` as an **array** of fiel
| `500` | Internal Server Error | Send failed at the WhatsApp engine or an unexpected server error; when WhatsApp Web itself threw during a whatsapp-web.js send or status post, the body carries `code: 'ENGINE_PAGE_ERROR'`, `pageError` (`name`, `message`) and, when the page could read it, the WhatsApp Web `build` |
| `501` | Not Implemented | The operation is not supported by the active engine (see the capability matrix, docs/29) |
| `502` | Bad Gateway | An engine transport failure (e.g. a dead Baileys socket; retryable), an upstream component returned something unusable (not retryable; each route section carries the exact wording), or, on a multi-node deployment, forwarding a session-scoped request to its owner node failed after the request may already have been sent, so a non-idempotent call must not be replayed blindly (see docs/13) |
-| `503` | Service Unavailable | A dependency or the session is not ready (boot draining, a datastore down, the engine reconnecting, a media `url` fetch through the session's egress proxy failing before any response, whether the proxy or the target is at fault); retryable. On a multi-node deployment a forwarded request answers `503` only when the owner node was never reached, so the request was not carried out (see docs/13) |
+| `503` | Service Unavailable | A dependency or the session is not ready (boot draining, a datastore down, the engine reconnecting, a media `url` fetch through the session's egress proxy failing before any response, whether the proxy or the target is at fault); retryable. On a multi-node deployment a forwarded request answers `503` only when the owner node was never reached, so the request was not carried out (see docs/13). Also answered, with `Retry-After`, when too many request bodies are in flight at once |
| `504` | Gateway Timeout | An upstream the gateway waits on did not answer in time: on a multi-node deployment, the owner node did not answer a forwarded session-scoped request within `SESSION_PROXY_TIMEOUT_MS`; on `POST /sessions/{sessionId}/start` (see §6.4.1), the engine did not finish starting within its timeout, either because initialization never finished at all (WhatsApp Web, the network or the session `proxyUrl` unreachable in a way that hangs the connection, or a browser stalled mid-startup, for example under a container memory or resource limit) or because the engine's own auth poll expired after the page loaded (typically an unreachable `proxyUrl`), and the message names which. On the forwarding path the request may still have been carried out on the owner node, so a non-idempotent call must not be replayed blindly (see docs/13) |
### Timestamp Conventions
diff --git a/src/config/inflight-body-budget.spec.ts b/src/config/inflight-body-budget.spec.ts
index b83511a89..25f6a54be 100644
--- a/src/config/inflight-body-budget.spec.ts
+++ b/src/config/inflight-body-budget.spec.ts
@@ -307,6 +307,114 @@ describe('createInflightBodyBudget middleware', () => {
});
});
+describe('anonymous tier', () => {
+ beforeEach(() => jest.useFakeTimers());
+ afterEach(() => jest.useRealTimers());
+
+ // Budget 1000 with a 250-byte body cap: the anonymous pool is max(2 x 250, 25% of 1000) = 500,
+ // each anonymous IP's share is 250 and each key's share is half the budget, 500.
+ const tiered = () => {
+ const classify = jest.fn((req: Request) => {
+ const key = req.headers['x-api-key'];
+ return typeof key === 'string' ? `hash-${key}` : undefined;
+ });
+ return { budget: createInflightBodyBudget(1000, { classify, bodyLimitBytes: 250 }), classify };
+ };
+ const from = (ip: string, headers: Record<string, string>): Request & EventEmitter => {
+ const req = makeReq(headers);
+ (req as unknown as { socket: { remoteAddress: string } }).socket.remoteAddress = ip;
+ return req;
+ };
+
+ it('keeps unrecognised requests inside their pool while a keyed request still gets in', () => {
+ const { budget } = tiered();
+ expect(run(budget, from('198.51.100.1', { 'content-length': '250' })).next).toHaveBeenCalledTimes(1);
+ expect(run(budget, from('198.51.100.2', { 'content-length': '250' })).next).toHaveBeenCalledTimes(1);
+ expect(budget.anonymousBytes()).toBe(500);
+
+ // The aggregate has 500 free, but none of it is open to a third anonymous client.
+ const refused = run(budget, from('198.51.100.3', { 'content-length': '10' }));
+ expect(refused.next).not.toHaveBeenCalled();
+ expect(refused.state.code).toBe(503);
+
+ const keyed = run(budget, from('198.51.100.3', { 'content-length': '400', 'x-api-key': 'k1' }));
+ expect(keyed.next).toHaveBeenCalledTimes(1);
+ expect(budget.currentBytes()).toBe(900);
+ expect(budget.anonymousBytes()).toBe(500);
+ });
+
+ it('leaves room for other unkeyed senders while one client holds a full-size body (defaults)', () => {
+ const budget = createInflightBodyBudget(100 * MB, { classify: () => undefined, bodyLimitBytes: 25 * MB });
+ const big = run(budget, from('198.51.100.1', { 'content-length': String(25 * MB) }));
+ expect(big.next).toHaveBeenCalledTimes(1);
+ // The same client gets no more than one full-size body.
+ expect(run(budget, from('198.51.100.1', { 'content-length': '2048' })).state.code).toBe(503);
+
+ // A small delivery from another address (an ingress webhook, say) still gets in.
+ const other = run(budget, from('203.0.113.9', { 'content-length': '2048' }));
+ expect(other.next).toHaveBeenCalledTimes(1);
+ expect(budget.anonymousBytes()).toBe(25 * MB + 2048);
+ });
+
+ it('charges an anonymous body no more than the body cap, so the parser can still answer 413', () => {
+ const { budget } = tiered();
+ const { next, res } = run(budget, from('198.51.100.1', { 'content-length': '900' }));
+ expect(next).toHaveBeenCalledTimes(1);
+ expect(budget.currentBytes()).toBe(250);
+ res.emit('finish');
+ expect(budget.currentBytes()).toBe(0);
+ expect(budget.anonymousBytes()).toBe(0);
+ });
+
+ it('shares per key across addresses, and per address never across keys', () => {
+ const { budget } = tiered();
+ expect(run(budget, from('198.51.100.1', { 'content-length': '400', 'x-api-key': 'k1' })).next).toHaveBeenCalled();
+ const sameKey = run(budget, from('198.51.100.2', { 'content-length': '200', 'x-api-key': 'k1' }));
+ expect(sameKey.state.code).toBe(503);
+ const otherKey = run(budget, from('198.51.100.1', { 'content-length': '400', 'x-api-key': 'k2' }));
+ expect(otherKey.next).toHaveBeenCalledTimes(1);
+ });
+
+ it('aborts an anonymous body whose growth crosses the pool, releasing every counter once', () => {
+ // Budget 8 MiB, body cap 2 MiB: pool 4 MiB, 2 MiB per anonymous IP, 1 MiB chunked placeholder.
+ const budget = createInflightBodyBudget(8 * MB, { classify: () => undefined, bodyLimitBytes: 2 * MB });
+ const slow = from('198.51.100.1', { 'transfer-encoding': 'chunked' });
+ const slowRes = run(budget, slow).res;
+ const second = run(budget, from('198.51.100.2', { 'content-length': String(2 * MB) }));
+ const third = run(budget, from('198.51.100.3', { 'content-length': String(MB) }));
+ expect(second.next).toHaveBeenCalledTimes(1);
+ expect(third.next).toHaveBeenCalledTimes(1);
+ expect(budget.anonymousBytes()).toBe(4 * MB);
+
+ // Within its own share and the aggregate, but past the pool.
+ (slow as unknown as { socket: { bytesRead: number } }).socket.bytesRead = 1.5 * MB;
+ jest.advanceTimersByTime(5_000);
+ expect((slow as unknown as { destroy: jest.Mock }).destroy).toHaveBeenCalledTimes(1);
+ slow.emit('close');
+ slowRes.emit('finish');
+ expect(budget.anonymousBytes()).toBe(3 * MB);
+ expect(budget.clientBytes(slow)).toBe(0);
+
+ second.res.emit('finish');
+ third.res.emit('finish');
+ expect(budget.anonymousBytes()).toBe(0);
+ expect(budget.currentBytes()).toBe(0);
+ });
+
+ it('does not classify a request without a body', () => {
+ const { budget, classify } = tiered();
+ run(budget, from('198.51.100.1', { 'x-api-key': 'k1' }));
+ expect(classify).not.toHaveBeenCalled();
+ });
+
+ it('keeps one pool with per-IP shares when no classifier is given', () => {
+ const budget = createInflightBodyBudget(1000, { bodyLimitBytes: 250 });
+ expect(run(budget, from('198.51.100.1', { 'content-length': '400' })).next).toHaveBeenCalledTimes(1);
+ expect(run(budget, from('198.51.100.2', { 'content-length': '400' })).next).toHaveBeenCalledTimes(1);
+ expect(budget.anonymousBytes()).toBe(0);
+ });
+});
+
describe('stall reaper', () => {
beforeEach(() => jest.useFakeTimers());
afterEach(() => jest.useRealTimers());
@@ -436,6 +544,41 @@ describe('stall reaper', () => {
expect(budget.currentBytes()).toBeGreaterThan(MB);
});
+ it('paces an anonymous chunked body against the smaller amount it is charged', () => {
+ const budget = createInflightBodyBudget(MB, { classify: () => undefined, bodyLimitBytes: 64 * 1024 });
+ const req = makeReq({ 'transfer-encoding': 'chunked' });
+ run(budget, req);
+ expect(budget.anonymousBytes()).toBe(64 * 1024);
+
+ // 64 KiB in 300 s is about 220 B/s; 1 KiB every 4 s keeps up, 1 MiB in 300 s would not.
+ trickle([req], 1024, 60_000);
+ expect(destroyMock(req)).not.toHaveBeenCalled();
+ });
+
+ it('does not let trickling unkeyed chunked bodies hold the unkeyed pool past the pace window', () => {
+ const budget = createInflightBodyBudget(100 * MB, { classify: () => undefined, bodyLimitBytes: 25 * MB });
+ const held: Request[] = [];
+ for (const ip of ['198.51.100.1', '198.51.100.2']) {
+ for (let i = 0; i < 25; i++) {
+ const req = makeReq({ 'transfer-encoding': 'chunked' });
+ socketOf(req).remoteAddress = ip;
+ expect(run(budget, req).next).toHaveBeenCalledTimes(1);
+ held.push(req);
+ }
+ }
+ expect(budget.anonymousBytes()).toBe(50 * MB);
+ const early = makeReq({ 'content-length': '2000' });
+ socketOf(early).remoteAddress = '203.0.113.9';
+ expect(run(budget, early).state.code).toBe(503);
+
+ trickle(held, 1, 20_000);
+ for (const r of held) expect(destroyMock(r)).toHaveBeenCalledTimes(1);
+ expect(budget.anonymousBytes()).toBe(0);
+ const late = makeReq({ 'content-length': '2000' });
+ socketOf(late).remoteAddress = '203.0.113.9';
+ expect(run(budget, late).next).toHaveBeenCalledTimes(1);
+ });
+
it('keeps a client share held by trickling requests', () => {
const budget = createInflightBodyBudget(1000); // share 0.5 -> cap 500
const a = makeReq({ 'content-length': '250' });
diff --git a/src/config/inflight-body-budget.ts b/src/config/inflight-body-budget.ts
index 218be8048..fee1c7c81 100644
--- a/src/config/inflight-body-budget.ts
+++ b/src/config/inflight-body-budget.ts
@@ -124,6 +124,20 @@ export interface InflightBodyBudgetOptions {
* a legitimate bulk uploader above the share gets 503 + Retry-After, not a hang.
*/
perClientShare?: number;
+ /**
+ * Turns on the anonymous tier. Returns a stable id (the stored key hash) when the request carries
+ * an API key known to be active, undefined otherwise. Unrecognised requests share a pool of
+ * ANONYMOUS_POOL_FRACTION of the budget (at least two body caps), so they can never take the rest
+ * away from keyed traffic; a recognised key draws on the whole budget with its own per-key share, wherever it
+ * connects from. Without this option every request draws on the whole budget with a per-IP share.
+ */
+ classify?: (req: Request, clientIp: string) => string | undefined;
+ /**
+ * The per-request body cap (BODY_SIZE_LIMIT) in bytes. An anonymous request is never charged
+ * more: the parser refuses a longer declared body with 413 without buffering it. Default 25 MiB,
+ * the parser's own default.
+ */
+ bodyLimitBytes?: number;
/**
* Node's request timeout (REQUEST_TIMEOUT_MS) in milliseconds. A body that falls behind the pace
* needed to deliver its reservation within it (the declared size, or a chunked body's
@@ -137,17 +151,34 @@ export interface InflightBodyBudget {
middleware: (req: Request, res: Response, next: NextFunction) => void;
/** Aggregate bytes currently attributed to in-flight request bodies (observability/tests). */
currentBytes: () => number;
- /** Bytes currently attributed to one client's in-flight bodies (observability/tests). */
+ /** Bytes currently attributed to one client IP's in-flight bodies (observability/tests). */
clientBytes: (req: Request) => number;
+ /** Bytes currently charged to the anonymous tier (observability/tests). */
+ anonymousBytes: () => number;
}
+/** Share of the budget open to requests without a recognised API key (see the classify option). */
+const ANONYMOUS_POOL_FRACTION = 0.25;
+
export function createInflightBodyBudget(budgetBytes: number, options?: InflightBodyBudgetOptions): InflightBodyBudget {
let inFlightBytes = 0;
const retryAfter = String(options?.retryAfterSeconds ?? 1);
const trustedProxies = options?.trustedProxies ?? [];
const share = Math.min(1, Math.max(Number.EPSILON, options?.perClientShare ?? 0.5));
const perClientCap = Math.max(1, Math.floor(budgetBytes * share));
+ const classify = options?.classify;
+ const bodyLimitBytes = options?.bodyLimitBytes ?? FALLBACK_LIMIT_BYTES;
const requestTimeoutMs = options?.requestTimeoutMs ?? DEFAULT_REQUEST_TIMEOUT_MS;
+ // The anonymous pool holds at least two full-size bodies, so with the default half share one
+ // client never fills it and other unkeyed senders, ingress deliveries among them, keep room. A
+ // budget below three body caps leaves keyed traffic less than a full-size body, which the docs
+ // call out.
+ const anonPool = Math.min(
+ budgetBytes,
+ Math.max(2 * bodyLimitBytes, Math.floor(budgetBytes * ANONYMOUS_POOL_FRACTION)),
+ );
+ const anonClientCap = Math.max(Math.min(bodyLimitBytes, anonPool), Math.floor(anonPool * share));
+ let anonInFlight = 0;
// Per-client in-flight bytes, keyed on the resolved client IP (an IPv6 client on its /64).
// Entries are created lazily and deleted by the same exactly-once release that decrements the
// aggregate, so the map cannot leak a client that finished. Only body-carrying requests are
@@ -238,15 +269,32 @@ export function createInflightBodyBudget(budgetBytes: number, options?: Inflight
// Admission control on the RESERVED size: a request that would push the aggregate OR ITS
// CLIENT'S SHARE past the budget is refused before a single byte of its body is buffered.
// The per-client share bounds what one source can hold.
- const clientKey = limiterKeyForIp(resolveClientIp(req, trustedProxies));
+ // A recognised key is shared per key (its entries are bounded by the key count, so they are
+ // exempt from the map cap); anything else per client IP.
+ const clientIp = resolveClientIp(req, trustedProxies);
+ const keyId = classify?.(req, clientIp);
+ const anonymous = classify !== undefined && keyId === undefined;
+ const clientKey = keyId !== undefined ? `key:${keyId}` : limiterKeyForIp(clientIp);
+ const shareCap = anonymous ? anonClientCap : perClientCap;
+ const ceiling = anonymous ? bodyLimitBytes : Infinity;
const clientBusy = clientInFlight.get(clientKey) ?? 0;
- const mapAtCapacity = clientInFlight.size >= MAX_TRACKED_CLIENTS && !clientInFlight.has(clientKey);
- if (inFlightBytes + reserved > budgetBytes || clientBusy + reserved > perClientCap || mapAtCapacity) {
+ const mapAtCapacity =
+ keyId === undefined && clientInFlight.size >= MAX_TRACKED_CLIENTS && !clientInFlight.has(clientKey);
+ // The aggregate check uses the full declared size; the tier and share checks the charged size.
+ const charge = Math.min(reserved, ceiling);
+ if (
+ inFlightBytes + reserved > budgetBytes ||
+ clientBusy + charge > shareCap ||
+ (anonymous && anonInFlight + charge > anonPool) ||
+ mapAtCapacity
+ ) {
rejectBusy(req, res);
return;
}
+ reserved = charge;
inFlightBytes += reserved;
+ if (anonymous) anonInFlight += reserved;
clientInFlight.set(clientKey, clientBusy + reserved);
let released = false;
@@ -265,6 +313,7 @@ export function createInflightBodyBudget(budgetBytes: number, options?: Inflight
released = true;
disarmStallReaper();
inFlightBytes -= reserved;
+ if (anonymous) anonInFlight -= reserved;
const busy = (clientInFlight.get(clientKey) ?? reserved) - reserved;
if (busy > 0) clientInFlight.set(clientKey, busy);
else clientInFlight.delete(clientKey);
@@ -289,21 +338,23 @@ export function createInflightBodyBudget(budgetBytes: number, options?: Inflight
// A declared body keeps its declared size until it is released. A chunked body is re-priced at
// the bytes that have arrived, never below its opening placeholder until it is complete: budget
// handed back mid-stream would be taken back, unchecked, by a body that then completes between
- // polls. Growth that crosses the aggregate or the client share aborts the
+ // polls. Growth that crosses the aggregate, the client share or the anonymous pool aborts the
// request mid-stream, the same bound a declared length gets at admission. A complete body is
// re-priced at its real size but never aborted: it is already buffered. reserved is updated
// BEFORE release() so the exactly-once decrement subtracts the reconciled size.
const reconcile = (complete: boolean): void => {
if (released || declared !== undefined) return;
const floor = complete ? 0 : undeclaredReservation;
- const actual = Math.max(floor, socket.bytesRead - startBytes);
+ const actual = Math.min(Math.max(floor, socket.bytesRead - startBytes), ceiling);
const delta = actual - reserved;
if (delta === 0) return;
inFlightBytes += delta;
+ if (anonymous) anonInFlight += delta;
reserved = actual;
const busy = (clientInFlight.get(clientKey) ?? 0) + delta;
clientInFlight.set(clientKey, busy);
- if (!complete && delta > 0 && (busy > perClientCap || inFlightBytes > budgetBytes)) {
+ const overTier = anonymous && anonInFlight > anonPool;
+ if (!complete && delta > 0 && (busy > shareCap || inFlightBytes > budgetBytes || overTier)) {
release();
req.destroy();
}
@@ -364,6 +415,7 @@ export function createInflightBodyBudget(budgetBytes: number, options?: Inflight
middleware,
currentBytes: () => inFlightBytes,
clientBytes: req => clientInFlight.get(limiterKeyForIp(resolveClientIp(req, trustedProxies))) ?? 0,
+ anonymousBytes: () => anonInFlight,
};
}
diff --git a/src/configure-app.ts b/src/configure-app.ts
index e66642116..4b030f0b1 100644
--- a/src/configure-app.ts
+++ b/src/configure-app.ts
@@ -1,11 +1,17 @@
import { INestApplication } from '@nestjs/common';
+import { ModuleRef } from '@nestjs/core';
import helmet from 'helmet';
import { Request, Response, NextFunction, json, urlencoded } from 'express';
import { randomBytes } from 'crypto';
import { existsSync, readFileSync } from 'fs';
import { extname, join } from 'path';
import { DASHBOARD_DIST, dashboardServingEnabled, dashboardBuildPresent } from './app.module';
-import { createInflightBodyBudget, resolveInflightBodyBudgetBytes } from './config/inflight-body-budget';
+import {
+ createInflightBodyBudget,
+ parseBodyLimitBytes,
+ resolveInflightBodyBudgetBytes,
+} from './config/inflight-body-budget';
+import { ActiveKeyIndex } from './modules/auth/active-key-index';
import { requestContextMiddleware } from './common/middleware/request-context.middleware';
import { injectDashboardCspNonce } from './config/dashboard-csp';
import { resolveCorsPolicy, isUpgradeInsecureRequestsEnabled, resolveBodyLimit } from './config/bootstrap-security';
@@ -60,19 +66,30 @@ export function configureApp(app: INestApplication, options: ConfigureAppOptions
process.env.INFLIGHT_BODY_BUDGET_BYTES,
process.env.BODY_SIZE_LIMIT,
);
+ // Cap request body size (DoS hardening). Media sends carry base64 in the JSON body,
+ // so the default is generous; tune with BODY_SIZE_LIMIT.
+ const bodyLimit = resolveBodyLimit(process.env.BODY_SIZE_LIMIT);
+ // Requests without a recognised API key share a pool of a quarter of the budget, never less than
+ // two BODY_SIZE_LIMIT bodies (half the default budget). With no AuthModule in the app (some test
+ // modules) every request is unrecognised, which is the stricter side.
+ // Looked up through ModuleRef: a failed app.get() aborts the process instead of throwing.
+ let keyIndex: ActiveKeyIndex | undefined;
+ try {
+ keyIndex = app.get(ModuleRef).get(ActiveKeyIndex, { strict: false });
+ } catch {
+ keyIndex = undefined;
+ }
app.use(
createInflightBodyBudget(inflightBudgetBytes, {
trustedProxies: (process.env.TRUSTED_PROXIES || '')
.split(',')
.map(p => p.trim())
.filter(Boolean),
+ classify: (req, clientIp) => keyIndex?.recognise(req.headers, clientIp),
+ bodyLimitBytes: parseBodyLimitBytes(bodyLimit),
requestTimeoutMs: resolveRequestTimeoutMs(process.env.REQUEST_TIMEOUT_MS),
}).middleware,
);
-
- // Cap request body size (DoS hardening). Media sends carry base64 in the JSON body,
- // so the default is generous; tune with BODY_SIZE_LIMIT.
- const bodyLimit = resolveBodyLimit(process.env.BODY_SIZE_LIMIT);
// The `verify` callback stashes the EXACT bytes json() received on req.rawBody, byte-identical to
// what a provider signed, so the @Public ingress controller can HMAC-verify over the raw body
// (JSON.stringify(req.body) is NOT byte-identical). Cheap for every route; non-ingress routes ignore it.
diff --git a/src/modules/auth/active-key-index.spec.ts b/src/modules/auth/active-key-index.spec.ts
new file mode 100644
index 000000000..6bbfa3f58
--- /dev/null
+++ b/src/modules/auth/active-key-index.spec.ts
@@ -0,0 +1,115 @@
+import { createHash, createHmac } from 'crypto';
+import { Repository } from 'typeorm';
+import { ActiveKeyIndex } from './active-key-index';
+import { ApiKey } from './entities/api-key.entity';
+
+const sha = (key: string) => createHash('sha256').update(key).digest('hex');
+
+type Row = Pick<ApiKey, 'keyHash' | 'allowedIps' | 'expiresAt'>;
+
+const indexOver = (rows: Row[] | (() => Promise<Row[]>)) => {
+ const find = jest.fn(typeof rows === 'function' ? rows : () => Promise.resolve(rows));
+ const index = new ActiveKeyIndex({ find } as unknown as Repository<ApiKey>);
+ return { index, find };
+};
+
+const row = (key: string, extra: Partial<Row> = {}): Row => ({
+ keyHash: sha(key),
+ allowedIps: null,
+ expiresAt: null,
+ ...extra,
+});
+
+describe('ActiveKeyIndex', () => {
+ const pepper = process.env.API_KEY_PEPPER;
+ afterEach(() => {
+ if (pepper === undefined) delete process.env.API_KEY_PEPPER;
+ else process.env.API_KEY_PEPPER = pepper;
+ jest.useRealTimers();
+ });
+
+ it('recognises an active key from X-API-Key or a Bearer token, trimmed', async () => {
+ const { index, find } = indexOver([row('k1')]);
+ await index.onApplicationBootstrap();
+ index.onModuleDestroy();
+
+ expect(find).toHaveBeenCalledWith(expect.objectContaining({ where: { isActive: true } }));
+ expect(index.recognise({ 'x-api-key': ' k1 ' }, '192.0.2.1')).toBe(sha('k1'));
+ expect(index.recognise({ authorization: 'Bearer k1' }, '192.0.2.1')).toBe(sha('k1'));
+ expect(index.recognise({ 'x-api-key': 'nope' }, '192.0.2.1')).toBeUndefined();
+ expect(index.recognise({}, '192.0.2.1')).toBeUndefined();
+ });
+
+ it('reads X-API-Key before the Bearer token, as the guard does', async () => {
+ const { index } = indexOver([row('k1')]);
+ await index.onApplicationBootstrap();
+ index.onModuleDestroy();
+ expect(index.recognise({ 'x-api-key': 'nope', authorization: 'Bearer k1' }, '192.0.2.1')).toBeUndefined();
+ });
+
+ it('hashes with the pepper when one is set', async () => {
+ process.env.API_KEY_PEPPER = 'pepper';
+ const peppered = createHmac('sha256', 'pepper').update('k1').digest('hex');
+ const { index } = indexOver([{ keyHash: peppered, allowedIps: null, expiresAt: null }]);
+ await index.onApplicationBootstrap();
+ index.onModuleDestroy();
+ expect(index.recognise({ 'x-api-key': 'k1' }, '192.0.2.1')).toBe(peppered);
+ });
+
+ it('ignores an expired key and one used from outside its allowed addresses', async () => {
+ const { index } = indexOver([
+ row('old', { expiresAt: new Date(Date.now() - 1000) }),
+ row('fenced', { allowedIps: ['10.0.0.0/8'] }),
+ ]);
+ await index.onApplicationBootstrap();
+ index.onModuleDestroy();
+ expect(index.recognise({ 'x-api-key': 'old' }, '192.0.2.1')).toBeUndefined();
+ expect(index.recognise({ 'x-api-key': 'fenced' }, '192.0.2.1')).toBeUndefined();
+ expect(index.recognise({ 'x-api-key': 'fenced' }, '10.1.2.3')).toBe(sha('fenced'));
+ });
+
+ it('keeps the previous view when a refresh fails', async () => {
+ let fail = false;
+ const { index } = indexOver(() => (fail ? Promise.reject(new Error('db down')) : Promise.resolve([row('k1')])));
+ await index.onApplicationBootstrap();
+ fail = true;
+ index.refreshSoon();
+ await (index as unknown as { loading: Promise<void> }).loading;
+ index.onModuleDestroy();
+ expect(index.recognise({ 'x-api-key': 'k1' }, '192.0.2.1')).toBe(sha('k1'));
+ });
+
+ it('runs one more load when asked to refresh during a load, and none after destroy', async () => {
+ let rows = [row('k1')];
+ const { index, find } = indexOver(() => Promise.resolve(rows));
+ await index.onApplicationBootstrap();
+ expect(find).toHaveBeenCalledTimes(1);
+
+ rows = [row('k2')];
+ index.refreshSoon();
+ index.refreshSoon();
+ index.refreshSoon();
+ await (index as unknown as { loading: Promise<void> }).loading;
+ expect(find).toHaveBeenCalledTimes(3);
+ expect(index.recognise({ 'x-api-key': 'k2' }, '192.0.2.1')).toBe(sha('k2'));
+ expect(index.recognise({ 'x-api-key': 'k1' }, '192.0.2.1')).toBeUndefined();
+
+ index.onModuleDestroy();
+ index.refreshSoon();
+ expect(find).toHaveBeenCalledTimes(3);
+ });
+
+ it('refreshes on an interval until destroyed', async () => {
+ jest.useFakeTimers();
+ const { index, find } = indexOver([row('k1')]);
+ await index.onApplicationBootstrap();
+ expect(find).toHaveBeenCalledTimes(1);
+
+ await jest.advanceTimersByTimeAsync(30_000);
+ expect(find).toHaveBeenCalledTimes(2);
+
+ index.onModuleDestroy();
+ await jest.advanceTimersByTimeAsync(60_000);
+ expect(find).toHaveBeenCalledTimes(2);
+ });
+});
diff --git a/src/modules/auth/active-key-index.ts b/src/modules/auth/active-key-index.ts
new file mode 100644
index 000000000..842c7d65e
--- /dev/null
+++ b/src/modules/auth/active-key-index.ts
@@ -0,0 +1,107 @@
+import { Injectable, OnApplicationBootstrap, OnModuleDestroy } from '@nestjs/common';
+import { InjectRepository } from '@nestjs/typeorm';
+import { Repository } from 'typeorm';
+import type { IncomingHttpHeaders } from 'http';
+import { ApiKey } from './entities/api-key.entity';
+import { hashApiKey } from './api-key-hash';
+import { bearerToken } from '../../common/security/bearer-token';
+import { ipMatches } from '../../common/utils/ip';
+import { createLogger } from '../../common/services/logger.service';
+
+const REFRESH_INTERVAL_MS = 30_000;
+
+interface IndexedKey {
+ allowedIps: string[] | null;
+ expiresAt: Date | null;
+}
+
+/**
+ * In-memory view of the active API keys, used BEFORE a request body is read to decide which tier
+ * of the in-flight body budget the request draws on. It is advisory only: it never authorises
+ * anything, the API-key guard still validates every request against the database. A key this
+ * node creates, changes or revokes is picked up at once; a direct write to this node's main
+ * database within REFRESH_INTERVAL_MS. Keys are per node, so a change made on another node never
+ * reaches this index.
+ */
+@Injectable()
+export class ActiveKeyIndex implements OnApplicationBootstrap, OnModuleDestroy {
+ private readonly logger = createLogger('ActiveKeyIndex');
+ private keys = new Map<string, IndexedKey>();
+ private timer: ReturnType<typeof setInterval> | undefined;
+ private loading: Promise<void> | undefined;
+ private reloadAgain = false;
+ private destroyed = false;
+
+ constructor(
+ @InjectRepository(ApiKey, 'main')
+ private readonly apiKeyRepository: Repository<ApiKey>,
+ ) {}
+
+ async onApplicationBootstrap(): Promise<void> {
+ this.refreshSoon();
+ await this.loading;
+ this.timer = setInterval(() => this.refreshSoon(), REFRESH_INTERVAL_MS);
+ this.timer.unref();
+ }
+
+ onModuleDestroy(): void {
+ this.destroyed = true;
+ clearInterval(this.timer);
+ this.timer = undefined;
+ }
+
+ /**
+ * Reload without blocking the caller. A call that lands while a load is running queues exactly
+ * one more load, so a write that committed after the running SELECT started is never missed.
+ */
+ refreshSoon(): void {
+ if (this.destroyed) return;
+ if (this.loading) {
+ this.reloadAgain = true;
+ return;
+ }
+ this.loading = (async () => {
+ do {
+ this.reloadAgain = false;
+ await this.load();
+ } while (this.reloadAgain && !this.destroyed);
+ })().finally(() => {
+ this.loading = undefined;
+ });
+ }
+
+ /**
+ * The stored hash of the request's key when it names an active, unexpired key that may be used
+ * from clientIp; undefined otherwise. Reads the key the way the guard does: X-API-Key first, then
+ * a Bearer token, trimmed.
+ */
+ recognise(headers: IncomingHttpHeaders, clientIp: string): string | undefined {
+ const header = headers['x-api-key'];
+ const raw = (typeof header === 'string' && header ? header : bearerToken(headers.authorization))?.trim();
+ if (!raw) return undefined;
+ const keyHash = hashApiKey(raw, process.env.API_KEY_PEPPER);
+ const key = this.keys.get(keyHash);
+ if (!key) return undefined;
+ // Expiry is checked here rather than in the query: SQLite stores dates as strings, and a key
+ // that expires between two refreshes must stop counting at once.
+ if (key.expiresAt && key.expiresAt < new Date()) return undefined;
+ if (key.allowedIps?.length && !key.allowedIps.some(entry => ipMatches(clientIp, entry))) return undefined;
+ return keyHash;
+ }
+
+ private async load(): Promise<void> {
+ try {
+ const rows = await this.apiKeyRepository.find({
+ select: { keyHash: true, allowedIps: true, expiresAt: true },
+ where: { isActive: true },
+ });
+ this.keys = new Map(rows.map(row => [row.keyHash, { allowedIps: row.allowedIps, expiresAt: row.expiresAt }]));
+ } catch (error) {
+ // Keep the last good view: a transient database error must not move every keyed request into
+ // the anonymous tier.
+ this.logger.warn('Could not refresh the active API key index; keeping the previous one', {
+ error: error instanceof Error ? error.message : String(error),
+ });
+ }
+ }
+}
diff --git a/src/modules/auth/auth.module.ts b/src/modules/auth/auth.module.ts
index 2eb4a80c8..e26181ab9 100644
--- a/src/modules/auth/auth.module.ts
+++ b/src/modules/auth/auth.module.ts
@@ -5,6 +5,7 @@ import { ApiKey } from './entities/api-key.entity';
import { AuthService } from './auth.service';
import { ApiKeyUsageTracker } from './api-key-usage-tracker.service';
import { ChatScopeService } from './chat-scope.service';
+import { ActiveKeyIndex } from './active-key-index';
import { AuthController } from './auth.controller';
import { AuthValidateController } from './auth-validate.controller';
import { ApiKeyGuard } from './guards/api-key.guard';
@@ -18,6 +19,7 @@ import { ProxyAwareThrottlerGuard } from '../../common/security/proxy-aware-thro
AuthService,
ApiKeyUsageTracker,
ChatScopeService,
+ ActiveKeyIndex,
{
provide: APP_GUARD,
useClass: ProxyAwareThrottlerGuard,
@@ -27,6 +29,6 @@ import { ProxyAwareThrottlerGuard } from '../../common/security/proxy-aware-thro
useClass: ApiKeyGuard,
},
],
- exports: [AuthService, ChatScopeService],
+ exports: [AuthService, ChatScopeService, ActiveKeyIndex],
})
export class AuthModule {}
diff --git a/src/modules/auth/auth.service.spec.ts b/src/modules/auth/auth.service.spec.ts
index 19b0cc6a3..2dd32145f 100644
--- a/src/modules/auth/auth.service.spec.ts
+++ b/src/modules/auth/auth.service.spec.ts
@@ -11,6 +11,7 @@ import * as fs from 'fs';
import { AuthService, resolveSeedApiKey, bannerKeyLine, UnresolvedApiKeyException } from './auth.service';
import { ApiKeyUsageTracker } from './api-key-usage-tracker.service';
import { getRequestActor, runWithRequestId } from '../../common/services/request-context';
+import { ActiveKeyIndex } from './active-key-index';
import { ApiKey, ApiKeyRole } from './entities/api-key.entity';
// Helpers
@@ -112,8 +113,10 @@ describe('AuthService', () => {
let committedWrites: Array<{ mode: 'update' | 'delete'; patch?: Record<string, unknown>; guarded: boolean }>;
/** The raw AND-fragment the last-admin guard binds, so the SQL predicate itself is asserted. */
let lastAdminFragments: string[];
+ let keyIndex: { refreshSoon: jest.Mock };
beforeEach(async () => {
+ keyIndex = { refreshSoon: jest.fn() };
repository = {
count: jest.fn(),
find: jest.fn(),
@@ -135,6 +138,7 @@ describe('AuthService', () => {
provide: getRepositoryToken(ApiKey, 'main'),
useValue: repository,
},
+ { provide: ActiveKeyIndex, useValue: keyIndex },
],
}).compile();
@@ -416,6 +420,22 @@ describe('AuthService', () => {
});
});
+ describe('active key index', () => {
+ it('is refreshed after every key write', async () => {
+ (repository.create as jest.Mock).mockImplementation((dto: Partial<ApiKey>) => ({ ...dto, id: 'uuid-new' }));
+ await service.createApiKey({ name: 'new' });
+ expect(keyIndex.refreshSoon).toHaveBeenCalledTimes(1);
+
+ setupKeys([createMockApiKey({ id: 'uuid-1' }), createMockApiKey({ id: 'uuid-2' })]);
+ await service.update('uuid-1', { name: 'renamed' });
+ expect(keyIndex.refreshSoon).toHaveBeenCalledTimes(2);
+ await service.revoke('uuid-1');
+ expect(keyIndex.refreshSoon).toHaveBeenCalledTimes(3);
+ await service.delete('uuid-2');
+ expect(keyIndex.refreshSoon).toHaveBeenCalledTimes(4);
+ });
+ });
+
describe('revoke', () => {
it('should set isActive to false', async () => {
setupKeys([createMockApiKey({ id: 'uuid-1', isActive: true })]);
diff --git a/src/modules/auth/auth.service.ts b/src/modules/auth/auth.service.ts
index 901419b7f..2692cea00 100644
--- a/src/modules/auth/auth.service.ts
+++ b/src/modules/auth/auth.service.ts
@@ -3,6 +3,7 @@ import {
ForbiddenException,
Injectable,
NotFoundException,
+ Optional,
UnauthorizedException,
OnModuleInit,
OnModuleDestroy,
@@ -19,6 +20,7 @@ import { createLogger } from '../../common/services/logger.service';
import { setRequestActor } from '../../common/services/request-context';
import { readBootstrapKey, removeBootstrapKey, writeBootstrapKey } from './bootstrap-key-file';
import { ApiKeyUsageTracker } from './api-key-usage-tracker.service';
+import { ActiveKeyIndex } from './active-key-index';
import { apiKeyAuthorizationFingerprint, normalizeScopeList } from './api-key-authorization';
import { normalizeChatAllowList } from '../../common/security/chat-scope';
import { EventsGateway, type ApiKeyEvictionReason } from '../events/events.gateway';
@@ -78,6 +80,7 @@ export class AuthService implements OnModuleInit, OnModuleDestroy {
private readonly apiKeyRepository: Repository<ApiKey>,
private readonly usageTracker: ApiKeyUsageTracker,
private readonly moduleRef: ModuleRef,
+ @Optional() private readonly keyIndex?: ActiveKeyIndex,
) {}
async onModuleInit(): Promise<void> {
@@ -209,6 +212,7 @@ export class AuthService implements OnModuleInit, OnModuleDestroy {
});
const saved = await this.apiKeyRepository.save(apiKey);
+ this.keyIndex?.refreshSoon();
this.logger.log(`API key created: ${saved.name}`, {
keyId: saved.id,
role: saved.role,
@@ -281,6 +285,7 @@ export class AuthService implements OnModuleInit, OnModuleDestroy {
// The row's post-write state, for the eviction comparison below.
saved = await this.findOne(id);
}
+ this.keyIndex?.refreshSoon();
// One fingerprint definition, two callers: this immediate eviction and the gateway's periodic
// re-validation sweep. Sharing it keeps the two from disagreeing about what an authorization
@@ -307,6 +312,7 @@ export class AuthService implements OnModuleInit, OnModuleDestroy {
this.usageTracker.forget(id);
this.removeBootstrapKeyFileIfMatching(apiKey);
this.evictActiveSockets(id, 'deleted');
+ this.keyIndex?.refreshSoon();
this.logger.log(`API key deleted: ${apiKey.name}`, {
keyId: id,
action: 'api_key_deleted',
@@ -335,6 +341,7 @@ export class AuthService implements OnModuleInit, OnModuleDestroy {
// Kick any WebSocket connections already authenticated with this key: without this, a revoked
// key keeps receiving events on already-subscribed sockets until they happen to disconnect.
this.evictActiveSockets(id, 'revoked');
+ this.keyIndex?.refreshSoon();
return saved;
}
diff --git a/test/configure-app.e2e-spec.ts b/test/configure-app.e2e-spec.ts
index fbdeb1597..227d6f38a 100644
--- a/test/configure-app.e2e-spec.ts
+++ b/test/configure-app.e2e-spec.ts
@@ -3,11 +3,13 @@ import { NestFactory } from '@nestjs/core';
import request from 'supertest';
import { App } from 'supertest/types';
import { mkdtempSync, mkdirSync, writeFileSync } from 'fs';
+import { request as httpRequest } from 'http';
import { tmpdir } from 'os';
import { join } from 'path';
import { configureApp } from '../src/configure-app';
import { applyGlobalValidation } from '../src/config/app-validation';
import { DASHBOARD_CSP_NONCE_PLACEHOLDER } from '../src/config/dashboard-csp';
+import { ActiveKeyIndex } from '../src/modules/auth/active-key-index';
/**
* The production HTTP surface, run for real. Every other e2e builds a bare Nest app, so the stack
@@ -161,3 +163,63 @@ describe('production HTTP surface (configureApp)', () => {
expect(res.status).toBe(415);
});
});
+
+describe('in-flight body budget tiers (configureApp)', () => {
+ let app: INestApplication<App>;
+ const previousBodyLimit = process.env.BODY_SIZE_LIMIT;
+
+ // Stands in for the real index: only 'known-key' is an active key.
+ const keyIndex = {
+ recognise: (headers: { 'x-api-key'?: unknown }) => (headers['x-api-key'] === 'known-key' ? 'h' : undefined),
+ };
+
+ @Module({ controllers: [EchoController], providers: [{ provide: ActiveKeyIndex, useValue: keyIndex }] })
+ class TieredSurfaceModule {}
+
+ beforeAll(async () => {
+ // 1mb cap: a 4mb budget whose anonymous pool is 2mb, 1mb per client address.
+ process.env.BODY_SIZE_LIMIT = '1mb';
+ app = await NestFactory.create<INestApplication<App>>(TieredSurfaceModule, { bodyParser: false, logger: false });
+ configureApp(app);
+ applyGlobalValidation(app);
+ await app.listen(0, '127.0.0.1');
+ });
+
+ afterAll(async () => {
+ await app?.close();
+ if (previousBodyLimit === undefined) delete process.env.BODY_SIZE_LIMIT;
+ else process.env.BODY_SIZE_LIMIT = previousBodyLimit;
+ });
+
+ it('keeps room for a recognised key while an unrecognised client holds its whole share', async () => {
+ const { port } = new URL(await app.getUrl());
+ // Declares a full-size body, this address's whole share, and never finishes sending it.
+ const held = httpRequest({
+ host: '127.0.0.1',
+ port,
+ path: '/api/echo',
+ method: 'POST',
+ headers: { 'Content-Type': 'application/json', 'Content-Length': String(1024 * 1024) },
+ });
+ held.on('error', () => undefined);
+ held.write('{"a":"');
+ await new Promise(resolve => setTimeout(resolve, 200));
+
+ try {
+ const anonymous = await request(app.getHttpServer())
+ .post('/api/echo')
+ .set('Content-Type', 'application/json')
+ .send({ small: true });
+ expect(anonymous.status).toBe(503);
+
+ const keyed = await request(app.getHttpServer())
+ .post('/api/echo')
+ .set('Content-Type', 'application/json')
+ .set('X-API-Key', 'known-key')
+ .send({ small: true });
+ expect(keyed.status).toBe(201);
+ } finally {
+ held.destroy();
+ }
+ });
+});