Free, Open Source, Self-Hosted WhatsApp API Gateway
TypeScript apibotgatewayself-hostedwhatsappwhatsapp-apiwhatsapp-automationwhatsapp-botwhatsapp-gatewaywhatsapp-gateway-apiwhatsapp-webwhatsapp-web-api
fix(ingress): count only verified deliveries against the instance limit
Yudhi Armyndharisd906cf6a6842
diff --git a/.env.example b/.env.exampleindex 75684a4c0..f1878c5d6 100644--- a/.env.example+++ b/.env.example@@ -693,12 +693,13 @@ WEBHOOK_SSRF_PROTECT=true# Providers deliver every tenant's webhooks from one shared egress IP, so this is keyed on# (pluginId, instanceId) instead of IP — a noisy tenant gets 429'd without throttling its neighbors.# Independent of, and in addition to, the IP-keyed tiers above.+# Counted only for deliveries that pass signature verification; requests that fail it count against+# INGRESS_IP_LIMIT alone. A scheme 'none' route (ALLOW_UNSIGNED_INGRESS=true) passes every request.# INGRESS_INSTANCE_LIMIT=120 # max requests per instance per window (default 120)# INGRESS_INSTANCE_TTL=60000 # window in ms (default 60000 = 60s)-# Second bucket on the same window, keyed on the client instead of the path. The per-instance key is-# built from the caller-supplied :pluginId/:instanceId, so varying them mints a fresh bucket; this-# bound is what an unauthenticated caller cannot walk around. Sized 10x the per-instance default so-# it never binds before it, raise it if one IP legitimately drives more than that.+# Second bucket on the same window, keyed on the client instead of the path. It is checked before+# anything else, so it is the only bound on requests that fail verification. Sized 10x the per-instance+# default so it never binds before it, raise it if one IP legitimately drives more than that.# INGRESS_IP_LIMIT=1200 # max ingress requests per client IP per window (default 1200)# Integration Fabric delivery retry + retention (the queued path taken when QUEUE_ENABLED=true).diff --git a/docs/04-security-design.md b/docs/04-security-design.mdindex 50633442f..1d20eaa41 100644--- a/docs/04-security-design.md+++ b/docs/04-security-design.md@@ -291,10 +291,10 @@ Exceeding a window returns `429 Too Many Requests`. Because the windows are **naThe ingress route (`ALL /api/ingress/:pluginId/:instanceId/*path`) is exempt from the global per-IP tiers (their 100/min medium window sits below the per-instance limit, so a provider fanning every tenant's webhooks through one egress IP was shed before the per-instance bound could fire). It carries its own two windows instead, both on `INGRESS_INSTANCE_TTL` (default 60000 ms):-- `instance`, keyed on `(pluginId, instanceId)`, env `INGRESS_INSTANCE_LIMIT`, default 120. Sheds one noisy tenant without touching its neighbours.-- `ingress-ip`, keyed on the client (proxy-aware, see `TRUSTED_PROXIES`), env `INGRESS_IP_LIMIT`, default 1200. The `instance` key is built from path segments the caller supplies, so varying them mints a fresh bucket; this window is the bound an unauthenticated caller cannot walk around. It is sized 10x the per-instance default so it never binds first for legitimate traffic.+- `instance`, keyed on `(pluginId, instanceId)`, env `INGRESS_INSTANCE_LIMIT`, default 120. Sheds one noisy tenant without touching its neighbours. Charged only once a delivery passes signature verification, so unknown-instance, failed-challenge and oversized requests never touch it. A route declaring signature scheme `none` (allowed only with `ALLOW_UNSIGNED_INGRESS=true`) passes every request, so there any caller can spend this bucket.+- `ingress-ip`, keyed on the client (proxy-aware, see `TRUSTED_PROXIES`), env `INGRESS_IP_LIMIT`, default 1200. It is checked before anything else and is the only bound on traffic that fails verification. It is sized 10x the per-instance default so it never binds first for legitimate traffic.-Responses therefore carry `X-RateLimit-*-instance` and `X-RateLimit-*-ingress-ip`, and on saturation the `Retry-After-*` of whichever window shed the request, mirrored into a plain `Retry-After`.+Every response the per-IP window admits carries `X-RateLimit-*-ingress-ip`, an acknowledged delivery also carries `X-RateLimit-*-instance`, and a `429` carries the `Retry-After-*` of whichever window shed the request, mirrored into a plain `Retry-After`.The API exposes the rate-limit headers via CORS (`exposedHeaders`) so browser clients can read them, the plain `Retry-After` included. The simplest backpressure signal remains the `429` status itself, with `Retry-After` as the retry delay.diff --git a/docs/06-api-specification.md b/docs/06-api-specification.mdindex 9231d7257..e8b492796 100644--- a/docs/06-api-specification.md+++ b/docs/06-api-specification.md@@ -6853,7 +6853,7 @@ first delivery, and is not enqueued again. The ack is rendered from the retry, san ack body is the retry's time and `{rawBody}` is the retry's body. A retry still passes the checksthat run before dedup: while a `session-alive` route's bound session is down it gets the `503`.-**Errors:** `401` signature verification failed (missing, stale, or wrong secret) · `403` `GET` verification challenge failed (`verifyToken` mismatch) · `404` unknown pluginId/instanceId, or no such claimed route · `413` body over the route's `maxBodyBytes` · `429` rate limit: the per-instance bucket (`INGRESS_INSTANCE_LIMIT`) or the per-client-IP bucket (`INGRESS_IP_LIMIT`), both per `INGRESS_INSTANCE_TTL`; the global per-IP tiers skip this route, so these two are its bounds, and `Retry-After-instance` / `Retry-After-ingress-ip` names the one that shed the request, alongside a plain `Retry-After` carrying the same delay · `503` a route declaring a `session-alive` preflight whose bound session is not connected: the delivery is not persisted, so the provider's retry is treated as a new delivery, and `Retry-After` carries the delay+**Errors:** `401` signature verification failed (missing, stale, or wrong secret) · `403` `GET` verification challenge failed (`verifyToken` mismatch) · `404` unknown pluginId/instanceId, or no such claimed route · `413` body over the route's `maxBodyBytes` · `429` rate limit: the per-instance bucket (`INGRESS_INSTANCE_LIMIT`, counted only for deliveries that pass signature verification) or the per-client-IP bucket (`INGRESS_IP_LIMIT`), both per `INGRESS_INSTANCE_TTL`; the global per-IP tiers skip this route, so these two are its bounds, and `Retry-After-instance` / `Retry-After-ingress-ip` names the one that shed the request, alongside a plain `Retry-After` carrying the same delay · `503` a route declaring a `session-alive` preflight whose bound session is not connected: the delivery is not persisted, so the provider's retry is treated as a new delivery, and `Retry-After` carries the delay## 6.5 Real-time API (WebSocket)diff --git a/docs/25-integration-fabric.md b/docs/25-integration-fabric.mdindex 329bf7fc8..e84545f54 100644--- a/docs/25-integration-fabric.md+++ b/docs/25-integration-fabric.md@@ -160,10 +160,11 @@ Four tables live on the data connection, each created by a hand-authored dual-dithe API-key guard but validates a **per-instance HMAC (or shared secret)** over the **raw** requestbytes with a constant-time comparison. The raw body is preserved by a verify callback on the body parserbecause a re-serialized payload is not byte-identical to what the provider signed. The route is exempt from- the global per-IP throttle and bounded by its own guard instead, on two keys, `(pluginId, instanceId)` and- the client IP: a provider delivering every tenant's webhooks from one egress address would otherwise be- shed at the global tier before the per-instance bound ever fired. The payload is intentionally not bound- to a DTO so strict validation cannot reject unknown provider fields.+ the global per-IP throttle and bounded on two keys instead: the client IP, checked by its own guard before+ anything else, and `(pluginId, instanceId)`, charged only once the delivery's signature verifies. A+ provider delivering every tenant's webhooks from one egress address would otherwise be shed at the global+ tier before the per-instance bound ever fired. The payload is intentionally not bound to a DTO so strict+ validation cannot reject unknown provider fields.- **Replay and duplication.** A signed-timestamp tolerance rejects stale deliveries, and`(pluginId, instanceId, providerDeliveryId)` deduplication plus a queue job id keyed on the delivery idprovides best-effort de-duplication when the provider supplies a stable delivery id. Standard Webhooks defaultsdiff --git a/src/common/security/proxy-aware-throttler.guard.ts b/src/common/security/proxy-aware-throttler.guard.tsindex bb3d740b4..df15b1a75 100644--- a/src/common/security/proxy-aware-throttler.guard.ts+++ b/src/common/security/proxy-aware-throttler.guard.ts@@ -36,8 +36,8 @@ export class ProxyAwareThrottlerGuard extends ThrottlerGuard {/*** The shared-bucket condition is deployment-wide, so the warning fires once per guard instance.* In practice that is once per process: this guard runs as the singleton APP_GUARD. The only other- * instance is InstanceThrottlerGuard, whose defensive super.getTracker fallback (missing route- * params) is unreachable on the real ingress route, so at most one extra line could ever appear.+ * instance is InstanceThrottlerGuard on the ingress route, which keys on this tracker too, so at+ * most one extra line can appear.*/private warnedSharedProxyBucket = false;@@ -89,11 +89,12 @@ export class ProxyAwareThrottlerGuard extends ThrottlerGuard {* Emit a plain `Retry-After` alongside the library's `Retry-After-<window>`.** The base guard suffixes the header with the tier name (`Retry-After-short` here, and- * `Retry-After-instance` / `Retry-After-ingress-ip` on the ingress route). That is useful for- * telling an operator which bucket shed a request and useless to every HTTP client, none of which- * look for those spellings. A shed request therefore advertised a retry hint nothing could read,- * and a caller that retries a 429 only when the response carries `Retry-After` gave up instead of- * coming back. Written BEFORE delegating, so the suffixed header and the exception are untouched.+ * `Retry-After-ingress-ip` on the ingress route, where IngressService sets `Retry-After-instance`+ * and the plain header itself). That is useful for telling an operator which bucket shed a+ * request and useless to every HTTP client, none of which look for those spellings. A shed+ * request therefore advertised a retry hint nothing could read, and a caller that retries a 429+ * only when the response carries `Retry-After` gave up instead of coming back. Written BEFORE+ * delegating, so the suffixed header and the exception are untouched.** Known ceiling: the first blocked window wins. Tiers are evaluated in order, so the value is the* shortest blocked window rather than the longest. That can cost a caller one wasted retry, but itdiff --git a/src/configure-app.ts b/src/configure-app.tsindex 87b0f138c..26e69da8a 100644--- a/src/configure-app.ts+++ b/src/configure-app.ts@@ -208,10 +208,11 @@ export function configureApp(app: INestApplication, options: ConfigureAppOptionscredentials: corsPolicy.credentials,methods: ['GET', 'POST', 'PUT', 'DELETE', 'PATCH', 'OPTIONS'],allowedHeaders: ['Content-Type', 'X-API-Key', 'Authorization', 'X-Request-ID'],- // The throttlers are named (short/medium/long, plus instance and ingress-ip on ingress), so- // @nestjs/throttler suffixes every rate-limit header with the throttler name. Expose the suffixed- // names so browser clients can actually read them, plus the plain `Retry-After` the guard adds- // on top of them, which is not CORS-safelisted either.+ // The throttlers are named (short/medium/long, plus ingress-ip on ingress), so @nestjs/throttler+ // suffixes every rate-limit header with the throttler name; IngressService emits the per-instance+ // bucket's headers under the same `-instance` suffix. Expose the suffixed names so browser clients+ // can actually read them, plus the plain `Retry-After` (added by the guard, or by IngressService+ // for the instance bucket), which is not CORS-safelisted either.exposedHeaders: ['X-RateLimit-Limit-short','X-RateLimit-Remaining-short',diff --git a/src/modules/integration/ingress-instance-limit.spec.ts b/src/modules/integration/ingress-instance-limit.spec.tsnew file mode 100644index 000000000..5c2283ef6--- /dev/null+++ b/src/modules/integration/ingress-instance-limit.spec.ts@@ -0,0 +1,47 @@+import { ThrottlerStorageService } from '@nestjs/throttler';+import { admitIngressInstance } from './ingress-instance-limit';++describe('admitIngressInstance', () => {+ let storage: ThrottlerStorageService;+ const env = { INGRESS_INSTANCE_LIMIT: '2', INGRESS_INSTANCE_TTL: '60000' };++ beforeEach(() => {+ storage = new ThrottlerStorageService();+ });+ afterEach(() => storage.onApplicationShutdown());++ it('admits up to the limit with rate headers, then sheds with Retry-After', async () => {+ const first = await admitIngressInstance(storage, 'chatwoot', 'acct1', env);+ expect(first).toEqual({+ ok: true,+ headers: {+ 'X-RateLimit-Limit-instance': '2',+ 'X-RateLimit-Remaining-instance': '1',+ 'X-RateLimit-Reset-instance': expect.any(String) as string,+ },+ });+ const second = await admitIngressInstance(storage, 'chatwoot', 'acct1', env);+ expect(second.ok).toBe(true);+ expect(second.headers['X-RateLimit-Remaining-instance']).toBe('0');++ // The block must hold with the in-memory storage too: a block duration of 0 there never blocks.+ const third = await admitIngressInstance(storage, 'chatwoot', 'acct1', env);+ expect(third.ok).toBe(false);+ expect(Number(third.headers['Retry-After'])).toBeGreaterThan(0);+ expect(third.headers['Retry-After-instance']).toBe(third.headers['Retry-After']);+ });++ it('keeps an independent bucket per instance', async () => {+ for (let i = 0; i < 3; i++) await admitIngressInstance(storage, 'chatwoot', 'acct1', env);+ await expect(admitIngressInstance(storage, 'chatwoot', 'acct2', env)).resolves.toMatchObject({ ok: true });+ });++ it('falls back to 120 per 60 s when the variables are blank', async () => {+ const result = await admitIngressInstance(storage, 'chatwoot', 'acct1', {+ INGRESS_INSTANCE_LIMIT: ' ',+ INGRESS_INSTANCE_TTL: '',+ });+ expect(result.headers['X-RateLimit-Limit-instance']).toBe('120');+ expect(Number(result.headers['X-RateLimit-Reset-instance'])).toBeLessThanOrEqual(60);+ });+});diff --git a/src/modules/integration/ingress-instance-limit.ts b/src/modules/integration/ingress-instance-limit.tsnew file mode 100644index 000000000..4eed50357--- /dev/null+++ b/src/modules/integration/ingress-instance-limit.ts@@ -0,0 +1,41 @@+import type { ThrottlerStorage } from '@nestjs/throttler';+import { resolveNonNegativeIntEnv } from '../../config/configuration';++export interface IngressAdmission {+ ok: boolean;+ headers: Record<string, string>;+}++/**+ * The per-instance ingress rate bucket, charged by IngressService only once a delivery has passed+ * signature verification. Providers deliver every tenant's webhooks from one egress IP, so this is+ * what keeps a noisy (pluginId, instanceId) from starving its neighbours. Traffic that fails earlier+ * is bounded by the per-client-IP tier in InstanceThrottlerGuard alone.+ *+ * Header names and values match what the throttler guard emitted for this tier. Every value is a+ * string, because safeAckHeaders drops anything else on the way to the wire.+ */+export async function admitIngressInstance(+ storage: ThrottlerStorage,+ pluginId: string,+ instanceId: string,+ env: NodeJS.ProcessEnv = process.env,+): Promise<IngressAdmission> {+ // Blank-as-unset, like the guard's tiers: a blank compose forward must not become a limit of 0.+ const limit = resolveNonNegativeIntEnv(env.INGRESS_INSTANCE_LIMIT, 120);+ const ttl = resolveNonNegativeIntEnv(env.INGRESS_INSTANCE_TTL, 60000);+ // The block lasts one window, as the guard's `blockDuration || ttl` did; 0 would never block.+ const record = await storage.increment(`ingress-instance:${pluginId}:${instanceId}`, ttl, limit, ttl, 'instance');+ if (record.isBlocked) {+ const wait = String(record.timeToBlockExpire);+ return { ok: false, headers: { 'Retry-After-instance': wait, 'Retry-After': wait } };+ }+ return {+ ok: true,+ headers: {+ 'X-RateLimit-Limit-instance': String(limit),+ 'X-RateLimit-Remaining-instance': String(Math.max(0, limit - record.totalHits)),+ 'X-RateLimit-Reset-instance': String(record.timeToExpire),+ },+ };+}diff --git a/src/modules/integration/ingress-preflight.ts b/src/modules/integration/ingress-preflight.tsindex f4dec9f2b..0101fe1c7 100644--- a/src/modules/integration/ingress-preflight.ts+++ b/src/modules/integration/ingress-preflight.ts@@ -8,7 +8,8 @@ export type PreflightRejection = { status: number; body: string; headers?: Recor* matter re-attempt a 503 only when the header is set and do not parse its value (supabase/auth checks* `Get("retry-after") != ""`). A session that is merely still coming up already passes through to the* normal 202 + enqueue path, so a rejection means no live engine or FAILED, and what bounds a provider- * hammering a dead session is InstanceThrottlerGuard's per-instance bucket, not this number.+ * hammering a dead session is the per-instance bucket IngressService charges after verification, not+ * this number.*/const PREFLIGHT_RETRY_AFTER_SECONDS = 5;diff --git a/src/modules/integration/ingress.controller.ts b/src/modules/integration/ingress.controller.tsindex 43368eaa2..e165e4b6c 100644--- a/src/modules/integration/ingress.controller.ts+++ b/src/modules/integration/ingress.controller.ts@@ -16,8 +16,9 @@ import { InstanceThrottlerGuard } from './instance-throttler.guard';@Public()// The global per-IP throttle SKIPS this route (its medium tier, 100/min by default, sits below the// per-instance limit's 120/min, so a provider delivering every tenant's webhooks from one shared-// egress IP was 429'd at the IP tier before the instance bound ever fired). Fairness here is the-// InstanceThrottlerGuard's job: keyed on (pluginId, instanceId), a noisy tenant sheds alone.+// egress IP was 429'd at the IP tier before the instance bound ever fired). InstanceThrottlerGuard+// bounds each client instead, and IngressService charges a per-(pluginId, instanceId) bucket once a+// delivery's signature verifies, so a noisy tenant sheds alone.@SkipThrottle()@Controller('ingress')export class IngressController {@@ -26,17 +27,14 @@ export class IngressController {// Express 5 (path-to-regexp v8) has no bare `*` — Nest's route converter rewrites it to the named// wildcard `*path`, so the trailing segments land in req.params.path (an array), not req.params[0].//- // InstanceThrottlerGuard carries this route's rate bounds: the global per-IP guard skips this- // controller (@SkipThrottle above) because its medium tier (100/min) sits below this guard's- // per-instance default (120/min), which 429'd every tenant of a shared-egress-IP provider at- // the IP tier before the instance bound ever fired. This guard ignores the bare @SkipThrottle- // (see its shouldSkip) and enforces TWO buckets: one keyed on (pluginId, instanceId), so a noisy- // tenant sheds alone, and one keyed on the client IP, because the first key comes from the path- // the caller supplies and would otherwise leave this @Public route with no bound it cannot walk- // around. Their limits/ttl (INGRESS_INSTANCE_LIMIT / INGRESS_INSTANCE_TTL / INGRESS_IP_LIMIT) are- // read directly by the guard itself, NOT via @Throttle: @Throttle metadata is reflected on the- // route and read by every ThrottlerGuard subclass that walks a tier of that name. See- // InstanceThrottlerGuard's onModuleInit for how it keeps its tiers fully independent.+ // Two rate bounds, neither from the global per-IP guard (skipped via @SkipThrottle above, because+ // its medium tier of 100/min sits below the per-instance default of 120/min and 429'd every+ // tenant of a shared-egress-IP provider). InstanceThrottlerGuard ignores the bare @SkipThrottle+ // (see its shouldSkip) and bounds each client IP before anything else (INGRESS_IP_LIMIT). The+ // per-(pluginId, instanceId) bucket (INGRESS_INSTANCE_LIMIT) is charged by IngressService only+ // after the signature verifies. Both windows use INGRESS_INSTANCE_TTL. The limits are read from the+ // environment, NOT via @Throttle: @Throttle metadata is reflected on the route and read by every+ // ThrottlerGuard subclass that walks a tier of that name.@UseGuards(InstanceThrottlerGuard)@All(':pluginId/:instanceId/*path')// The wildcard segment is part of the published path template, so it needs a parameter of its own —diff --git a/src/modules/integration/ingress.service.spec.ts b/src/modules/integration/ingress.service.spec.tsindex ade6995b0..70ca27a98 100644--- a/src/modules/integration/ingress.service.spec.ts+++ b/src/modules/integration/ingress.service.spec.ts@@ -790,3 +790,101 @@ describe('IngressService.handle — response contract', () => {expect(dup.enqueue).not.toHaveBeenCalled();});});++describe('IngressService per-instance rate bucket', () => {+ const post = (overrides: Record<string, unknown> = {}) => ({+ pluginId: 'chatwoot',+ instanceId: 'acct1',+ route: 'chatwoot',+ method: 'POST',+ headers: { 'x-delivery': 'd1' } as Record<string, string>,+ query: {} as Record<string, string>,+ rawBody: '{}',+ ...overrides,+ });+ const admitted = {+ ok: true,+ headers: { 'X-RateLimit-Limit-instance': '120', 'X-RateLimit-Remaining-instance': '119' },+ };+ const blocked = { ok: false, headers: { 'Retry-After-instance': '7', 'Retry-After': '7' } };++ it.each([+ ['an unknown instance', { instances: { resolve: jest.fn().mockResolvedValue(null) } }, post(), 404],+ ['an unknown route', { manifestRoute: jest.fn().mockReturnValue(undefined) }, post(), 404],+ ['an oversized body', {}, post({ rawBody: 'x'.repeat(2048) }), 413],+ [+ 'a failed signature',+ {+ manifestRoute: jest.fn().mockReturnValue({+ route: 'chatwoot',+ maxBodyBytes: 1024,+ signature: { scheme: 'hmac-sha256', header: 'x-sig' },+ }),+ },+ post({ headers: { 'x-sig': 'sha256=bad' } }),+ 401,+ ],+ [+ 'a GET challenge',+ {+ manifestRoute: jest.fn().mockReturnValue({+ route: 'chatwoot',+ maxBodyBytes: 1024,+ signature: { scheme: 'none' },+ challenge: { tokenParam: 'hub.verify_token', echoParam: 'hub.challenge' },+ }),+ },+ post({ method: 'GET', query: { 'hub.verify_token': 'wrong' } }),+ 403,+ ],+ ])('does not charge the bucket for %s', async (_label, overrides, req, status) => {+ const admitInstance = jest.fn().mockResolvedValue(blocked);+ const svc = new IngressService(deps({ ...overrides, admitInstance }));+ const res = await svc.handle(req);+ expect(res.status).toBe(status);+ expect(admitInstance).not.toHaveBeenCalled();+ });++ it('charges a verified delivery once and sheds it with the throttler 429 when the bucket is full', async () => {+ const admitInstance = jest.fn().mockResolvedValue(blocked);+ const d = deps({ admitInstance });+ const res = await new IngressService(d).handle(post());+ expect(admitInstance).toHaveBeenCalledTimes(1);+ expect(admitInstance).toHaveBeenCalledWith('chatwoot', 'acct1');+ expect(res).toEqual({+ status: 429,+ body: JSON.stringify({ statusCode: 429, message: 'ThrottlerException: Too Many Requests' }),+ headers: { 'content-type': 'application/json', 'Retry-After-instance': '7', 'Retry-After': '7' },+ });+ expect(d.events.recordOrSkip).not.toHaveBeenCalled();+ expect(d.enqueue).not.toHaveBeenCalled();+ });++ it('charges before the session-alive preflight, so a full bucket answers 429 rather than 503', async () => {+ const admitInstance = jest.fn().mockResolvedValue(blocked);+ const svc = new IngressService(+ deps({+ admitInstance,+ sessionStatus: () => undefined,+ manifestRoute: jest.fn().mockReturnValue({+ route: 'chatwoot',+ maxBodyBytes: 1024,+ signature: { scheme: 'none' },+ response: { preflight: ['session-alive'], ack: { status: 200 } },+ }),+ }),+ );+ await expect(svc.handle(post())).resolves.toMatchObject({ status: 429 });+ });++ it.each([+ ['a new delivery', true],+ ['a re-delivery', false],+ ])('adds the rate headers to the ack of %s', async (_label, isNew) => {+ const d = deps({ admitInstance: jest.fn().mockResolvedValue(admitted) });+ d.events.recordOrSkip = jest.fn().mockResolvedValue(isNew);+ const res = await new IngressService(d).handle(post());+ expect(res.status).toBe(202);+ expect(res.headers).toEqual(admitted.headers);+ });+});diff --git a/src/modules/integration/ingress.service.ts b/src/modules/integration/ingress.service.tsindex 9ecd14f16..392680e9f 100644--- a/src/modules/integration/ingress.service.ts+++ b/src/modules/integration/ingress.service.ts@@ -1,10 +1,12 @@import { createHash } from 'node:crypto';+import { throttlerMessage } from '@nestjs/throttler';import { safeEqualStr, verifyIngressSignature } from './ingress-signature';import { PluginIngressRoute } from '../../core/plugins/plugin.interfaces';import { IngressJobData } from '../queue/processors/ingress.processor';import type { EngineStatus } from '../../engine/interfaces/whatsapp-engine.interface';import { evaluatePreflight } from './ingress-preflight';import { renderAck } from './ingress-ack';+import type { IngressAdmission } from './ingress-instance-limit';import { parseBodyLimitBytes } from '../../config/inflight-body-budget';import { resolveBodyLimit } from '../../config/bootstrap-security';import { createLogger } from '../../common/services/logger.service';@@ -58,13 +60,17 @@ export interface IngressDeps {// Optional structured sink for preflight rejections, so operators can audit deliveries that were// rejected host-side (and therefore leave no dedup/DLQ row). Absent in pure unit tests.log?: (event: string, meta: Record<string, unknown>) => void;+ // Optional per-instance rate bucket (admitIngressInstance), charged only for a verified delivery.+ // Absent in pure unit tests: every delivery is admitted and no rate headers are added.+ admitInstance?: (pluginId: string, instanceId: string) => Promise<IngressAdmission>;now: () => number;}/*** The fast-ack ingress pipeline. Pure orchestration over injected deps so it is unit-testable without* Nest DI: resolve the instance → answer a GET challenge host-side → size cap → verify over the RAW- * body → dedup (persist-before-ack) → best-effort conversation id → enqueue (or inline) → 202.+ * body → per-instance rate bucket → dedup (persist-before-ack) → best-effort conversation id →+ * enqueue (or inline) → 202.*/export class IngressService {private readonly logger = createLogger('IngressService');@@ -142,6 +148,20 @@ export class IngressService {});if (!verdict.ok) return { status: 401, body: verdict.reason ?? 'signature verification failed' };+ // The per-instance bucket is charged here, after verification; everything above is bounded by the+ // guard's per-client-IP tier.+ // Before the preflight, so a signed provider hammering a dead session is still bounded.+ const admission = this.deps.admitInstance ? await this.deps.admitInstance(req.pluginId, req.instanceId) : undefined;+ if (admission && !admission.ok) {+ return {+ status: 429,+ // The body the global filter writes for a guard's ThrottlerException, so the wire shape is+ // the same one this route answered with before the charge moved here.+ body: JSON.stringify({ statusCode: 429, message: throttlerMessage }),+ headers: { 'content-type': 'application/json', ...admission.headers },+ };+ }+// Host-side preflight (e.g. session-alive). AFTER signature verify (so an unauthenticated caller// cannot probe liveness) and BEFORE the dedup persist (so a 5xx-rejected delivery never writes a// dedup row that would swallow the provider's retry as a 200 'duplicate' — the dedup trap). A@@ -195,11 +215,12 @@ export class IngressService {// hand (rawBody, deliveryId, now), never stored state, which is what makes rendering it on a dedup// hit sound: an ack field that had to reflect the PERSISTED row would echo the retry's values as the// original's. Keep it that way.- const ack = renderAck(route.response?.ack, {+ const rendered = renderAck(route.response?.ack, {rawBody: req.rawBody,timestamp: String(Math.floor(this.deps.now() / 1000)),id: deliveryId,});+ const ack = admission ? { ...rendered, headers: { ...rendered.headers, ...admission.headers } } : rendered;const isNew = await this.deps.events.recordOrSkip({instanceId: req.instanceId,diff --git a/src/modules/integration/instance-throttler.guard.spec.ts b/src/modules/integration/instance-throttler.guard.spec.tsindex e9e83a896..f0cc7475f 100644--- a/src/modules/integration/instance-throttler.guard.spec.ts+++ b/src/modules/integration/instance-throttler.guard.spec.ts@@ -1,31 +1,23 @@import { InstanceThrottlerGuard } from './instance-throttler.guard';describe('InstanceThrottlerGuard', () => {- it('keys the bucket on (pluginId, instanceId) from the route params, not the client IP', async () => {+ // The per-instance bucket is charged by IngressService after signature verification; this guard+ // keys only on the client IP.+ it('keys on the client IP even when the ingress route params are present', async () => {const guard = Object.create(InstanceThrottlerGuard.prototype) as InstanceThrottlerGuard & {getTracker(req: unknown): Promise<string>;};- const req = { params: { pluginId: 'chatwoot', instanceId: 'acct1' }, ip: '203.0.113.9' };- await expect(guard.getTracker(req)).resolves.toBe('ingress:chatwoot:acct1');- });-- it('falls back to the client IP when params are missing (defensive)', async () => {- const guard = Object.create(InstanceThrottlerGuard.prototype) as InstanceThrottlerGuard & {- getTracker(req: unknown): Promise<string>;- };- await expect(guard.getTracker({ params: {}, ip: '203.0.113.9' })).resolves.toContain('203.0.113.9');- });-- it('falls back to the client IP when only one of pluginId/instanceId is present', async () => {- const guard = Object.create(InstanceThrottlerGuard.prototype) as InstanceThrottlerGuard & {- getTracker(req: unknown): Promise<string>;- };- const req = { params: { pluginId: 'chatwoot' }, ip: '203.0.113.9' };- await expect(guard.getTracker(req)).resolves.toContain('203.0.113.9');+ const tracked = await guard.getTracker({+ params: { pluginId: 'chatwoot', instanceId: 'acct1' },+ ip: '203.0.113.9',+ headers: {},+ });+ expect(tracked).toContain('203.0.113.9');+ expect(tracked).not.toContain('acct1');});// A blank compose `${KEY:-}` forward must never become a limit of 0: the throttler would then- // reject the first hit on every instance, silently 429ing all inbound webhooks.+ // reject the first hit, silently 429ing all inbound webhooks.describe('tier resolution from the environment', () => {const KEYS = ['INGRESS_INSTANCE_LIMIT', 'INGRESS_INSTANCE_TTL', 'INGRESS_IP_LIMIT'] as const;const saved: Array<[string, string | undefined]> = [];@@ -40,7 +32,7 @@ describe('InstanceThrottlerGuard', () => {}});- type Tier = { name: string; limit: number; ttl: number; getTracker?: (req: unknown) => Promise<string> };+ type Tier = { name: string; limit: number; ttl: number; getTracker?: unknown };const resolveTiers = async (): Promise<Tier[]> => {const guard = Object.create(InstanceThrottlerGuard.prototype) as InstanceThrottlerGuard & {@@ -53,50 +45,18 @@ describe('InstanceThrottlerGuard', () => {return guard.throttlers;};- const sizes = (tiers: Tier[]): Array<{ name: string; limit: number; ttl: number }> =>- tiers.map(({ name, limit, ttl }) => ({ name, limit, ttl }));-it.each(['', ' '])('treats a blank value (%p) as unset instead of a limit of 0', async blank => {process.env.INGRESS_INSTANCE_LIMIT = blank;process.env.INGRESS_INSTANCE_TTL = blank;process.env.INGRESS_IP_LIMIT = blank;- expect(sizes(await resolveTiers())).toEqual([- { name: 'instance', limit: 120, ttl: 60000 },- { name: 'ingress-ip', limit: 1200, ttl: 60000 },- ]);+ expect(await resolveTiers()).toEqual([{ name: 'ingress-ip', limit: 1200, ttl: 60000 }]);});- it('honors real values', async () => {+ it('carries only the client-IP tier, sized from the environment', async () => {process.env.INGRESS_INSTANCE_LIMIT = '5';process.env.INGRESS_INSTANCE_TTL = '1000';process.env.INGRESS_IP_LIMIT = '9';- expect(sizes(await resolveTiers())).toEqual([- { name: 'instance', limit: 5, ttl: 1000 },- { name: 'ingress-ip', limit: 9, ttl: 1000 },- ]);- });-- /**- * The point of the second tier: the per-instance bucket is keyed on caller-supplied path- * segments, so varying them mints a fresh bucket per request. The IP tier must stay keyed on the- * client even when those params ARE present, or the route has no bound an unauthenticated caller- * cannot walk around.- */- it('keys the ip tier on the client even when instance params are present', async () => {- const tiers = await resolveTiers();- const ipTier = tiers.find(t => t.name === 'ingress-ip');- expect(ipTier?.getTracker).toBeDefined();- const req = { params: { pluginId: 'chatwoot', instanceId: 'acct1' }, ip: '203.0.113.9' };- const tracked = await ipTier!.getTracker!(req);- expect(tracked).toContain('203.0.113.9');- expect(tracked).not.toContain('acct1');- });-- it('leaves the instance tier keyed on the instance', async () => {- const tiers = await resolveTiers();- const instanceTier = tiers.find(t => t.name === 'instance');- // No per-tier override: it inherits the class getTracker, proven by the first tests above.- expect(instanceTier?.getTracker).toBeUndefined();+ expect(await resolveTiers()).toEqual([{ name: 'ingress-ip', limit: 9, ttl: 1000 }]);});});});diff --git a/src/modules/integration/instance-throttler.guard.ts b/src/modules/integration/instance-throttler.guard.tsindex 2dccc1abb..c4aa04f79 100644--- a/src/modules/integration/instance-throttler.guard.ts+++ b/src/modules/integration/instance-throttler.guard.ts@@ -3,90 +3,48 @@ import { ProxyAwareThrottlerGuard } from '../../common/security/proxy-aware-throimport { resolveNonNegativeIntEnv } from '../../config/configuration';/**- * Rate-limit bucket keyed on the ingress route's (pluginId, instanceId) instead of the client IP.+ * The ingress route's pre-authentication rate bound, keyed on the client IP.*- * Providers (Chatwoot, Meta, etc.) deliver every tenant's webhooks from the SAME egress IP, so the- * global `ProxyAwareThrottlerGuard` — keyed on IP — lumps all instances into one bucket: a noisy- * tenant on one instance would rate-limit every other instance sharing that provider's IP. Keying on- * the instance instead sheds a single misbehaving `(pluginId, instanceId)` at the edge before it fills- * the shared ingress queue, without punishing its neighbors.+ * The route is `@Public`, so this is the only bound on traffic that has not proven anything yet:+ * unknown instances, failed challenges, oversized bodies and bad signatures. The per-instance bound+ * that keeps one noisy tenant from starving the others on a provider's shared egress IP is charged+ * by IngressService only once a delivery's signature verifies (see admitIngressInstance).*- * Applied alongside (not instead of) the global IP guard — see ingress.controller.ts. It deliberately- * does NOT use `@Throttle()` to size its limit: `@Throttle` metadata is reflected on the route/handler- * and is read by EVERY `ThrottlerGuard` instance that walks a tier of that name — including the global- * per-IP guard, which shares the exact same `short`/`medium`/`long` tier list from the one process-wide- * `ThrottlerModule.forRootAsync` config. Overriding one of those tiers here would silently retarget the- * global guard's tolerance for this route too (proven by an earlier version of this guard's e2e test:- * a second, unrelated instance got 429'd purely for sharing the test client's IP with a throttled one).- * Instead, `onModuleInit` below replaces `this.throttlers` with its own self-contained tiers that only- * this guard instance evaluates, so their limits are fully independent of the global guard's tiers.- *- * It carries TWO tiers on the same window: `instance`, keyed on the route params, and `ingress-ip`,- * keyed on the client. The instance key is caller-supplied, so the ip tier is the only bound a caller- * cannot walk around by varying the path. See `onModuleInit` for the sizing.+ * It deliberately does NOT use `@Throttle()` to size its limit: `@Throttle` metadata is reflected on+ * the route/handler and is read by EVERY `ThrottlerGuard` instance that walks a tier of that name,+ * including the global per-IP guard, which shares the `short`/`medium`/`long` tier list from the one+ * process-wide `ThrottlerModule.forRootAsync` config. Overriding one of those tiers here would silently+ * retarget the global guard's tolerance for this route too. Instead, `onModuleInit` below replaces+ * `this.throttlers` with its own tier that only this guard instance evaluates.*/@Injectable()export class InstanceThrottlerGuard extends ProxyAwareThrottlerGuard {async onModuleInit(): Promise<void> {await super.onModuleInit();- // `??` only defaults on undefined, so a blank compose `${KEY:-}` forward used to reach- // `Number('')` === 0 — a limit of 0 rejects the very first hit, silently 429ing EVERY inbound- // webhook. Boot validation cannot catch it either (env.validation treats a blank value as- // unset). resolveNonNegativeIntEnv treats blank as unset and only accepts plain decimals; an- // explicit 0 is still rejected at boot by the positive-int check on INGRESS_INSTANCE_LIMIT.- const ttl = resolveNonNegativeIntEnv(process.env.INGRESS_INSTANCE_TTL, 60000);+ // Blank-as-unset: a blank compose `${KEY:-}` forward must not become a limit of 0, which rejects+ // the very first hit. An explicit 0 is still rejected at boot by env.validation.this.throttlers = [{- name: 'instance',- limit: resolveNonNegativeIntEnv(process.env.INGRESS_INSTANCE_LIMIT, 120),- ttl,- },- {- // Second bucket, same window, keyed on the CLIENT rather than the route params. The instance- // bucket alone cannot bound this route: its key comes from `:pluginId/:instanceId`, which the- // caller supplies, so varying them mints a fresh bucket per request. An unauthenticated- // caller (the route is `@Public`) therefore walks around the limit entirely, and grows the- // throttler's key space while doing it. Sized well ABOVE the per-instance limit so it never- // becomes the binding constraint for a legitimate provider fanning many tenants through one- // egress IP (the case the instance bucket exists for): 10x the instance default. Raise it- // with `INGRESS_IP_LIMIT` when one IP legitimately drives more than that.+ // Sized well ABOVE the per-instance limit so it never becomes the binding constraint for a+ // legitimate provider fanning many tenants through one egress IP: 10x the instance default.+ // Raise it with `INGRESS_IP_LIMIT` when one IP legitimately drives more than that. The tier+ // keeps the inherited proxy-aware client-IP tracker.name: 'ingress-ip',limit: resolveNonNegativeIntEnv(process.env.INGRESS_IP_LIMIT, 1200),- ttl,- getTracker: req => this.trackClientIp(req),+ ttl: resolveNonNegativeIntEnv(process.env.INGRESS_INSTANCE_TTL, 60000),},];}- /**- * The inherited proxy-aware, IP-keyed tracker, reachable from the tier list above (an arrow- * function there cannot say `super`). `getTracker` below is overridden for the instance tier, so- * the ip tier has to reach past that override deliberately.- */- private trackClientIp(req: Record<string, unknown>): Promise<string> {- return super.getTracker(req);- }-/*** This guard does NOT honour a bare `@SkipThrottle()`. The ingress controller carries one so the- * GLOBAL per-IP guard skips the route: its medium tier (default 100/min) sits BELOW this guard's- * per-instance default (120/min), so a provider delivering every tenant's webhooks from one- * shared egress IP - the exact scenario this guard exists for - was 429'd at the IP tier before- * the instance bound ever fired, and sustained traffic hit the 1000/h long tier at ~16/min. This- * guard carries the route's own better-keyed limits instead, and both of its tiers stay- * unconditional: skipping them would leave a `@Public` route with no rate bound at all.+ * GLOBAL per-IP guard skips the route: its medium tier (default 100/min) sits BELOW the+ * per-instance default (120/min), so a provider delivering every tenant's webhooks from one shared+ * egress IP was 429'd at the IP tier before the instance bound ever fired. This guard carries the+ * route's own limit instead, and its tier stays unconditional: skipping it would leave a `@Public`+ * route with no rate bound at all.*/protected shouldSkip(): Promise<boolean> {return Promise.resolve(false);}-- protected async getTracker(req: Record<string, unknown>): Promise<string> {- const params = (req.params ?? {}) as { pluginId?: string; instanceId?: string };- if (params.pluginId && params.instanceId) {- return `ingress:${params.pluginId}:${params.instanceId}`;- }- // Defensive fallback: params should always be present on the ingress route, but if this guard- // is ever reused elsewhere (or Nest fails to populate params), don't silently share one bucket.- return super.getTracker(req);- }}diff --git a/src/modules/integration/integration.module.ts b/src/modules/integration/integration.module.tsindex 451eefc81..610484692 100644--- a/src/modules/integration/integration.module.ts+++ b/src/modules/integration/integration.module.ts@@ -1,6 +1,7 @@import { Module, DynamicModule, Type } from '@nestjs/common';import { TypeOrmModule, getRepositoryToken } from '@nestjs/typeorm';import { Repository } from 'typeorm';+import { ThrottlerStorage } from '@nestjs/throttler';import { PluginInstance } from './entities/plugin-instance.entity';import { IngressEvent } from './entities/ingress-event.entity';import { IntegrationDeliveryFailure } from './entities/integration-delivery-failure.entity';@@ -8,6 +9,7 @@ import { PluginInstanceService } from './plugin-instance.service';import { IngressEventService } from './ingress-event.service';import { IngressService, IngressRouteDescriptor } from './ingress.service';import { IngressController } from './ingress.controller';+import { admitIngressInstance } from './ingress-instance-limit';import { IngressEnqueueService, buildIngressDeadLetterRow } from './ingress-enqueue.service';import { IngressReconcilerService } from './ingress-reconciler.service';import { RedriveService } from './redrive.service';@@ -69,6 +71,7 @@ if (process.env.QUEUE_ENABLED === 'true') {IngressEnqueueService,getRepositoryToken(IntegrationDeliveryFailure, 'data'),EngineRegistry,+ ThrottlerStorage,],useFactory: (instances: PluginInstanceService,@@ -77,6 +80,7 @@ if (process.env.QUEUE_ENABLED === 'true') {ingressEnqueue: IngressEnqueueService,failures: Repository<IntegrationDeliveryFailure>,registry: EngineRegistry,+ throttlerStorage: ThrottlerStorage,) => {const dlqLogger = createLogger('IngressEnqueue');const ingressLogger = createLogger('Ingress');@@ -92,6 +96,8 @@ if (process.env.QUEUE_ENABLED === 'true') {// Audit sink for preflight rejections (they leave no dedup/DLQ row). The Prometheus counter is// a separate follow-up (see plan notes); the structured log is the MVP audit surface.log: (event, meta) => ingressLogger.warn(event, meta),+ // The app's global throttler storage (Redis when enabled), so the bucket is shared across nodes.+ admitInstance: (pluginId, instanceId) => admitIngressInstance(throttlerStorage, pluginId, instanceId),// Live ingress delivery: record the dispatch outcome on the event row (so the reconciler// can tell a stranded 'pending' row from one that reached the dispatch tier) and, on a// swallowed inline-dispatch failure, persist a dead-letter row so RedriveService can replaydiff --git a/test/ingress-instance-throttle.e2e-spec.ts b/test/ingress-instance-throttle.e2e-spec.tsindex b63f98ac3..ab35affc5 100644--- a/test/ingress-instance-throttle.e2e-spec.ts+++ b/test/ingress-instance-throttle.e2e-spec.ts@@ -2,9 +2,8 @@// AppModule boots; stub it so ts-jest (CommonJS) can load the module graph.jest.mock('archiver', () => ({ TarArchive: jest.fn() }));-// Set BEFORE AppModule is imported: InstanceThrottlerGuard reads these directly in its onModuleInit-// (a lifecycle hook that runs once at boot), not per-request, so they must be in place before the-// Nest testing module is compiled below.+// IngressService reads these per delivery; set them before AppModule is imported anyway, since the+// ingress guard sizes its own tier from INGRESS_INSTANCE_TTL once at boot.process.env.INGRESS_INSTANCE_LIMIT = '3';process.env.INGRESS_INSTANCE_TTL = '60000';@@ -24,11 +23,11 @@ import { PluginLoaderService } from './../src/core/plugins/plugin-loader.serviceimport { PluginInstance } from './../src/modules/integration/entities/plugin-instance.entity';/**- * Deterministic proof that InstanceThrottlerGuard actually enforces its per-route limit, and that two- * different instances get independent buckets (a noisy tenant doesn't starve its neighbor). Uses its- * own low INGRESS_INSTANCE_LIMIT (3) set before AppModule boots, rather than the shared e2e app in- * integration-fabric.e2e-spec.ts, so hammering past the limit is fast and doesn't perturb the golden- * 202/401 coverage there (which relies on the default limit of 120 never being hit).+ * Deterministic proof that the per-instance limit is enforced, that two different instances get+ * independent buckets (a noisy tenant doesn't starve its neighbor), and that only verified deliveries+ * are counted. Uses its own low INGRESS_INSTANCE_LIMIT (3) set before AppModule boots, rather than the+ * shared e2e app in integration-fabric.e2e-spec.ts, so hammering past the limit is fast and doesn't+ * perturb the golden 202/401 coverage there (which relies on the default limit of 120 never being hit).*/describe('Ingress per-instance fairness throttle (e2e)', () => {let app: INestApplication<App>;@@ -92,7 +91,7 @@ describe('Ingress per-instance fairness throttle (e2e)', () => {});jest.spyOn(loader, 'dispatchWebhookForInstance').mockResolvedValue(undefined);- for (const instanceId of ['acct1', 'acct2']) {+ for (const instanceId of ['acct1', 'acct2', 'acct3']) {await instanceRepo.save(instanceRepo.create({id: `chatwoot:${instanceId}`,@@ -133,10 +132,8 @@ describe('Ingress per-instance fairness throttle (e2e)', () => {const blocked = await send('acct1', 'acct1-overflow');expect(blocked.status).toBe(429);- // ThrottlerGuard runs as a guard, before the handler, so it throws ThrottlerException before the- // controller's @Res() is ever reached — the header is set on the real response object directly- // from the guard layer and the global exception filter formats the 429 body. Confirm both survive- // the @Res() route (not just the status code).+ // IngressService answers the 429 itself, through the controller's @Res(): confirm the headers and+ // the throttler's JSON body both reach the wire, not just the status code.expect(blocked.headers['retry-after-instance']).toBeDefined();// The suffixed name says WHICH bucket shed it; the plain one is the only spelling a client reads.expect(blocked.headers['retry-after']).toBe(blocked.headers['retry-after-instance']);@@ -147,4 +144,19 @@ describe('Ingress per-instance fairness throttle (e2e)', () => {const other = await send('acct2', 'acct2-0');expect(other.status).toBe(202);});++ it('does not count unsigned requests against the instance limit', async () => {+ // More unsigned requests than INGRESS_INSTANCE_LIMIT: each is rejected on its signature, never shed.+ for (let i = 0; i < 5; i++) {+ const res = await request(app.getHttpServer())+ .post('/api/ingress/chatwoot/acct3/chatwoot')+ .set('X-Chatwoot-Delivery', `acct3-unsigned-${i}`)+ .set('Content-Type', 'application/json')+ .send(raw);+ expect(res.status).toBe(401);+ }+ const signed = await send('acct3', 'acct3-0');+ expect(signed.status).toBe(202);+ expect(signed.headers['x-ratelimit-remaining-instance']).toBe('2');+ });});diff --git a/test/ingress-ip-throttle.e2e-spec.ts b/test/ingress-ip-throttle.e2e-spec.tsindex 5ca354f2f..c4b10221c 100644--- a/test/ingress-ip-throttle.e2e-spec.ts+++ b/test/ingress-ip-throttle.e2e-spec.ts@@ -3,9 +3,8 @@jest.mock('archiver', () => ({ TarArchive: jest.fn() }));// Set BEFORE AppModule is imported: InstanceThrottlerGuard reads these in onModuleInit (once, at-// boot), not per request. A low ip limit with the per-instance limit left at its default isolates-// the tier under test: the instance bucket can never bind here, because every request below uses a-// different (pluginId, instanceId) pair and therefore its own instance bucket.+// boot), not per request. The per-instance limit is left at its default and never binds here: it is+// charged only after signature verification, and every request below names an unknown instance.process.env.INGRESS_IP_LIMIT = '4';process.env.INGRESS_INSTANCE_LIMIT = '120';process.env.INGRESS_INSTANCE_TTL = '60000';@@ -20,10 +19,9 @@ import { applyGlobalValidation } from './../src/config/app-validation';/*** The ingress route is `@Public` and carries `@SkipThrottle()` so the global per-IP tiers leave it* alone (their 100/min sits below the per-instance 120/min and would 429 a shared-egress provider- * before the instance bound ever fired). That left the per-instance bucket as the only limiter, and- * its key is built from the caller-supplied `:pluginId/:instanceId` path segments, so walking those- * segments minted a fresh bucket per request: an unauthenticated client had no effective bound at- * all. This proves the client-keyed tier now binds regardless of the path.+ * before the instance bound ever fired). The per-instance bucket is charged only once a delivery's+ * signature verifies, so the client-keyed tier of the ingress guard is the only bound on+ * unauthenticated traffic. This proves it binds regardless of the path.** The pairs below name no registered instance, so each request 404s at the handler. That is the* point: the guard runs BEFORE the handler, so a 404 still consumes budget, and an unknown-instance