Skip to content

Free, Open Source, Self-Hosted WhatsApp API Gateway

fix(ingress): carry Retry-After on a retryable rejection (#1642)

Yudhi Armyndharis4f5605f7d471
diff --git a/CHANGELOG.md b/CHANGELOG.md
index b8b762776..9a2f1b4f3 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -14,6 +14,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
+- The `session-alive` preflight's `503` carries a `Retry-After`, so a provider that retries a 503 only when that header is present comes back instead of failing the call; the rejection writes no dedup row, so the retry is treated as a new delivery ([#1639](https://github.com/rmyndharis/OpenWA/issues/1639)).
+- A `429` from a rate-limit window carries a plain `Retry-After` in seconds alongside the existing `Retry-After-<window>`, which no HTTP client reads. The suffixed names stay, since they are what identify which window shed the request ([#1639](https://github.com/rmyndharis/OpenWA/issues/1639)).
- An ingress route's declared ack `content-type` reaches the provider instead of being overwritten with `text/plain`, so a provider that requires `application/json` on a 200 or 202 accepts the ack; a type a browser could execute is still forced to `text/plain` ([#1637](https://github.com/rmyndharis/OpenWA/issues/1637)). Thanks @wesamdev for the report.
- Restoring a data archive into PostgreSQL from a gateway that does not run in UTC no longer shifts every timestamp by the host offset, and no longer shifts it again on each further restore ([#1624](https://github.com/rmyndharis/OpenWA/issues/1624)).
- Retention sweeps on PostgreSQL delete the rows their window names instead of taking up to the host's UTC offset of younger rows with them, and the `today` message counts cover the host's local day.
@@ -31,6 +33,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Upgrade notes (behavior changes)
+- A client that honours `Retry-After` now sees one on every rate-limit `429`, where before it saw only the suffixed `Retry-After-<window>` and typically ignored it. When the `long` window sheds a request that value can be up to an hour, so a client that sleeps for it will now wait rather than retry immediately.
- PostgreSQL deployments: the data connection issues `SET TIME ZONE 'UTC'` per connection and verifies the result at boot. A deployment where that cannot hold (a pooler that drops session state) now fails to start, naming the effective zone; set the default instead with `ALTER DATABASE "<database>" SET TimeZone='UTC'`. SQLite deployments, and any gateway already running in UTC, are unaffected and no data moves.
- PostgreSQL deployments whose **gateway** ran outside UTC before this release: the twelve columns the app writes itself hold that host's local wall time and now read as UTC, so they appear shifted by the offset. They are `sessions.connectedAt`, `sessions.lastActiveAt`, `sessions.claimedAt`, `sessions.leaseExpiresAt`, `webhooks.lastTriggeredAt`, `webhook_outbox_events.lastAttemptAt`, `ingress_events.lastDispatchAt`, `message_batches.started_at`, `message_batches.completed_at`, `lid_mappings.updatedAt`, `chat_states.updatedAt` and `baileys_stored_messages.createdAt`. The last three carry a `DEFAULT now()` that never fires, because their only writer passes the value. With the gateway stopped, convert each with the host's old zone, which resolves daylight saving per row: `UPDATE sessions SET "connectedAt" = ("connectedAt" AT TIME ZONE 'Asia/Jakarta') AT TIME ZONE 'UTC' WHERE "connectedAt" IS NOT NULL;`. `claimedAt` and `leaseExpiresAt` are cluster runtime state: clear them, do not convert them, with every node stopped and before the first start on this release, or the lease reads shifted by the old offset and the session is unusable until it lapses. East of UTC it reads hours into the future, so every node treats the session as held elsewhere and `POST /sessions/{id}/start` answers `409`; west of UTC it reads already lapsed, so a peer can adopt a session that is still running. `UPDATE sessions SET "nodeId" = NULL, "claimedAt" = NULL, "leaseExpiresAt" = NULL, "nodeUrl" = NULL;`. Leaving `lid_mappings.updatedAt` and `chat_states.updatedAt` unconverted also mis-ranks the boot preload of both caches, which orders by that column under a cap.
- The remaining eighteen `createdAt`/`updatedAt` columns are written by PostgreSQL itself (`DEFAULT now()`) in the **server's** zone, not the gateway's. On a UTC server, which is the image default and what the bundled Compose file starts, they are already correct and must not be converted; convert them only if the server itself ran outside UTC, with the server's old zone.
diff --git a/docs/04-security-design.md b/docs/04-security-design.md
index 771e218da..f8934b3d9 100644
--- a/docs/04-security-design.md
+++ b/docs/04-security-design.md
@@ -271,19 +271,19 @@ TTL values are in milliseconds. The `/api/metrics` and `/api/health*` routes are
### Response on limit
-Exceeding a window returns `429 Too Many Requests`. Because the windows are **named** throttlers (`short` / `medium` / `long`), `@nestjs/throttler` suffixes every rate-limit header with the throttler name — there are no unsuffixed `Retry-After` or `X-RateLimit-*` headers:
+Exceeding a window returns `429 Too Many Requests`. Because the windows are **named** throttlers (`short` / `medium` / `long`), `@nestjs/throttler` suffixes every rate-limit header with the throttler name, so there are no unsuffixed `X-RateLimit-*` headers. `Retry-After` is the exception: on a `429` the guard emits the plain spelling too, because no HTTP client reads a suffixed one.
- On success, each response carries one header triple per window: `X-RateLimit-Limit-short` / `-Remaining-short` / `-Reset-short`, plus the `-medium` and `-long` equivalents.
-- On `429`, the exceeded window sets `Retry-After-short`, `Retry-After-medium`, or `Retry-After-long` (seconds until the block expires) — read whichever is present rather than a plain `Retry-After`.
+- On `429`, the exceeded window sets `Retry-After-short`, `Retry-After-medium`, or `Retry-After-long` (seconds until the block expires), and the guard mirrors that value into a plain `Retry-After`. Read the plain one; the suffixed name is what tells an operator which window shed the request.
The ingress route (`ALL /api/ingress/:pluginId/:instanceId/*path`) is exempt from the global per-IP tiers (their 100/min medium window sits below the per-instance limit, so a provider fanning every tenant's webhooks through one egress IP was shed before the per-instance bound could fire). It carries its own two windows instead, both on `INGRESS_INSTANCE_TTL` (default 60000 ms):
- `instance`, keyed on `(pluginId, instanceId)`, env `INGRESS_INSTANCE_LIMIT`, default 120. Sheds one noisy tenant without touching its neighbours.
- `ingress-ip`, keyed on the client (proxy-aware, see `TRUSTED_PROXIES`), env `INGRESS_IP_LIMIT`, default 1200. The `instance` key is built from path segments the caller supplies, so varying them mints a fresh bucket; this window is the bound an unauthenticated caller cannot walk around. It is sized 10x the per-instance default so it never binds first for legitimate traffic.
-Responses therefore carry `X-RateLimit-*-instance` and `X-RateLimit-*-ingress-ip`, and on saturation the `Retry-After-*` of whichever window shed the request.
+Responses therefore carry `X-RateLimit-*-instance` and `X-RateLimit-*-ingress-ip`, and on saturation the `Retry-After-*` of whichever window shed the request, mirrored into a plain `Retry-After`.
-The API exposes the rate-limit headers via CORS (`exposedHeaders`) so browser clients can read them. The simplest backpressure signal remains the `429` status itself, with the suffixed `Retry-After-*` as the retry delay.
+The API exposes the rate-limit headers via CORS (`exposedHeaders`) so browser clients can read them, the plain `Retry-After` included. The simplest backpressure signal remains the `429` status itself, with `Retry-After` as the retry delay.
### WebSocket (`/events`) limits
diff --git a/docs/25-integration-fabric.md b/docs/25-integration-fabric.md
index aa7eb907c..c1f0b69b7 100644
--- a/docs/25-integration-fabric.md
+++ b/docs/25-integration-fabric.md
@@ -101,7 +101,8 @@ Alongside this async pipeline, a route may additionally declare a `response` con
declare a host-side `response` contract that shapes that synchronous reply without making the plugin
inline. Its `preflight` checks (today: `session-alive`) run **after** signature verification and
**before** the dedup persist — returning `503` only for a definitively-dead concrete-scoped WhatsApp
- session (no live engine or `FAILED`); recoverable statuses and `READY` pass through to a normal
+ session (no live engine or `FAILED`), with a `Retry-After` so a provider that retries a 503 only when
+ that header is present comes back; recoverable statuses and `READY` pass through to a normal
`202`+enqueue so the worker can still fail fast and the dedup row still holds the delivery. A declared
`ack` (`status`/`body`/`headers`) replaces the default `202 accepted`. For a route declaring `response`,
the ack is returned without awaiting enqueue so a queue-disabled deployment cannot block the provider's
diff --git a/src/common/security/proxy-aware-throttler.guard.spec.ts b/src/common/security/proxy-aware-throttler.guard.spec.ts
index 3fd14cdb3..e9a0a6bb9 100644
--- a/src/common/security/proxy-aware-throttler.guard.spec.ts
+++ b/src/common/security/proxy-aware-throttler.guard.spec.ts
@@ -176,3 +176,56 @@ describe('ProxyAwareThrottlerGuard.shouldSkip', () => {
expect(await skip()).toBe(false);
});
});
+
+/**
+ * The library writes only `Retry-After-<tier>`, a spelling no HTTP client reads, so a shed request
+ * advertised a retry hint nothing could act on. The plain header must go out with the same value.
+ */
+describe('ProxyAwareThrottlerGuard.throwThrottlingException', () => {
+ const invoke = async (
+ setHeaders: boolean | undefined,
+ ): Promise<{ headers: Record<string, string>; threw: boolean }> => {
+ const headers: Record<string, string> = {};
+ const guard = Object.create(ProxyAwareThrottlerGuard.prototype) as ProxyAwareThrottlerGuard;
+ Object.assign(guard, { commonOptions: { setHeaders }, errorMessage: 'ThrottlerException: Too Many Requests' });
+ (guard as unknown as { getRequestResponse(c: unknown): unknown }).getRequestResponse = () => ({
+ req: {},
+ res: { header: (name: string, value: string) => void (headers[name] = value) },
+ });
+ let threw = false;
+ try {
+ await (
+ guard as unknown as {
+ throwThrottlingException(c: unknown, d: unknown): Promise<void>;
+ }
+ ).throwThrottlingException(
+ {},
+ {
+ limit: 10,
+ ttl: 60,
+ key: 'k',
+ tracker: '1.2.3.4',
+ totalHits: 11,
+ timeToExpire: 42,
+ isBlocked: true,
+ timeToBlockExpire: 37,
+ },
+ );
+ } catch {
+ threw = true;
+ }
+ return { headers, threw };
+ };
+
+ it('emits a plain Retry-After carrying the blocked window, and still throws', async () => {
+ const { headers, threw } = await invoke(undefined);
+ expect(headers['Retry-After']).toBe('37');
+ expect(threw).toBe(true);
+ });
+
+ it('respects setHeaders: false, matching the flag the base guard gates its own header on', async () => {
+ const { headers, threw } = await invoke(false);
+ expect(headers['Retry-After']).toBeUndefined();
+ expect(threw).toBe(true);
+ });
+});
diff --git a/src/common/security/proxy-aware-throttler.guard.ts b/src/common/security/proxy-aware-throttler.guard.ts
index 615d81685..166f6c76c 100644
--- a/src/common/security/proxy-aware-throttler.guard.ts
+++ b/src/common/security/proxy-aware-throttler.guard.ts
@@ -1,5 +1,5 @@
import { ExecutionContext, Injectable } from '@nestjs/common';
-import { ThrottlerGuard } from '@nestjs/throttler';
+import { ThrottlerGuard, ThrottlerLimitDetail } from '@nestjs/throttler';
import { resolveClientIp, RequestLike } from '../utils/ip';
import { createLogger } from '../services/logger.service';
@@ -78,4 +78,35 @@ export class ProxyAwareThrottlerGuard extends ThrottlerGuard {
]);
return Promise.resolve(skip === true);
}
+
+ /**
+ * Emit a plain `Retry-After` alongside the library's `Retry-After-<window>`.
+ *
+ * The base guard suffixes the header with the tier name (`Retry-After-short` here, and
+ * `Retry-After-instance` / `Retry-After-ingress-ip` on the ingress route). That is useful for
+ * telling an operator which bucket shed a request and useless to every HTTP client, none of which
+ * look for those spellings. A shed request therefore advertised a retry hint nothing could read,
+ * and a caller that retries a 429 only when the response carries `Retry-After` gave up instead of
+ * coming back. Written BEFORE delegating, so the suffixed header and the exception are untouched.
+ *
+ * Known ceiling: the first blocked window wins. Tiers are evaluated in order, so the value is the
+ * shortest blocked window rather than the longest. That can cost a caller one wasted retry, but it
+ * never advertises a wait shorter than the window that actually answered. Reporting the maximum
+ * would mean evaluating every tier before answering, which is a much larger change.
+ */
+ protected async throwThrottlingException(
+ context: ExecutionContext,
+ throttlerLimitDetail: ThrottlerLimitDetail,
+ ): Promise<void> {
+ // The same flag the base guard gates its own header on. A per-tier `setHeaders` is not visible
+ // from here; this application configures neither, so both default to true.
+ if (this.commonOptions.setHeaders ?? true) {
+ const { res } = this.getRequestResponse(context);
+ (res as { header: (name: string, value: string) => void }).header(
+ 'Retry-After',
+ String(throttlerLimitDetail.timeToBlockExpire),
+ );
+ }
+ await super.throwThrottlingException(context, throttlerLimitDetail);
+ }
}
diff --git a/src/configure-app.ts b/src/configure-app.ts
index 1b0d29093..87b0f138c 100644
--- a/src/configure-app.ts
+++ b/src/configure-app.ts
@@ -208,9 +208,10 @@ export function configureApp(app: INestApplication, options: ConfigureAppOptions
credentials: corsPolicy.credentials,
methods: ['GET', 'POST', 'PUT', 'DELETE', 'PATCH', 'OPTIONS'],
allowedHeaders: ['Content-Type', 'X-API-Key', 'Authorization', 'X-Request-ID'],
- // The throttlers are named (short/medium/long, plus instance on ingress), so @nestjs/throttler
- // suffixes every rate-limit header with the throttler name — the unsuffixed variants are never
- // sent. Expose the suffixed names so browser clients can actually read them.
+ // The throttlers are named (short/medium/long, plus instance and ingress-ip on ingress), so
+ // @nestjs/throttler suffixes every rate-limit header with the throttler name. Expose the suffixed
+ // names so browser clients can actually read them, plus the plain `Retry-After` the guard adds
+ // on top of them, which is not CORS-safelisted either.
exposedHeaders: [
'X-RateLimit-Limit-short',
'X-RateLimit-Remaining-short',
@@ -227,6 +228,7 @@ export function configureApp(app: INestApplication, options: ConfigureAppOptions
'X-RateLimit-Limit-ingress-ip',
'X-RateLimit-Remaining-ingress-ip',
'X-RateLimit-Reset-ingress-ip',
+ 'Retry-After',
'Retry-After-short',
'Retry-After-medium',
'Retry-After-long',
diff --git a/src/modules/integration/ingress-preflight.spec.ts b/src/modules/integration/ingress-preflight.spec.ts
index a480c29f7..43239998e 100644
--- a/src/modules/integration/ingress-preflight.spec.ts
+++ b/src/modules/integration/ingress-preflight.spec.ts
@@ -41,6 +41,7 @@ describe('evaluatePreflight', () => {
).toEqual({
status: 503,
body: 'session not ready',
+ headers: { 'Retry-After': '5' },
});
});
@@ -48,6 +49,7 @@ describe('evaluatePreflight', () => {
expect(evaluatePreflight(route({ preflight: [{ type: 'session-alive' }] }), 'sess-1', () => undefined)).toEqual({
status: 503,
body: 'session not ready',
+ headers: { 'Retry-After': '5' },
});
});
diff --git a/src/modules/integration/ingress-preflight.ts b/src/modules/integration/ingress-preflight.ts
index 2c3fd1061..f4dec9f2b 100644
--- a/src/modules/integration/ingress-preflight.ts
+++ b/src/modules/integration/ingress-preflight.ts
@@ -1,7 +1,16 @@
import { EngineStatus } from '../../engine/interfaces/whatsapp-engine.interface';
import type { IngressRouteDescriptor } from './ingress.service';
-export type PreflightRejection = { status: number; body: string };
+export type PreflightRejection = { status: number; body: string; headers?: Record<string, string> };
+
+/**
+ * Retry-After on the session-alive 503. Presence is what buys the retry: the provider dispatchers that
+ * matter re-attempt a 503 only when the header is set and do not parse its value (supabase/auth checks
+ * `Get("retry-after") != ""`). A session that is merely still coming up already passes through to the
+ * normal 202 + enqueue path, so a rejection means no live engine or FAILED, and what bounds a provider
+ * hammering a dead session is InstanceThrottlerGuard's per-instance bucket, not this number.
+ */
+const PREFLIGHT_RETRY_AFTER_SECONDS = 5;
/**
* Evaluates a route's host-side preflight checks. Returns null to PASS (proceed to dedup + ack + enqueue),
@@ -26,7 +35,13 @@ export function evaluatePreflight(
if (!sessionStatus) continue; // unwired (pure unit mode): skip rather than false-reject
const status = sessionStatus(sessionScope);
if (status === undefined || status === EngineStatus.FAILED) {
- return { status: 503, body: 'session not ready' };
+ // Attached at the return site, not to every rejection: a future non-retryable check must not
+ // inherit a header that tells the provider to come back.
+ return {
+ status: 503,
+ body: 'session not ready',
+ headers: { 'Retry-After': String(PREFLIGHT_RETRY_AFTER_SECONDS) },
+ };
}
}
}
diff --git a/src/modules/integration/ingress.service.spec.ts b/src/modules/integration/ingress.service.spec.ts
index b7e6158e4..d301dae9f 100644
--- a/src/modules/integration/ingress.service.spec.ts
+++ b/src/modules/integration/ingress.service.spec.ts
@@ -567,6 +567,9 @@ describe('IngressService.handle — response contract', () => {
});
const res = await new IngressService(d).handle(baseReq);
expect(res.status).toBe(503);
+ // The provider retries a 503 only when it carries Retry-After, so re-packing the rejection into
+ // {status, body} silently turned a retryable rejection into a lost delivery.
+ expect(res.headers).toEqual({ 'Retry-After': '5' });
expect(d.events.recordOrSkip).not.toHaveBeenCalled();
expect(d.enqueue).not.toHaveBeenCalled();
expect(d.log).toHaveBeenCalledWith(
diff --git a/src/modules/integration/ingress.service.ts b/src/modules/integration/ingress.service.ts
index a048f3257..169fc446b 100644
--- a/src/modules/integration/ingress.service.ts
+++ b/src/modules/integration/ingress.service.ts
@@ -155,7 +155,9 @@ export class IngressService {
status: preflight.status,
sessionScope: instance.sessionScope,
});
- return { status: preflight.status, body: preflight.body };
+ // Returned whole, so a rejection's headers reach the wire. Re-packing the two fields dropped the
+ // Retry-After that decides whether the provider retries at all.
+ return preflight;
}
// Standard Webhooks signs and requires webhook-id, so it is the stable, authenticated retry id.
diff --git a/test/ingress-instance-throttle.e2e-spec.ts b/test/ingress-instance-throttle.e2e-spec.ts
index 4cbdc1322..b63f98ac3 100644
--- a/test/ingress-instance-throttle.e2e-spec.ts
+++ b/test/ingress-instance-throttle.e2e-spec.ts
@@ -138,6 +138,8 @@ describe('Ingress per-instance fairness throttle (e2e)', () => {
// from the guard layer and the global exception filter formats the 429 body. Confirm both survive
// the @Res() route (not just the status code).
expect(blocked.headers['retry-after-instance']).toBeDefined();
+ // The suffixed name says WHICH bucket shed it; the plain one is the only spelling a client reads.
+ expect(blocked.headers['retry-after']).toBe(blocked.headers['retry-after-instance']);
expect(blocked.body).toMatchObject({ statusCode: 429 });
// A different (pluginId, instanceId) shares the provider's egress IP but gets an independent
diff --git a/test/ingress-ip-throttle.e2e-spec.ts b/test/ingress-ip-throttle.e2e-spec.ts
index 7b522757e..5ca354f2f 100644
--- a/test/ingress-ip-throttle.e2e-spec.ts
+++ b/test/ingress-ip-throttle.e2e-spec.ts
@@ -62,6 +62,8 @@ describe('Ingress per-IP rate bound (e2e)', () => {
expect(shed.status).toBe(429);
// Names the tier that shed it, so an operator can tell this from the per-instance bound.
expect(shed.headers['retry-after-ingress-ip']).toBeDefined();
+ // The suffixed name says WHICH bucket shed it; the plain one is the only spelling a client reads.
+ expect(shed.headers['retry-after']).toBe(shed.headers['retry-after-ingress-ip']);
expect(shed.body).toMatchObject({ statusCode: 429 });
});
});