Free, Open Source, Self-Hosted WhatsApp API Gateway
TypeScript apibotgatewayself-hostedwhatsappwhatsapp-apiwhatsapp-automationwhatsapp-botwhatsapp-gatewaywhatsapp-gateway-apiwhatsapp-webwhatsapp-web-api
fix(ingress): answer a re-delivery with the route's declared ack (#1644)
Yudhi Armyndharis1a568633ce9c
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 9a2f1b4f3..031697b68 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -14,6 +14,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Fixed+- A re-delivery of an already-persisted ingress event is answered with the route's declared ack, the same response the first delivery received, instead of a hardcoded `200 duplicate` that bypassed the ack entirely. A provider that validates the ack no longer fails on the retry path dedup exists for ([#1638](https://github.com/rmyndharis/OpenWA/issues/1638)).- The `session-alive` preflight's `503` carries a `Retry-After`, so a provider that retries a 503 only when that header is present comes back instead of failing the call; the rejection writes no dedup row, so the retry is treated as a new delivery ([#1639](https://github.com/rmyndharis/OpenWA/issues/1639)).- A `429` from a rate-limit window carries a plain `Retry-After` in seconds alongside the existing `Retry-After-<window>`, which no HTTP client reads. The suffixed names stay, since they are what identify which window shed the request ([#1639](https://github.com/rmyndharis/OpenWA/issues/1639)).- An ingress route's declared ack `content-type` reaches the provider instead of being overwritten with `text/plain`, so a provider that requires `application/json` on a 200 or 202 accepts the ack; a type a browser could execute is still forced to `text/plain` ([#1637](https://github.com/rmyndharis/OpenWA/issues/1637)). Thanks @wesamdev for the report.@@ -33,6 +34,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Upgrade notes (behavior changes)+- A repeat ingress delivery is no longer distinguishable by its response. A route with no declared `response` answers `202 accepted` where it answered `200 duplicate`, and a route with a declared ack answers that ack, including a non-2xx one, so a provider retrying such a route no longer stops after the first retry. The `ingress_events` dedup row remains the record of which deliveries were repeats.- A client that honours `Retry-After` now sees one on every rate-limit `429`, where before it saw only the suffixed `Retry-After-<window>` and typically ignored it. When the `long` window sheds a request that value can be up to an hour, so a client that sleeps for it will now wait rather than retry immediately.- PostgreSQL deployments: the data connection issues `SET TIME ZONE 'UTC'` per connection and verifies the result at boot. A deployment where that cannot hold (a pooler that drops session state) now fails to start, naming the effective zone; set the default instead with `ALTER DATABASE "<database>" SET TimeZone='UTC'`. SQLite deployments, and any gateway already running in UTC, are unaffected and no data moves.- PostgreSQL deployments whose **gateway** ran outside UTC before this release: the twelve columns the app writes itself hold that host's local wall time and now read as UTC, so they appear shifted by the offset. They are `sessions.connectedAt`, `sessions.lastActiveAt`, `sessions.claimedAt`, `sessions.leaseExpiresAt`, `webhooks.lastTriggeredAt`, `webhook_outbox_events.lastAttemptAt`, `ingress_events.lastDispatchAt`, `message_batches.started_at`, `message_batches.completed_at`, `lid_mappings.updatedAt`, `chat_states.updatedAt` and `baileys_stored_messages.createdAt`. The last three carry a `DEFAULT now()` that never fires, because their only writer passes the value. With the gateway stopped, convert each with the host's old zone, which resolves daylight saving per row: `UPDATE sessions SET "connectedAt" = ("connectedAt" AT TIME ZONE 'Asia/Jakarta') AT TIME ZONE 'UTC' WHERE "connectedAt" IS NOT NULL;`. `claimedAt` and `leaseExpiresAt` are cluster runtime state: clear them, do not convert them, with every node stopped and before the first start on this release, or the lease reads shifted by the old offset and the session is unusable until it lapses. East of UTC it reads hours into the future, so every node treats the session as held elsewhere and `POST /sessions/{id}/start` answers `409`; west of UTC it reads already lapsed, so a peer can adopt a session that is still running. `UPDATE sessions SET "nodeId" = NULL, "claimedAt" = NULL, "leaseExpiresAt" = NULL, "nodeUrl" = NULL;`. Leaving `lid_mappings.updatedAt` and `chat_states.updatedAt` unconverted also mis-ranks the boot preload of both caches, which orders by that column under a cap.diff --git a/docs/06-api-specification.md b/docs/06-api-specification.mdindex d2b1f9306..e1006bb08 100644--- a/docs/06-api-specification.md+++ b/docs/06-api-specification.md@@ -6670,9 +6670,10 @@ is read raw, never DTO-bound, so the signed bytes reach the verifier unchanged).the provider's verification handshake and answers only when its `verifyToken` matches.**Response** — the primary success path is `202`: the delivery is persisted and queued for async-plugin processing. `200` means the `GET` verification-challenge echo, or a duplicate delivery-already persisted (idempotent re-delivery). A route may shape the synchronous status/body/headers-the provider sees via its declarative `ack` config (doc 25); the plugin itself always runs async.+plugin processing. `200` means the `GET` verification-challenge echo, or a route whose declared+`ack` sets it. A route may shape the synchronous status/body/headers the provider sees via its+declarative `ack` config (doc 25); the plugin itself always runs async. A re-delivery of an+already-persisted event is answered with that same ack, byte for byte, and is not enqueued again.**Errors:** `401` signature verification failed (missing, stale, or wrong secret) · `403` `GET` verification challenge failed (`verifyToken` mismatch) · `404` unknown pluginId/instanceId, or no such claimed route · `413` body over the route's `maxBodyBytes` · `429` rate limit: the per-instance bucket (`INGRESS_INSTANCE_LIMIT`) or the per-client-IP bucket (`INGRESS_IP_LIMIT`), both per `INGRESS_INSTANCE_TTL`; the global per-IP tiers skip this route, so these two are its bounds, and `Retry-After-instance` / `Retry-After-ingress-ip` names the one that shed the requestdiff --git a/openapi.json b/openapi.jsonindex dd77ebc6e..6704c07e9 100644--- a/openapi.json+++ b/openapi.json@@ -8827,7 +8827,7 @@],"responses": {"200": {- "description": "GET verification challenge echo, or a duplicate delivery already persisted (idempotent re-delivery). Not the primary success path — see 202."+ "description": "GET verification challenge echo, or a route whose declared ack sets 200. Not the primary success path; see 202. A re-delivery is answered with the same ack as the first delivery, so it is not distinguishable by status."},"202": {"description": "Webhook accepted and queued for async plugin processing (the primary success path)."@@ -8885,7 +8885,7 @@],"responses": {"200": {- "description": "GET verification challenge echo, or a duplicate delivery already persisted (idempotent re-delivery). Not the primary success path — see 202."+ "description": "GET verification challenge echo, or a route whose declared ack sets 200. Not the primary success path; see 202. A re-delivery is answered with the same ack as the first delivery, so it is not distinguishable by status."},"202": {"description": "Webhook accepted and queued for async plugin processing (the primary success path)."@@ -8943,7 +8943,7 @@],"responses": {"200": {- "description": "GET verification challenge echo, or a duplicate delivery already persisted (idempotent re-delivery). Not the primary success path — see 202."+ "description": "GET verification challenge echo, or a route whose declared ack sets 200. Not the primary success path; see 202. A re-delivery is answered with the same ack as the first delivery, so it is not distinguishable by status."},"202": {"description": "Webhook accepted and queued for async plugin processing (the primary success path)."@@ -9001,7 +9001,7 @@],"responses": {"200": {- "description": "GET verification challenge echo, or a duplicate delivery already persisted (idempotent re-delivery). Not the primary success path — see 202."+ "description": "GET verification challenge echo, or a route whose declared ack sets 200. Not the primary success path; see 202. A re-delivery is answered with the same ack as the first delivery, so it is not distinguishable by status."},"202": {"description": "Webhook accepted and queued for async plugin processing (the primary success path)."@@ -9059,7 +9059,7 @@],"responses": {"200": {- "description": "GET verification challenge echo, or a duplicate delivery already persisted (idempotent re-delivery). Not the primary success path — see 202."+ "description": "GET verification challenge echo, or a route whose declared ack sets 200. Not the primary success path; see 202. A re-delivery is answered with the same ack as the first delivery, so it is not distinguishable by status."},"202": {"description": "Webhook accepted and queued for async plugin processing (the primary success path)."@@ -9117,7 +9117,7 @@],"responses": {"200": {- "description": "GET verification challenge echo, or a duplicate delivery already persisted (idempotent re-delivery). Not the primary success path — see 202."+ "description": "GET verification challenge echo, or a route whose declared ack sets 200. Not the primary success path; see 202. A re-delivery is answered with the same ack as the first delivery, so it is not distinguishable by status."},"202": {"description": "Webhook accepted and queued for async plugin processing (the primary success path)."@@ -9175,7 +9175,7 @@],"responses": {"200": {- "description": "GET verification challenge echo, or a duplicate delivery already persisted (idempotent re-delivery). Not the primary success path — see 202."+ "description": "GET verification challenge echo, or a route whose declared ack sets 200. Not the primary success path; see 202. A re-delivery is answered with the same ack as the first delivery, so it is not distinguishable by status."},"202": {"description": "Webhook accepted and queued for async plugin processing (the primary success path)."diff --git a/src/modules/integration/ingress-enqueue.service.ts b/src/modules/integration/ingress-enqueue.service.tsindex 4af0c66fb..cea1a0da0 100644--- a/src/modules/integration/ingress-enqueue.service.ts+++ b/src/modules/integration/ingress-enqueue.service.ts@@ -163,7 +163,7 @@ export class IngressEnqueueService implements OnApplicationBootstrap {await this.loader.dispatchWebhookForInstance(data);return { outcome: 'dispatched' };} catch (err) {- // A duplicate delivery already 200s before this point, so a failure here is a real dispatch error.+ // A duplicate delivery is already acked before this point, so a failure here is a real dispatch error.// Log and swallow so the provider still gets its 202 (at-least-once, like the webhook fallback).// enqueue() intentionally does NOT write a dead-letter row here — it is shared with RedriveService// (a failed replay must not spawn a second DLQ row). The 'failed' outcome + error is returned so thediff --git a/src/modules/integration/ingress-reconciler.service.ts b/src/modules/integration/ingress-reconciler.service.tsindex baac1b986..2890e97dc 100644--- a/src/modules/integration/ingress-reconciler.service.ts+++ b/src/modules/integration/ingress-reconciler.service.ts@@ -46,7 +46,7 @@ export interface IngressReconcileStats {* Closes the last silent-loss window of the fast-ack ingress pipeline. persist-before-ack makes the* ingress_events row durable, but durability alone is not delivery: a crash between the persist and* the enqueue, or a fire-and-forget enqueue whose outcome never gets recorded, strands the row- * 'pending' forever — a provider retry only hits the dedup oracle and 200s as 'duplicate'.+ * 'pending' forever — a provider retry only hits the dedup oracle and is answered with the route's ack.** The reconciler sweeps small batches of stale 'pending' rows and re-dispatches them through the* exact same IngressEnqueueService the live path uses (same deliveryId as BullMQ jobId, so a replaydiff --git a/src/modules/integration/ingress.controller.ts b/src/modules/integration/ingress.controller.tsindex b2c30a242..1a048e7d2 100644--- a/src/modules/integration/ingress.controller.ts+++ b/src/modules/integration/ingress.controller.ts@@ -50,7 +50,7 @@ export class IngressController {})@ApiOkResponse({description:- 'GET verification challenge echo, or a duplicate delivery already persisted (idempotent re-delivery). Not the primary success path — see 202.',+ 'GET verification challenge echo, or a route whose declared ack sets 200. Not the primary success path; see 202. A re-delivery is answered with the same ack as the first delivery, so it is not distinguishable by status.',})@ApiResponse({status: 202,diff --git a/src/modules/integration/ingress.service.spec.ts b/src/modules/integration/ingress.service.spec.tsindex d301dae9f..0328439fd 100644--- a/src/modules/integration/ingress.service.spec.ts+++ b/src/modules/integration/ingress.service.spec.ts@@ -163,12 +163,13 @@ describe('IngressService.handle', () => {expect(d.enqueue).toHaveBeenCalledWith(expect.objectContaining({ deliveryId: webhookId }), webhookId);});- it('short-circuits a duplicate delivery with 200 and no enqueue', async () => {+ it('short-circuits a duplicate delivery with the route ack and no enqueue', async () => {const d = deps({ events: { recordOrSkip: jest.fn().mockResolvedValue(false) } });const svc = new IngressService(d);const res = await svc.handle(req);expect(d.enqueue).not.toHaveBeenCalled();- expect(res.status).toBe(200);+ // No declared `response`, so this is the default ack: byte-identical to a first delivery's.+ expect(res).toEqual({ status: 202, body: 'accepted' });});it('rejects an oversized body with 413 before any dedup or enqueue', async () => {@@ -662,22 +663,30 @@ describe('IngressService.handle — response contract', () => {);});- it('keeps the duplicate path as 200 "duplicate" regardless of a declared ack', async () => {- const d = depsWith({+ it('answers a re-delivery with the same ack as the first delivery', async () => {+ // The whole point of dedup is that a retry is indistinguishable. A route declaring a 204 used to+ // answer a bare 204 first and a 200 text/plain 'duplicate' on the retry, which a provider that+ // validates the ack rejects on the exact path dedup exists for.+ const route = {+ route: 'send-sms',+ mode: 'async',+ verify: 'core',+ maxBodyBytes: 1024,+ signature: { scheme: 'none' },+ dedupHeader: 'x-delivery',+ response: { ack: { status: 204, headers: { 'content-type': 'application/json' } } },+ };+ const first = await new IngressService(depsWith({ manifestRoute: jest.fn().mockReturnValue(route) })).handle(+ baseReq,+ );+ const dup = depsWith({events: { recordOrSkip: jest.fn().mockResolvedValue(false) },- manifestRoute: jest.fn().mockReturnValue({- route: 'send-sms',- mode: 'async',- verify: 'core',- maxBodyBytes: 1024,- signature: { scheme: 'none' },- dedupHeader: 'x-delivery',- response: { ack: { status: 200, body: '{"ok":true}' } },- }),+ manifestRoute: jest.fn().mockReturnValue(route),});- const res = await new IngressService(d).handle(baseReq);- expect(res.status).toBe(200);- expect(res.body).toBe('duplicate');- expect(d.enqueue).not.toHaveBeenCalled();+ const retry = await new IngressService(dup).handle(baseReq);++ expect(first).toEqual({ status: 204, headers: { 'content-type': 'application/json' } });+ expect(retry).toEqual(first);+ expect(dup.enqueue).not.toHaveBeenCalled();});});diff --git a/src/modules/integration/ingress.service.ts b/src/modules/integration/ingress.service.tsindex 169fc446b..403a21f1d 100644--- a/src/modules/integration/ingress.service.ts+++ b/src/modules/integration/ingress.service.ts@@ -177,6 +177,16 @@ export class IngressService {body: req.rawBody,rawBody: req.rawBody,};+ // Rendered BEFORE the dedup check so a provider retry gets the byte-identical ack rather than a+ // second contract on the same route. The ctx is the request in hand (rawBody, deliveryId, now),+ // never stored state, which is what makes rendering it on a dedup hit sound: an ack field that had+ // to reflect the PERSISTED row would echo the retry's values as the original's. Keep it that way.+ const ack = renderAck(route.response?.ack, {+ rawBody: req.rawBody,+ timestamp: String(Math.floor(this.deps.now() / 1000)),+ id: deliveryId,+ });+const isNew = await this.deps.events.recordOrSkip({instanceId: req.instanceId,pluginId: req.pluginId,@@ -187,7 +197,7 @@ export class IngressService {payloadHash: createHash('sha256').update(req.rawBody).digest('hex'),sessionId: instance.sessionScope,});- if (!isNew) return { status: 200, body: 'duplicate' }; // already persisted/acked+ if (!isNew) return ack; // already persisted/acked; a retry gets the first delivery's ack// Best-effort conversation id for P1 ordering. Never throws — a malformed body just yields undefined.const providerConversationId = extractConversationId(route.conversationId, req.headers, req.rawBody);@@ -203,12 +213,6 @@ export class IngressService {payload,};- const ack = renderAck(route.response?.ack, {- rawBody: req.rawBody,- timestamp: String(Math.floor(this.deps.now() / 1000)),- id: deliveryId,- });-if (route.response) {// Sync-response route: the ack is host-side and final; enqueue (queued or inline) must NOT block// it — a queue-disabled deployment otherwise holds the HTTP response for up to the inline dispatch