Free, Open Source, Self-Hosted WhatsApp API Gateway
TypeScript apibotgatewayself-hostedwhatsappwhatsapp-apiwhatsapp-automationwhatsapp-botwhatsapp-gatewaywhatsapp-gateway-apiwhatsapp-webwhatsapp-web-api
fix(session): stop answering 503 after a forward was dispatched
Yudhi Armyndharis7c8040afa273
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 3ce9b6e97..733a73ba9 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -17,6 +17,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Fixed+- A request forwarded to the node that owns its session answers `504` or `502`, not `503`, when the forward times out or breaks after the request was sent, so a client retrying on `503` no longer repeats a send the owner may have carried out; `503` remains for an owner that could not be reached at all.- A session whose automatic reconnect fails to relaunch the engine (a network, DNS or browser launch error) keeps retrying with backoff instead of stopping in `failed` until restarted by hand; an authentication failure or a stale browser profile still ends in `failed` ([#1580](https://github.com/rmyndharis/OpenWA/issues/1580)).- A session that runs out of reconnect attempts fires the `session:error` plugin hook when it lands in `failed`.- An explicit `maxReconnectAttempts` is honoured and the reconnect delay is capped at 5 minutes; the budget used to restart once the backoff passed 5 minutes, so the limit was never reached and the documented 1-hour cap never applied.diff --git a/docs/13-horizontal-scaling.md b/docs/13-horizontal-scaling.mdindex d062304b7..44f1efc84 100644--- a/docs/13-horizontal-scaling.md+++ b/docs/13-horizontal-scaling.md@@ -60,6 +60,13 @@> which is exactly how a takeover begins. Without `NODE_URL` the whole path is inert and> single-node deployments pay nothing.>+> **A failed forward says whether the owner could have acted.** When the owner cannot be+> reached at all (connection refused, unresolvable or unusable `NODE_URL`), the answer is+> `503` and the request was not carried out, so it is safe to retry. Once the request has+> been sent, a failure answers `504` (no reply within `SESSION_PROXY_TIMEOUT_MS`) or `502`+> (the connection broke mid-request): the owner may already have carried it out, so a+> non-idempotent call such as a message send must not be repeated blindly on either.+>> **The lease compares timestamps written by different nodes, so their clocks must agree.** Each> node writes `leaseExpiresAt` from its own clock and reads every other node's the same way, so a> node whose clock runs more than one lease TTL (default 60s) ahead sees healthy peers as lapsed anddiff --git a/sdk/go/errors.go b/sdk/go/errors.goindex d6a8ed622..701a1f69e 100644--- a/sdk/go/errors.go+++ b/sdk/go/errors.go@@ -37,7 +37,11 @@ var (// than a refusal: WhatsApp never replied, the socket was down, or the// request budget ran out. Unlike every other sentinel here it is// RETRYABLE. The non-idempotent sends are deliberately left unbounded by- // the gateway so they never answer one.+ // the gateway so a slow WhatsApp reply never answers one, and in a+ // multi-node deployment a forwarded request answers 503 only when the+ // owner node was never reached. A forward that fails after the request+ // was sent answers 502 or 504 instead: the owner may already have carried+ // it out, so do not repeat a non-idempotent send on those unchecked.ErrServiceUnavailable = errors.New("openwa: service unavailable"))diff --git a/sdk/java/src/main/java/com/rmyndharis/openwa/errors/OpenWAServiceUnavailableError.java b/sdk/java/src/main/java/com/rmyndharis/openwa/errors/OpenWAServiceUnavailableError.javaindex 8fa511e80..bee77b52f 100644--- a/sdk/java/src/main/java/com/rmyndharis/openwa/errors/OpenWAServiceUnavailableError.java+++ b/sdk/java/src/main/java/com/rmyndharis/openwa/errors/OpenWAServiceUnavailableError.java@@ -5,8 +5,11 @@ package com.rmyndharis.openwa.errors;** <p>The gateway answers this when the engine did not confirm the operation in time: WhatsApp never* replied, the socket was down, or the request budget ran out. <b>Retryable</b>, unlike every other- * typed error here. The non-idempotent sends are deliberately left unbounded by the gateway so they- * never answer one.+ * typed error here. The non-idempotent sends are deliberately left unbounded by the gateway so a slow+ * WhatsApp reply never answers one, and in a multi-node deployment a forwarded request answers 503+ * only when the owner node was never reached. A forward that fails after the request was sent+ * answers 502 or 504 instead (a plain {@link OpenWAApiError}): the owner may already have carried it+ * out, so do not repeat a non-idempotent send on those unchecked.*/public class OpenWAServiceUnavailableError extends OpenWAApiError {public OpenWAServiceUnavailableError(String message, int status, Object body, String errorKind) {diff --git a/sdk/javascript/src/errors.ts b/sdk/javascript/src/errors.tsindex ae09257b6..72b82b5a4 100644--- a/sdk/javascript/src/errors.ts+++ b/sdk/javascript/src/errors.ts@@ -87,7 +87,11 @@ export class OpenWANotImplementedError extends OpenWAApiError {}* request budget ran out. **Retryable**, unlike every other typed error here.** Not every 503 is safe to repeat blindly: the non-idempotent sends (group create, channel create,- * media send) are deliberately left unbounded by the gateway precisely so they never answer one.+ * media send) are deliberately left unbounded by the gateway so a slow WhatsApp reply never answers+ * one, and in a multi-node deployment a forwarded request answers 503 only when the owner node was+ * never reached. A forward that fails after the request was sent answers 502 or 504 instead (a plain+ * `OpenWAApiError`): the owner may already have carried it out, so do not repeat a non-idempotent+ * send on those unchecked.*/export class OpenWAServiceUnavailableError extends OpenWAApiError {}diff --git a/sdk/php/src/Exceptions/OpenWAServiceUnavailableException.php b/sdk/php/src/Exceptions/OpenWAServiceUnavailableException.phpindex 0f48172f3..4c30f3b21 100644--- a/sdk/php/src/Exceptions/OpenWAServiceUnavailableException.php+++ b/sdk/php/src/Exceptions/OpenWAServiceUnavailableException.php@@ -9,8 +9,11 @@ namespace OpenWA\Exceptions;** The gateway answers this when the engine did not confirm the operation in time: WhatsApp never* replied, the socket was down, or the request budget ran out. Retryable, unlike every other typed- * exception here. The non-idempotent sends are deliberately left unbounded by the gateway so they- * never answer one.+ * exception here. The non-idempotent sends are deliberately left unbounded by the gateway so a slow+ * WhatsApp reply never answers one, and in a multi-node deployment a forwarded request answers 503+ * only when the owner node was never reached. A forward that fails after the request was sent+ * answers 502 or 504 instead (a plain OpenWAApiException): the owner may already have carried it+ * out, so do not repeat a non-idempotent send on those unchecked.*/class OpenWAServiceUnavailableException extends OpenWAApiException{diff --git a/sdk/python/openwa/errors.py b/sdk/python/openwa/errors.pyindex 93c912f95..383922fc6 100644--- a/sdk/python/openwa/errors.py+++ b/sdk/python/openwa/errors.py@@ -83,8 +83,11 @@ class OpenWAServiceUnavailableError(OpenWAApiError):The gateway answers this when the engine did not confirm the operation in time: WhatsApp neverreplied, the socket was down, or the request budget ran out. Retryable, unlike every other typed- error here. The non-idempotent sends are deliberately left unbounded by the gateway so they never- answer one.+ error here. The non-idempotent sends are deliberately left unbounded by the gateway so a slow+ WhatsApp reply never answers one, and in a multi-node deployment a forwarded request answers 503+ only when the owner node was never reached. A forward that fails after the request was sent+ answers 502 or 504 instead (a plain OpenWAApiError): the owner may already have carried it out, so+ do not repeat a non-idempotent send on those unchecked."""diff --git a/src/modules/session/session-proxy.interceptor.spec.ts b/src/modules/session/session-proxy.interceptor.spec.tsindex e2ed31db1..ea56e7d1f 100644--- a/src/modules/session/session-proxy.interceptor.spec.ts+++ b/src/modules/session/session-proxy.interceptor.spec.ts@@ -43,6 +43,15 @@ describe('SessionProxyInterceptor', () => {...over,});+ /** A port nothing listens on: connecting to it is refused before any request is sent. */+ const closedPort = async (): Promise<number> => {+ const probe = http.createServer();+ await new Promise<void>(resolve => probe.listen(0, '127.0.0.1', resolve));+ const { port } = probe.address() as AddressInfo;+ await new Promise<void>(resolve => probe.close(() => resolve()));+ return port;+ };+const makeResponse = () => {const res = {status: jest.fn(),@@ -305,7 +314,7 @@ describe('SessionProxyInterceptor', () => {it('an unreachable owner answers 503 with the owner named, never a hang or a crash', async () => {const { interceptor, context, next, handle, res } = build({- row: row({ nodeUrl: 'http://127.0.0.1:1' }),+ row: row({ nodeUrl: `http://127.0.0.1:${await closedPort()}` }),timeoutMs: 2000,});@@ -317,6 +326,70 @@ describe('SessionProxyInterceptor', () => {expect(errorBody.message).toContain('peer-node');expect(handle).not.toHaveBeenCalled();});++ // Once the request has been sent the owner may have acted on it, and clients replay a POST on+ // 503 as declined-before-acting. A send that timed out or broke mid-flight must not look like+ // that, or a retry sends the WhatsApp message twice.+ const withOwner = async (handler: http.RequestListener, run: (url: string) => Promise<void>): Promise<void> => {+ const owner = http.createServer(handler);+ await new Promise<void>(resolve => owner.listen(0, '127.0.0.1', resolve));+ try {+ await run(`http://127.0.0.1:${(owner.address() as AddressInfo).port}`);+ } finally {+ owner.closeAllConnections();+ await new Promise<void>(resolve => owner.close(() => resolve()));+ }+ };+ const sendRequest = () => request({ method: 'POST', originalUrl: `/api/sessions/${SID}/messages/send-text` });++ it('answers 504, not 503, when the owner received the send but did not answer in time', async () => {+ await withOwner(+ () => undefined,+ async url => {+ const { interceptor, context, next, res } = build({+ row: row({ nodeUrl: url }),+ req: sendRequest(),+ timeoutMs: 200,+ });+ await interceptor.intercept(context, next);++ expect(res.status).toHaveBeenCalledTimes(1);+ expect(res.status).toHaveBeenCalledWith(504);+ expect((res.json.mock.calls[0] as [{ error: string }])[0].error).toBe('Gateway Timeout');+ },+ );+ });++ it('answers 502, not 503, when the connection breaks after the send was dispatched', async () => {+ await withOwner(+ req => req.socket.destroy(),+ async url => {+ const { interceptor, context, next, res } = build({ row: row({ nodeUrl: url }), req: sendRequest() });+ await interceptor.intercept(context, next);++ expect(res.status).toHaveBeenCalledTimes(1);+ expect(res.status).toHaveBeenCalledWith(502);+ },+ );+ });++ it('answers 502 without relaying the owner status or headers when the body read fails', async () => {+ await withOwner(+ (req, res) => {+ res.writeHead(201, { 'content-type': 'application/json', 'content-length': '100' });+ res.write('{"id":');+ setTimeout(() => req.socket.destroy(), 20);+ },+ async url => {+ const { interceptor, context, next, res } = build({ row: row({ nodeUrl: url }), req: sendRequest() });+ await interceptor.intercept(context, next);++ expect(res.status).toHaveBeenCalledTimes(1);+ expect(res.status).toHaveBeenCalledWith(502);+ expect(res.setHeader).not.toHaveBeenCalled();+ },+ );+ });});});diff --git a/src/modules/session/session-proxy.interceptor.ts b/src/modules/session/session-proxy.interceptor.tsindex 00dc2eef1..950d98e75 100644--- a/src/modules/session/session-proxy.interceptor.ts+++ b/src/modules/session/session-proxy.interceptor.ts@@ -45,6 +45,19 @@ const RELAYED_RESPONSE_HEADERS = ['x-ratelimit-reset-long',] as const;+/**+ * Connection-stage failure codes: the owner never received the request, so nothing ran there and a+ * retry cannot repeat an action. Any other failure may come after the owner started acting on it.+ */+const NOT_DISPATCHED_CODES = new Set([+ 'ECONNREFUSED',+ 'ENOTFOUND',+ 'EAI_AGAIN',+ 'EHOSTUNREACH',+ 'ENETUNREACH',+ 'UND_ERR_CONNECT_TIMEOUT',+]);+/** Marks a request as already forwarded once. Whatever happens, it is never forwarded again. */export const FORWARDED_HEADER = 'x-openwa-forwarded';@@ -198,11 +211,12 @@ export class SessionProxyInterceptor implements NestInterceptor {}const hasBody = !['GET', 'HEAD'].includes(request.method);+ let target: string | undefined;try {// Inside the try: a NODE_URL that is not a usable absolute URL makes this throw, and that is- // an unreachable-owner condition (the 503 below names the node and the setting) — not a 500+ // an unreachable-owner condition (the 503 below names the node and the setting), not a 500// on a request that had nothing wrong with it. Boot validation rejects such a value too.- const target = forwardTarget(request.originalUrl, ownerNodeUrl);+ target = forwardTarget(request.originalUrl, ownerNodeUrl);const upstream = await fetch(target, {method: request.method,headers,@@ -212,6 +226,9 @@ export class SessionProxyInterceptor implements NestInterceptor {signal: AbortSignal.timeout(timeoutMs),redirect: 'manual',});+ // Read the whole body before touching the response, so a read that fails midway leaves no+ // relayed status or headers behind on the error answer below.+ const body = Buffer.from(await upstream.arrayBuffer());response.status(upstream.status);for (const name of RELAYED_RESPONSE_HEADERS) {@@ -219,7 +236,6 @@ export class SessionProxyInterceptor implements NestInterceptor {if (value) response.setHeader(name, value);}response.setHeader('x-openwa-served-by', ownerNodeId);- const body = Buffer.from(await upstream.arrayBuffer());if (body.length > 0) response.send(body);else response.end();} catch (error) {@@ -227,12 +243,34 @@ export class SessionProxyInterceptor implements NestInterceptor {ownerNodeUrl,error: error instanceof Error ? error.message : String(error),});- response.status(503).json({- statusCode: 503,- message:- `this session is hosted on node '${ownerNodeId}', which could not be reached from this node — ` +- 'check the owner node and its NODE_URL',- error: 'Service Unavailable',+ // Duck-typed rather than `instanceof`: fetch rejects with a DOMException on timeout, and its+ // errors are not guaranteed to share this context's Error class.+ const failure = (error ?? {}) as { name?: unknown; cause?: { code?: unknown } };+ const code = failure.cause?.code;+ if (target === undefined || (typeof code === 'string' && NOT_DISPATCHED_CODES.has(code))) {+ // Nothing reached the owner: 503 is the honest, retryable answer.+ response.status(503).json({+ statusCode: 503,+ message:+ `this session is hosted on node '${ownerNodeId}', which could not be reached from this node — ` ++ 'check the owner node and its NODE_URL',+ error: 'Service Unavailable',+ });+ return;+ }+ // Anything else may have happened after the request was sent, so the owner may have acted on+ // it (a message may be out). Never 503 here: clients treat 503 as declined-before-acting and+ // replay non-idempotent sends on it.+ const timedOut = failure.name === 'TimeoutError' || failure.name === 'AbortError';+ const status = timedOut ? 504 : 502;+ response.status(status).json({+ statusCode: status,+ message: timedOut+ ? `node '${ownerNodeId}', which hosts this session, did not answer within ${timeoutMs}ms; ` ++ 'the request may still have been carried out there'+ : `forwarding to node '${ownerNodeId}', which hosts this session, failed after the request was sent; ` ++ 'it may have been carried out there',+ error: timedOut ? 'Gateway Timeout' : 'Bad Gateway',});}}diff --git a/test/session-proxy.e2e-spec.ts b/test/session-proxy.e2e-spec.tsindex e26d8d1e8..9e2ea8431 100644--- a/test/session-proxy.e2e-spec.ts+++ b/test/session-proxy.e2e-spec.ts@@ -145,7 +145,12 @@ describe('Session request forwarding (e2e)', () => {it('an unreachable live owner answers 503 naming the owner, not a hang', async () => {const session = await seedOwnedElsewhere();- await sessions.update(session.id, { nodeUrl: 'http://127.0.0.1:1' });+ // A port that was just released: the connection is refused, so nothing reached the owner.+ const probe = http.createServer();+ await new Promise<void>(resolve => probe.listen(0, '127.0.0.1', resolve));+ const { port } = probe.address() as AddressInfo;+ await new Promise<void>(resolve => probe.close(() => resolve()));+ await sessions.update(session.id, { nodeUrl: `http://127.0.0.1:${port}` });const res = await request(app.getHttpServer()).get(`/api/sessions/${session.id}`)