Skip to content

Free, Open Source, Self-Hosted WhatsApp API Gateway

fix(session): stop answering 503 after a forward was dispatched

Yudhi Armyndharis7c8040afa273
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 3ce9b6e97..733a73ba9 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -17,6 +17,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
+- A request forwarded to the node that owns its session answers `504` or `502`, not `503`, when the forward times out or breaks after the request was sent, so a client retrying on `503` no longer repeats a send the owner may have carried out; `503` remains for an owner that could not be reached at all.
- A session whose automatic reconnect fails to relaunch the engine (a network, DNS or browser launch error) keeps retrying with backoff instead of stopping in `failed` until restarted by hand; an authentication failure or a stale browser profile still ends in `failed` ([#1580](https://github.com/rmyndharis/OpenWA/issues/1580)).
- A session that runs out of reconnect attempts fires the `session:error` plugin hook when it lands in `failed`.
- An explicit `maxReconnectAttempts` is honoured and the reconnect delay is capped at 5 minutes; the budget used to restart once the backoff passed 5 minutes, so the limit was never reached and the documented 1-hour cap never applied.
diff --git a/docs/13-horizontal-scaling.md b/docs/13-horizontal-scaling.md
index d062304b7..44f1efc84 100644
--- a/docs/13-horizontal-scaling.md
+++ b/docs/13-horizontal-scaling.md
@@ -60,6 +60,13 @@
> which is exactly how a takeover begins. Without `NODE_URL` the whole path is inert and
> single-node deployments pay nothing.
>
+> **A failed forward says whether the owner could have acted.** When the owner cannot be
+> reached at all (connection refused, unresolvable or unusable `NODE_URL`), the answer is
+> `503` and the request was not carried out, so it is safe to retry. Once the request has
+> been sent, a failure answers `504` (no reply within `SESSION_PROXY_TIMEOUT_MS`) or `502`
+> (the connection broke mid-request): the owner may already have carried it out, so a
+> non-idempotent call such as a message send must not be repeated blindly on either.
+>
> **The lease compares timestamps written by different nodes, so their clocks must agree.** Each
> node writes `leaseExpiresAt` from its own clock and reads every other node's the same way, so a
> node whose clock runs more than one lease TTL (default 60s) ahead sees healthy peers as lapsed and
diff --git a/sdk/go/errors.go b/sdk/go/errors.go
index d6a8ed622..701a1f69e 100644
--- a/sdk/go/errors.go
+++ b/sdk/go/errors.go
@@ -37,7 +37,11 @@ var (
// than a refusal: WhatsApp never replied, the socket was down, or the
// request budget ran out. Unlike every other sentinel here it is
// RETRYABLE. The non-idempotent sends are deliberately left unbounded by
- // the gateway so they never answer one.
+ // the gateway so a slow WhatsApp reply never answers one, and in a
+ // multi-node deployment a forwarded request answers 503 only when the
+ // owner node was never reached. A forward that fails after the request
+ // was sent answers 502 or 504 instead: the owner may already have carried
+ // it out, so do not repeat a non-idempotent send on those unchecked.
ErrServiceUnavailable = errors.New("openwa: service unavailable")
)
diff --git a/sdk/java/src/main/java/com/rmyndharis/openwa/errors/OpenWAServiceUnavailableError.java b/sdk/java/src/main/java/com/rmyndharis/openwa/errors/OpenWAServiceUnavailableError.java
index 8fa511e80..bee77b52f 100644
--- a/sdk/java/src/main/java/com/rmyndharis/openwa/errors/OpenWAServiceUnavailableError.java
+++ b/sdk/java/src/main/java/com/rmyndharis/openwa/errors/OpenWAServiceUnavailableError.java
@@ -5,8 +5,11 @@ package com.rmyndharis.openwa.errors;
*
* <p>The gateway answers this when the engine did not confirm the operation in time: WhatsApp never
* replied, the socket was down, or the request budget ran out. <b>Retryable</b>, unlike every other
- * typed error here. The non-idempotent sends are deliberately left unbounded by the gateway so they
- * never answer one.
+ * typed error here. The non-idempotent sends are deliberately left unbounded by the gateway so a slow
+ * WhatsApp reply never answers one, and in a multi-node deployment a forwarded request answers 503
+ * only when the owner node was never reached. A forward that fails after the request was sent
+ * answers 502 or 504 instead (a plain {@link OpenWAApiError}): the owner may already have carried it
+ * out, so do not repeat a non-idempotent send on those unchecked.
*/
public class OpenWAServiceUnavailableError extends OpenWAApiError {
public OpenWAServiceUnavailableError(String message, int status, Object body, String errorKind) {
diff --git a/sdk/javascript/src/errors.ts b/sdk/javascript/src/errors.ts
index ae09257b6..72b82b5a4 100644
--- a/sdk/javascript/src/errors.ts
+++ b/sdk/javascript/src/errors.ts
@@ -87,7 +87,11 @@ export class OpenWANotImplementedError extends OpenWAApiError {}
* request budget ran out. **Retryable**, unlike every other typed error here.
*
* Not every 503 is safe to repeat blindly: the non-idempotent sends (group create, channel create,
- * media send) are deliberately left unbounded by the gateway precisely so they never answer one.
+ * media send) are deliberately left unbounded by the gateway so a slow WhatsApp reply never answers
+ * one, and in a multi-node deployment a forwarded request answers 503 only when the owner node was
+ * never reached. A forward that fails after the request was sent answers 502 or 504 instead (a plain
+ * `OpenWAApiError`): the owner may already have carried it out, so do not repeat a non-idempotent
+ * send on those unchecked.
*/
export class OpenWAServiceUnavailableError extends OpenWAApiError {}
diff --git a/sdk/php/src/Exceptions/OpenWAServiceUnavailableException.php b/sdk/php/src/Exceptions/OpenWAServiceUnavailableException.php
index 0f48172f3..4c30f3b21 100644
--- a/sdk/php/src/Exceptions/OpenWAServiceUnavailableException.php
+++ b/sdk/php/src/Exceptions/OpenWAServiceUnavailableException.php
@@ -9,8 +9,11 @@ namespace OpenWA\Exceptions;
*
* The gateway answers this when the engine did not confirm the operation in time: WhatsApp never
* replied, the socket was down, or the request budget ran out. Retryable, unlike every other typed
- * exception here. The non-idempotent sends are deliberately left unbounded by the gateway so they
- * never answer one.
+ * exception here. The non-idempotent sends are deliberately left unbounded by the gateway so a slow
+ * WhatsApp reply never answers one, and in a multi-node deployment a forwarded request answers 503
+ * only when the owner node was never reached. A forward that fails after the request was sent
+ * answers 502 or 504 instead (a plain OpenWAApiException): the owner may already have carried it
+ * out, so do not repeat a non-idempotent send on those unchecked.
*/
class OpenWAServiceUnavailableException extends OpenWAApiException
{
diff --git a/sdk/python/openwa/errors.py b/sdk/python/openwa/errors.py
index 93c912f95..383922fc6 100644
--- a/sdk/python/openwa/errors.py
+++ b/sdk/python/openwa/errors.py
@@ -83,8 +83,11 @@ class OpenWAServiceUnavailableError(OpenWAApiError):
The gateway answers this when the engine did not confirm the operation in time: WhatsApp never
replied, the socket was down, or the request budget ran out. Retryable, unlike every other typed
- error here. The non-idempotent sends are deliberately left unbounded by the gateway so they never
- answer one.
+ error here. The non-idempotent sends are deliberately left unbounded by the gateway so a slow
+ WhatsApp reply never answers one, and in a multi-node deployment a forwarded request answers 503
+ only when the owner node was never reached. A forward that fails after the request was sent
+ answers 502 or 504 instead (a plain OpenWAApiError): the owner may already have carried it out, so
+ do not repeat a non-idempotent send on those unchecked.
"""
diff --git a/src/modules/session/session-proxy.interceptor.spec.ts b/src/modules/session/session-proxy.interceptor.spec.ts
index e2ed31db1..ea56e7d1f 100644
--- a/src/modules/session/session-proxy.interceptor.spec.ts
+++ b/src/modules/session/session-proxy.interceptor.spec.ts
@@ -43,6 +43,15 @@ describe('SessionProxyInterceptor', () => {
...over,
});
+ /** A port nothing listens on: connecting to it is refused before any request is sent. */
+ const closedPort = async (): Promise<number> => {
+ const probe = http.createServer();
+ await new Promise<void>(resolve => probe.listen(0, '127.0.0.1', resolve));
+ const { port } = probe.address() as AddressInfo;
+ await new Promise<void>(resolve => probe.close(() => resolve()));
+ return port;
+ };
+
const makeResponse = () => {
const res = {
status: jest.fn(),
@@ -305,7 +314,7 @@ describe('SessionProxyInterceptor', () => {
it('an unreachable owner answers 503 with the owner named, never a hang or a crash', async () => {
const { interceptor, context, next, handle, res } = build({
- row: row({ nodeUrl: 'http://127.0.0.1:1' }),
+ row: row({ nodeUrl: `http://127.0.0.1:${await closedPort()}` }),
timeoutMs: 2000,
});
@@ -317,6 +326,70 @@ describe('SessionProxyInterceptor', () => {
expect(errorBody.message).toContain('peer-node');
expect(handle).not.toHaveBeenCalled();
});
+
+ // Once the request has been sent the owner may have acted on it, and clients replay a POST on
+ // 503 as declined-before-acting. A send that timed out or broke mid-flight must not look like
+ // that, or a retry sends the WhatsApp message twice.
+ const withOwner = async (handler: http.RequestListener, run: (url: string) => Promise<void>): Promise<void> => {
+ const owner = http.createServer(handler);
+ await new Promise<void>(resolve => owner.listen(0, '127.0.0.1', resolve));
+ try {
+ await run(`http://127.0.0.1:${(owner.address() as AddressInfo).port}`);
+ } finally {
+ owner.closeAllConnections();
+ await new Promise<void>(resolve => owner.close(() => resolve()));
+ }
+ };
+ const sendRequest = () => request({ method: 'POST', originalUrl: `/api/sessions/${SID}/messages/send-text` });
+
+ it('answers 504, not 503, when the owner received the send but did not answer in time', async () => {
+ await withOwner(
+ () => undefined,
+ async url => {
+ const { interceptor, context, next, res } = build({
+ row: row({ nodeUrl: url }),
+ req: sendRequest(),
+ timeoutMs: 200,
+ });
+ await interceptor.intercept(context, next);
+
+ expect(res.status).toHaveBeenCalledTimes(1);
+ expect(res.status).toHaveBeenCalledWith(504);
+ expect((res.json.mock.calls[0] as [{ error: string }])[0].error).toBe('Gateway Timeout');
+ },
+ );
+ });
+
+ it('answers 502, not 503, when the connection breaks after the send was dispatched', async () => {
+ await withOwner(
+ req => req.socket.destroy(),
+ async url => {
+ const { interceptor, context, next, res } = build({ row: row({ nodeUrl: url }), req: sendRequest() });
+ await interceptor.intercept(context, next);
+
+ expect(res.status).toHaveBeenCalledTimes(1);
+ expect(res.status).toHaveBeenCalledWith(502);
+ },
+ );
+ });
+
+ it('answers 502 without relaying the owner status or headers when the body read fails', async () => {
+ await withOwner(
+ (req, res) => {
+ res.writeHead(201, { 'content-type': 'application/json', 'content-length': '100' });
+ res.write('{"id":');
+ setTimeout(() => req.socket.destroy(), 20);
+ },
+ async url => {
+ const { interceptor, context, next, res } = build({ row: row({ nodeUrl: url }), req: sendRequest() });
+ await interceptor.intercept(context, next);
+
+ expect(res.status).toHaveBeenCalledTimes(1);
+ expect(res.status).toHaveBeenCalledWith(502);
+ expect(res.setHeader).not.toHaveBeenCalled();
+ },
+ );
+ });
});
});
diff --git a/src/modules/session/session-proxy.interceptor.ts b/src/modules/session/session-proxy.interceptor.ts
index 00dc2eef1..950d98e75 100644
--- a/src/modules/session/session-proxy.interceptor.ts
+++ b/src/modules/session/session-proxy.interceptor.ts
@@ -45,6 +45,19 @@ const RELAYED_RESPONSE_HEADERS = [
'x-ratelimit-reset-long',
] as const;
+/**
+ * Connection-stage failure codes: the owner never received the request, so nothing ran there and a
+ * retry cannot repeat an action. Any other failure may come after the owner started acting on it.
+ */
+const NOT_DISPATCHED_CODES = new Set([
+ 'ECONNREFUSED',
+ 'ENOTFOUND',
+ 'EAI_AGAIN',
+ 'EHOSTUNREACH',
+ 'ENETUNREACH',
+ 'UND_ERR_CONNECT_TIMEOUT',
+]);
+
/** Marks a request as already forwarded once. Whatever happens, it is never forwarded again. */
export const FORWARDED_HEADER = 'x-openwa-forwarded';
@@ -198,11 +211,12 @@ export class SessionProxyInterceptor implements NestInterceptor {
}
const hasBody = !['GET', 'HEAD'].includes(request.method);
+ let target: string | undefined;
try {
// Inside the try: a NODE_URL that is not a usable absolute URL makes this throw, and that is
- // an unreachable-owner condition (the 503 below names the node and the setting) — not a 500
+ // an unreachable-owner condition (the 503 below names the node and the setting), not a 500
// on a request that had nothing wrong with it. Boot validation rejects such a value too.
- const target = forwardTarget(request.originalUrl, ownerNodeUrl);
+ target = forwardTarget(request.originalUrl, ownerNodeUrl);
const upstream = await fetch(target, {
method: request.method,
headers,
@@ -212,6 +226,9 @@ export class SessionProxyInterceptor implements NestInterceptor {
signal: AbortSignal.timeout(timeoutMs),
redirect: 'manual',
});
+ // Read the whole body before touching the response, so a read that fails midway leaves no
+ // relayed status or headers behind on the error answer below.
+ const body = Buffer.from(await upstream.arrayBuffer());
response.status(upstream.status);
for (const name of RELAYED_RESPONSE_HEADERS) {
@@ -219,7 +236,6 @@ export class SessionProxyInterceptor implements NestInterceptor {
if (value) response.setHeader(name, value);
}
response.setHeader('x-openwa-served-by', ownerNodeId);
- const body = Buffer.from(await upstream.arrayBuffer());
if (body.length > 0) response.send(body);
else response.end();
} catch (error) {
@@ -227,12 +243,34 @@ export class SessionProxyInterceptor implements NestInterceptor {
ownerNodeUrl,
error: error instanceof Error ? error.message : String(error),
});
- response.status(503).json({
- statusCode: 503,
- message:
- `this session is hosted on node '${ownerNodeId}', which could not be reached from this node — ` +
- 'check the owner node and its NODE_URL',
- error: 'Service Unavailable',
+ // Duck-typed rather than `instanceof`: fetch rejects with a DOMException on timeout, and its
+ // errors are not guaranteed to share this context's Error class.
+ const failure = (error ?? {}) as { name?: unknown; cause?: { code?: unknown } };
+ const code = failure.cause?.code;
+ if (target === undefined || (typeof code === 'string' && NOT_DISPATCHED_CODES.has(code))) {
+ // Nothing reached the owner: 503 is the honest, retryable answer.
+ response.status(503).json({
+ statusCode: 503,
+ message:
+ `this session is hosted on node '${ownerNodeId}', which could not be reached from this node — ` +
+ 'check the owner node and its NODE_URL',
+ error: 'Service Unavailable',
+ });
+ return;
+ }
+ // Anything else may have happened after the request was sent, so the owner may have acted on
+ // it (a message may be out). Never 503 here: clients treat 503 as declined-before-acting and
+ // replay non-idempotent sends on it.
+ const timedOut = failure.name === 'TimeoutError' || failure.name === 'AbortError';
+ const status = timedOut ? 504 : 502;
+ response.status(status).json({
+ statusCode: status,
+ message: timedOut
+ ? `node '${ownerNodeId}', which hosts this session, did not answer within ${timeoutMs}ms; ` +
+ 'the request may still have been carried out there'
+ : `forwarding to node '${ownerNodeId}', which hosts this session, failed after the request was sent; ` +
+ 'it may have been carried out there',
+ error: timedOut ? 'Gateway Timeout' : 'Bad Gateway',
});
}
}
diff --git a/test/session-proxy.e2e-spec.ts b/test/session-proxy.e2e-spec.ts
index e26d8d1e8..9e2ea8431 100644
--- a/test/session-proxy.e2e-spec.ts
+++ b/test/session-proxy.e2e-spec.ts
@@ -145,7 +145,12 @@ describe('Session request forwarding (e2e)', () => {
it('an unreachable live owner answers 503 naming the owner, not a hang', async () => {
const session = await seedOwnedElsewhere();
- await sessions.update(session.id, { nodeUrl: 'http://127.0.0.1:1' });
+ // A port that was just released: the connection is refused, so nothing reached the owner.
+ const probe = http.createServer();
+ await new Promise<void>(resolve => probe.listen(0, '127.0.0.1', resolve));
+ const { port } = probe.address() as AddressInfo;
+ await new Promise<void>(resolve => probe.close(() => resolve()));
+ await sessions.update(session.id, { nodeUrl: `http://127.0.0.1:${port}` });
const res = await request(app.getHttpServer())
.get(`/api/sessions/${session.id}`)