Skip to content

Free, Open Source, Self-Hosted WhatsApp API Gateway

fix(chart): give the probes timeouts their handlers can meet

Yudhi Armyndharis3f37078f47dc
diff --git a/charts/openwa/templates/statefulset.yaml b/charts/openwa/templates/statefulset.yaml
index 5e349466c..80978627c 100644
--- a/charts/openwa/templates/statefulset.yaml
+++ b/charts/openwa/templates/statefulset.yaml
@@ -48,16 +48,16 @@ spec:
# connect retry (up to 10 attempts, 3s apart), plugin load and the backfills — so during
# boot the endpoint is not unhealthy, it is closed, and every probe is a connection
# refusal. Without a startupProbe the liveness block below governs that window and allows
- # 30s + 10s x (3-1) = 50s before the kubelet restarts the container, which the connect
- # retry alone can consume.
+ # 30s + 10s x (6-1) = 80s before the kubelet restarts the container, which the connect
+ # retry plus migrations can consume.
#
# Budget here is periodSeconds x (failureThreshold - 1) = 5s x (60-1) = 295s — the same
- # arithmetic as the 50s above, because the first probe fires at initialDelaySeconds and the
+ # arithmetic as the 80s above, because the first probe fires at initialDelaySeconds and the
# verdict lands on the Nth consecutive failure, not one period after it. The Kubernetes
# docs' shorthand (period x threshold) would say 300s; it is one period long, and applying
- # it to the liveness probe above would give 60s rather than the 50s stated there. Do not
+ # it to the liveness probe above would give 90s rather than the 80s stated there. Do not
# "correct" this back without changing that one too. The point of either figure is that it
- # converts "killed at 50s by default" into "killed at a budget we chose" — a large
+ # converts "killed at 80s by default" into "killed at a budget we chose": a large
# improvement, and not a guarantee for a boot that is genuinely unbounded.
#
# initialDelaySeconds is deliberately left at its default of 0: the first probe fires
@@ -68,11 +68,17 @@ spec:
# the values below keep their meaning for steady state and stop governing boot. (That
# suspension is documented kubelet behaviour, not something this chart's tests observe —
# rendering a manifest cannot demonstrate a runtime scheduling rule.)
+ #
+ # timeoutSeconds is explicit on every probe. The kubelet default of 1s is missed by a
+ # CPU-throttled pod (the limit is shared with every Chromium) even on the static /live route,
+ # and by /ready by design: it bounds each database probe at READINESS_PROBE_TIMEOUT_MS
+ # (3000 ms, src/modules/health/health.controller.ts) so it can answer its own 503 first.
startupProbe:
httpGet:
path: /api/health/live
port: http
periodSeconds: 5
+ timeoutSeconds: 5
failureThreshold: 60
livenessProbe:
httpGet:
@@ -80,12 +86,15 @@ spec:
port: http
initialDelaySeconds: 30
periodSeconds: 10
+ timeoutSeconds: 5
+ failureThreshold: 6
readinessProbe:
httpGet:
path: /api/health/ready
port: http
initialDelaySeconds: 10
periodSeconds: 5
+ timeoutSeconds: 5
resources:
{{- toYaml .Values.resources | nindent 12 }}
securityContext:
diff --git a/docs/13-horizontal-scaling.md b/docs/13-horizontal-scaling.md
index 64185d032..9110d3e0f 100644
--- a/docs/13-horizontal-scaling.md
+++ b/docs/13-horizontal-scaling.md
@@ -452,18 +452,31 @@ spec:
mountPath: /app/data
- name: tmp
mountPath: /tmp
+ # Same probes as charts/openwa/templates/statefulset.yaml. The startupProbe holds liveness
+ # off during boot (migrations, connect retry, plugin load, backfills), and every timeoutSeconds is
+ # explicit because the 1s kubelet default is shorter than /ready's 3s database bound.
+ startupProbe:
+ httpGet:
+ path: /api/health/live
+ port: 2785
+ periodSeconds: 5
+ timeoutSeconds: 5
+ failureThreshold: 60
livenessProbe:
httpGet:
- path: /api/health
+ path: /api/health/live
port: 2785
initialDelaySeconds: 30
periodSeconds: 10
+ timeoutSeconds: 5
+ failureThreshold: 6
readinessProbe:
httpGet:
path: /api/health/ready
port: 2785
initialDelaySeconds: 10
periodSeconds: 5
+ timeoutSeconds: 5
volumes:
- name: tmp
emptyDir: {}
diff --git a/scripts/check-chart-behaviour.mjs b/scripts/check-chart-behaviour.mjs
index d9d1af0d0..f8f8ecbc8 100644
--- a/scripts/check-chart-behaviour.mjs
+++ b/scripts/check-chart-behaviour.mjs
@@ -21,6 +21,7 @@
* Run locally: `npm run check:chart`. Runs in CI (Helm chart and workflows job). Needs Docker.
*/
import { execFileSync } from 'node:child_process';
+import { readFileSync } from 'node:fs';
import { fileURLToPath } from 'node:url';
// Pinned in step with the `helm lint` / `helm template` steps of .github/workflows/ci.yml. A version
@@ -132,9 +133,10 @@ const check = (id, ok, detail) => results.push({ id, ok, detail });
);
}
-// Boot is long and varies with the number of sessions to restore. That belongs to a startupProbe: it
-// suspends the liveness probe until it succeeds, so the boot window and the running-health window can
-// be set independently. Without one, the liveness budget alone decides how long boot may take.
+// Boot is long (migrations, the database connect retry, plugin load, backfills). That belongs to a
+// startupProbe: it suspends the liveness probe until it succeeds, so the boot window and the
+// running-health window can be set independently. Without one, the liveness budget alone
+// decides how long boot may take.
{
const sts = byKind(render(), 'StatefulSet')[0] ?? '';
const startup = mapAt(sts, ['startupProbe']);
@@ -150,6 +152,31 @@ const check = (id, ok, detail) => results.push({ id, ok, detail });
);
}
+// A probe that times out counts as a failure. The kubelet default is 1s, which /ready misses by
+// design (it bounds each database probe at READINESS_PROBE_TIMEOUT_MS so it can answer its own 503)
+// and which a CPU-throttled pod can miss even on the static /live route. The constant is read from
+// the controller so the two cannot drift; a rename fails here rather than skipping the check.
+{
+ const source = readFileSync(new URL('../src/modules/health/health.controller.ts', import.meta.url), 'utf8');
+ const match = /READINESS_PROBE_TIMEOUT_MS\s*=\s*([\d_]+)/.exec(source);
+ const handlerMs = match ? Number(match[1].replace(/_/g, '')) : NaN;
+ const sts = byKind(render(), 'StatefulSet')[0] ?? '';
+ const timeout = name => Number(mapAt(sts, [name])?.timeoutSeconds ?? 1);
+ const readiness = timeout('readinessProbe');
+ const short = ['livenessProbe', 'startupProbe'].filter(name => timeout(name) < 2);
+ check(
+ 'probe-timeouts-cover-handlers',
+ Number.isFinite(handlerMs) && readiness * 1000 > handlerMs && short.length === 0,
+ !Number.isFinite(handlerMs)
+ ? 'READINESS_PROBE_TIMEOUT_MS not found in src/modules/health/health.controller.ts'
+ : readiness * 1000 <= handlerMs
+ ? `readinessProbe times out at ${readiness}s, not above the ${handlerMs}ms the handler may take`
+ : short.length
+ ? `${short.join(' and ')} time out below 2s`
+ : `readiness ${readiness}s exceeds the ${handlerMs}ms handler bound; liveness and startup allow ${timeout('livenessProbe')}s and ${timeout('startupProbe')}s`,
+ );
+}
+
// One scrape target per pod per endpoint. Prometheus Operator yields a target for every address of
// every Service matching the selector and de-duplicates nothing by pod, so a selector matching N
// Services produces N series sets separated only by the `service` label.
diff --git a/src/common/ci-privilege-gate.spec.ts b/src/common/ci-privilege-gate.spec.ts
index 50a4ba592..a6b0c09b4 100644
--- a/src/common/ci-privilege-gate.spec.ts
+++ b/src/common/ci-privilege-gate.spec.ts
@@ -257,3 +257,35 @@ describe('a job that can mint a publish credential pins global npm installs and
expect(pythonInstallsInOidcJobs(file)).toEqual([]);
});
});
+
+/**
+ * The operator-facing text around the entrypoint, the probes and the backup scripts states what they
+ * do. Each of these was once true of an earlier version and outlived the change that made it false.
+ */
+describe('deployment docs describe what the entrypoint, probes and backup scripts do', () => {
+ const root = path.join(__dirname, '..', '..');
+ const read = (file: string): string => fs.readFileSync(path.join(root, file), 'utf8');
+
+ // Session auto-start is detached: boot does not wait for it, so no probe covers it.
+ it('does not claim the startupProbe covers session restore', () => {
+ expect(read('docs/13-horizontal-scaling.md')).not.toMatch(/off during boot \([^)]*session/);
+ expect(read('scripts/check-chart-behaviour.mjs')).not.toMatch(/sessions to restore/);
+ });
+
+ // The chart's liveness budget lives in statefulset.yaml; a copied figure goes stale.
+ it('does not restate the chart liveness budget in session comments', () => {
+ for (const file of ['src/modules/session/session.service.ts', 'src/modules/session/session.service.spec.ts']) {
+ expect(read(file)).not.toMatch(/the chart's (?:budget )?is ~\d+s/);
+ }
+ });
+
+ // The startupProbe suspends liveness until it first succeeds, so a closed port at boot meets it, not liveness.
+ it('names the startupProbe as the budget a closed port at boot runs against', () => {
+ for (const file of ['src/modules/session/session.service.ts', 'src/modules/session/session.service.spec.ts']) {
+ const text = read(file).replace(/\n\s*\/\/ ?/g, ' ');
+ expect(text).not.toMatch(/every liveness\s+probe/i);
+ expect(text).not.toMatch(/the chart's liveness budget/);
+ expect(text).toMatch(/the chart's startupProbe budget/);
+ }
+ });
+});
diff --git a/src/modules/session/session.service.spec.ts b/src/modules/session/session.service.spec.ts
index 91be30c1b..2b88a1363 100644
--- a/src/modules/session/session.service.spec.ts
+++ b/src/modules/session/session.service.spec.ts
@@ -8527,8 +8527,9 @@ describe('SessionService', () => {
// Nest binds the HTTP listener only after every bootstrap hook settles, and a launch is a
// Chromium start bounded by resolveEngineInitTimeoutMs (>= 60s) with a 2s throttle between
- // sessions. Awaiting the loop here kept the port CLOSED for that whole time, so every liveness
- // probe in the window was a connection refusal — the chart's budget is ~50s.
+ // sessions. Awaiting the loop here kept the port CLOSED for that whole time, so every probe in
+ // the window was a connection refusal, and the chart's startupProbe budget
+ // (charts/openwa/templates/statefulset.yaml) is a fixed number of seconds.
it('returns without waiting for a launch that has not finished', async () => {
process.env.AUTO_START_SESSIONS = 'true';
(repository.find as jest.Mock).mockResolvedValue([{ id: 'a', name: 'A' }]);
diff --git a/src/modules/session/session.service.ts b/src/modules/session/session.service.ts
index 626eb88e4..400004d1f 100644
--- a/src/modules/session/session.service.ts
+++ b/src/modules/session/session.service.ts
@@ -214,9 +214,10 @@ export class SessionService implements OnModuleDestroy, OnModuleInit, OnApplicat
// hook has settled, and this loop's duration is unbounded: one engine initialization is at least
// 60s (resolveEngineInitTimeoutMs) and there is a 2s throttle between sessions, so a host with
// ten authenticated sessions kept the port CLOSED — not unhealthy, closed — for ten minutes.
- // Every liveness probe in that window is a connection refusal, and no probe budget can cover a
- // bound that scales with the session count: the chart's is ~50s and the Dockerfile HEALTHCHECK
- // encodes the same expectation. Awaited on shutdown so a launch in flight is accounted for.
+ // Every probe in that window is a connection refusal, and no probe budget can cover a bound that
+ // scales with the session count: the chart's startupProbe budget (statefulset.yaml), which governs
+ // boot, is a fixed number of seconds, and the Dockerfile HEALTHCHECK encodes the same expectation.
+ // Awaited on shutdown so a launch in flight is accounted for.
this.autoStartRun = this.autoStartSessions().catch((error: unknown) => {
// Previously this rejected out of the hook and aborted boot, so a transient database error
// during the session scan took the whole gateway down rather than the auto-start.