Free, Open Source, Self-Hosted WhatsApp API Gateway
TypeScript apibotgatewayself-hostedwhatsappwhatsapp-apiwhatsapp-automationwhatsapp-botwhatsapp-gatewaywhatsapp-gateway-apiwhatsapp-webwhatsapp-web-api
fix(chart): give the probes timeouts their handlers can meet
Yudhi Armyndharis3f37078f47dc
diff --git a/charts/openwa/templates/statefulset.yaml b/charts/openwa/templates/statefulset.yamlindex 5e349466c..80978627c 100644--- a/charts/openwa/templates/statefulset.yaml+++ b/charts/openwa/templates/statefulset.yaml@@ -48,16 +48,16 @@ spec:# connect retry (up to 10 attempts, 3s apart), plugin load and the backfills — so during# boot the endpoint is not unhealthy, it is closed, and every probe is a connection# refusal. Without a startupProbe the liveness block below governs that window and allows- # 30s + 10s x (3-1) = 50s before the kubelet restarts the container, which the connect- # retry alone can consume.+ # 30s + 10s x (6-1) = 80s before the kubelet restarts the container, which the connect+ # retry plus migrations can consume.## Budget here is periodSeconds x (failureThreshold - 1) = 5s x (60-1) = 295s — the same- # arithmetic as the 50s above, because the first probe fires at initialDelaySeconds and the+ # arithmetic as the 80s above, because the first probe fires at initialDelaySeconds and the# verdict lands on the Nth consecutive failure, not one period after it. The Kubernetes# docs' shorthand (period x threshold) would say 300s; it is one period long, and applying- # it to the liveness probe above would give 60s rather than the 50s stated there. Do not+ # it to the liveness probe above would give 90s rather than the 80s stated there. Do not# "correct" this back without changing that one too. The point of either figure is that it- # converts "killed at 50s by default" into "killed at a budget we chose" — a large+ # converts "killed at 80s by default" into "killed at a budget we chose": a large# improvement, and not a guarantee for a boot that is genuinely unbounded.## initialDelaySeconds is deliberately left at its default of 0: the first probe fires@@ -68,11 +68,17 @@ spec:# the values below keep their meaning for steady state and stop governing boot. (That# suspension is documented kubelet behaviour, not something this chart's tests observe —# rendering a manifest cannot demonstrate a runtime scheduling rule.)+ #+ # timeoutSeconds is explicit on every probe. The kubelet default of 1s is missed by a+ # CPU-throttled pod (the limit is shared with every Chromium) even on the static /live route,+ # and by /ready by design: it bounds each database probe at READINESS_PROBE_TIMEOUT_MS+ # (3000 ms, src/modules/health/health.controller.ts) so it can answer its own 503 first.startupProbe:httpGet:path: /api/health/liveport: httpperiodSeconds: 5+ timeoutSeconds: 5failureThreshold: 60livenessProbe:httpGet:@@ -80,12 +86,15 @@ spec:port: httpinitialDelaySeconds: 30periodSeconds: 10+ timeoutSeconds: 5+ failureThreshold: 6readinessProbe:httpGet:path: /api/health/readyport: httpinitialDelaySeconds: 10periodSeconds: 5+ timeoutSeconds: 5resources:{{- toYaml .Values.resources | nindent 12 }}securityContext:diff --git a/docs/13-horizontal-scaling.md b/docs/13-horizontal-scaling.mdindex 64185d032..9110d3e0f 100644--- a/docs/13-horizontal-scaling.md+++ b/docs/13-horizontal-scaling.md@@ -452,18 +452,31 @@ spec:mountPath: /app/data- name: tmpmountPath: /tmp+ # Same probes as charts/openwa/templates/statefulset.yaml. The startupProbe holds liveness+ # off during boot (migrations, connect retry, plugin load, backfills), and every timeoutSeconds is+ # explicit because the 1s kubelet default is shorter than /ready's 3s database bound.+ startupProbe:+ httpGet:+ path: /api/health/live+ port: 2785+ periodSeconds: 5+ timeoutSeconds: 5+ failureThreshold: 60livenessProbe:httpGet:- path: /api/health+ path: /api/health/liveport: 2785initialDelaySeconds: 30periodSeconds: 10+ timeoutSeconds: 5+ failureThreshold: 6readinessProbe:httpGet:path: /api/health/readyport: 2785initialDelaySeconds: 10periodSeconds: 5+ timeoutSeconds: 5volumes:- name: tmpemptyDir: {}diff --git a/scripts/check-chart-behaviour.mjs b/scripts/check-chart-behaviour.mjsindex d9d1af0d0..f8f8ecbc8 100644--- a/scripts/check-chart-behaviour.mjs+++ b/scripts/check-chart-behaviour.mjs@@ -21,6 +21,7 @@* Run locally: `npm run check:chart`. Runs in CI (Helm chart and workflows job). Needs Docker.*/import { execFileSync } from 'node:child_process';+import { readFileSync } from 'node:fs';import { fileURLToPath } from 'node:url';// Pinned in step with the `helm lint` / `helm template` steps of .github/workflows/ci.yml. A version@@ -132,9 +133,10 @@ const check = (id, ok, detail) => results.push({ id, ok, detail }););}-// Boot is long and varies with the number of sessions to restore. That belongs to a startupProbe: it-// suspends the liveness probe until it succeeds, so the boot window and the running-health window can-// be set independently. Without one, the liveness budget alone decides how long boot may take.+// Boot is long (migrations, the database connect retry, plugin load, backfills). That belongs to a+// startupProbe: it suspends the liveness probe until it succeeds, so the boot window and the+// running-health window can be set independently. Without one, the liveness budget alone+// decides how long boot may take.{const sts = byKind(render(), 'StatefulSet')[0] ?? '';const startup = mapAt(sts, ['startupProbe']);@@ -150,6 +152,31 @@ const check = (id, ok, detail) => results.push({ id, ok, detail }););}+// A probe that times out counts as a failure. The kubelet default is 1s, which /ready misses by+// design (it bounds each database probe at READINESS_PROBE_TIMEOUT_MS so it can answer its own 503)+// and which a CPU-throttled pod can miss even on the static /live route. The constant is read from+// the controller so the two cannot drift; a rename fails here rather than skipping the check.+{+ const source = readFileSync(new URL('../src/modules/health/health.controller.ts', import.meta.url), 'utf8');+ const match = /READINESS_PROBE_TIMEOUT_MS\s*=\s*([\d_]+)/.exec(source);+ const handlerMs = match ? Number(match[1].replace(/_/g, '')) : NaN;+ const sts = byKind(render(), 'StatefulSet')[0] ?? '';+ const timeout = name => Number(mapAt(sts, [name])?.timeoutSeconds ?? 1);+ const readiness = timeout('readinessProbe');+ const short = ['livenessProbe', 'startupProbe'].filter(name => timeout(name) < 2);+ check(+ 'probe-timeouts-cover-handlers',+ Number.isFinite(handlerMs) && readiness * 1000 > handlerMs && short.length === 0,+ !Number.isFinite(handlerMs)+ ? 'READINESS_PROBE_TIMEOUT_MS not found in src/modules/health/health.controller.ts'+ : readiness * 1000 <= handlerMs+ ? `readinessProbe times out at ${readiness}s, not above the ${handlerMs}ms the handler may take`+ : short.length+ ? `${short.join(' and ')} time out below 2s`+ : `readiness ${readiness}s exceeds the ${handlerMs}ms handler bound; liveness and startup allow ${timeout('livenessProbe')}s and ${timeout('startupProbe')}s`,+ );+}+// One scrape target per pod per endpoint. Prometheus Operator yields a target for every address of// every Service matching the selector and de-duplicates nothing by pod, so a selector matching N// Services produces N series sets separated only by the `service` label.diff --git a/src/common/ci-privilege-gate.spec.ts b/src/common/ci-privilege-gate.spec.tsindex 50a4ba592..a6b0c09b4 100644--- a/src/common/ci-privilege-gate.spec.ts+++ b/src/common/ci-privilege-gate.spec.ts@@ -257,3 +257,35 @@ describe('a job that can mint a publish credential pins global npm installs andexpect(pythonInstallsInOidcJobs(file)).toEqual([]);});});++/**+ * The operator-facing text around the entrypoint, the probes and the backup scripts states what they+ * do. Each of these was once true of an earlier version and outlived the change that made it false.+ */+describe('deployment docs describe what the entrypoint, probes and backup scripts do', () => {+ const root = path.join(__dirname, '..', '..');+ const read = (file: string): string => fs.readFileSync(path.join(root, file), 'utf8');++ // Session auto-start is detached: boot does not wait for it, so no probe covers it.+ it('does not claim the startupProbe covers session restore', () => {+ expect(read('docs/13-horizontal-scaling.md')).not.toMatch(/off during boot \([^)]*session/);+ expect(read('scripts/check-chart-behaviour.mjs')).not.toMatch(/sessions to restore/);+ });++ // The chart's liveness budget lives in statefulset.yaml; a copied figure goes stale.+ it('does not restate the chart liveness budget in session comments', () => {+ for (const file of ['src/modules/session/session.service.ts', 'src/modules/session/session.service.spec.ts']) {+ expect(read(file)).not.toMatch(/the chart's (?:budget )?is ~\d+s/);+ }+ });++ // The startupProbe suspends liveness until it first succeeds, so a closed port at boot meets it, not liveness.+ it('names the startupProbe as the budget a closed port at boot runs against', () => {+ for (const file of ['src/modules/session/session.service.ts', 'src/modules/session/session.service.spec.ts']) {+ const text = read(file).replace(/\n\s*\/\/ ?/g, ' ');+ expect(text).not.toMatch(/every liveness\s+probe/i);+ expect(text).not.toMatch(/the chart's liveness budget/);+ expect(text).toMatch(/the chart's startupProbe budget/);+ }+ });+});diff --git a/src/modules/session/session.service.spec.ts b/src/modules/session/session.service.spec.tsindex 91be30c1b..2b88a1363 100644--- a/src/modules/session/session.service.spec.ts+++ b/src/modules/session/session.service.spec.ts@@ -8527,8 +8527,9 @@ describe('SessionService', () => {// Nest binds the HTTP listener only after every bootstrap hook settles, and a launch is a// Chromium start bounded by resolveEngineInitTimeoutMs (>= 60s) with a 2s throttle between- // sessions. Awaiting the loop here kept the port CLOSED for that whole time, so every liveness- // probe in the window was a connection refusal — the chart's budget is ~50s.+ // sessions. Awaiting the loop here kept the port CLOSED for that whole time, so every probe in+ // the window was a connection refusal, and the chart's startupProbe budget+ // (charts/openwa/templates/statefulset.yaml) is a fixed number of seconds.it('returns without waiting for a launch that has not finished', async () => {process.env.AUTO_START_SESSIONS = 'true';(repository.find as jest.Mock).mockResolvedValue([{ id: 'a', name: 'A' }]);diff --git a/src/modules/session/session.service.ts b/src/modules/session/session.service.tsindex 626eb88e4..400004d1f 100644--- a/src/modules/session/session.service.ts+++ b/src/modules/session/session.service.ts@@ -214,9 +214,10 @@ export class SessionService implements OnModuleDestroy, OnModuleInit, OnApplicat// hook has settled, and this loop's duration is unbounded: one engine initialization is at least// 60s (resolveEngineInitTimeoutMs) and there is a 2s throttle between sessions, so a host with// ten authenticated sessions kept the port CLOSED — not unhealthy, closed — for ten minutes.- // Every liveness probe in that window is a connection refusal, and no probe budget can cover a- // bound that scales with the session count: the chart's is ~50s and the Dockerfile HEALTHCHECK- // encodes the same expectation. Awaited on shutdown so a launch in flight is accounted for.+ // Every probe in that window is a connection refusal, and no probe budget can cover a bound that+ // scales with the session count: the chart's startupProbe budget (statefulset.yaml), which governs+ // boot, is a fixed number of seconds, and the Dockerfile HEALTHCHECK encodes the same expectation.+ // Awaited on shutdown so a launch in flight is accounted for.this.autoStartRun = this.autoStartSessions().catch((error: unknown) => {// Previously this rejected out of the hook and aborted boot, so a transient database error// during the session scan took the whole gateway down rather than the auto-start.