Free, Open Source, Self-Hosted WhatsApp API Gateway
TypeScript apibotgatewayself-hostedwhatsappwhatsapp-apiwhatsapp-automationwhatsapp-botwhatsapp-gatewaywhatsapp-gateway-apiwhatsapp-webwhatsapp-web-api
docs(ops): update metrics, webhook, release and upgrade notes
Yudhi Armyndharis4dff8e063339
diff --git a/docs/04-security-design.md b/docs/04-security-design.mdindex 357c066c0..f298e9c44 100644--- a/docs/04-security-design.md+++ b/docs/04-security-design.md@@ -382,7 +382,7 @@ The windows above run at the routing layer, after a request body has been buffer- Requests without an active API key (public ingress webhooks, unauthenticated callers) share a pool of a quarter of the budget, never less than twice `BODY_SIZE_LIMIT` (so half of the default budget), and are charged at most `BODY_SIZE_LIMIT` each. The rest is kept for requests that carry an active key. With `INFLIGHT_BODY_BUDGET_BYTES` below three times `BODY_SIZE_LIMIT` that reserve is smaller than one full-size body, and at twice `BODY_SIZE_LIMIT` or less there is none.- A request with an active key may hold at most half of the budget per key, wherever it connects from. An unkeyed client (IPv6 on its /64) may hold at most half of the unkeyed pool, or one `BODY_SIZE_LIMIT` if that is larger, so one full-size body from one address leaves room for other unkeyed senders once `INFLIGHT_BODY_BUDGET_BYTES` is at least twice `BODY_SIZE_LIMIT`; below that the room left is the budget minus one `BODY_SIZE_LIMIT`, and at a budget equal to `BODY_SIZE_LIMIT` a single full-size unkeyed body refuses every other body-carrying request, ingress deliveries included, with `503` until it finishes. Behind a reverse proxy without `TRUSTED_PROXIES`, every unkeyed request, ingress included, keys on the proxy address and draws on that one share, so at the defaults a single full-size unkeyed upload refuses ingress deliveries with `503` until it finishes.- Keys are recognised from an in-memory index of active keys, refreshed at once when this node changes a key and every 30 seconds for a direct write to this node's main database. Keys are per node, so a change made on another node never reaches it. The index only picks the tier; the API-key guard still authorises every request.-- A request holds its declared `Content-Length` until it finishes. After a 15-second grace it must keep up the pace that lands the whole body within `REQUEST_TIMEOUT_MS` (by default 25 MiB in 300 seconds, about 85 KiB/s); a request that falls behind is dropped and its reservation released. A chunked body holds a 1 MiB placeholder, or the bytes that have arrived once they pass it, and must deliver the placeholder at the same pace (about 3.4 KiB/s by default). If that growth crosses the budget, its client's share, or (without a key) the unkeyed pool, that request is aborted.+- A request holds its declared `Content-Length` until its response has been sent or the connection closes, not until its body has been read, so a slow handler (an engine send, a typing simulation) keeps its share charged while it works. After a 15-second grace it must keep up the pace that lands the whole body within `REQUEST_TIMEOUT_MS` (by default 25 MiB in 300 seconds, about 85 KiB/s); a request that falls behind is dropped and its reservation released. A chunked body holds a 1 MiB placeholder, or the bytes that have arrived once they pass it, and must deliver the placeholder at the same pace (about 3.4 KiB/s by default). If that growth crosses the budget, its client's share, or (without a key) the unkeyed pool, that request is aborted.- A body that sends nothing for 15 seconds is dropped and its reservation released.- Requests without a body are never refused for lack of budget.diff --git a/docs/08-development-guidelines.md b/docs/08-development-guidelines.mdindex 3996696b8..473097d17 100644--- a/docs/08-development-guidelines.md+++ b/docs/08-development-guidelines.md@@ -919,7 +919,7 @@ const sessions = await sessionRepo### Caching StrategyMemoize a hot, expensive read in-process with a TTL, as `StatsService` does for its aggregate-responses (`STATS_CACHE_TTL_MS`, default 30000; `0` disables the memo). `CacheService` is an optional+responses (`STATS_CACHE_TTL_MS`, default 30000; `0` disables the memo). Each finished computation also drops the entries that have expired, including those of deleted sessions nobody asks for again. `CacheService` is an optionalRedis client that stores no values and only answers `isAvailable()` for the infra status probe, so norequest path reads from it (see [3.13.3 Cache Service](./03-system-architecture.md#3133-cache-service)).diff --git a/docs/09-testing-strategy.md b/docs/09-testing-strategy.mdindex 58a227200..7a03c4568 100644--- a/docs/09-testing-strategy.md+++ b/docs/09-testing-strategy.md@@ -259,7 +259,7 @@ request's superseded run; push runs on `main` are never cancelled or queued behi| `lint` | backend ESLint, full-program TypeScript check, formatting, version consistency, .dockerignore context, OpenAPI snapshot, SDK routes and webhook events against the contract, contract coverage per SDK, SDK docs against the shipped client surface, client wire shapes (`check:contract-shapes` — the JavaScript SDK's, dashboard's, Python's, Go's and Java's hand-written types against the OpenAPI schemas; the PHP client returns untyped arrays and has no types layer to gate) || `audit` | dependency security audit of all three npm trees (root, `dashboard/` and `sdk/javascript/`); when npm's audit endpoint cannot answer, the root audit skips with a warning annotation here while the dashboard and SDK audits, a plain `npm audit`, fail closed, so the job still goes red until a re-run; the release gate and the weekly scan fail (`CHECK_AUDIT_REQUIRED=1`) || `test` | backend coverage run, repo-file drift gates (`test:docs`), script unit tests (node:test), e2e smoke tests |-| `test-postgres` | real PostgreSQL 16 service, backend build, migration smoke, and the PostgreSQL specs (FTS provider, boot-migration lock, message list ordering, entity synchronize, UTC pin, webhook recovery, send idempotency keys, replace-all import, session content lifetimes, two-node session delete, lid mapping ownership) |+| `test-postgres` | real PostgreSQL 16 service, backend build, migration smoke, and the PostgreSQL specs (FTS provider, boot-migration lock, message list ordering, entity synchronize, UTC pin, webhook recovery, send idempotency keys, replace-all import, restore without optional tables, message selection restore, session content lifetimes, two-node session delete, lid mapping ownership) || `dashboard` | dashboard install, lint, formatting, type-check, i18n parity, build, unit tests || `scripts-smoke` | shellcheck on `docker-entrypoint.sh` and every `scripts/*.sh`, plus the backup/restore smoke test || `chart` | helm lint, helm template with default and fully-toggled values, kubeconform on both renders, the rendered-behaviour check, actionlint on the workflows |diff --git a/docs/10-devops-infrastructure.md b/docs/10-devops-infrastructure.mdindex 7c297eeaa..179fca743 100644--- a/docs/10-devops-infrastructure.md+++ b/docs/10-devops-infrastructure.md@@ -436,7 +436,8 @@ LOG_FORMAT=json# REDIS_ENABLED=false# REDIS_HOST=localhost# REDIS_PORT=6379-# REDIS_TLS=false # true for a Redis that requires TLS (not the built-in container); every client+# REDIS_TLS=false # true for a Redis that requires TLS (not the built-in container: the boot refuses REDIS_TLS=true+# # and a named REDIS_USERNAME beside it, as it serves plain TCP and only the default user); every client# # (cache, rate limits, queue, WebSocket fan-out) uses it. Private CA: NODE_EXTRA_CA_CERTS in the launch# # environment (shell, systemd, compose environment:), not in .env# REDIS_CACHE_DB=1 # logical database for the cache@@ -506,13 +507,12 @@ WEBHOOK_DISPATCH_MAX_QUEUED=1000# quarter of WEBHOOK_DISPATCH_MAX_QUEUED behind that limit and sheds the rest, so other sessions# keep room. Opt-in events such as message.receipt are bounded per receiving webhook instead: a# quarter of WEBHOOK_DISPATCH_CONCURRENCY deliveries run at once, divided among the session's-# webhooks that name the event and one at least for each, and three quarters of+# webhooks that name the event, and three quarters of# WEBHOOK_DISPATCH_MAX_QUEUED events wait; further ones are dropped, not replayed, and counted in# openwa_webhook_delivery_failures_total. A webhook whose receiver never answers holds only its own-# places, but the pool is shared: a session holds up to four at once (one per webhook past four), and-# another session's events wait for a slot once such sessions together hold all of them, which-# is four sessions at the defaults. A webhook's share is fixed when its queue is created and kept while-# it has events, so receipt webhooks added meanwhile can briefly raise a session above four. There is+# places, but the pool is shared: a session holds at most a quarter of it at once (four at the+# defaults) however many of its webhooks name the event, and another session's events wait for a slot+# once such sessions together hold all of them, which is four sessions at the defaults. There is# no process-wide cap on the events waiting.# Each running, parked or backoff-waiting inline delivery holds its event's inline media# (up to 4/3 x WEBHOOK_MEDIA_INLINE_MAX_BYTES as base64), about 1.3 GiB per process at the defaults@@ -930,6 +930,12 @@ aggregate overview from `StatsService` plus `process.memoryUsage()`, memoizes thshort TTL (~5s, so back-to-back scrapes don't repeat the DB scan), and exposes it at`GET /api/metrics`.+A scrape waits at most 2 seconds for that overview. When the read fails or is still pending after that,+the database-derived series are left out and `openwa_stats_available` reads 0, while the process, HTTP,+webhook and queue series are served as usual. A read still pending is shared by the scrapes that arrive+meanwhile, so they answer without waiting again and add no query of their own, whatever+`STATS_CACHE_TTL_MS` is. The 2 seconds bound the wait, not the query: nothing cancels it in the database.+Access is **disabled by default**: the endpoint returns **404** unless `METRICS_TOKEN` is set. Whenset, scrapers must send `Authorization: Bearer <token>` (compared with `timingSafeEqual`); a missing orwrong token returns 401. After 10 failed attempts from one client IP within a minute, that client is@@ -950,7 +956,7 @@ export class MetricsService {async render(): Promise<string> {// Guarded: an unreachable data database must cost the DB-derived series, not the whole scrape.- // `overview` is null on failure, which is what openwa_stats_available reports.+ // `overview` is null on failure or after the 2-second wait, which is what openwa_stats_available reports.const overview = await this.readOverviewOrNull();const mem = process.memoryUsage();const lines: string[] = [];@@ -987,7 +993,7 @@ export class MetricsService {| `openwa_event_loop_delay_max_seconds` | gauge | none | Maximum event-loop delay since the previous uncached scrape || `openwa_unhandled_rejections_total` | counter | `kind` | Unhandled promise rejections since process start (`other` or `page_context_lost`) || `openwa_queue_jobs` | gauge | `queue`, `state` | BullMQ jobs per queue in `wait`/`active`/`delayed`/`failed` (cluster-wide, from Redis) |-| `openwa_stats_available` | gauge | — | 1 when the last overview read of the database-derived series below succeeded, 0 if it failed |+| `openwa_stats_available` | gauge | — | 1 when the last overview read of the database-derived series below succeeded, 0 if it failed or took over 2 s || `openwa_sessions_total` | gauge | — | Configured sessions || `openwa_sessions_active` | gauge | — | READY (active) sessions || `openwa_sessions` | gauge | `status` | Session count per status |@@ -1007,7 +1013,7 @@ without knowing anything about OpenWA. They come from `src/common/metrics/requesNot every row appears on every scrape, and the difference matters when you write alerts. Thedatabase-derived series (`openwa_sessions*`, `openwa_messages*`) are **omitted entirely** when the-overview cannot be read — `openwa_stats_available` is what tells the two cases apart, so alert on it+overview cannot be read within 2 seconds — `openwa_stats_available` is what tells the two cases apart, so alert on itrather than reading a missing series as zero. `openwa_send_pacing_refusals_total` appears only oncethe governor has refused something. `openwa_queue_jobs` appears only with `QUEUE_ENABLED=true`, and aqueue whose counts cannot be read within 2 s is left out of that scrape. For these, `absent()` is the@@ -1027,9 +1033,9 @@ seen, so keep new renderers on that composition.> **The database-derived series can be absent.** `openwa_sessions_*`, `openwa_messages_*` and the per-status> breakdown come from `StatsService.getOverview()`, which is memoized for `STATS_CACHE_TTL_MS` (default 30 s,-> shared with `GET /api/stats/overview`) behind the 5 s render cache. If that read fails — an outage, a statement-> timeout, pool exhaustion, a `SQLITE_BUSY` under load — they are OMITTED rather than reported as zero, and-> `openwa_stats_available` goes to 0. The process, HTTP and webhook series keep being served, so `up` stays 1+> shared with `GET /api/stats/overview`) behind the 5 s render cache. If that read fails or takes over 2 s — an+> outage, a statement timeout, pool exhaustion, a `SQLITE_BUSY` under load — they are OMITTED rather than reported+> as zero, and `openwa_stats_available` goes to 0. The process, HTTP and webhook series keep being served, so `up` stays 1> and still means "the process is alive". Alert on `openwa_stats_available == 0` for the degradation itself;> an alert written as `openwa_sessions_active == 0` would never fire for it, and one written with `absent()`> would. Because of the two caches, `openwa_stats_available` can keep reporting 1, and the series their lastdiff --git a/docs/14-migration-guide.md b/docs/14-migration-guide.mdindex a334f0bd7..7ecbf3b8f 100644--- a/docs/14-migration-guide.md+++ b/docs/14-migration-guide.md@@ -895,6 +895,12 @@ skips: check `pg_index.indisvalid` and drop the invalid index before building it| `next` | A production boot refuses a `METRICS_TOKEN` that is the `.env.example` placeholder or a weak value the other secrets are checked against (`test`, `change-me`, `changeme`, `password`, `secret`, `admin`, `demo` and a few more), trimmed and case-insensitive | Set a long random token, or unset `METRICS_TOKEN` to keep `/api/metrics` disabled (it answers `404`); there is no length floor, so a short value that is not on the list still boots || `next` | Session deletion commits bounded batches (at most 2000 message rows on SQLite or 500 on PostgreSQL) before the final transaction. Current local fixtures took 23.2 s for 299,102 messages on SQLite and 11.9 s for 1,000,007 on PostgreSQL, each with about 2,000 outbox rows and 500 delivery failures. These are examples, not limits or comparisons with `0.25.0`; large outbox and failure histories can add substantially more time. Earlier batches remain committed after a later failure | Set client and proxy timeouts for the session size. Repeat DELETE to finish, or explicitly `POST /start` to keep the remaining session. Without a newer owner or explicit start, a failure after removal begins keeps the session stopped || `next` | `restore.sh` stops before it copies anything, naming the key, when a path from the archive's own `.env.generated` leads out of the data directory, or into a directory the restore refills | Set that key in the environment or `./.env` to restore there; the default paths and a leftover `./uploads` restore as before |+| `next` | A boot with `REDIS_TLS=true`, or a `REDIS_USERNAME` other than `default`, beside the built-in Redis (`REDIS_BUILTIN=true` with the bundled host) is refused, and so is an Infrastructure page save that would produce it; the container serves plain TCP and only its default user | Set `REDIS_TLS=false` and clear `REDIS_USERNAME` for the built-in Redis, or switch to an external Redis host with `REDIS_BUILTIN=false` |+| `next` | `POST .../groups/:groupId/membership-requests/approve` and `/reject` answer `400` for a body that is a JSON list or is sent in a content type other than `application/json` or `application/x-www-form-urlencoded`; both used to act on every pending request | Send `{}` or no body to act on every pending request, and name requesters in a JSON or form body |+| `next` | On whatsapp-web.js, `POST /api/sessions/:sessionId/pairing-code` answers `409` for a number that is the stored phone of another session of the gateway | Log that session out or delete it first, or link by QR |+| `next` | On SQLite the first boot checks the full-text index against all stored message text and rebuilds it when they differ, so the boot takes longer on a large database while the port is closed | Allow for it in the startup probe; later boots skip the check unless `DATABASE_SYNCHRONIZE=true` |+| `next` | The idempotency key of recurring webhook events (`message.edited`, `message.reaction`, `session.*`, `group.*`, `presence.update`) ends in a random id minted per dispatch instead of an ISO timestamp | Dedupe on the whole key; a receiver that reads its last segment as a time must stop |+| `next` | A `conversationId` JSON pointer of an ingress route is read by RFC 6901 tokens: a trailing or doubled slash names an empty key, which the old reader skipped | Remove a stray slash from such a pointer; a delivery it no longer resolves falls back to the per-instance lane || `0.25.0` | Once teardown begins (SIGTERM, SIGINT, `POST /api/infra/restart`) every request except `/api/health`, `/api/health/*`, the dashboard's HTML pages and CORS preflights answers `503` with `Connection: close`, `GET /api/infra/health` included | Probe `/api/health/live` for liveness that survives teardown, and retry a `503` once the instance is up again || `0.25.0` | `stop_grace_period` (Docker) and `terminationGracePeriodSeconds` (Kubernetes) go from 45 to 75 s, and a stop that runs 67 s past the shutdown delay exits with status 1 instead of 137 | Raise a pinned 45 s value to at least 75 s, plus the excess of `SHUTDOWN_DELAY_MS` over 3 s and of `WEBHOOK_SHUTDOWN_DRAIN_MS` over 5 s; on Helm upgrade with `--reset-then-reuse-values` (3.14 or later) or `-f values.yaml`, since plain `--reuse-values` keeps the 0.24.0 defaults; a container that drops all capabilities and starts as root must add `KILL` to the capabilities 0.24.0 shipped, or a stop never reaches the app; update exit-code alerts || `0.25.0` | `400` instead of `201` or `200` for a phone number WhatsApp reports as unregistered (Baileys; `503` when the lookup gets no answer) and for `vote-poll` with no matching option text (whatsapp-web.js) | Handle `400` (`POLL_OPTION_NOT_FOUND` carries `validOptions`) and `503` on these routes; an empty `options` list still clears a vote |diff --git a/docs/15-project-roadmap.md b/docs/15-project-roadmap.mdindex 6eb78f279..37d4a1e91 100644--- a/docs/15-project-roadmap.md+++ b/docs/15-project-roadmap.md@@ -739,8 +739,10 @@ flowchart TBT --> DD --> E[boot-smoke: run the image on amd64 + arm64]D --> F[image-scan: Trivy, CRITICAL/HIGH, fixable only]+ D --> R[release-tags: drop X.Y and latest when a newer release holds them]E --> G[promote: apply X.Y.Z, X.Y, latest to GHCR + Docker Hub]F --> G+ R --> GG --> H[verify-published: resolve every ref anonymously]H --> I[GitHub Release, notes from the CHANGELOG section]```@@ -754,6 +756,12 @@ GitHub Release is gated on all of it.A tag with any `-` suffix (for example `-rc.1`, `-beta.1`, `-alpha.1`) is a prerelease: it skips themutable `X.Y` and `latest` channels and is flagged prerelease on GitHub.+For a stable tag, `release-tags` reads which release `X.Y` and `latest` serve on each registry and leaves+a channel alone when it already serves a newer one, so releasing a patch to an older line or re-running an+old tag never moves a channel backwards; the exact `X.Y.Z` tag is still applied. A channel it cannot read+fails the release rather than counting as absent; only the registry's own "manifest unknown" answer does.+`verify-published` then expects only the tags `release-tags` selected.+### When a gate failsEverything from `promote` onward is skipped, so no release tag reaches either registry and no GitHub