Skip to content

Free, Open Source, Self-Hosted WhatsApp API Gateway

39.2 KB
Raw Blame

13 - Horizontal Scaling Guide

⚠️ PARTIALLY IMPLEMENTED — STILL DEPLOY replicas: 1

Supported topology remains exactly one API instance per session-data volume. Do not run the multi-replica examples below yet.

What now exists. Sessions carry an owner (nodeId) and a renewed lease (leaseExpiresAt). A process claims a session before starting its engine and refuses when another node holds a live claim, so two replicas can no longer both launch the same session — which is what corrupted the shared LocalAuth directory. A booting process also resets only the sessions it may claim, instead of reporting a live peer's sessions as disconnected. Claims are released on a clean shutdown and expire otherwise, so failover does not depend on one. NODE_ID names the process; it defaults to the hostname and must be stable across restarts and unique per running process: two processes sharing a hostname (host networking, pm2 cluster mode) must each set it, and a node that sees another process renewing leases under its NODE_ID logs duplicate_node_id.

A node that loses its claim gives up the engine. A lease can lapse while the process is perfectly healthy — a slow query is enough — after which a peer may legitimately take the session; while the holder can reach the database, renewal detects the loss and tears the local engine down at the next heartbeat, so any two-engine overlap is bounded to roughly one heartbeat interval (and a teardown failure is logged as an error rather than silently retried). A failed renewal is deliberately not read as a loss: the TTL is sized to absorb a database blip, and concluding otherwise would stop every healthy engine on the node. The price is the partition case: a holder that cannot reach the database keeps its engines, a peer may adopt the session once the lease lapses, and both engines run until the holder's first successful renewal after it reconnects, so the overlap lasts as long as the outage plus up to one heartbeat.

Bulk-send batches follow the same rule. A batch is only ever driven by the process holding its session's engine, so a booting replica now reaps only the batches whose sessions it may claim, instead of declaring a peer's in-flight batches FAILED while they are still sending. A data import likewise reports the sessions it could not reconcile because another node is running them.

Failover now completes on its own. A periodic takeover sweep (default every 30s, SESSION_TAKEOVER_SWEEP_MS) adopts sessions whose holder's lease lapsed — a crashed peer, or a recreated container whose new identity boots before its old lease expires. Only authenticated sessions in a running-or-should-be state are adopted; mid-pairing and operator-failed ones are left alone, and a cleanly stopped session releases its claim so it is never "lapsed". Adopting a session fails its stuck in-flight batches (no auto-resume — the dead node's already-sent messages are unknowable), and so does an explicit POST /start or POST /stop of a session whose holder's lease lapsed. The sweep's adoption is gated by the same AUTO_START_SESSIONS flag as boot auto-start; the sweep itself is not, and runs on every node, because it also has a job that starts nothing: a row a vanished node left ready, initializing, authenticating or action_required is marked disconnected once its lease expired more than two TTLs ago, which is three TTLs after the holder's last renewal plus up to one sweep interval. A qr_ready row is marked too while it has no phone, since the sweep never adopts a row without one; a qr_ready row with a phone keeps its status, so the correction never makes it adoptable. Nothing else revisits such a row, since the boot reset skips a foreign claim that is still live. A session an operator stopped (POST /stop or /force-kill), or whose DELETE failed part-way or died with its node, is never adopted, even when its claim did lapse, until an explicit POST /start. While versions are mixed during an upgrade, a node still on 0.25.0 does not read a DELETE's record: it can adopt such a session, and a POST /start it serves leaves the record in place. A stop during a delete preserves its claim, including repeated stops after its lease expires or becomes NULL. If another node takes over an expired claim, each later removal transaction checks the new owner and refuses the old delete. No transaction remains open across a hook. Successful adoption cleanup commits with its claim or expired-claim release, so delayed cleanup cannot fail a later owner's new batches. Older nodes do not preserve these delete holds, fence each removal transaction, or serialize adoption cleanup with ownership changes. Upgrade every node before relying on them; older backup importers also treat the combined delete/stop record as unknown. Older nodes also lack the final ownership check on a stop write, so they can still report a stop as successful while a peer starts the session. Upgrade every node before relying on that fence.

Known limitation: a holder that is alive but cannot reach the database for more than three lease TTLs minus one heartbeat (160s at defaults) is marked disconnected by a peer too, and does not write its status back when it reconnects. Its renewal still finds its own claim, so it detects no loss, and a steadily connected engine emits no new status. The row reads disconnected until that engine's status next changes.

Request routing now exists, opt-in via NODE_URL. When every node sets its own reachable URL (e.g. NODE_URL=http://10.0.0.5:2785), a session-scoped request landing on a non-owner is forwarded to the live owner and the owner's response is relayed back (x-openwa-served-by names it). The forward happens after API-key auth, carries the caller's credentials, is bounded by SESSION_PROXY_TIMEOUT_MS (default 60s), and is one hop only — a forwarded request is never forwarded again; one that still lands on a live non-owner (stale ownership, or a client-forged hop marker) is refused with a retryable 409 rather than executed there. A lapsed owner is deliberately NOT forwarded to: the local node handles the request, which is exactly how a takeover begins. Without NODE_URL the whole path is inert and single-node deployments pay nothing.

A path on NODE_URL is kept (0.23.7 and later) and put in front of the forwarded request's own path, which already starts with /api. Give one only when the node is reached through a reverse proxy under that prefix (NODE_URL=https://gw.example.com/node-a); a node reached directly takes none. A NODE_URL ending in /api forwards to /api/api/..., and every routed request answers 404.

The owner authenticates a forwarded request again, against its own key store. API keys and the audit log live in each node's main SQLite file (MAIN_DATABASE_NAME, default ./data/main.sqlite), not in the shared data database, and nothing replicates them. A forwarded key must therefore also exist on the owner. The receiving node checks its own copy's role, allowedIps, allowedSessions and allowedChats before forwarding, and the owner checks its copy again; a filtered list, such as a session's chats, is filtered by the owner's copy alone. A missing or narrower copy on the owner answers 401 or 403 (or a shorter list). A broader one is accepted and widens those lists, so keep the copies identical. A key created through the API or the dashboard gets a fresh random value and exists only on the node that created it. Matching keys come only from the same API_MASTER_KEY seeded on each node's first boot (it seeds only while no key exists), or from a copy of one node's main.sqlite taken while it is stopped (never one file shared over network storage), which authenticates only on nodes with the same API_KEY_PEPPER, since the copied hashes carry the source node's pepper. Each node's audit log records only the requests it handled.

List and stats routes answer from the node that received them. GET /api/sessions and GET /api/sessions/stats/overview name no session, so they are never forwarded. lastError, restriction, the stats active count and memoryUsage are that node's own view, not the owner's: the first two come from what this node's engines recorded, active counts only this node's engines, and memoryUsage is this process's. engineLoaded is the exception, since it also counts a live claim by another node. With routing on, GET /api/sessions/:sessionId is forwarded to the owner, so its answer is the owner's; start, stop, logout and force-kill are forwarded the same way, so a true means stop, logout and force-kill can act (and start answers 400, as it does on the owner). Without NODE_URL nothing is forwarded, so a node that does not hold the session still reports true but cannot act on it: start and stop answer 409 there, and logout and force-kill answer 400.

A failed forward says whether the owner could have acted. When the owner cannot be reached at all (connection refused, unresolvable or unusable NODE_URL), the answer is 503 and the request was not carried out, so it is safe to retry. A timeout answers 504 (no reply within SESSION_PROXY_TIMEOUT_MS), and any other failure answers 502 (the connection broke, possibly after the request was sent; a TLS certificate the forwarding node does not trust also lands here, with the cause code in its warning log): the owner may already have carried it out, so a non-idempotent call such as a message send must not be repeated blindly on either.

The lease compares timestamps written by different nodes, so their clocks must agree. Each node writes leaseExpiresAt from its own clock and reads every other node's the same way. Just before each renewal the holder's lease is only one TTL minus one heartbeat ahead, so a node whose clock runs more than about SESSION_LEASE_TTL_MS - SESSION_LEASE_HEARTBEAT_MS ahead (40s at defaults, 20s once one renewal is missed) sees healthy peers as lapsed and will take their sessions over. Run NTP (or any time sync) on every node — the default on ordinary server images — and treat any skew near that margin as a misconfiguration. The status correction reads the same timestamps: a node whose clock runs more than three TTLs minus one heartbeat ahead (160s at defaults) marks a healthy peer's sessions disconnected, even with AUTO_START_SESSIONS off, and nothing writes them back. The zone each node runs in is no longer part of this: on PostgreSQL the data connection binds, parses and defaults every timestamp in UTC (05 - Database Design), so two nodes in different zones, or one zone that observes daylight saving, still read each other's leases as the instants they were written at. Only the clocks have to agree. One exception, during an upgrade: a node still on 0.23.5 or earlier writes the lease in its own local wall time, so while versions are mixed the old cross-zone error above is back for as long as the older node keeps renewing. Running every node in TZ=UTC (the image default) removes it.

A forwarded request is throttled on both nodes. The receiving node counts it before forwarding, and the owner counts it again on arrival; with REDIS_ENABLED=true, and each peer node listed in the owner's TRUSTED_PROXIES (below), both counts land in the same shared bucket. Without that, the owner counts every request forwarded by a peer in one bucket keyed on that peer's address. Size the rate limits with that in mind for a routed deployment.

Forwards carry the client address in x-forwarded-for (inbound chain preserved, the observed peer appended). For an allowedIps-restricted key or the per-IP throttler to see the REAL client on forwarded calls, each node must list its peer nodes' addresses in its TRUSTED_PROXIES — otherwise the owner (correctly) ignores the chain and every forwarded request appears to come from the peer node itself.

WebSocket events now fan out across replicas when Redis is enabled (REDIS_ENABLED=true, the same flag the throttler and cache already use). The gateway broadcasts to rooms; a Redis pub/sub adapter attached to Socket.IO relays those broadcasts to every replica, so a client connected to node A receives an event raised on node B. Scope honestly: this distributes event fan-out only. The per-key WS rate-limit buckets (counted per replica) and the engine registry are still process-local. Without REDIS_ENABLED the adapter is inert and delivery is single-node, exactly as before. With it, every event of a one-to-one chat also costs the emitting node one or two lid_mappings lookups (one for a @lid chat, none for a group), even when no chat-restricted key exists: it cannot know whether another replica holds a restricted socket, so it names the chat in every form for the rooms to match.

The fan-out to other replicas is best effort and keeps no replay queue. A Redis that pauses for less than 10 s (a snapshot fork, a slow fsync) answers afterwards, and every event raised meanwhile reaches the other replicas late, at up to about 1,200 events a second; above that the cap described below applies first and drops the frames beyond it. A longer silence, a restart, a failover or an outage drops the connection a node publishes on, and what was written to it may be lost with it, or may still reach the other replicas late and after newer events, which their clients then apply out of order. The events that node raises while it is disconnected, plus its reconnect backoff of up to 5 s once Redis is back, are dropped for the clients connected to the other replicas: they are not delivered late and not twice. Clients connected to the node that raised an event are served directly, and webhooks do not use this path, so neither is affected. Memory is bounded in two ways. A Redis that sends nothing for 10 s has the connection destroyed, and the node holds nothing more for it; what it had already held, at most 10 s of events, is freed once the node has reconnected. A Redis that keeps answering, but slower than events are raised, is never silent for 10 s, so a cap bounds it instead: once the node holds 12,000 unanswered events (10 s at 1,000 events a second, with a fifth to spare) and the backlog has outlived 2 s, meaning the count has not been below 6,000 for that long, it drops further frames for the other replicas until the count is back below 6,000, with a warning when dropping starts and another, with the count, when it ends. The node can hold the traffic of those 2 s on top of the 12,000. A burst raised in a single tick is not cut as long as raising it past its 6,000th frame takes under 2 s: a healthy Redis answers it right after, and the node holds it until then. A synchronous burst that takes longer than that to raise past 6,000 frames is cut at 12,000 even against a healthy Redis, and the warning names the unanswered count. A replica whose own Redis connection is reconnecting also misses what the others publish meanwhile. The connection a replica receives on only listens, so it is pinged every 15 s and replaced when an answer takes over 30 s: one that stops answering without being closed (a proxy in the path that keeps its side open, a NAT or a firewall that drops the flow) costs that replica at most about 45 s of the others' events before it subscribes again. A Redis that pauses for up to 30 s keeps that connection, so the replica still receives late what the others had written before their own 10 s deadline; a longer silence while a ping waits replaces it, and what the others had published into that silence is then lost for that replica.

Mid-connection key eviction converges on a timer, not a broadcast, and only on one node. The node that processes a revoke, delete, or narrowing tears down that key's sockets synchronously, in the same request. Nothing is published to peers. Every node re-validates the keys behind its own live sockets against its own key store once a minute (EventsGateway.sweepApiKeyAuthorization, one batched read of the key ids currently holding sockets) and evicts on a row that is gone, inactive, expired, or whose role, allowedIps, allowedSessions, allowedChats or expiry no longer matches the snapshot the socket authenticated with. That catches an expiry, and a change written to the node's main database other than through its own API. Before this sweep, a revoke, delete or expiry that reached a node that way waited for the client's next subscribe, and a narrowing was never caught at all: it leaves the key valid, so only the new subscribe is rejected while every room joined earlier stays joined. REST reads the row per request, so the node that processed a change rejects the key's REST calls at once. Every other node keeps accepting the key, on REST and on its sockets, until the same change is made there: repeat a revoke, delete, expiry change or narrowing on every node.

What does not exist yet, and is why one replica is still the answer. The cross-replica gap just named (WS rate-limit state) remains process-local. API keys and the audit log are per node: each node keeps them in its own main SQLite file, so a key created, revoked or narrowed on one node is unchanged on the others (see the routing paragraph above). Leases are timed by each node's own clock, not the database's (see the clock paragraph above). Not every lifecycle path is fenced: the liveness watchdog and reconnect timers still act on whatever is in the local registry. BulkMessageService keeps its live batch state in process, so a takeover cannot resume a batch — only fail it. MCP/agent tool invocations execute on the node that received them rather than being forwarded.

Everything below (node affinity, replicas: 3) remains a design sketch until those land.

This guide explains a proposed design for deploying OpenWA in a horizontally scaled environment for high availability and increased capacity.

13.1 Architecture Overview

flowchart TB
    subgraph LB["Load Balancer"]
        NGINX[Nginx/Traefik]
    end

    subgraph Nodes["OpenWA Nodes"]
        N1[OpenWA Node 1]
        N2[OpenWA Node 2]
        N3[OpenWA Node 3]
    end

    subgraph Storage["Shared Storage"]
        PG[(PostgreSQL)]
        REDIS[(Redis)]
        S3[S3/MinIO<br/>Media Storage]
    end

    LB --> N1
    LB --> N2
    LB --> N3

    N1 --> PG
    N2 --> PG
    N3 --> PG

    N1 --> REDIS
    N2 --> REDIS
    N3 --> REDIS

    N1 --> S3
    N2 --> S3
    N3 --> S3

Key Principles

Principle Description
Session Affinity WhatsApp sessions are stateful and must stay on the same node
Shared Database PostgreSQL stores all persistent data across nodes
Redis for State Shared cache and queue coordination
Sticky Sessions Load balancer routes session requests to the correct node

13.2 Session Affinity Strategy

Since WhatsApp sessions maintain active connections (a browser instance for whatsapp-web.js, or a WebSocket for baileys — set via ENGINE_TYPE), they cannot be freely moved between nodes.

Strategy 1: Session-to-Node Mapping (Recommended)

This mapping ships: each sessions row records its owner in nodeId, nodeUrl, claimedAt and leaseExpiresAt (migrations AddSessionOwnership and AddSessionNodeUrl). No load balancer reads it: a node that receives a request for a session it does not own forwards it to the owner's NODE_URL, as the routing paragraph at the top of this guide describes.

Strategy 2: Consistent Hashing

Route sessions based on session ID hash. (Not implemented — no such routing helper exists in code; the sketch below is illustrative of the future design.)

function getNodeForSession(sessionId: string, nodes: string[]): string {
  const hash = crypto.createHash('md5').update(sessionId).digest('hex');
  const index = parseInt(hash.substring(0, 8), 16) % nodes.length;
  return nodes[index];
}

Strategy 3: Session Claim

Each node "claims" sessions on startup and releases them on shutdown. This is implemented: see the claim, lease and takeover description at the top of this guide.

13.3 Docker Swarm Deployment

docker-compose.swarm.yml

version: '3.8'

services:
  openwa:
    image: ghcr.io/rmyndharis/openwa:latest
    # Teardown takes up to 63 s with every datastore on; raise this by the amount SHUTDOWN_DELAY_MS or
    # WEBHOOK_SHUTDOWN_DRAIN_MS exceeds its default, or the 10 s default kills the task mid-teardown.
    stop_grace_period: 75s
    deploy:
      replicas: 1 # MUST stay 1 until multi-replica is supported — multiple replicas on one session volume corrupt WhatsApp auth
      update_config:
        parallelism: 1
        delay: 30s
      restart_policy:
        condition: on-failure
        max_attempts: 3
      resources:
        limits:
          memory: 2G
        reservations:
          memory: 512M
    environment:
      - NODE_ENV=production
      - DATABASE_TYPE=postgres
      - DATABASE_HOST=postgres
      - DATABASE_NAME=openwa
      - DATABASE_USERNAME=openwa
      - DATABASE_PASSWORD=${DB_PASSWORD}
      - REDIS_HOST=redis
      - QUEUE_ENABLED=true
      # The session-ownership identity: it names which process holds each session's engine, and
      # it must be STABLE across restarts (a value that changes makes the restarted process a new
      # node, which has to wait out its own previous lease). Defaults to the container hostname.
      - NODE_ID={{.Node.Hostname}}-{{.Task.Slot}}
    volumes:
      # The whole data tree, not only sessions/: main.sqlite (API keys, audit log), baileys/,
      # media and the generated secrets live there too, and a replaced task would lose them.
      - openwa-data:/app/data
    networks:
      - openwa-net
    depends_on:
      - postgres
      - redis

  postgres:
    image: postgres:16-alpine
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.role == manager
    environment:
      - POSTGRES_DB=openwa
      - POSTGRES_USER=openwa
      - POSTGRES_PASSWORD=${DB_PASSWORD}
    volumes:
      - postgres-data:/var/lib/postgresql/data
    networks:
      - openwa-net

  redis:
    image: redis:7-alpine
    deploy:
      replicas: 1
    command: redis-server --appendonly yes --maxmemory-policy noeviction
    volumes:
      - redis-data:/data
    networks:
      - openwa-net

  # NOTE (v0.4.0): OpenWA no longer ships a bundled Traefik container.
  # For TLS / public exposure, bring your own reverse proxy (Traefik, nginx,
  # Caddy, a cloud load balancer, etc.) and point it at openwa:2785.
  # See section 13.5 for Traefik / nginx config examples.

volumes:
  postgres-data:
  redis-data:
  openwa-data:

networks:
  openwa-net:
    driver: overlay

Deploy to Swarm

# Initialize swarm (if not already)
docker swarm init

# Deploy stack
docker stack deploy -c docker-compose.swarm.yml openwa

# Check status
docker service ls
docker service ps openwa_openwa

Do not scale the openwa service (docker service scale openwa_openwa=N). The openwa-data volume above is declared with the default local driver (not external), so Swarm creates one per node: replicas co-located on a single node share that directory and corrupt the WhatsApp auth state, while replicas placed on other nodes each get a fresh empty volume and start an unauthenticated engine instead. Either way the deployment breaks — see the warning at the top of this guide. Scaling only becomes safe once the gaps listed at the top of this guide are closed.

13.4 Kubernetes Deployment

k8s/namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: openwa

k8s/configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: openwa-config
  namespace: openwa
data:
  NODE_ENV: 'production'
  DATABASE_TYPE: 'postgres'
  DATABASE_HOST: 'postgres-service'
  DATABASE_PORT: '5432'
  DATABASE_NAME: 'openwa'
  REDIS_HOST: 'redis-service'
  REDIS_PORT: '6379'
  QUEUE_ENABLED: 'true'
  PORT: '2785'

k8s/secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: openwa-secrets
  namespace: openwa
type: Opaque
stringData:
  DATABASE_USERNAME: openwa
  DATABASE_PASSWORD: your-secure-password

k8s/deployment.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: openwa
  namespace: openwa
spec:
  serviceName: openwa-headless # must match the headless Service declared in k8s/service.yaml
  replicas: 1 # MUST stay 1 until multi-replica is supported — see the warning at the top of this guide
  selector:
    matchLabels:
      app: openwa
  template:
    metadata:
      labels:
        app: openwa
    spec:
      # The teardown can take up to 63 s on a clustered replica (src/common/services/shutdown-budget.ts), so
      # Kubernetes' default grace of 30 s would SIGKILL a slow stop half way. Match the shipped chart.
      terminationGracePeriodSeconds: 75
      # OS-level containment is defence in depth for the plugin sandbox (see docs/30-plugin-sandboxing.md):
      # it limits the privileges the API process holds, not what a plugin inside that process can reach.
      # Without it the API process, and every plugin loaded into it, keeps whatever the container runtime
      # grants by default, so a plugin that abuses Node built-ins (fs, net) can read host files / open raw
      # sockets outside the capability model. The shipped compose file runs the image read-only +
      # cap_drop:ALL, and the node process runs as the non-root openwa user; the manifest below mirrors
      # that so a k8s deploy is not silently weaker. By default the entrypoint starts as root to chown
      # /app/data, then drops to openwa (uid/gid 997) via gosu. To run non-root from the start instead,
      # add runAsNonRoot: true, runAsUser: 997 and runAsGroup: 997 here and remove the capabilities `add`
      # list below; the entrypoint then skips the chown and the drop (see podSecurityContext in
      # charts/openwa/values.yaml). fsGroup makes the volume writable by openwa in either mode;
      # OnRootMismatch re-owns it only when its root is wrong, not every file on every mount.
      securityContext:
        fsGroup: 997
        fsGroupChangePolicy: OnRootMismatch
      containers:
        - name: openwa
          image: ghcr.io/rmyndharis/openwa:latest
          ports:
            - containerPort: 2785
              name: http
          envFrom:
            - configMapRef:
                name: openwa-config
            - secretRef:
                name: openwa-secrets
          env:
            # The session-ownership identity — see the compose example above. It must be STABLE
            # across restarts, which a Deployment's pod name is NOT: use a StatefulSet (whose pod
            # names are ordinal and stable) for a routed multi-node deployment, or pin a value per
            # replica. A changing NODE_ID still converges (the old lease lapses and the sweep
            # adopts), but every restart then costs one lease TTL of downtime for its sessions.
            - name: NODE_ID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
          # Container-level hardening. readOnlyRootFilesystem requires every writable path to be an
          # explicitly mounted volume — /app/data (SQLite, sessions, media, plugin storage) and /tmp
          # (Chromium needs a writable HOME/XDG for whatsapp-web.js).
          securityContext:
            readOnlyRootFilesystem: true
            allowPrivilegeEscalation: false
            capabilities:
              drop: ['ALL']
              # The root entrypoint's chown, the gosu privilege drop, and init forwarding signals after it.
              add: ['CHOWN', 'DAC_OVERRIDE', 'FOWNER', 'KILL', 'SETGID', 'SETUID']
          resources:
            requests:
              memory: '512Mi'
              cpu: '250m'
            limits:
              memory: '2Gi'
              cpu: '1000m'
          volumeMounts:
            - name: openwa-data
              mountPath: /app/data
            - name: tmp
              mountPath: /tmp
          # Same probes as charts/openwa/templates/statefulset.yaml. The startupProbe holds liveness
          # off during boot (migrations, connect retry, plugin load, backfills), and every timeoutSeconds is
          # explicit because the 1s kubelet default is shorter than /ready's 3s database bound.
          startupProbe:
            httpGet:
              path: /api/health/live
              port: 2785
            periodSeconds: 5
            timeoutSeconds: 5
            failureThreshold: 60
          livenessProbe:
            httpGet:
              path: /api/health/live
              port: 2785
            initialDelaySeconds: 30
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 6
          readinessProbe:
            httpGet:
              path: /api/health/ready
              port: 2785
            initialDelaySeconds: 10
            periodSeconds: 5
            timeoutSeconds: 5
      volumes:
        - name: tmp
          emptyDir: {}
  volumeClaimTemplates:
    - metadata:
        name: openwa-data
      spec:
        accessModes: ['ReadWriteOnce']
        resources:
          requests:
            storage: 10Gi

k8s/service.yaml

apiVersion: v1
kind: Service
metadata:
  name: openwa-service
  namespace: openwa
spec:
  type: ClusterIP
  selector:
    app: openwa
  ports:
    - port: 80
      targetPort: 2785
      name: http
---
apiVersion: v1
kind: Service
metadata:
  name: openwa-headless
  namespace: openwa
spec:
  clusterIP: None
  selector:
    app: openwa
  ports:
    - port: 2785
      name: http

k8s/ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: openwa-ingress
  namespace: openwa
  annotations:
    nginx.ingress.kubernetes.io/affinity: 'cookie'
    nginx.ingress.kubernetes.io/session-cookie-name: 'openwa-session'
    nginx.ingress.kubernetes.io/session-cookie-max-age: '172800'
    # ingress-nginx refuses a request body over 1m by default, which base64 media sends exceed.
    # Keep this at or above BODY_SIZE_LIMIT (25mb by default).
    nginx.ingress.kubernetes.io/proxy-body-size: '25m'
spec:
  ingressClassName: nginx
  rules:
    - host: openwa.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: openwa-service
                port:
                  number: 80
  tls:
    - hosts:
        - openwa.example.com
      secretName: openwa-tls

Deploy to Kubernetes

The maintained chart is charts/openwa, a single-instance StatefulSet that already encodes the replicaCount: 1 constraint below; prefer it over hand-applied manifests. The raw manifests here stay for operators who do not use Helm.

# Apply the namespace first: a directory is applied in file-name order, and the other files name it
kubectl apply -f k8s/namespace.yaml

# Apply all manifests
kubectl apply -f k8s/

# Check pods
kubectl get pods -n openwa

# Check logs
kubectl logs -f statefulset/openwa -n openwa

Do not raise replicas above 1 (kubectl scale statefulset openwa --replicas=N). Each pod gets its own PVC, so extra replicas do not share a session directory — they each start their own unauthenticated engine, and with AUTO_START_SESSIONS=true every pod tries to drive the same configured sessions from the shared database. See the warning at the top of this guide.

13.5 Load Balancer Configuration

Traefik Dynamic Config

# traefik/dynamic-scaling.yml
http:
  routers:
    openwa:
      rule: 'Host(`openwa.example.com`)'
      service: openwa
      middlewares:
        - sticky-session

  middlewares:
    sticky-session:
      headers:
        customResponseHeaders:
          X-OpenWA-Node: '{{.Node}}'

  services:
    openwa:
      loadBalancer:
        sticky:
          cookie:
            name: openwa_node
            secure: true
            httpOnly: true
        servers:
          - url: 'http://openwa-1:2785'
          - url: 'http://openwa-2:2785'
          - url: 'http://openwa-3:2785'
        healthCheck:
          path: /api/health
          interval: 10s
          timeout: 3s

Nginx Upstream Config

upstream openwa {
    ip_hash;  # Sticky sessions based on client IP

    server openwa-1:2785 weight=1 max_fails=3 fail_timeout=30s;
    server openwa-2:2785 weight=1 max_fails=3 fail_timeout=30s;
    server openwa-3:2785 weight=1 max_fails=3 fail_timeout=30s;
}

server {
    listen 80;
    server_name openwa.example.com;

    # nginx refuses a request body over 1m by default, below the 25mb OpenWA itself accepts
    # (BODY_SIZE_LIMIT), so base64 media sends would answer 413 before reaching a node.
    client_max_body_size 25m;

    location / {
        proxy_pass http://openwa;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        # Session affinity cookie
        proxy_cookie_path / "/; SameSite=Strict; HttpOnly";
    }

    location /api/health {
        proxy_pass http://openwa;
        proxy_connect_timeout 5s;
        proxy_read_timeout 5s;
    }
}

13.6 Capacity Planning

Resource Requirements per Node

Sessions Memory CPU Disk
1-5 1 GB 0.5 vCPU 5 GB
5-10 2 GB 1 vCPU 10 GB
10-25 4 GB 2 vCPU 25 GB
25-50 8 GB 4 vCPU 50 GB

Scaling Guidelines

The replica count stays at 1 (see 13.3 and 13.4), so the only levers available today are vertical — adjust the resources limits/reservations in the manifests above — or run a second single-instance deployment with its own session volume and split sessions between them. Horizontal scale up / scale down becomes an option only once the gaps listed at the top of this guide are closed.

On one Docker daemon, the built-in datastores Dashboard > Infrastructure starts are shared, not per instance: their container names (openwa-postgres, openwa-redis, openwa-minio), their volumes and the openwa-network network are fixed, so a second instance that turns on Use Built-in attaches to the first instance's container and data. Let at most one instance on a daemon use them, and point the others at datastores of their own with their Use Built-in toggles off.

Each instance finds a bundled container by its com.openwa.service label anywhere on the daemon. Its status reads a running one as the datastore it uses whenever its own host for that datastore is spelled like the bundled one: postgres, redis, minio or the container name, in any letter case, with or without a trailing dot. A dashboard restart that then switches that datastore off or to another host stops the first container on the daemon with that label, which can be the other instance's. So give the other instances' datastores host names of their own and no com.openwa.* labels; a second copy of the shipped compose file needs its own project name and container names, and its app must reach its datastores under names other than postgres, redis and minio. An instance set up that way whose *_BUILTIN flags are not true leaves the other instance's containers running: its dashboard counts a container OpenWA created as its own only beside its own flag, and asks to stop none of them.

Metric Threshold Action
CPU > 80% 5 minutes Raise limits.cpu (StatefulSet); the Swarm block above declares no CPU constraint, so add one
Memory > 85% 5 minutes Raise the memory limit
CPU < 30% 15 minutes Lower requests.cpu — that is what the scheduler reserves; lowering limits.cpu only tightens throttling
Active sessions per instance > 20 - Move sessions to a second instance with its own volume

Throughput Projections

Design targets for 2 vCPU / 4GB RAM nodes, not measurements — no benchmark artifact in this repository backs any row, including the 1-node one, and multi-node operation is not implemented (see the warning at the top of this guide), so the 3- and 5-node rows could not have been run at all:

Nodes Sessions Messages/sec p95 Latency
1 10 50 150ms
3 30 150 180ms
5 50 250 200ms

13.7 Monitoring

Prometheus Metrics

OpenWA exports Prometheus text exposition at GET /api/metrics (openwa_* gauges and counters). The endpoint returns 404 until METRICS_TOKEN is set, and then requires that token as a Bearer:

# prometheus/prometheus.yml
scrape_configs:
  - job_name: 'openwa'
    static_configs:
      # Swarm service name (13.3). On Kubernetes there is no Service called `openwa` — scrape the
      # pod through the headless Service instead, e.g.
      # openwa-0.openwa-headless.openwa.svc.cluster.local:2785
      - targets: ['openwa:2785']
    metrics_path: '/api/metrics'
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/metrics_token
# prometheus/openwa-rules.yaml
groups:
  - name: openwa
    rules:
      - alert: HighMemoryUsage
        expr: container_memory_usage_bytes{container="openwa"} > 1.8e9
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'OpenWA node high memory usage'

      - alert: NodeDown
        expr: up{job="openwa"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: 'OpenWA node is down'

Health Check Endpoints

Endpoint Purpose
/api/health Basic health check — returns status and timestamp; version only for an authenticated caller
/api/health/live Liveness probe (static ok; reflects process liveness only)
/api/health/ready Readiness probe — verifies the main + data databases respond (returns 503 while draining or if a DB is down)