Free, Open Source, Self-Hosted WhatsApp API Gateway
TypeScript apibotgatewayself-hostedwhatsappwhatsapp-apiwhatsapp-automationwhatsapp-botwhatsapp-gatewaywhatsapp-gateway-apiwhatsapp-webwhatsapp-web-api
fix(scripts): make restore work on the compose volume and Helm PVC
Yudhi Armyndharisa307ad940a8c
diff --git a/docs/11-operational-runbooks.md b/docs/11-operational-runbooks.mdindex 089117a2e..6d737e66f 100644--- a/docs/11-operational-runbooks.md+++ b/docs/11-operational-runbooks.md@@ -433,7 +433,7 @@ curl -H "X-API-Key: $API_KEY" \# Started with docker-compose.dev.yml (the README Quick Start)? Add `-f docker-compose.dev.yml`# to every docker compose command in this runbook, the Rollback block included, and write `openwa`-# wherever a command names the `openwa-api` service (steps 6 and 7, rollback step 2).+# wherever a command names the `openwa-api` service (steps 6 and 7, rollback steps 2 and 3).# 4. Stop servicesdocker compose down@@ -503,15 +503,20 @@ curl -H "X-API-Key: $API_KEY" \# 1. Stop servicesdocker compose down-# 2. Check out the previous release and rebuild the image+# 2. Restore from the pre-upgrade backup (both DBs + sessions). The archive upgrade step 2 produced is+# "$BACKUP_DIR/openwa-backup-<timestamp>.tar.gz". The databases in place still hold the failed+# upgrade's data, so the restore refuses to touch them without --force. It runs in the image+# because the data lives in the openwa-data volume (see Runbook: Restore from Backup), and before+# the checkout below because an image older than 0.23.7 cannot move its safety snapshot off the+# read-only container root+docker compose run --rm --no-deps --entrypoint /app/scripts/restore.sh \+ -v "$BACKUP_DIR:/backups" -e OPENWA_RESTORE_SNAPSHOT_DIR=/backups -e TMPDIR=/backups \+ openwa-api /backups/openwa-backup-<timestamp>.tar.gz --force++# 3. Check out the previous release and rebuild the imagegit checkout v<old-version>docker compose build openwa-api-# 3. Restore from the pre-upgrade backup (both DBs + sessions) — the archive step 2 produced is-# "$BACKUP_DIR/openwa-backup-<timestamp>.tar.gz". The databases in place still hold the failed-# upgrade's data, so the restore refuses to touch them without --force-./scripts/restore.sh "$BACKUP_DIR/openwa-backup-<timestamp>.tar.gz" --force-# 4. Start with old versiondocker compose up -d@@ -668,6 +673,67 @@ docker compose up -dcurl -s -X POST -H "X-API-Key: <an-existing-key>" http://localhost:2785/api/auth/validate```+> Step 2 as written restores into `./data` in the checkout. The app reads that directory only on a+> bare-metal install or under `docker-compose.dev.yml`, which bind-mounts it. The production compose+> file keeps the data in the **named volume** `openwa-data` and the Helm chart in a PVC, so a host run+> there fills a directory the container never reads and still reports success. Run the script from the+> image against the volume instead, in place of step 2. Both mount the container root read-only, so+> `OPENWA_RESTORE_SNAPSHOT_DIR` (0.23.7 or later) puts the pre-restore snapshot of the data dir on a+> writable, persistent path, and `TMPDIR` keeps the extracted archive there too; allow free space for+> about twice the data plus the archive. `--force` is included because the volume of an existing+> install still holds its databases:+>+> ```bash+> # Compose: the entrypoint override runs the script as root, which can read the archive and write+> # the volume; the next start hands the restored files back to the app user.+> docker compose run --rm --no-deps --entrypoint /app/scripts/restore.sh \+> -v "$PWD/backups:/backups" -e OPENWA_RESTORE_SNAPSHOT_DIR=/backups -e TMPDIR=/backups \+> openwa-api /backups/openwa-backup-<timestamp>.tar.gz --force+>+> # Helm, for a release named openwa (`kubectl get statefulset,configmap,pvc` shows the names of+> # another): stop the pod, then run the script in a helper pod on the same PVC, with the release's image.+> kubectl scale statefulset/openwa --replicas=0+> kubectl wait --for=delete pod/openwa-0 --timeout=120s+> kubectl apply -f - <<'EOF'+> apiVersion: v1+> kind: Pod+> metadata:+> name: openwa-restore+> spec:+> restartPolicy: Never+> containers:+> - name: restore+> image: ghcr.io/rmyndharis/openwa:<version>+> command: ['sleep', 'infinity']+> envFrom:+> - configMapRef:+> name: openwa+> volumeMounts:+> - { name: data, mountPath: /app/data }+> - { name: work, mountPath: /restore }+> volumes:+> - name: data+> persistentVolumeClaim:+> claimName: data-openwa-0+> - name: work+> emptyDir: {}+> EOF+> kubectl wait --for=condition=Ready pod/openwa-restore --timeout=300s+> kubectl cp ./backups/openwa-backup-<timestamp>.tar.gz openwa-restore:/restore/backup.tar.gz+> # The image sets HOME=/app/data, and the script refuses a data dir that is the home directory, so+> # HOME is moved off it here (the compose files already set HOME=/tmp).+> kubectl exec openwa-restore -- env HOME=/tmp OPENWA_RESTORE_SNAPSHOT_DIR=/restore TMPDIR=/restore \+> ./scripts/restore.sh /restore/backup.tar.gz --force+> # The emptyDir goes away with the pod: copy off the snapshot the script named first.+> kubectl cp openwa-restore:/restore/data.pre-restore-<ts> ./backups/data.pre-restore-<ts>+> kubectl delete pod openwa-restore+> kubectl scale statefulset/openwa --replicas=1+> ```+>+> The PostgreSQL import in step 3 then reads the dump from the archive, because `./data` on the host+> is not the volume: `tar -xzOf ./backups/openwa-backup-<timestamp>.tar.gz ./database.sql | psql "$DATABASE_URL"`.+> On Helm, run the step 4 check through `kubectl port-forward` to the release's Service.+> **PostgreSQL restores are read as UTC.** From 0.23.6 the data connection binds, parses and defaults> every timestamp in UTC, and refuses to boot when its session is not on UTC> ([05 - Database Design](./05-database-design.md#timestamps-on-postgresql-are-utc)). A `database.sql`diff --git a/scripts/restore.sh b/scripts/restore.shindex c5b336934..19baf04b3 100755--- a/scripts/restore.sh+++ b/scripts/restore.sh@@ -24,6 +24,9 @@# OPENWA_DATA_DIR data directory to restore non-DB state into (default: ./data)# SESSION_DATA_PATH, BAILEYS_AUTH_DIR, STORAGE_LOCAL_PATH, PLUGINS_DIR# override the corresponding state directories+# OPENWA_RESTORE_SNAPSHOT_DIR+# where the data-dir safety snapshot goes (default: next to the data dir);+# needed when that parent is read-only, as in the shipped container## Stop the OpenWA app before restoring. A snapshot of the current data dir is taken# first so a bad restore can be undone.@@ -273,10 +276,14 @@ if [ "$FORCE" -ne 1 ]; thenfifi-# Safety snapshot of whatever is there now.+# Safety snapshot of whatever is there now, next to the data dir unless OPENWA_RESTORE_SNAPSHOT_DIR+# names another directory. The shipped compose file and Helm chart mount the data dir as a volume+# under a read-only root, where that sibling cannot be written.if [ -d "$DATA_DIR" ] && [ -n "$(ls -A "$DATA_DIR" 2>/dev/null || true)" ]; then- SAFETY="${DATA_DIR%/}.pre-restore-$RESTORE_TIMESTAMP"+ SAFETY_DIR="${OPENWA_RESTORE_SNAPSHOT_DIR:-$(dirname "$DATA_DIR")}"+ SAFETY="${SAFETY_DIR%/}/$(basename "$DATA_DIR").pre-restore-$RESTORE_TIMESTAMP"log "Snapshotting current data dir -> $SAFETY"+ mkdir -p "$SAFETY_DIR"cp -pR "$DATA_DIR" "$SAFETY"fidiff --git a/scripts/smoke-test-backup-restore.sh b/scripts/smoke-test-backup-restore.shindex f30ba762e..0242d9a12 100755--- a/scripts/smoke-test-backup-restore.sh+++ b/scripts/smoke-test-backup-restore.sh@@ -15,6 +15,8 @@# (i) the data-store half of that guard refuses on its own# (j) a probe that fails or prints no usable count leaves the target counted as live# (k) an operator's sqlite3 rc file changes neither answer of the guard (skipped without sqlite3)+# (l) an unwritable BACKUP_DIR fails before anything is staged (skipped as root)+# (m) OPENWA_RESTORE_SNAPSHOT_DIR takes the data-dir snapshot off a read-only parent (skipped as root)## Usage: ./scripts/smoke-test-backup-restore.sh# Requires: bash, tar, node (restore.sh path resolution). sqlite3 is optional (see (c) and (k)).@@ -24,7 +26,7 @@ set -euo pipefail# would aim a case at a real install, and restore replaces the state directories wholesale, so every# case starts from none of them and sets exactly the paths it uses.unset OPENWA_DATA_DIR BACKUP_DIR DATABASE_TYPE MAIN_DATABASE_NAME DATABASE_NAME SESSION_DATA_PATH \- BAILEYS_AUTH_DIR STORAGE_LOCAL_PATH PLUGINS_DIR PLUGIN_STATE_DIR+ BAILEYS_AUTH_DIR STORAGE_LOCAL_PATH PLUGINS_DIR PLUGIN_STATE_DIR OPENWA_RESTORE_SNAPSHOT_DIRREPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"BACKUP="$REPO_ROOT/scripts/backup.sh"@@ -500,5 +502,34 @@ elseecho "SKIP: (l) running as root, which ignores the permission bits this case relies on"fi+echo ""+echo "==> (m) OPENWA_RESTORE_SNAPSHOT_DIR takes the data-dir snapshot off a read-only parent"+# The shipped compose file and Helm chart mount the data dir as a volume under a read-only root, so+# the snapshot's default place next to it cannot be written and the restore stopped there.+if [ "$(id -u)" -ne 0 ]; then+ M="$WORK/m"+ mkdir -p "$M/root/data" "$M/snapshots"+ printf 'mike-before\n' >"$M/root/data/.api-key"+ chmod a-w "$M/root"+ set +e+ OUT_M="$(cd "$M" && MAIN_DATABASE_NAME="$M/root/data/main.sqlite" DATABASE_NAME="$M/root/data/openwa.sqlite" \+ OPENWA_DATA_DIR="$M/root/data" OPENWA_RESTORE_SNAPSHOT_DIR="$M/snapshots" "$RESTORE" "$ARCHIVE_H" 2>&1)"+ RC_M=$?+ set -e+ chmod u+w "$M/root"+ if [ "$RC_M" -ne 0 ]; then+ fail "(m) restore failed with a writable OPENWA_RESTORE_SNAPSHOT_DIR: $OUT_M"+ fi+ if [ "$(cat "$M"/snapshots/data.pre-restore-*/.api-key 2>/dev/null || true)" != "mike-before" ]; then+ fail "(m) the data-dir snapshot is not under OPENWA_RESTORE_SNAPSHOT_DIR"+ fi+ if [ "$(db_fingerprint "$M/root/data/main.sqlite")" != "hotel-archive-main" ]; then+ fail "(m) the restore did not put the archived main DB in place"+ fi+ pass "(m) data-dir snapshot written under OPENWA_RESTORE_SNAPSHOT_DIR, restore completes"+else+ echo "SKIP: (m) running as root, which ignores the permission bits this case relies on"+fi+echo ""echo "All smoke tests passed!"