Skip to content

Free, Open Source, Self-Hosted WhatsApp API Gateway

docs(security): cover demoted keys and Kubernetes in admin key recovery (#1820)

Yudhi Armyndharise1092d63f647
diff --git a/CHANGELOG.md b/CHANGELOG.md
index e6a4c81a0..bad41243e 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -114,7 +114,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Document the memory media messages and parked webhook deliveries hold during a media burst, and the settings that limit it.
- The storage migration guide says to check the store size and file count against the import caps before switching backends, and how to re-run an aborted import.
- The README and architecture docs describe the Local/S3 storage backend as the live media store, not a backup target.
-- Document how to recover a lost admin API key without revoking the other keys.
+- Document how to recover a lost admin API key without revoking the other keys, on source, Compose and Helm installs.
- The integration docs say `integration_delivery_failures` holds only failed inbound deliveries.
- The webhook runbook documents the ADMIN delivery-failure redrive and no longer says failed deliveries retry on their own.
diff --git a/docs/04-security-design.md b/docs/04-security-design.md
index c81def4b1..41ec9c25f 100644
--- a/docs/04-security-design.md
+++ b/docs/04-security-design.md
@@ -106,24 +106,30 @@ Keys are stored as one-way hashes, so a lost key cannot be read back from the da
found where it was saved, or replaced. Try these in order. If every key stopped working at once after
`API_KEY_PEPPER` was set or changed, follow the pepper recovery in §4.4 instead.
-1. **The bootstrap key file or `API_MASTER_KEY`.** The key seeded on first boot is kept in
- `data/.api-key` (or the `BOOTSTRAP_KEY_FILE` path) until that key is no longer active (revoked,
- expired or deleted). While the file holds a live key, the startup banner ends with
- `(full key in <path>)`. Read it with `cat data/.api-key`, or with
- `docker compose exec openwa-api cat /app/data/.api-key` on the compose deployment. If
- `API_MASTER_KEY` was set before first boot, the seeded key is that value, and it still works
- unless it was revoked since.
+1. **The bootstrap key file or `API_MASTER_KEY`.** Whenever `api_keys` is empty (first boot, or
+ after step 4 or the §4.4 pepper reset), the gateway seeds an ADMIN key, set to `API_MASTER_KEY`
+ when that is set, and keeps it in `data/.api-key` (or the `BOOTSTRAP_KEY_FILE` path) until that
+ key is no longer active (revoked, expired or deleted). While the file holds a live key, the
+ startup banner ends with `(full key in <path>)`. Read it with `cat data/.api-key`, with
+ `docker compose exec openwa-api cat /app/data/.api-key` on the compose deployment, or with
+ `kubectl exec openwa-0 -- cat /app/data/.api-key` on the Helm chart (release `openwa`). The key
+ manages keys only while it still meets step 2's conditions. Once it is demoted from `admin`,
+ given any `allowedSessions` or `allowedChats`, or given `allowedIps` that exclude your address,
+ the file and the banner line remain but key management is refused; go on to step 2 or 3.
2. **Another ADMIN key.** Any active, unexpired `admin` key without `allowedSessions` or
- `allowedChats` can manage keys: mint a replacement with `POST /api/auth/api-keys`
- (`{"name": "Admin", "role": "admin"}`; the raw key is returned once, in `apiKey`), then revoke the
- lost one with `POST /api/auth/api-keys/:id/revoke`, or do both from the dashboard's API Keys page.
+ `allowedChats`, used from an address its `allowedIps` (when set) admits, can manage keys: mint a
+ replacement with `POST /api/auth/api-keys` (`{"name": "Admin", "role": "admin"}`; the raw key is
+ returned once, in `apiKey`), then revoke the lost one with `POST /api/auth/api-keys/:id/revoke`,
+ or do both from the dashboard's API Keys page.
3. **Replace the key in the main database.** With no usable ADMIN key left, give the lost key's row
a new key; every other key keeps working. Stop the instance and copy the main database aside. Then
generate a key and its hash with the gateway's own hash function and configuration (process env,
`.env` and `data/.env.generated`, so the same `API_KEY_PEPPER`). Run it from the gateway's install
- directory, the one holding `dist/` and `data/`, so it reads that install's configuration:
+ directory, the one holding `dist/` and `data/`, so it reads that install's configuration. When a
+ service manager (systemd, pm2) sets `API_KEY_PEPPER` only in its unit, export the same value in
+ the shell first, or the hash will not match:
```bash
GEN="require('./dist/config/load-env'); const key = 'owa_k1_' + require('crypto').randomBytes(32).toString('hex'); console.log('key:', key); console.log('prefix:', key.slice(0, 12)); console.log('hash:', require('./dist/modules/auth/api-key-hash').hashApiKey(key, process.env.API_KEY_PEPPER))"
@@ -140,6 +146,15 @@ found where it was saved, or replaced. Try these in order. If every key stopped
sqlite3 data/main.sqlite "UPDATE api_keys SET keyHash = '<hash>', keyPrefix = '<prefix>', isActive = 1, expiresAt = NULL, allowedIps = NULL, allowedSessions = NULL, allowedChats = NULL WHERE id = '<id>'"
```
+ On the Helm chart (release `openwa`), run the generator in the running pod before stopping it,
+ since only that pod has the release's Secret and so its `API_KEY_PEPPER`:
+ `kubectl exec openwa-0 -- node -e "$GEN"`. Then scale the StatefulSet to 0, start the
+ `openwa-restore` helper pod on the release's data PVC as
+ [11 - Runbook: Restore from Backup](./11-operational-runbooks.md#runbook-restore-from-backup)
+ does, copy the database aside and run both `sqlite3` commands there with
+ `kubectl exec openwa-restore --` against `/app/data/main.sqlite`, then delete the pod and scale
+ back to 1.
+
Start the instance and use the new key. The update also clears the row's revoked state, expiry,
and IP, session and chat restrictions, so it can manage keys again; narrow it afterwards through
the API if it carried any. The old key no longer authenticates.
@@ -247,7 +262,7 @@ OpenWA serves plain HTTP on its port; terminate **TLS at your reverse proxy / lo
**Hardening you can apply today:** set `API_KEY_PEPPER`; restrict the data volume and database to the app's user; and encrypt at the infrastructure layer (LUKS / cloud-provider encrypted volumes / an encrypted managed Postgres) rather than relying on application-level field encryption, which is not implemented.
-**Setting or changing `API_KEY_PEPPER` on an existing install invalidates every key, the admin key included.** No stored hash matches any more, minting a key needs a valid ADMIN key, and a key is seeded only into an empty `api_keys` table, so nothing can be re-issued through the API. Set the pepper before first boot. On a running install: stop the instance, set the pepper, empty the table in the main database (the `MAIN_DATABASE_NAME` path, `data/main.sqlite` by default; on a source install `sqlite3 data/main.sqlite "DELETE FROM api_keys"`, on the compose deployment `docker compose run --rm --no-deps openwa-api sqlite3 /app/data/main.sqlite "DELETE FROM api_keys"`), start the instance, take the new ADMIN key from the startup banner or `data/.api-key` (or set `API_MASTER_KEY` beforehand to choose it), then re-issue the other keys.
+**Setting or changing `API_KEY_PEPPER` on an existing install invalidates every key, the admin key included.** No stored hash matches any more, minting a key needs a valid ADMIN key, and a key is seeded only into an empty `api_keys` table, so nothing can be re-issued through the API. Set the pepper before first boot. On a running install: stop the instance, set the pepper, empty the table in the main database (the `MAIN_DATABASE_NAME` path, `data/main.sqlite` by default; on a source install `sqlite3 data/main.sqlite "DELETE FROM api_keys"`, on the compose deployment `docker compose run --rm --no-deps openwa-api sqlite3 /app/data/main.sqlite "DELETE FROM api_keys"`), start the instance, take the new ADMIN key from the startup banner or `data/.api-key` (or set `API_MASTER_KEY` beforehand to choose it), then re-issue the other keys. On the Helm chart, empty the table while the StatefulSet is at 0 from the `openwa-restore` helper pod of [11 - Runbook: Restore from Backup](./11-operational-runbooks.md#runbook-restore-from-backup) (`kubectl exec openwa-restore -- sqlite3 /app/data/main.sqlite "DELETE FROM api_keys"`), delete the pod, and only then set the pepper in the release's Secret (`secretEnv.API_KEY_PEPPER`, or the `existingSecret`) and scale back to 1; a `helm upgrade` that sets the pepper also scales back to `replicaCount`. Read the new key with `kubectl exec openwa-0 -- cat /app/data/.api-key`.
## 4.5 Input Validation
diff --git a/docs/12-troubleshooting-faq.md b/docs/12-troubleshooting-faq.md
index 5d4f7262d..243923c76 100644
--- a/docs/12-troubleshooting-faq.md
+++ b/docs/12-troubleshooting-faq.md
@@ -1417,7 +1417,7 @@ retention/encryption to completed archives externally; see the [backup and resto
**Q: I lost the admin API key. How do I get back in?**
-> A: Keys are stored hashed, so the database cannot give it back. Check `data/.api-key` (or `BOOTSTRAP_KEY_FILE`) and the `API_MASTER_KEY` value the instance was first started with, then use another unscoped ADMIN key to mint a replacement. With neither, stop the instance and write a new key's hash into the lost key's row in the main database, which keeps every other key working. The steps are in [Recovering a lost admin key](./04-security-design.md#recovering-a-lost-admin-key).
+> A: Keys are stored hashed, so the database cannot give it back. Try the key in `data/.api-key` (or `BOOTSTRAP_KEY_FILE`), or the `API_MASTER_KEY` value set when the key table was last seeded, which still works while that key is an active, unscoped ADMIN key. Failing that, use another unscoped ADMIN key to mint a replacement. With neither, stop the instance and write a new key's hash into the lost key's row in the main database, which keeps every other key working. The steps are in [Recovering a lost admin key](./04-security-design.md#recovering-a-lost-admin-key).
### Webhook Questions
diff --git a/src/common/docs-governance.spec.ts b/src/common/docs-governance.spec.ts
index 6341b3d7f..816883c8f 100644
--- a/src/common/docs-governance.spec.ts
+++ b/src/common/docs-governance.spec.ts
@@ -297,6 +297,28 @@ describe('governance docs match the repository', () => {
);
});
+ // The bootstrap key file outlives a demotion or a scope change, and the Helm chart's StatefulSet
+ // leaves no pod to exec into once it is scaled to 0 for the database write.
+ it('gives the admin key recovery its Helm commands and the bootstrap key caveat', () => {
+ const security = read('docs/04-security-design.md');
+ const recovery = between(security, '### Recovering a lost admin key', '\n## ');
+ expect(read('charts/openwa/templates/NOTES.txt')).toContain('default "/app/data/.api-key"');
+ expect(recovery).toContain('`kubectl exec openwa-0 -- cat /app/data/.api-key`');
+ expect(recovery).toMatch(/demoted from `admin`[\s\S]+go on to step 2 or 3/);
+ expect(recovery).toMatch(/given any `allowedSessions` or `allowedChats`, or given `allowedIps` that exclude/);
+ expect(recovery).toMatch(/`allowedIps` \(when set\) admits/);
+ expect(recovery).toContain('`kubectl exec openwa-0 -- node -e "$GEN"`');
+ expect(recovery).toContain('`kubectl exec openwa-restore --`');
+ const restore = between(read('docs/11-operational-runbooks.md'), '### Runbook: Restore from Backup', '\n### ');
+ expect(restore).toContain('name: openwa-restore');
+ expect(restore).toContain('claimName: data-openwa-0');
+ const pepper = between(security, '**Setting or changing `API_KEY_PEPPER`', '\n\n');
+ expect(pepper).toContain('`kubectl exec openwa-restore -- sqlite3 /app/data/main.sqlite "DELETE FROM api_keys"`');
+ const faq = between(read('docs/12-troubleshooting-faq.md'), '**Q: I lost the admin API key.', '\n\n### ');
+ expect(faq).not.toMatch(/first started with/);
+ expect(faq).toMatch(/last seeded, which still works while that key is an active, unscoped ADMIN key/);
+ });
+
it('keeps the README non-affiliation disclaimer', () => {
const section = between(read('README.md'), '## Disclaimer', '\n## ');
expect(section).toMatch(/not affiliated/);