Free, Open Source, Self-Hosted WhatsApp API Gateway
TypeScript apibotgatewayself-hostedwhatsappwhatsapp-apiwhatsapp-automationwhatsapp-botwhatsapp-gatewaywhatsapp-gateway-apiwhatsapp-webwhatsapp-web-api
docs(storage): explain the import caps before the backend switch (#1828)
Yudhi Armyndharis9dc6d2f86ef2
diff --git a/.env.example b/.env.exampleindex 0677761a0..d34686e85 100644--- a/.env.example+++ b/.env.example@@ -706,7 +706,7 @@ WEBHOOK_SSRF_PROTECT=true# Storage import/export limits (ADMIN /infra/storage/* endpoints):# STORAGE_IMPORT_MAX_BYTES=209715200 # per-entry cap for a tar.gz import; aborts on overflow (default 200 MiB)# STORAGE_IMPORT_MAX_ENTRIES=100000 # max entries in an import archive; aborts beyond this (default 100000)-# STORAGE_IMPORT_MAX_TOTAL_BYTES=10737418240 # total bytes one import may write; aborts on overflow (default 10 GiB)+# STORAGE_IMPORT_MAX_TOTAL_BYTES=10737418240 # total bytes one import may extract; aborts on overflow (default 10 GiB)# The two STORAGE_EXPORT_* values are raw milliseconds: a value that is not a positive integer# (`1h` included) fails the boot, and so does a TTL above 2147483647 ms (Node's timer ceiling).# STORAGE_EXPORT_TTL_MS=3600000 # auto-delete an export archive after this long (default 1h)diff --git a/CHANGELOG.md b/CHANGELOG.mdindex f67104f9e..e6a4c81a0 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -28,6 +28,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0- JavaScript SDK CI and release jobs install with `--engine-strict` wherever they run the tests, so an unsupported Node fails at install.- The JavaScript SDK smoke check sends requests through both built clients, so the Node 18 CI lane covers the request path.- Retire the whatsapp-web.js download-mimetype install patch; inbound media downloads now pass the mimetype themselves.+- Storage import abort messages name the `STORAGE_IMPORT_MAX_BYTES` or `STORAGE_IMPORT_MAX_ENTRIES` setting that stopped them.- CI and the weekly security scan run `npm audit` over the JavaScript SDK, and Dependabot watches its lockfile.### Fixed@@ -56,7 +57,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0- Use the `OPENWA_DATA_DIR` volume's `media/`, even before it exists, for a leftover `STORAGE_LOCAL_PATH=./uploads` in host-side `backup.sh` and `restore.sh` runs, instead of the working directory's `./uploads`.- Read the `./data` database defaults and `./data/...` paths in `./.env` under `OPENWA_DATA_DIR` in host-side `backup.sh` and `restore.sh` runs, instead of the working directory's `./data`.- Probe the configured `PORT` in the Docker image healthcheck instead of a fixed 2785.-- Cap the total bytes one storage import writes at `STORAGE_IMPORT_MAX_TOTAL_BYTES`.+- Cap the total bytes one storage import extracts at `STORAGE_IMPORT_MAX_TOTAL_BYTES`.- Close the Baileys session proxy's fetch dispatcher when the session disconnects, logs out or is destroyed.- Java SDK (next SDK release after 0.5.1): the request timeout also bounds the response body, so a stalled body raises `OpenWATimeoutError`.- Keep a new dashboard template draft when an earlier template save finishes.@@ -111,6 +112,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0- The API reference, database design, runbook and glossary describe when the webhook outbox copy is retired after a sweep replay or a redrive, and which rows neither path recovers.- Document the memory media messages and parked webhook deliveries hold during a media burst, and the settings that limit it.+- The storage migration guide says to check the store size and file count against the import caps before switching backends, and how to re-run an aborted import.- The README and architecture docs describe the Local/S3 storage backend as the live media store, not a backup target.- Document how to recover a lost admin API key without revoking the other keys.- The integration docs say `integration_delivery_failures` holds only failed inbound deliveries.diff --git a/README.md b/README.mdindex 9c13ecd2a..ec55af301 100644--- a/README.md+++ b/README.md@@ -32,7 +32,7 @@**OpenWA** is a free, open-source WhatsApp API Gateway designed for developers who need full control over their messaging infrastructure—without vendor lock-in or hidden paywalls.-Built on a **pluggable architecture**, OpenWA lets you select database engines (SQLite/PostgreSQL), media storage backends (Local/S3), and cache layers (disabled/Redis) through configuration rather than application-code changes. The storage backend is the live store for status media and, when chat-media archiving is enabled, archived chat media; other message media is returned inline to API and webhook consumers. `scripts/backup.sh` copies only the local media directory, so an S3 bucket needs a backup of its own.+Built on a **pluggable architecture**, OpenWA lets you select database engines (SQLite/PostgreSQL), media storage backends (Local/S3), and cache layers (disabled/Redis) through configuration rather than application-code changes. The storage backend is the live store for status media and, when chat-media archiving is enabled, archived chat media; other message media is returned inline to API and webhook consumers. Of the media, `scripts/backup.sh` archives only the local media directory, so an S3 bucket needs a backup of its own.| | || ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |diff --git a/docs/03-system-architecture.md b/docs/03-system-architecture.mdindex 7d147db89..4273f1ecd 100644--- a/docs/03-system-architecture.md+++ b/docs/03-system-architecture.md@@ -1309,10 +1309,11 @@ internally on `storageType` — there is no `I*Adapter` interface, separate adap`StorageFactory`. The two backends are `local` (the default; files under `./data/media`) and `s3`.It is the live media store: the status store writes status media through `putFile` (under`statuses/`) and sweeps orphans back out with `deleteFile`, and the storage export/import endpoints-move its contents between backends. `scripts/backup.sh` copies only the local media directory, never-the S3 bucket. Incoming and outgoing message media is returned inline to REST/webhook consumers and is-**not** written through `StorageService` unless `CHAT_MEDIA_ARCHIVE_ENABLED=true`, which archives a copy-under `chat-media/<sessionId>/` (media this account sent also needs `CHAT_MEDIA_ARCHIVE_OUTBOUND=true`).+move its contents between backends. Of the media, `scripts/backup.sh` archives only the local media+directory, never the S3 bucket. Incoming and outgoing message media is returned inline to REST/webhook+consumers and is **not** written through `StorageService` unless `CHAT_MEDIA_ARCHIVE_ENABLED=true`, which+archives a copy under `chat-media/<sessionId>/` (media this account sent also needs+`CHAT_MEDIA_ARCHIVE_OUTBOUND=true`).That copy is in addition to the inline one unless `MESSAGE_INLINE_MEDIA=archive`, which replaces therow's inline base64 with an `{ omitted: true, archived: true }` marker once the file reads back intact.On S3 every key sits under the `S3_KEY_PREFIX` root (default `media/`).diff --git a/docs/06-api-specification.md b/docs/06-api-specification.mdindex 7bc0e4160..be21b76ba 100644--- a/docs/06-api-specification.md+++ b/docs/06-api-specification.md@@ -6010,7 +6010,7 @@ Import storage files from a `tar.gz` located inside the `data/` directory.`failed` counts archive entries the store refused to write; a bad or traversing entry is skipped without failing the rest. `imported` is `false` when entries failed and none was written.-**Errors:** `400` missing/out-of-`data/`/not-found path, or `Storage import failed: <reason>` when the file is not a readable gzip tar archive or exceeds the import resource caps (an abort keeps the entries written before it, since there is no rollback; re-run the import once the archive is fixed, as each entry overwrites) · `401` · `403` · `500` · `503` S3 configured but not reachable since boot (a later outage answers `200` with `imported: false` and the entries in `failed`)+**Errors:** `400` missing/out-of-`data/`/not-found path, or `Storage import failed: <reason>` when the file is not a readable gzip tar archive or exceeds the import resource caps `STORAGE_IMPORT_MAX_BYTES` (per file), `STORAGE_IMPORT_MAX_ENTRIES` or `STORAGE_IMPORT_MAX_TOTAL_BYTES` (an abort keeps the entries written before it, since there is no rollback, and each entry overwrites on a re-run; a re-run under the same caps stops at the same entry, so for a cap abort first raise the matching setting on the destination as the migration guide describes, and for an unreadable archive re-run with a sound copy) · `401` · `403` · `500` · `503` S3 configured but not reachable since boot (a later outage answers `200` with `imported: false` and the entries in `failed`)---diff --git a/docs/14-migration-guide.md b/docs/14-migration-guide.mdindex 1ea9a5200..54108298e 100644--- a/docs/14-migration-guide.md+++ b/docs/14-migration-guide.md@@ -234,11 +234,31 @@ does not go back to `false` until a restart, so it shows the bucket came up, notoutage after that makes the count and the export fail with `500`, and the import answer `imported: false`with the entries counted in `failed`.+Check the import caps before the switch. The import counts every byte it extracts from the archive and+stops before it would write past `STORAGE_IMPORT_MAX_TOTAL_BYTES` (default `10737418240`, 10 GiB), or+past `STORAGE_IMPORT_MAX_ENTRIES` files (default `100000`), so when Step 1's `sizeBytes` or `count` is+larger, set a higher value in the `.env` with the Step 3 changes. Compose passes `.env` values to the+container only when it creates it, so the Step 4 `docker compose up -d` applies the value and a+dashboard restart, which reuses the container, does not: on the dashboard route run+`docker compose up -d` before Step 5, or set the value in `data/.env.generated` instead, which every+restart reads (under compose it lives in the `openwa-data` volume, so edit it through+`docker compose exec openwa-api`). The import also stops at any single file above+`STORAGE_IMPORT_MAX_BYTES` (default `209715200`, 200 MiB). It holds each file in memory, about twice its+size at peak, so raise that cap only while `OPENWA_MEM_LIMIT` (default `2g`) leaves that much headroom+beside the running sessions, or the container can be killed mid-import. The export logs a warning for+each of these three caps the archive is above, measured against the lower of this deployment's value and+the shipped default, so a warning whose "this deployment" value is already above the archive's figure+needs no change on this gateway. An aborted import keeps what it wrote, and a re-run under the same caps+stops at the same file, so raise the matching setting on the destination as above (`.env` plus+`docker compose up -d`, or `data/.env.generated` plus a restart) before re-running Step 5.+```bash# Step 1: Check current storage file countcurl -s 'http://localhost:2785/api/infra/storage/files/count' \-H 'X-API-Key: YOUR_KEY'# Response: { "storageType": "local", "count": 150, "sizeBytes": 15000000 }+# Compare sizeBytes with STORAGE_IMPORT_MAX_TOTAL_BYTES and count with STORAGE_IMPORT_MAX_ENTRIES+# (see above) before Step 3.# Step 2: Export all files as tar.gzcurl -s 'http://localhost:2785/api/infra/storage/export' \@@ -246,6 +266,9 @@ curl -s 'http://localhost:2785/api/infra/storage/export' \# Response: { "message": "Storage export completed", "download": "data/exports/storage-export-xxx.tar.gz" }# The archive is auto-removed after STORAGE_EXPORT_TTL_MS (default 1h), so re-import it before then.# It is written under data/ so it survives the restart in Step 4 and stays import-able.+# Before Step 3, check `docker compose logs openwa-api` for an "Export contains" warning: one about+# the per-entry import limit means raising STORAGE_IMPORT_MAX_BYTES with the Step 3 changes,+# unless its "this deployment" value is already above the file's size.# Step 3: Change the storage configuration# Dashboard: Infrastructure > Amazon S3 + "Use Built-in MinIO Container"; save, then@@ -269,11 +292,6 @@ curl -X POST 'http://localhost:2785/api/infra/storage/import' \-d '{"filePath": "data/exports/storage-export-xxx.tar.gz"}'```-The import aborts once it has written more than `STORAGE_IMPORT_MAX_TOTAL_BYTES` (default 10 GiB). For a-larger store, raise it on the destination before Step 5; the export logs a warning when it exceeds that-limit. The import also aborts on any single file above `STORAGE_IMPORT_MAX_BYTES` (default 200 MiB), and-the export warns about that too. An aborted import keeps what it wrote and can be re-run.-| Scenario | Support | Method || ---------------------------- | ------- | ------------------------ || Local → Built-in MinIO | ✅ | Export → Config → Import |diff --git a/src/common/docs-governance.spec.ts b/src/common/docs-governance.spec.tsindex 6325c42ae..6341b3d7f 100644--- a/src/common/docs-governance.spec.ts+++ b/src/common/docs-governance.spec.ts@@ -262,6 +262,27 @@ describe('governance docs match the repository', () => {expect(read(file)).not.toMatch(/backup\s*\/\s*migration|media backup/i);expect(read(file)).toMatch(/live media store/i);}+ // The table cell also says "live media store", so pin the README sentence carrying the warning.+ expect(read('README.md')).toMatch(+ /backup\.sh` archives only the local media directory, so an S3 bucket needs a backup/,+ );+ });++ // The export writes one archive entry per file, so Step 1's count meets the entry cap at import.+ it('has the storage migration check both import caps before the backend switch', () => {+ const guide = read('docs/14-migration-guide.md');+ const caps = between(guide, 'Check the import caps before the switch.', '\n\n').replace(/\s+/g, ' ');+ expect(caps).toContain('`STORAGE_IMPORT_MAX_ENTRIES` files (default `100000`)');+ expect(caps).toContain("Step 1's `sizeBytes` or `count`");+ // The import buffers each file whole, so a raised per-entry cap must fit the container's memory.+ expect(caps).toContain('`OPENWA_MEM_LIMIT` (default `2g`)');+ const step1 = between(guide, '# Step 1: Check current storage file count', '# Step 2').replace(/\s+/g, ' ');+ expect(step1).toContain('sizeBytes with STORAGE_IMPORT_MAX_TOTAL_BYTES');+ expect(step1).toContain('count with STORAGE_IMPORT_MAX_ENTRIES');+ // Step 1 cannot show the largest file, so the export warning is the only early per-file signal.+ const step2 = between(guide, '# Step 2: Export all files as tar.gz', '# Step 3').replace(/\s+/g, ' ');+ expect(step2).toContain('docker compose logs openwa-api');+ expect(step2).toContain('STORAGE_IMPORT_MAX_BYTES');});// The Java transport waits on the whole exchange with the client timeout, so a slow media downloaddiff --git a/src/common/storage/storage-transfer.spec.ts b/src/common/storage/storage-transfer.spec.tsindex dae8b131f..7a8ff90f1 100644--- a/src/common/storage/storage-transfer.spec.ts+++ b/src/common/storage/storage-transfer.spec.ts@@ -265,7 +265,7 @@ describe('importFromStream reports partial counts on an abort', () => {).catch((err: unknown) => err);expect(rejection).toBeInstanceOf(Error);- expect((rejection as Error).message).toMatch(/2-entry limit/);+ expect((rejection as Error).message).toMatch(/2-entry limit \(STORAGE_IMPORT_MAX_ENTRIES\)/);expect(rejection).toMatchObject({ imported: 2, failed: 0 });expect(written).toEqual(['a.jpg', 'b.jpg']);} finally {@@ -275,7 +275,7 @@ describe('importFromStream reports partial counts on an abort', () => {});});-describe('importFromStream caps the total bytes one import writes', () => {+describe('importFromStream caps the total bytes one import extracts', () => {const withTotalCap = async <T>(cap: string, run: () => Promise<T>): Promise<T> => {const prev = process.env.STORAGE_IMPORT_MAX_TOTAL_BYTES;process.env.STORAGE_IMPORT_MAX_TOTAL_BYTES = cap;diff --git a/src/common/storage/storage-transfer.ts b/src/common/storage/storage-transfer.tsindex 56c976e4c..b0e5ead61 100644--- a/src/common/storage/storage-transfer.ts+++ b/src/common/storage/storage-transfer.ts@@ -9,7 +9,7 @@ import { LoggerService } from '../services/logger.service';const DEFAULT_IMPORT_MAX_BYTES = 200 * 1024 * 1024;/** Max number of entries an import archive may contain. Bounds an entry-count DoS. */const DEFAULT_IMPORT_MAX_ENTRIES = 100_000;-/** Cap on the bytes one import writes in total (10 GiB). The two caps above still allow terabytes. */+/** Cap on the bytes one import extracts in total (10 GiB). The two caps above still allow terabytes. */const DEFAULT_IMPORT_MAX_TOTAL_BYTES = 10 * 1024 * 1024 * 1024;/**@@ -258,7 +258,7 @@ export async function importFromStream(}if (++entryCount > maxEntries) {stream.resume();- fail(new Error(`Import aborted: archive exceeds the ${maxEntries}-entry limit`));+ fail(new Error(`Import aborted: archive exceeds the ${maxEntries}-entry limit (STORAGE_IMPORT_MAX_ENTRIES)`));return;}// Only regular files are media. A directory or link entry carries no content, and writing it@@ -284,7 +284,11 @@ export async function importFromStream(if (entryBytes > maxEntryBytes) {entryAborted = true;stream.resume(); // drain the remainder so the source can end- fail(new Error(`Import aborted: entry "${key}" exceeds the ${maxEntryBytes}-byte per-entry cap`));+ fail(+ new Error(+ `Import aborted: entry "${key}" exceeds the ${maxEntryBytes}-byte per-entry cap (STORAGE_IMPORT_MAX_BYTES)`,+ ),+ );} else if (totalBytes > maxTotalBytes) {// The entry that crosses the cap is dropped, so no import writes more than maxTotalBytes.entryAborted = true;diff --git a/src/common/storage/storage.service.spec.ts b/src/common/storage/storage.service.spec.tsindex 68c03565a..7304d010b 100644--- a/src/common/storage/storage.service.spec.ts+++ b/src/common/storage/storage.service.spec.ts@@ -219,7 +219,9 @@ describe('StorageService import resource caps (decompression-bomb defense)', ()process.env.STORAGE_IMPORT_MAX_BYTES = '8';const gz = await makeTarGz([{ name: 'bomb.bin', data: 'far-more-than-eight-bytes' }]);- await expect(service.importFromStream(Readable.from(gz))).rejects.toThrow(/byte|cap|exceed|large/i);+ await expect(service.importFromStream(Readable.from(gz))).rejects.toThrow(+ /8-byte per-entry cap \(STORAGE_IMPORT_MAX_BYTES\)/,+ );expect(fs.existsSync(path.join(localPath, 'bomb.bin'))).toBe(false);});