Skip to content

Free, Open Source, Self-Hosted WhatsApp API Gateway

docs(storage): explain the import caps before the backend switch (#1828)

Yudhi Armyndharis9dc6d2f86ef2
diff --git a/.env.example b/.env.example
index 0677761a0..d34686e85 100644
--- a/.env.example
+++ b/.env.example
@@ -706,7 +706,7 @@ WEBHOOK_SSRF_PROTECT=true
# Storage import/export limits (ADMIN /infra/storage/* endpoints):
# STORAGE_IMPORT_MAX_BYTES=209715200 # per-entry cap for a tar.gz import; aborts on overflow (default 200 MiB)
# STORAGE_IMPORT_MAX_ENTRIES=100000 # max entries in an import archive; aborts beyond this (default 100000)
-# STORAGE_IMPORT_MAX_TOTAL_BYTES=10737418240 # total bytes one import may write; aborts on overflow (default 10 GiB)
+# STORAGE_IMPORT_MAX_TOTAL_BYTES=10737418240 # total bytes one import may extract; aborts on overflow (default 10 GiB)
# The two STORAGE_EXPORT_* values are raw milliseconds: a value that is not a positive integer
# (`1h` included) fails the boot, and so does a TTL above 2147483647 ms (Node's timer ceiling).
# STORAGE_EXPORT_TTL_MS=3600000 # auto-delete an export archive after this long (default 1h)
diff --git a/CHANGELOG.md b/CHANGELOG.md
index f67104f9e..e6a4c81a0 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -28,6 +28,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- JavaScript SDK CI and release jobs install with `--engine-strict` wherever they run the tests, so an unsupported Node fails at install.
- The JavaScript SDK smoke check sends requests through both built clients, so the Node 18 CI lane covers the request path.
- Retire the whatsapp-web.js download-mimetype install patch; inbound media downloads now pass the mimetype themselves.
+- Storage import abort messages name the `STORAGE_IMPORT_MAX_BYTES` or `STORAGE_IMPORT_MAX_ENTRIES` setting that stopped them.
- CI and the weekly security scan run `npm audit` over the JavaScript SDK, and Dependabot watches its lockfile.
### Fixed
@@ -56,7 +57,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Use the `OPENWA_DATA_DIR` volume's `media/`, even before it exists, for a leftover `STORAGE_LOCAL_PATH=./uploads` in host-side `backup.sh` and `restore.sh` runs, instead of the working directory's `./uploads`.
- Read the `./data` database defaults and `./data/...` paths in `./.env` under `OPENWA_DATA_DIR` in host-side `backup.sh` and `restore.sh` runs, instead of the working directory's `./data`.
- Probe the configured `PORT` in the Docker image healthcheck instead of a fixed 2785.
-- Cap the total bytes one storage import writes at `STORAGE_IMPORT_MAX_TOTAL_BYTES`.
+- Cap the total bytes one storage import extracts at `STORAGE_IMPORT_MAX_TOTAL_BYTES`.
- Close the Baileys session proxy's fetch dispatcher when the session disconnects, logs out or is destroyed.
- Java SDK (next SDK release after 0.5.1): the request timeout also bounds the response body, so a stalled body raises `OpenWATimeoutError`.
- Keep a new dashboard template draft when an earlier template save finishes.
@@ -111,6 +112,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- The API reference, database design, runbook and glossary describe when the webhook outbox copy is retired after a sweep replay or a redrive, and which rows neither path recovers.
- Document the memory media messages and parked webhook deliveries hold during a media burst, and the settings that limit it.
+- The storage migration guide says to check the store size and file count against the import caps before switching backends, and how to re-run an aborted import.
- The README and architecture docs describe the Local/S3 storage backend as the live media store, not a backup target.
- Document how to recover a lost admin API key without revoking the other keys.
- The integration docs say `integration_delivery_failures` holds only failed inbound deliveries.
diff --git a/README.md b/README.md
index 9c13ecd2a..ec55af301 100644
--- a/README.md
+++ b/README.md
@@ -32,7 +32,7 @@
**OpenWA** is a free, open-source WhatsApp API Gateway designed for developers who need full control over their messaging infrastructure—without vendor lock-in or hidden paywalls.
-Built on a **pluggable architecture**, OpenWA lets you select database engines (SQLite/PostgreSQL), media storage backends (Local/S3), and cache layers (disabled/Redis) through configuration rather than application-code changes. The storage backend is the live store for status media and, when chat-media archiving is enabled, archived chat media; other message media is returned inline to API and webhook consumers. `scripts/backup.sh` copies only the local media directory, so an S3 bucket needs a backup of its own.
+Built on a **pluggable architecture**, OpenWA lets you select database engines (SQLite/PostgreSQL), media storage backends (Local/S3), and cache layers (disabled/Redis) through configuration rather than application-code changes. The storage backend is the live store for status media and, when chat-media archiving is enabled, archived chat media; other message media is returned inline to API and webhook consumers. Of the media, `scripts/backup.sh` archives only the local media directory, so an S3 bucket needs a backup of its own.
| | |
| ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
diff --git a/docs/03-system-architecture.md b/docs/03-system-architecture.md
index 7d147db89..4273f1ecd 100644
--- a/docs/03-system-architecture.md
+++ b/docs/03-system-architecture.md
@@ -1309,10 +1309,11 @@ internally on `storageType` — there is no `I*Adapter` interface, separate adap
`StorageFactory`. The two backends are `local` (the default; files under `./data/media`) and `s3`.
It is the live media store: the status store writes status media through `putFile` (under
`statuses/`) and sweeps orphans back out with `deleteFile`, and the storage export/import endpoints
-move its contents between backends. `scripts/backup.sh` copies only the local media directory, never
-the S3 bucket. Incoming and outgoing message media is returned inline to REST/webhook consumers and is
-**not** written through `StorageService` unless `CHAT_MEDIA_ARCHIVE_ENABLED=true`, which archives a copy
-under `chat-media/<sessionId>/` (media this account sent also needs `CHAT_MEDIA_ARCHIVE_OUTBOUND=true`).
+move its contents between backends. Of the media, `scripts/backup.sh` archives only the local media
+directory, never the S3 bucket. Incoming and outgoing message media is returned inline to REST/webhook
+consumers and is **not** written through `StorageService` unless `CHAT_MEDIA_ARCHIVE_ENABLED=true`, which
+archives a copy under `chat-media/<sessionId>/` (media this account sent also needs
+`CHAT_MEDIA_ARCHIVE_OUTBOUND=true`).
That copy is in addition to the inline one unless `MESSAGE_INLINE_MEDIA=archive`, which replaces the
row's inline base64 with an `{ omitted: true, archived: true }` marker once the file reads back intact.
On S3 every key sits under the `S3_KEY_PREFIX` root (default `media/`).
diff --git a/docs/06-api-specification.md b/docs/06-api-specification.md
index 7bc0e4160..be21b76ba 100644
--- a/docs/06-api-specification.md
+++ b/docs/06-api-specification.md
@@ -6010,7 +6010,7 @@ Import storage files from a `tar.gz` located inside the `data/` directory.
`failed` counts archive entries the store refused to write; a bad or traversing entry is skipped without failing the rest. `imported` is `false` when entries failed and none was written.
-**Errors:** `400` missing/out-of-`data/`/not-found path, or `Storage import failed: <reason>` when the file is not a readable gzip tar archive or exceeds the import resource caps (an abort keeps the entries written before it, since there is no rollback; re-run the import once the archive is fixed, as each entry overwrites) · `401` · `403` · `500` · `503` S3 configured but not reachable since boot (a later outage answers `200` with `imported: false` and the entries in `failed`)
+**Errors:** `400` missing/out-of-`data/`/not-found path, or `Storage import failed: <reason>` when the file is not a readable gzip tar archive or exceeds the import resource caps `STORAGE_IMPORT_MAX_BYTES` (per file), `STORAGE_IMPORT_MAX_ENTRIES` or `STORAGE_IMPORT_MAX_TOTAL_BYTES` (an abort keeps the entries written before it, since there is no rollback, and each entry overwrites on a re-run; a re-run under the same caps stops at the same entry, so for a cap abort first raise the matching setting on the destination as the migration guide describes, and for an unreadable archive re-run with a sound copy) · `401` · `403` · `500` · `503` S3 configured but not reachable since boot (a later outage answers `200` with `imported: false` and the entries in `failed`)
---
diff --git a/docs/14-migration-guide.md b/docs/14-migration-guide.md
index 1ea9a5200..54108298e 100644
--- a/docs/14-migration-guide.md
+++ b/docs/14-migration-guide.md
@@ -234,11 +234,31 @@ does not go back to `false` until a restart, so it shows the bucket came up, not
outage after that makes the count and the export fail with `500`, and the import answer `imported: false`
with the entries counted in `failed`.
+Check the import caps before the switch. The import counts every byte it extracts from the archive and
+stops before it would write past `STORAGE_IMPORT_MAX_TOTAL_BYTES` (default `10737418240`, 10 GiB), or
+past `STORAGE_IMPORT_MAX_ENTRIES` files (default `100000`), so when Step 1's `sizeBytes` or `count` is
+larger, set a higher value in the `.env` with the Step 3 changes. Compose passes `.env` values to the
+container only when it creates it, so the Step 4 `docker compose up -d` applies the value and a
+dashboard restart, which reuses the container, does not: on the dashboard route run
+`docker compose up -d` before Step 5, or set the value in `data/.env.generated` instead, which every
+restart reads (under compose it lives in the `openwa-data` volume, so edit it through
+`docker compose exec openwa-api`). The import also stops at any single file above
+`STORAGE_IMPORT_MAX_BYTES` (default `209715200`, 200 MiB). It holds each file in memory, about twice its
+size at peak, so raise that cap only while `OPENWA_MEM_LIMIT` (default `2g`) leaves that much headroom
+beside the running sessions, or the container can be killed mid-import. The export logs a warning for
+each of these three caps the archive is above, measured against the lower of this deployment's value and
+the shipped default, so a warning whose "this deployment" value is already above the archive's figure
+needs no change on this gateway. An aborted import keeps what it wrote, and a re-run under the same caps
+stops at the same file, so raise the matching setting on the destination as above (`.env` plus
+`docker compose up -d`, or `data/.env.generated` plus a restart) before re-running Step 5.
+
```bash
# Step 1: Check current storage file count
curl -s 'http://localhost:2785/api/infra/storage/files/count' \
-H 'X-API-Key: YOUR_KEY'
# Response: { "storageType": "local", "count": 150, "sizeBytes": 15000000 }
+# Compare sizeBytes with STORAGE_IMPORT_MAX_TOTAL_BYTES and count with STORAGE_IMPORT_MAX_ENTRIES
+# (see above) before Step 3.
# Step 2: Export all files as tar.gz
curl -s 'http://localhost:2785/api/infra/storage/export' \
@@ -246,6 +266,9 @@ curl -s 'http://localhost:2785/api/infra/storage/export' \
# Response: { "message": "Storage export completed", "download": "data/exports/storage-export-xxx.tar.gz" }
# The archive is auto-removed after STORAGE_EXPORT_TTL_MS (default 1h), so re-import it before then.
# It is written under data/ so it survives the restart in Step 4 and stays import-able.
+# Before Step 3, check `docker compose logs openwa-api` for an "Export contains" warning: one about
+# the per-entry import limit means raising STORAGE_IMPORT_MAX_BYTES with the Step 3 changes,
+# unless its "this deployment" value is already above the file's size.
# Step 3: Change the storage configuration
# Dashboard: Infrastructure > Amazon S3 + "Use Built-in MinIO Container"; save, then
@@ -269,11 +292,6 @@ curl -X POST 'http://localhost:2785/api/infra/storage/import' \
-d '{"filePath": "data/exports/storage-export-xxx.tar.gz"}'
```
-The import aborts once it has written more than `STORAGE_IMPORT_MAX_TOTAL_BYTES` (default 10 GiB). For a
-larger store, raise it on the destination before Step 5; the export logs a warning when it exceeds that
-limit. The import also aborts on any single file above `STORAGE_IMPORT_MAX_BYTES` (default 200 MiB), and
-the export warns about that too. An aborted import keeps what it wrote and can be re-run.
-
| Scenario | Support | Method |
| ---------------------------- | ------- | ------------------------ |
| Local → Built-in MinIO | ✅ | Export → Config → Import |
diff --git a/src/common/docs-governance.spec.ts b/src/common/docs-governance.spec.ts
index 6325c42ae..6341b3d7f 100644
--- a/src/common/docs-governance.spec.ts
+++ b/src/common/docs-governance.spec.ts
@@ -262,6 +262,27 @@ describe('governance docs match the repository', () => {
expect(read(file)).not.toMatch(/backup\s*\/\s*migration|media backup/i);
expect(read(file)).toMatch(/live media store/i);
}
+ // The table cell also says "live media store", so pin the README sentence carrying the warning.
+ expect(read('README.md')).toMatch(
+ /backup\.sh` archives only the local media directory, so an S3 bucket needs a backup/,
+ );
+ });
+
+ // The export writes one archive entry per file, so Step 1's count meets the entry cap at import.
+ it('has the storage migration check both import caps before the backend switch', () => {
+ const guide = read('docs/14-migration-guide.md');
+ const caps = between(guide, 'Check the import caps before the switch.', '\n\n').replace(/\s+/g, ' ');
+ expect(caps).toContain('`STORAGE_IMPORT_MAX_ENTRIES` files (default `100000`)');
+ expect(caps).toContain("Step 1's `sizeBytes` or `count`");
+ // The import buffers each file whole, so a raised per-entry cap must fit the container's memory.
+ expect(caps).toContain('`OPENWA_MEM_LIMIT` (default `2g`)');
+ const step1 = between(guide, '# Step 1: Check current storage file count', '# Step 2').replace(/\s+/g, ' ');
+ expect(step1).toContain('sizeBytes with STORAGE_IMPORT_MAX_TOTAL_BYTES');
+ expect(step1).toContain('count with STORAGE_IMPORT_MAX_ENTRIES');
+ // Step 1 cannot show the largest file, so the export warning is the only early per-file signal.
+ const step2 = between(guide, '# Step 2: Export all files as tar.gz', '# Step 3').replace(/\s+/g, ' ');
+ expect(step2).toContain('docker compose logs openwa-api');
+ expect(step2).toContain('STORAGE_IMPORT_MAX_BYTES');
});
// The Java transport waits on the whole exchange with the client timeout, so a slow media download
diff --git a/src/common/storage/storage-transfer.spec.ts b/src/common/storage/storage-transfer.spec.ts
index dae8b131f..7a8ff90f1 100644
--- a/src/common/storage/storage-transfer.spec.ts
+++ b/src/common/storage/storage-transfer.spec.ts
@@ -265,7 +265,7 @@ describe('importFromStream reports partial counts on an abort', () => {
).catch((err: unknown) => err);
expect(rejection).toBeInstanceOf(Error);
- expect((rejection as Error).message).toMatch(/2-entry limit/);
+ expect((rejection as Error).message).toMatch(/2-entry limit \(STORAGE_IMPORT_MAX_ENTRIES\)/);
expect(rejection).toMatchObject({ imported: 2, failed: 0 });
expect(written).toEqual(['a.jpg', 'b.jpg']);
} finally {
@@ -275,7 +275,7 @@ describe('importFromStream reports partial counts on an abort', () => {
});
});
-describe('importFromStream caps the total bytes one import writes', () => {
+describe('importFromStream caps the total bytes one import extracts', () => {
const withTotalCap = async <T>(cap: string, run: () => Promise<T>): Promise<T> => {
const prev = process.env.STORAGE_IMPORT_MAX_TOTAL_BYTES;
process.env.STORAGE_IMPORT_MAX_TOTAL_BYTES = cap;
diff --git a/src/common/storage/storage-transfer.ts b/src/common/storage/storage-transfer.ts
index 56c976e4c..b0e5ead61 100644
--- a/src/common/storage/storage-transfer.ts
+++ b/src/common/storage/storage-transfer.ts
@@ -9,7 +9,7 @@ import { LoggerService } from '../services/logger.service';
const DEFAULT_IMPORT_MAX_BYTES = 200 * 1024 * 1024;
/** Max number of entries an import archive may contain. Bounds an entry-count DoS. */
const DEFAULT_IMPORT_MAX_ENTRIES = 100_000;
-/** Cap on the bytes one import writes in total (10 GiB). The two caps above still allow terabytes. */
+/** Cap on the bytes one import extracts in total (10 GiB). The two caps above still allow terabytes. */
const DEFAULT_IMPORT_MAX_TOTAL_BYTES = 10 * 1024 * 1024 * 1024;
/**
@@ -258,7 +258,7 @@ export async function importFromStream(
}
if (++entryCount > maxEntries) {
stream.resume();
- fail(new Error(`Import aborted: archive exceeds the ${maxEntries}-entry limit`));
+ fail(new Error(`Import aborted: archive exceeds the ${maxEntries}-entry limit (STORAGE_IMPORT_MAX_ENTRIES)`));
return;
}
// Only regular files are media. A directory or link entry carries no content, and writing it
@@ -284,7 +284,11 @@ export async function importFromStream(
if (entryBytes > maxEntryBytes) {
entryAborted = true;
stream.resume(); // drain the remainder so the source can end
- fail(new Error(`Import aborted: entry "${key}" exceeds the ${maxEntryBytes}-byte per-entry cap`));
+ fail(
+ new Error(
+ `Import aborted: entry "${key}" exceeds the ${maxEntryBytes}-byte per-entry cap (STORAGE_IMPORT_MAX_BYTES)`,
+ ),
+ );
} else if (totalBytes > maxTotalBytes) {
// The entry that crosses the cap is dropped, so no import writes more than maxTotalBytes.
entryAborted = true;
diff --git a/src/common/storage/storage.service.spec.ts b/src/common/storage/storage.service.spec.ts
index 68c03565a..7304d010b 100644
--- a/src/common/storage/storage.service.spec.ts
+++ b/src/common/storage/storage.service.spec.ts
@@ -219,7 +219,9 @@ describe('StorageService import resource caps (decompression-bomb defense)', ()
process.env.STORAGE_IMPORT_MAX_BYTES = '8';
const gz = await makeTarGz([{ name: 'bomb.bin', data: 'far-more-than-eight-bytes' }]);
- await expect(service.importFromStream(Readable.from(gz))).rejects.toThrow(/byte|cap|exceed|large/i);
+ await expect(service.importFromStream(Readable.from(gz))).rejects.toThrow(
+ /8-byte per-entry cap \(STORAGE_IMPORT_MAX_BYTES\)/,
+ );
expect(fs.existsSync(path.join(localPath, 'bomb.bin'))).toBe(false);
});