Skip to content

The only tool you need to know what is happening and how to fix it.

feat: added storage docs (#197)

narrow-adventureae0f185d3ca1
diff --git a/docs/pages/client/js-sdk/sourcemap-upload.mdx b/docs/pages/client/js-sdk/sourcemap-upload.mdx
index 9f243d8c..40531eab 100644
--- a/docs/pages/client/js-sdk/sourcemap-upload.mdx
+++ b/docs/pages/client/js-sdk/sourcemap-upload.mdx
@@ -10,6 +10,12 @@ npm install -D @tracewayapp/sourcemap-upload
This provides the `traceway-sourcemaps` CLI command.
+## Getting an Upload Token
+
+Uploads authenticate with a dedicated **upload token**, not the project token your SDK uses. Generate it in the dashboard: open the **Connection** page for your project, scroll to the **Source Maps** section, and click **Generate Upload Token**. The token is per-project and can be copied from the same place later.
+
+Treat it like any other CI secret. **Regenerate** issues a new token and invalidates the current one immediately, so any pipeline still using the old token fails to upload until its secret is updated. Members with the `readonly` role cannot generate or regenerate tokens; ask an organization admin.
+
## Usage
```bash
@@ -21,6 +27,8 @@ traceway-sourcemaps \
The CLI recursively finds all `*.map` files in the specified directory and uploads them to Traceway via `POST /api/sourcemaps/upload`, together with the sibling `.js`/`.cjs`/`.mjs` bundle next to each map when one exists. The bundle is what lets the backend resolve the enclosing function name for each frame; without it, symbolication still resolves file, line, and column but leaves function names as the browser reported them.
+On self-hosted instances, uploaded files land in the backend's [blob storage](/server/storage); configure S3 or a persistent volume there before wiring uploads into CI, or the maps disappear when the container is recreated.
+
## CLI Options
| Flag | Required | Description |
@@ -90,6 +98,38 @@ Add the upload step to your `package.json` scripts:
Make sure `TRACEWAY_URL` and `TRACEWAY_SOURCEMAP_TOKEN` are set in your environment.
+## HTTP API
+
+The CLI is a thin wrapper around a single endpoint, so any CI pipeline can upload without Node.js:
+
+```
+POST /api/sourcemaps/upload
+Authorization: Bearer {upload token}
+Content-Type: multipart/form-data
+```
+
+Files go in the repeatable multipart field `files`. Upload each bundle together with its map:
+
+```bash
+curl --fail -X POST "https://your-traceway-instance.com/api/sourcemaps/upload" \
+ -H "Authorization: Bearer $TRACEWAY_SOURCEMAP_TOKEN" \
+ -F "files=@dist/assets/index-DZ8aDdLF.js" \
+ -F "files=@dist/assets/index-DZ8aDdLF.js.map"
+```
+
+The contract:
+
+| Behavior | Detail |
+|----------|--------|
+| Accepted extensions | `.map`, `.js`, `.cjs`, `.mjs`. Files with any other extension are silently skipped, not rejected. |
+| Size limit | 50MB per file. A larger file fails the request with `400` and an error message. |
+| Auth failure | Missing or invalid token returns `401`. |
+| Response | `200` with `{"uploaded": N}`, counting only the stored files (skipped files are excluded). |
+| Debug IDs | Detected server-side in the uploaded content; no extra fields or flags needed. |
+| Effect | Immediate. Cached resolvers for the uploaded filenames are evicted before the response, and precompiled [`.tw` resolvers](/learn/symbolication-js#the-tw-resolver-format) are rebuilt in the background so the first exception after a deploy finds a warm cache. |
+
+The multipart filename becomes the stored name, and stack frames match against the basename of the frame's file, so send plain filenames without directory paths. `curl -F` already does this: it uses the basename of the local path.
+
## How Maps Are Matched
There is nothing to configure at runtime. When an exception is captured, Traceway resolves each minified frame in two steps:
diff --git a/docs/pages/server/_meta.json b/docs/pages/server/_meta.json
index 968770d6..782f8e30 100644
--- a/docs/pages/server/_meta.json
+++ b/docs/pages/server/_meta.json
@@ -5,6 +5,7 @@
"minimal": "Minimal Container",
"sqlite": "SQLite",
"local-setup": "Local Setup",
+ "storage": "Blob Storage",
"haloy": "Haloy",
"capacity-monitoring": "Capacity & Self-Monitoring",
"upgrade-1-8": "Upgrading to 1.8"
diff --git a/docs/pages/server/all-in-one.mdx b/docs/pages/server/all-in-one.mdx
index 21a5ea52..c1a64ed0 100644
--- a/docs/pages/server/all-in-one.mdx
+++ b/docs/pages/server/all-in-one.mdx
@@ -13,11 +13,14 @@ docker run -d --name traceway \
-p 80:80 \
-v traceway-ch:/var/lib/clickhouse \
-v traceway-pg:/var/lib/postgresql/data \
+ -v traceway-storage:/app/storage \
ghcr.io/tracewayapp/traceway:latest
```
After starting, open `http://localhost/register` to create your first account.
+The `traceway-storage` volume holds [blob storage](/server/storage): uploaded source maps, session recordings, and AI traces. Without it those files are lost when the container is recreated. To use S3 instead of a volume, see [Blob Storage](/server/storage).
+
## Build from Source
Only needed if you want to run a custom or modified build:
@@ -29,6 +32,7 @@ docker run -d --name traceway \
-p 80:80 \
-v traceway-ch:/var/lib/clickhouse \
-v traceway-pg:/var/lib/postgresql/data \
+ -v traceway-storage:/app/storage \
traceway:latest
```
@@ -52,6 +56,8 @@ All environment variables have working defaults, no configuration required. Over
| `JWT_SECRET` | *(built-in default)* | JWT signing secret (min 32 chars) |
| `GIN_MODE` | `release` | Gin framework mode |
| `APP_BASE_URL` | *(none)* | Public URL of your Traceway instance (e.g. `https://traceway.example.com`). Used for SDK setup instructions, email links, and OAuth callback URLs. |
+| `STORAGE_TYPE` | `local` | Blob storage backend for source maps, session recordings, and AI traces: `local` or `s3`. See [Blob Storage](/server/storage) for the S3 variables. |
+| `STORAGE_PATH` | `./storage` | Folder for local blob storage, `/app/storage` in this image. Mount a volume there. Ignored when `STORAGE_TYPE=s3`. |
| `SESSION_RECORDING_RETENTION_DAYS` | `30` | Days to keep session recording files on disk under `STORAGE_PATH/recordings/`. Files older than the TTL are deleted hourly and on startup. No effect when `STORAGE_TYPE=s3`. Set to `0` to disable. |
> ClickHouse data retention is controlled per-table at the schema level (TTLs on `metric_points`, `metric_points_1m`, `metric_points_1h`, `log_records`); other telemetry tables are retained indefinitely until you drop monthly partitions manually. The `SQLITE_RETENTION_DAYS` setting only applies in SQLite mode and has no effect in this image.
diff --git a/docs/pages/server/docker-compose.mdx b/docs/pages/server/docker-compose.mdx
index 63625d1b..9cd5f3ac 100644
--- a/docs/pages/server/docker-compose.mdx
+++ b/docs/pages/server/docker-compose.mdx
@@ -25,6 +25,8 @@ services:
POSTGRES_PASSWORD: "traceway"
POSTGRES_SSLMODE: "disable"
JWT_SECRET: "change-this-to-a-secure-secret-at-least-32-chars"
+ volumes:
+ - traceway-storage:/app/storage
depends_on:
clickhouse:
condition: service_healthy
@@ -66,6 +68,7 @@ services:
volumes:
clickhouse-data:
postgres-data:
+ traceway-storage:
```
```bash
@@ -74,6 +77,8 @@ docker compose up -d
After starting, open `http://localhost/register` to create your first account.
+The `traceway-storage` volume holds [blob storage](/server/storage): uploaded source maps, session recordings, and AI traces. Without it those files are lost when the container is recreated. To use S3 instead of a volume, see [Blob Storage](/server/storage).
+
The `docker-compose.yml` in the repository root builds from source instead. Use that if you are actively developing Traceway or need to run a modified build (`docker compose up --build`).
## Configuration
@@ -95,6 +100,8 @@ Edit the environment variables in your compose file as needed:
| `POSTGRES_SSLMODE` | `disable` | PostgreSQL SSL mode |
| `JWT_SECRET` | `change-this-to-a-secure-secret-at-least-32-chars` | JWT signing secret (min 32 chars) |
| `APP_BASE_URL` | *(none)* | Public URL of your Traceway instance (e.g. `https://traceway.example.com`). Used for SDK setup instructions, email links, and OAuth callback URLs. |
+| `STORAGE_TYPE` | `local` | Blob storage backend for source maps, session recordings, and AI traces: `local` or `s3`. See [Blob Storage](/server/storage) for the S3 variables. |
+| `STORAGE_PATH` | `./storage` | Folder for local blob storage, `/app/storage` in the minimal image. Mount a volume there. Ignored when `STORAGE_TYPE=s3`. |
| `SESSION_RECORDING_RETENTION_DAYS` | `30` | Days to keep session recording files on disk under `STORAGE_PATH/recordings/`. A worker deletes older files hourly and on startup. No effect when `STORAGE_TYPE=s3`. Set to `0` to disable. |
> ClickHouse table TTLs (`metric_points`, `log_records`, etc.) are managed at the schema level; this stack does not run the SQLite retention worker.
diff --git a/docs/pages/server/haloy.mdx b/docs/pages/server/haloy.mdx
index 68418dba..89d040eb 100644
--- a/docs/pages/server/haloy.mdx
+++ b/docs/pages/server/haloy.mdx
@@ -181,6 +181,8 @@ Point your SDK's connection string at your Traceway domain:
| `GITHUB_CLIENT_ID` | *(unset)* | Optional. GitHub OAuth App client ID. Setting both GitHub variables enables the **Continue with GitHub** button. |
| `GITHUB_CLIENT_SECRET` | *(unset)* | Optional. GitHub OAuth App client secret. |
| `OAUTH_SESSION_SECRET` | *(falls back to `JWT_SECRET`)* | Optional. Cookie signing secret for the OAuth round-trip. |
+| `STORAGE_TYPE` | `local` | Blob storage backend for source maps, session recordings, and AI traces: `local` or `s3`. See [Blob Storage](/server/storage) for the S3 variables and persistence notes. |
+| `STORAGE_PATH` | `./storage` | Folder for local blob storage. Point it at a persistent mount, or use `STORAGE_TYPE=s3`. Ignored when `STORAGE_TYPE=s3`. |
| `SESSION_RECORDING_RETENTION_DAYS` | `30` | Optional. Days to keep on-disk session recordings under `STORAGE_PATH/recordings/`. Worker runs hourly and on startup. `0` disables; no effect when `STORAGE_TYPE=s3`. |
When configuring providers, set the callback URL on the provider side to `https://traceway.yourdomain.com/api/auth/callback/{google|github}`.
diff --git a/docs/pages/server/minimal.mdx b/docs/pages/server/minimal.mdx
index b3ea2697..558c69e3 100644
--- a/docs/pages/server/minimal.mdx
+++ b/docs/pages/server/minimal.mdx
@@ -11,6 +11,7 @@ docker pull ghcr.io/tracewayapp/traceway:minimal
docker run -d --name traceway \
-p 80:80 \
+ -v traceway-storage:/app/storage \
-e CLICKHOUSE_SERVER="clickhouse-host:9000" \
-e CLICKHOUSE_DATABASE="traceway" \
-e CLICKHOUSE_USERNAME="default" \
@@ -28,6 +29,8 @@ docker run -d --name traceway \
After starting, open `http://localhost/register` to create your first account.
+The `traceway-storage` volume holds [blob storage](/server/storage): uploaded source maps, session recordings, and AI traces. Without it those files are lost when the container is recreated. To use S3 instead of a volume, see [Blob Storage](/server/storage).
+
## Build from Source
Only needed if you want to run a custom or modified build:
@@ -55,6 +58,8 @@ Then use `traceway:minimal` instead of `ghcr.io/tracewayapp/traceway:minimal` in
| `POSTGRES_SSLMODE` | PostgreSQL SSL mode | `disable` |
| `JWT_SECRET` | JWT signing secret (min 32 chars) | `your-secret-here` |
| `APP_BASE_URL` | Public URL of your instance. Used for SDK setup instructions, email links, and OAuth callback URLs. | `https://traceway.example.com` |
+| `STORAGE_TYPE` | Blob storage backend for source maps, session recordings, and AI traces: `local` (default) or `s3`. See [Blob Storage](/server/storage) for the S3 variables. | `local` |
+| `STORAGE_PATH` | Folder for local blob storage. Defaults to `./storage`, which is `/app/storage` in this image. Mount a volume there. Ignored when `STORAGE_TYPE=s3`. | `/app/storage` |
| `SESSION_RECORDING_RETENTION_DAYS` | Days to keep on-disk session recordings under `STORAGE_PATH/recordings/`. Defaults to `30`. Worker runs hourly and on startup. `0` disables; no effect when `STORAGE_TYPE=s3`. | `30` |
## SSO (optional)
diff --git a/docs/pages/server/storage.mdx b/docs/pages/server/storage.mdx
new file mode 100644
index 00000000..5d4409d2
--- /dev/null
+++ b/docs/pages/server/storage.mdx
@@ -0,0 +1,80 @@
+# Blob Storage
+
+The backend keeps binary artifacts outside the databases, behind a single storage interface with two backends: the local filesystem (default) and S3. Everything in this page applies to all deployment flavors ([All-in-One](/server/all-in-one), [Minimal](/server/minimal), [Docker Compose](/server/docker-compose), [SQLite](/server/sqlite), [Haloy](/server/haloy)).
+
+What lives in blob storage:
+
+| Artifact | Keys | Written by |
+|----------|------|-----------|
+| Source maps and bundles | `sourcemaps/{projectId}/{filename}` and `sourcemaps/{projectId}/by-debug-id/{debugId}.js[.map]` | [`POST /api/sourcemaps/upload`](/client/js-sdk/sourcemap-upload), typically from CI |
+| Compiled `.tw` resolvers | same keys with the `.map`/`.js` suffix replaced by `.tw` | the [symbolication cache](/learn/symbolication-js#cache-backends-memory-and-disk), automatically |
+| Session recording segments | `recordings/{projectId}/...` | the [recording uploader](/server/capacity-monitoring) |
+| AI trace payloads | `ai-traces/{projectId}/...` | OTLP ingest |
+
+If you upload source maps from CI, this is where they land. Configure either a persistent volume (local mode) or S3 before relying on those uploads; with neither, the files sit inside the container and are lost when it is recreated, and symbolication silently stops matching.
+
+## Local Storage (default)
+
+With `STORAGE_TYPE` unset or set to `local`, blobs are written under `STORAGE_PATH`, which defaults to `./storage` relative to the backend working directory.
+
+Where that resolves to per image:
+
+| Image | Path | Persistence |
+|-------|------|-------------|
+| All-in-One (`traceway:latest`) | `/app/storage` | not a declared volume, mount one yourself |
+| Minimal (`traceway:minimal`) | `/app/storage` | not a declared volume, mount one yourself |
+| SQLite (`traceway:sqlite`) | `/data/storage` | covered by the image's `/data` volume |
+
+For the all-in-one and minimal images, add a mount so uploads survive container recreation:
+
+```bash
+docker run -d --name traceway \
+ ... \
+ -v traceway-storage:/app/storage \
+ ghcr.io/tracewayapp/traceway:minimal
+```
+
+Alternatively set `STORAGE_PATH` to a path that is already on a mounted volume.
+
+## S3 Storage
+
+Set `STORAGE_TYPE=s3` to offload blobs to S3 or any S3-compatible service (MinIO, Cloudflare R2, DigitalOcean Spaces). No volume is needed for blobs in this mode, and multiple backend instances share the same artifacts.
+
+```bash
+docker run -d --name traceway \
+ ... \
+ -e STORAGE_TYPE="s3" \
+ -e S3_BUCKET="your-bucket" \
+ -e S3_REGION="us-east-1" \
+ -e S3_ACCESS_KEY="your-access-key" \
+ -e S3_SECRET_KEY="your-secret-key" \
+ ghcr.io/tracewayapp/traceway:minimal
+```
+
+For non-AWS providers add `-e S3_ENDPOINT="https://s3.example.com"`, which also switches the client to path-style URLs. If `S3_ACCESS_KEY` and `S3_SECRET_KEY` are omitted, the AWS SDK falls back to the default credential chain (IAM role, environment, shared config), so on EC2/ECS/EKS you can rely on instance or task roles and pass no static keys at all.
+
+The backend needs `GetObject`, `PutObject`, and `DeleteObject` on the bucket. Startup fails fast if `S3_BUCKET` or `S3_REGION` is missing.
+
+## Configuration Reference
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `STORAGE_TYPE` | `local` | Blob storage backend. `local` or `s3` |
+| `STORAGE_PATH` | `./storage` | Folder for local blob storage (`/data/storage` in the SQLite image). Ignored when `STORAGE_TYPE=s3` |
+| `S3_BUCKET` | *(unset)* | Required when `STORAGE_TYPE=s3` |
+| `S3_REGION` | *(unset)* | Required when `STORAGE_TYPE=s3` |
+| `S3_ACCESS_KEY` | *(unset)* | Optional. Falls back to the default AWS credential chain when unset |
+| `S3_SECRET_KEY` | *(unset)* | Optional. Pair with `S3_ACCESS_KEY` |
+| `S3_ENDPOINT` | *(unset)* | Optional. Set for S3-compatible providers (MinIO, R2, Spaces). Enables path-style URLs |
+
+## Retention
+
+The `SESSION_RECORDING_RETENTION_DAYS` worker prunes recording files on local disk only; it is a no-op when `STORAGE_TYPE=s3`. Use a bucket lifecycle rule on the `recordings/` prefix instead.
+
+Source maps are retained indefinitely in both modes: the most recent upload of each filename is what symbolication uses, and stale `.tw` resolvers are replaced automatically on upload. If you upgraded from 1.7, the old versioned `sourcemaps/{projectId}/{version}/...` paths are dead data once the [flatten migration](/server/upgrade-1-8) completes and can be deleted.
+
+## Related Pages
+
+- [Source Maps](/client/js-sdk/sourcemap-upload): uploading maps from CI
+- [Symbolication (JS)](/learn/symbolication-js): how `.tw` resolvers and the cache tiers use this storage
+- [SQLite](/server/sqlite): single-container deployment with the same storage options