Skip to content

The only tool you need to know what is happening and how to fix it.

docs updated

ddux7c92bb22eb1c
diff --git a/CLAUDE.md b/CLAUDE.md
index bc4f99c6..1180cfbe 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -244,6 +244,10 @@ POSTGRES_DATABASE=traceway
POSTGRES_USERNAME=traceway
POSTGRES_PASSWORD=
POSTGRES_SSLMODE=disable
+
+# Retention (see "Data Retention" section below)
+SQLITE_RETENTION_DAYS=30 # 0 to disable; only applies in SQLite mode
+SESSION_RECORDING_RETENTION_DAYS=30 # 0 to disable; only applies when STORAGE_TYPE=local
```
---
@@ -652,7 +656,9 @@ func (c *ReportController) Report(ctx *gin.Context) {
| `metric_records` | Time-series system metrics | Monthly |
| `endpoints` | Endpoint aggregates (materialized) | None |
| `archived_exceptions` | Archived/resolved exceptions | None |
-| `log_records` | OTel logs with 3 attribute maps (resource/scope/log), 30-day TTL | Daily (`toDate(timestamp)`) |
+| `log_records` | OTel logs with 3 attribute maps (resource/scope/log) | Daily (`toDate(timestamp)`) |
+
+> Data retention/TTLs are documented separately under **Data Retention** below.
#### Tables (PostgreSQL)
| Table | Purpose |
@@ -716,6 +722,46 @@ for _, item := range items {
- `SQLiteJSONMap` — implements `sql.Scanner`/`driver.Valuer` for `map[string]string` ↔ SQLite JSON TEXT
- Row types (e.g., `endpointRow`, `taskRow`) wrap domain models with these types for lit compatibility
+#### Data Retention
+
+Retention is handled in three different ways depending on the deployment.
+
+**1. ClickHouse — `TTL` clauses on the table itself.** Only a few tables have a TTL; everything else is kept indefinitely (operators can drop monthly partitions manually if needed).
+
+| Table | Retention | Source migration |
+|-------|-----------|------------------|
+| `metric_points` (raw) | **7 days** | `0034_add_ttl_metric_points.up.sql` |
+| `metric_points_1m` (1-min rollup) | **30 days** | `0035_add_ttl_metric_points_1m.up.sql` |
+| `metric_points_1h` (1-hour rollup) | **1 year** | `0036_add_ttl_metric_points_1h.up.sql` |
+| `log_records` | **30 days** | `0045_create_log_records.up.sql` |
+| All other CH tables (`transactions`, `exception_stack_traces`, `tasks`, `spans`, `sessions`, `ai_traces`, `session_recordings`, `fired_notifications`, `archived_exceptions`, `slow_endpoints`, `endpoints`, etc.) | **No TTL — retained indefinitely** | — |
+
+**2. SQLite — `retention.Start` worker** (`backend/app/retention/sqlite.go`). In SQLite mode, neither `db.DB` nor `db.TelemetryDB` has any built-in expiry, so a background worker fires once at startup and then every hour and runs a `DELETE FROM <table> WHERE <time_column> < cutoff` against each telemetry table plus `notification_history`.
+
+| Variable | Default | Notes |
+|----------|---------|-------|
+| `SQLITE_RETENTION_DAYS` | `30` | TTL in days. Set to `0` to disable the worker entirely. Has no effect outside SQLite mode. |
+
+Tables it prunes (and the column used):
+
+| Database | Table | Time column |
+|----------|-------|-------------|
+| Telemetry (`db.TelemetryDB`) | `endpoints`, `tasks`, `exception_stack_traces`, `spans`, `metric_points`, `session_recordings`, `ai_traces` | `recorded_at` |
+| Telemetry | `log_records` | `timestamp` |
+| Telemetry | `sessions` | `started_at` |
+| Telemetry | `fired_notifications` | `fired_at` |
+| Main (`db.DB`) | `notification_history` | `created_at` |
+
+`archived_exceptions` (per-hash flags) and `slow_endpoints` (per-endpoint config) are intentionally skipped — they are not time-series data.
+
+**3. On-disk session recordings — `retention.Start` worker** (`backend/app/retention/recordings.go`). Session recordings written to local disk (`STORAGE_TYPE=local`) accumulate under `<STORAGE_PATH>/recordings/`. A second worker walks that directory once at startup and then every hour and removes files whose `mtime` is older than the TTL, then prunes any directories left empty. The worker is a no-op when `STORAGE_TYPE=s3`.
+
+| Variable | Default | Notes |
+|----------|---------|-------|
+| `SESSION_RECORDING_RETENTION_DAYS` | `30` | TTL in days. Set to `0` to disable the worker. Only runs when `STORAGE_TYPE=local` (default). |
+
+The DB rows in `session_recordings` are pruned by the SQLite retention worker (above) or by ClickHouse TTL — they are intentionally not coupled to the disk cleanup. Controllers that read recordings already log a non-fatal `traceway.CaptureException` when a referenced file is missing.
+
#### Organization Roles
| Role | Description |
|------|-------------|
diff --git a/DOCKER.md b/DOCKER.md
index 9a3fefa9..535e98b0 100644
--- a/DOCKER.md
+++ b/DOCKER.md
@@ -45,6 +45,7 @@ All have working defaults for the all-in-one image. Override as needed:
| `POSTGRES_SSLMODE` | `disable` | PostgreSQL SSL mode |
| `JWT_SECRET` | *(built-in default)* | JWT signing secret (min 32 chars) |
| `GIN_MODE` | `release` | Gin framework mode |
+| `SESSION_RECORDING_RETENTION_DAYS` | `30` | Days to keep on-disk session recordings under `STORAGE_PATH/recordings/`. Worker runs hourly and on startup. `0` disables; no effect when `STORAGE_TYPE=s3`. |
### Access Points
@@ -82,6 +83,7 @@ docker build -f Dockerfile.minimal -t traceway:minimal .
| `POSTGRES_PASSWORD` | PostgreSQL password | `password` |
| `POSTGRES_SSLMODE` | PostgreSQL SSL mode | `disable` |
| `JWT_SECRET` | JWT signing secret (min 32 chars) | `your-secret-here` |
+| `SESSION_RECORDING_RETENTION_DAYS` | Optional. Days to keep on-disk session recordings (default `30`, `0` disables, no effect when `STORAGE_TYPE=s3`). | `30` |
### Run
@@ -125,6 +127,8 @@ services:
POSTGRES_PASSWORD: "traceway"
POSTGRES_SSLMODE: "disable"
JWT_SECRET: "change-this-to-a-secure-secret-at-least-32-chars"
+ # Optional. Days to keep on-disk session recordings (default 30, 0 disables, no effect when STORAGE_TYPE=s3).
+ # SESSION_RECORDING_RETENTION_DAYS: "30"
depends_on:
clickhouse:
condition: service_healthy
diff --git a/backend/.env.docker b/backend/.env.docker
index 303cfa02..e9799105 100644
--- a/backend/.env.docker
+++ b/backend/.env.docker
@@ -11,3 +11,9 @@ POSTGRES_PASSWORD=
POSTGRES_SSLMODE=disable
JWT_SECRET=Xt7Kj2mP9qLwNc4rVb8sYd3hFgAeZuCpDnRoWxMvBa5k
GIN_MODE=release
+
+# Optional retention knobs (defaults shown). Set to 0 to disable.
+# SESSION_RECORDING_RETENTION_DAYS only takes effect when STORAGE_TYPE=local.
+# SQLITE_RETENTION_DAYS only takes effect when DB_TYPE=sqlite.
+#SESSION_RECORDING_RETENTION_DAYS=30
+#SQLITE_RETENTION_DAYS=30
diff --git a/docker-compose.sqlite.yml b/docker-compose.sqlite.yml
index 89013066..de4b42b6 100644
--- a/docker-compose.sqlite.yml
+++ b/docker-compose.sqlite.yml
@@ -9,6 +9,13 @@ services:
JWT_SECRET: "change-this-to-a-secure-secret-at-least-32-chars"
# APP_BASE_URL: "https://traceway.example.com"
+ # Optional retention knobs. Defaults are 30 days; set to "0" to disable.
+ # SQLITE_RETENTION_DAYS prunes telemetry rows; SESSION_RECORDING_RETENTION_DAYS
+ # cleans on-disk recordings under STORAGE_PATH/recordings/. Both run hourly
+ # and on startup. The recording cleanup has no effect when STORAGE_TYPE=s3.
+ # SQLITE_RETENTION_DAYS: "30"
+ # SESSION_RECORDING_RETENTION_DAYS: "30"
+
# Optional: store blobs (source maps, session recordings, AI traces) in S3
# instead of the local /data/storage folder.
# STORAGE_TYPE: "s3"
diff --git a/docker-compose.yml b/docker-compose.yml
index c2bb33e7..df9e19ba 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -18,6 +18,9 @@ services:
POSTGRES_PASSWORD: "traceway"
POSTGRES_SSLMODE: "disable"
JWT_SECRET: "change-this-to-a-secure-secret-at-least-32-chars"
+ # Optional. Days to keep on-disk session recordings under STORAGE_PATH/recordings/.
+ # Worker runs hourly and on startup; 0 disables; no effect when STORAGE_TYPE=s3.
+ # SESSION_RECORDING_RETENTION_DAYS: "30"
depends_on:
clickhouse:
condition: service_healthy
diff --git a/docs/pages/server/all-in-one.mdx b/docs/pages/server/all-in-one.mdx
index c111c1d2..231b4bb2 100644
--- a/docs/pages/server/all-in-one.mdx
+++ b/docs/pages/server/all-in-one.mdx
@@ -40,6 +40,9 @@ All environment variables have working defaults, no configuration required. Over
| `JWT_SECRET` | *(built-in default)* | JWT signing secret (min 32 chars) |
| `GIN_MODE` | `release` | Gin framework mode |
| `APP_BASE_URL` | *(none)* | Public URL of your Traceway instance (e.g. `https://traceway.example.com`). Used for SDK setup instructions, email links, and OAuth callback URLs. |
+| `SESSION_RECORDING_RETENTION_DAYS` | `30` | Days to keep session recording files on disk under `STORAGE_PATH/recordings/`. Files older than the TTL are deleted hourly and on startup. No effect when `STORAGE_TYPE=s3`. Set to `0` to disable. |
+
+> ClickHouse data retention is controlled per-table at the schema level (TTLs on `metric_points`, `metric_points_1m`, `metric_points_1h`, `log_records`); other telemetry tables are retained indefinitely until you drop monthly partitions manually. The `SQLITE_RETENTION_DAYS` setting only applies in SQLite mode and has no effect in this image.
## SSO (optional)
diff --git a/docs/pages/server/docker-compose.mdx b/docs/pages/server/docker-compose.mdx
index 0fc72e9d..7e49c4f2 100644
--- a/docs/pages/server/docker-compose.mdx
+++ b/docs/pages/server/docker-compose.mdx
@@ -33,6 +33,9 @@ The `docker-compose.yml` in the repository root is ready to use out of the box.
| `POSTGRES_SSLMODE` | `disable` | PostgreSQL SSL mode |
| `JWT_SECRET` | `change-this-to-a-secure-secret-at-least-32-chars` | JWT signing secret (min 32 chars) |
| `APP_BASE_URL` | *(none)* | Public URL of your Traceway instance (e.g. `https://traceway.example.com`). Used for SDK setup instructions, email links, and OAuth callback URLs. |
+| `SESSION_RECORDING_RETENTION_DAYS` | `30` | Days to keep session recording files on disk under `STORAGE_PATH/recordings/`. A worker deletes older files hourly and on startup. No effect when `STORAGE_TYPE=s3`. Set to `0` to disable. |
+
+> ClickHouse table TTLs (`metric_points`, `log_records`, etc.) are managed at the schema level; this stack does not run the SQLite retention worker.
## SSO (optional)
diff --git a/docs/pages/server/haloy.mdx b/docs/pages/server/haloy.mdx
index b80b12d2..68418dba 100644
--- a/docs/pages/server/haloy.mdx
+++ b/docs/pages/server/haloy.mdx
@@ -181,6 +181,7 @@ Point your SDK's connection string at your Traceway domain:
| `GITHUB_CLIENT_ID` | *(unset)* | Optional. GitHub OAuth App client ID. Setting both GitHub variables enables the **Continue with GitHub** button. |
| `GITHUB_CLIENT_SECRET` | *(unset)* | Optional. GitHub OAuth App client secret. |
| `OAUTH_SESSION_SECRET` | *(falls back to `JWT_SECRET`)* | Optional. Cookie signing secret for the OAuth round-trip. |
+| `SESSION_RECORDING_RETENTION_DAYS` | `30` | Optional. Days to keep on-disk session recordings under `STORAGE_PATH/recordings/`. Worker runs hourly and on startup. `0` disables; no effect when `STORAGE_TYPE=s3`. |
When configuring providers, set the callback URL on the provider side to `https://traceway.yourdomain.com/api/auth/callback/{google|github}`.
diff --git a/docs/pages/server/local-setup.mdx b/docs/pages/server/local-setup.mdx
index 981f9deb..b7d503e3 100644
--- a/docs/pages/server/local-setup.mdx
+++ b/docs/pages/server/local-setup.mdx
@@ -79,6 +79,8 @@ POSTGRES_SSLMODE=disable
| `GITHUB_CLIENT_ID` | Optional. GitHub OAuth App client ID. Setting both GitHub variables enables the **Continue with GitHub** button. See the [SSO guide](/learn/sso). |
| `GITHUB_CLIENT_SECRET` | Optional. GitHub OAuth App client secret. |
| `OAUTH_SESSION_SECRET` | Optional. Cookie signing secret for the OAuth round-trip. Falls back to `JWT_SECRET` when unset. |
+| `SESSION_RECORDING_RETENTION_DAYS` | Optional. Days to keep session recording files on disk under `STORAGE_PATH/recordings/`. Defaults to `30`. Worker runs hourly and on startup. `0` disables; no effect when `STORAGE_TYPE=s3`. |
+| `SQLITE_RETENTION_DAYS` | Optional. Days to keep SQLite telemetry rows. Defaults to `30`. `0` disables. Only applies when running with `DB_TYPE=sqlite`. |
For local OAuth testing, register the provider's callback URL as `http://localhost:5173/api/auth/callback/github` (or `.../google`).
diff --git a/docs/pages/server/minimal.mdx b/docs/pages/server/minimal.mdx
index 758d33cc..404f7d68 100644
--- a/docs/pages/server/minimal.mdx
+++ b/docs/pages/server/minimal.mdx
@@ -47,6 +47,7 @@ After starting, open `http://localhost/register` to create your first account.
| `POSTGRES_SSLMODE` | PostgreSQL SSL mode | `disable` |
| `JWT_SECRET` | JWT signing secret (min 32 chars) | `your-secret-here` |
| `APP_BASE_URL` | Public URL of your instance. Used for SDK setup instructions, email links, and OAuth callback URLs. | `https://traceway.example.com` |
+| `SESSION_RECORDING_RETENTION_DAYS` | Days to keep on-disk session recordings under `STORAGE_PATH/recordings/`. Defaults to `30`. Worker runs hourly and on startup. `0` disables; no effect when `STORAGE_TYPE=s3`. | `30` |
## SSO (optional)
diff --git a/docs/pages/server/sqlite.mdx b/docs/pages/server/sqlite.mdx
index e57c3d2d..73091afc 100644
--- a/docs/pages/server/sqlite.mdx
+++ b/docs/pages/server/sqlite.mdx
@@ -62,6 +62,8 @@ If `STORAGE_TYPE` is unset (or set to `local`), blobs are written to `/data/stor
| `S3_ENDPOINT` | *(unset)* | Optional. Set for S3-compatible providers (MinIO, R2, Spaces). Enables path-style URLs |
| `PORTS` | `80,8082` | Comma-separated list of HTTP ports the server should listen on |
| `GIN_MODE` | `release` | Gin runtime mode |
+| `SQLITE_RETENTION_DAYS` | `30` | Days to keep telemetry rows (endpoints, exceptions, spans, metrics, logs, sessions, recordings, fired notifications, notification history). A background worker prunes older rows once at startup and then every hour. Set to `0` to disable. |
+| `SESSION_RECORDING_RETENTION_DAYS` | `30` | Days to keep session recording files on disk under `STORAGE_PATH/recordings/`. Files older than the TTL are deleted hourly and on startup; empty subdirectories are removed. No effect when `STORAGE_TYPE=s3`. Set to `0` to disable. |
## SSO (optional)