Skip to content

The only tool you need to know what is happening and how to fix it.

exposing the cache size for sqlite + defaulting docs to duckdb

narrow-adventurec7ab850bb3e4
diff --git a/CLAUDE.md b/CLAUDE.md
index b8a26b8f..c2981aca 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -257,6 +257,7 @@ if err != nil {
JWT_SECRET=<min 32 char secret for JWT signing> # the only variable with no default in the standalone binary. There is deliberately no fallback in the code: a key committed to the repo would be identical in every clone. The all-in-one Docker image is the exception, it copies `backend/.env.docker` with a public key that must be overridden in production. Unset or under 32 chars exits 1 with an actionable message rather than a panic.
DB_TYPE= # sqlite | postgres. Defaults to sqlite in every build EXCEPT -tags transactional_pg, where an empty value still selects PostgreSQL. Without that tag SQLite is the only supported main DB (the migration runner applies SQLite-dialect migrations unconditionally), so the default is set once in config.defaultDBType rather than at each DBType call site.
PORTS=80,8082 # comma-separated HTTP listen ports. Every port is bound up front; a port that fails to bind (typically :80 as a non-root user) logs a warning (and reports via CaptureException when monitoring is configured), and startup only fails if NO port binds. Order is not significant: a listener that later stops serving takes the process down whichever position it holds.
+SQLITE_CACHE_SIZE_MB=512 # PRAGMA cache_size of the SQLite TELEMETRY DB in MB, per connection (the main DB keeps SQLite's 2MB default; no effect on DuckDB or ClickHouse telemetry). The telemetry pool opens up to 4 connections (`telemetryMaxOpenConns`, db_sqlite_open.go) and modernc's allocator rounds each ~4.4KB page buffer up to an 8KB slot, so resident memory is about 2x the value per connection: worst case 8x the value, 4GB at the default. A cache fills from ingest, from the retention DELETE scan at startup and every hour, and from reads of pages still in the WAL (write transactions and WAL pages bypass the 1GB mmap window), and it is held for as long as the pooled connection lives. At the default a 1GiB container is OOM-killed on a 24h dashboard or under sustained ingest (issue #396). Rule of thumb: container MB / 32 (16 for 512MB, 32 for 1GB, 64 for 2GB). A non-positive or unparsable value falls back to 512 (`config.SizeMB`). The effective value is logged at startup, and the embedded `Run(opts...)` path honors the env var through `applyEnvOverrides`. Operator docs: `docs/pages/server/sqlite.mdx#memory-sizing`.
SQLITE_PATH= # main DB path, defaults to ./traceway.db in the working directory. The telemetry DB derives from it (_telemetry.db, or _telemetry.duckdb under -tags telemetry_duckdb).
APP_BASE_URL= # public origin of this server (e.g. https://traceway.example.com). Used as the OAuth issuer / device verification URL and SSO redirect base, and to absolutize the deep links in notifications. If unset, the device-auth + well-known endpoints derive it per-request from the Host / X-Forwarded-* headers; set it explicitly behind a proxy that doesn't forward Host.
TRUSTED_PROXIES= # comma-separated IPs/CIDRs whose X-Forwarded-For/X-Real-IP gin trusts for c.ClientIP() (every per-IP rate limiter keys on it, and `/api/report` stores it on every session as `client.ip`). Unset = loopback only, so no peer can spoof its IP. Private ranges are deliberately NOT trusted by default: on a Docker/k8s network every peer has a private address, the clients included when the port is published directly, so trusting RFC1918 would let any of them forge XFF past every per-IP limiter. Set it to the CIDRs of whatever sits in front (a proxy container's compose network, the ingress pod range, the CDN ranges); otherwise every visitor shares one rate-limit bucket keyed on the proxy. The value REPLACES the default, so it must name every hop in the chain, not just the outermost (a CDN-only list leaves a private ingress untrusted and disables XFF entirely); `*` trusts every peer. A malformed entry exits 1 at startup with a FATAL message, before the database is opened (`trustedProxyNets`, cmd/run.go). The first request carrying X-Forwarded-For/X-Real-IP from a peer outside the list logs a one-time warning naming that peer (`warnOnUntrustedForwardedFor`, cmd/run.go), which is how a forgotten proxy hop shows up. Parsing lives in `Cfg.TrustedProxyList` (config.go), covered by config_test.go.
@@ -275,7 +276,7 @@ POSTGRES_USERNAME=traceway
POSTGRES_PASSWORD=
POSTGRES_SSLMODE=disable
-# DuckDB telemetry backend (only with -tags telemetry_duckdb build; see "DuckDB Telemetry Backend" below)
+# DuckDB telemetry backend (only with -tags telemetry_duckdb build; see "DuckDB Telemetry Backend" below). The embedded Run(opts...) path honors these three and DUCKDB_RETENTION_DAYS through applyEnvOverrides.
DUCKDB_MEMORY_LIMIT= # e.g. 4GB. Unset = DuckDB auto-tunes (~80% RAM). Set explicitly in memory-capped containers to avoid OOM.
DUCKDB_THREADS= # e.g. 4. Unset = DuckDB auto-tunes (= cores). Cap in constrained/shared environments.
DUCKDB_CHECKPOINT_THRESHOLD= # e.g. 256MB. Unset = DuckDB default (16MB). Raise under sustained ingest to reduce WAL checkpoint stalls; costs a larger WAL and longer restart replay.
diff --git a/backend/app/config/config.go b/backend/app/config/config.go
index 6df44bda..6f055d7a 100644
--- a/backend/app/config/config.go
+++ b/backend/app/config/config.go
@@ -20,6 +20,8 @@ type Cfg struct {
PostgresSSLMode string
SQLitePath string
+ SQLiteCacheSizeMB string
+
DuckDBMemoryLimit string
DuckDBThreads string
DuckDBCheckpointThreshold string
@@ -218,6 +220,8 @@ func LoadFromEnv() *Cfg {
PostgresSSLMode: os.Getenv("POSTGRES_SSLMODE"),
SQLitePath: os.Getenv("SQLITE_PATH"),
+ SQLiteCacheSizeMB: os.Getenv("SQLITE_CACHE_SIZE_MB"),
+
DuckDBMemoryLimit: os.Getenv("DUCKDB_MEMORY_LIMIT"),
DuckDBThreads: os.Getenv("DUCKDB_THREADS"),
DuckDBCheckpointThreshold: os.Getenv("DUCKDB_CHECKPOINT_THRESHOLD"),
diff --git a/backend/app/db/db_sqlite_open.go b/backend/app/db/db_sqlite_open.go
index 4051a5ee..296a89c6 100644
--- a/backend/app/db/db_sqlite_open.go
+++ b/backend/app/db/db_sqlite_open.go
@@ -7,9 +7,24 @@ import (
"fmt"
"strings"
+ "github.com/tracewayapp/traceway/backend/app/config"
_ "modernc.org/sqlite"
)
+const (
+ defaultTelemetryCacheMB = 512
+ telemetryMaxOpenConns = 4
+)
+
+func telemetryCacheMB() int64 {
+ value := ""
+ if config.Config != nil {
+ value = config.Config.SQLiteCacheSizeMB
+ }
+ const bytesPerMB = 1024 * 1024
+ return config.SizeMB(value, defaultTelemetryCacheMB) / bytesPerMB
+}
+
// Pragmas go on the DSN (modernc.org/sqlite supports `_pragma=name(value)`)
// rather than via db.Exec because Exec only configures the first pooled
// connection — once SetMaxOpenConns goes above 1 new connections would
@@ -26,7 +41,7 @@ func openSQLite(path string, telemetry bool) (*sql.DB, error) {
if telemetry {
params = append(params,
"_pragma=synchronous(NORMAL)",
- "_pragma=cache_size(-524288)",
+ fmt.Sprintf("_pragma=cache_size(-%d)", telemetryCacheMB()*1024),
"_pragma=temp_store(MEMORY)",
"_pragma=mmap_size(1073741824)",
"_pragma=wal_autocheckpoint(50000)",
@@ -62,7 +77,7 @@ func openSQLite(path string, telemetry bool) (*sql.DB, error) {
} else if telemetry {
// WAL allows concurrent readers; SQLite still serializes writes at the
// file level and busy_timeout absorbs short contention windows.
- d.SetMaxOpenConns(4)
+ d.SetMaxOpenConns(telemetryMaxOpenConns)
} else {
d.SetMaxOpenConns(1)
}
diff --git a/backend/app/db/db_telemetry_sqlite.go b/backend/app/db/db_telemetry_sqlite.go
index f7ae7f85..457dc923 100644
--- a/backend/app/db/db_telemetry_sqlite.go
+++ b/backend/app/db/db_telemetry_sqlite.go
@@ -27,7 +27,7 @@ func initTelemetryDB() error {
return err
}
TelemetryDB = telDB
- config.Logf("SQLite telemetry database opened at %s", telemetryPath)
+ config.Logf("SQLite telemetry database opened at %s (page cache: %d MB per connection, up to %d connections)", telemetryPath, telemetryCacheMB(), telemetryMaxOpenConns)
return nil
}
diff --git a/backend/cmd/run.go b/backend/cmd/run.go
index 5789ea94..20494fa0 100644
--- a/backend/cmd/run.go
+++ b/backend/cmd/run.go
@@ -462,6 +462,11 @@ func applyEnvOverrides(cfg *config.Cfg) {
{"TWILIO_MESSAGING_SERVICE_SID", &cfg.TwilioMessagingServiceSID},
{"ALLOW_PRIVATE_NOTIFICATION_TARGETS", &cfg.AllowPrivateNotificationTargets},
{"REPORT_MAX_BODY_MB", &cfg.ReportMaxBodyMB},
+ {"SQLITE_CACHE_SIZE_MB", &cfg.SQLiteCacheSizeMB},
+ {"DUCKDB_MEMORY_LIMIT", &cfg.DuckDBMemoryLimit},
+ {"DUCKDB_THREADS", &cfg.DuckDBThreads},
+ {"DUCKDB_CHECKPOINT_THRESHOLD", &cfg.DuckDBCheckpointThreshold},
+ {"DUCKDB_RETENTION_DAYS", &cfg.DuckDBRetentionDays},
{"OAUTH_SESSION_SECRET", &cfg.OAuthSessionSecret},
{"GOOGLE_CLIENT_ID", &cfg.GoogleClientID},
{"GOOGLE_CLIENT_SECRET", &cfg.GoogleClientSecret},
diff --git a/docker-compose.sqlite.yml b/docker-compose.sqlite.yml
index b5b99c32..2d198553 100644
--- a/docker-compose.sqlite.yml
+++ b/docker-compose.sqlite.yml
@@ -14,6 +14,11 @@ services:
# TRUSTED_PROXIES: "172.18.0.0/16"
# TRUSTED_PROXY_HEADER: "X-Real-IP"
+ # The telemetry page cache is per connection (up to 4) and costs about twice
+ # its size, so the 512 MB default can take up to 4 GB. On a memory-capped
+ # host set it to the container's memory in MB divided by 32: 32 for 1 GB.
+ # SQLITE_CACHE_SIZE_MB: "32"
+
# Optional retention knobs. Defaults are 30 days; set to "0" to disable.
# SQLITE_RETENTION_DAYS prunes telemetry rows; SESSION_RECORDING_RETENTION_DAYS
# cleans on-disk recordings under STORAGE_PATH/recordings/. Both run hourly
diff --git a/docs/pages/learn/embedded-mode.mdx b/docs/pages/learn/embedded-mode.mdx
index cf45c725..482f813f 100644
--- a/docs/pages/learn/embedded-mode.mdx
+++ b/docs/pages/learn/embedded-mode.mdx
@@ -2,11 +2,13 @@
import { Callout, Steps } from "nextra/components";
-Embedded mode runs a full Traceway server inside your Go process, using **SQLite** for all storage (relational data and telemetry). No Docker, no infrastructure: just one `tracewaybackend.Run()` call and your app has local observability.
+Embedded mode runs a full Traceway server inside your Go process. Users, projects and settings live in SQLite, and telemetry lives in **DuckDB**. Both run in your process, in memory or as local files. No Docker, no infrastructure: just one `tracewaybackend.Run()` call and your app has local observability.
This is intended for **local development only**. For production, use the [Docker Compose](/server/docker-compose) or [minimal](/server/minimal) deployment.
-Telemetry storage follows your build tags: a plain `go build` embeds SQLite, and building your app with `-tags telemetry_duckdb` (CGO required) swaps the embedded telemetry store to DuckDB's columnar engine. See [Build Tags](/learn/build-tags).
+<Callout type="warning">
+ **Build your app with `-tags telemetry_duckdb`.** The tag selects the DuckDB telemetry store, which is the one we recommend and the one this page documents. A build without the tag still works, but it falls back to SQLite telemetry, which is being phased out. See [Builds without the tag](#builds-without-the-tag).
+</Callout>
<Callout type="info">
**Want to see it in action?** Check out the [full working example on
@@ -51,7 +53,7 @@ go get github.com/tracewayapp/traceway/backend
### Start Traceway inside your app
-Call `tracewaybackend.Run()` in a goroutine during startup. It boots an in-memory SQLite server and seeds a default user and project:
+Call `tracewaybackend.Run()` in a goroutine during startup. It boots an in-memory server and seeds a default user and project:
```go
package main
@@ -169,9 +171,25 @@ The key parts:
- **`otel.SetTextMapPropagator`** is required in Go. The global propagator ships as a no-op, so without this call an HTTP call your app makes to another service shows up as its own separate trace instead of a child span
- **`defer shutdown()`** flushes the batch processor on exit. Without it the last few seconds of spans never leave the process
+### Run your app with the DuckDB tag
+
+```bash
+go run -tags telemetry_duckdb .
+```
+
+The tag needs CGO, so a C compiler has to be installed. [DuckDB telemetry](#duckdb-telemetry) below covers the requirements.
+
+The startup log tells you which store you got. This is the line to look for:
+
+```
+DuckDB telemetry database opened at
+```
+
+A path follows it once you keep data on disk. If the line says `SQLite telemetry database opened` instead, the tag did not reach the build.
+
### Open the dashboard
-Run your app, then open **[http://localhost:8082](http://localhost:8082)** in your browser.
+Open **[http://localhost:8082](http://localhost:8082)** in your browser.
Log in with the credentials from `WithDefaultUser` (`[email protected]` / `admin`).
@@ -179,15 +197,52 @@ Hit your app at `http://localhost:8080/ping` a few times. Traces will appear in
</Steps>
+## DuckDB telemetry
+
+DuckDB is a columnar engine that runs inside your process, the same way SQLite does. Traceway stores traces, issues, logs and metrics in it because dashboard queries stay fast as data grows.
+
+### Build requirements
+
+- **The build tag.** Pass `-tags telemetry_duckdb` to every `go build`, `go run` and `go test` of your app, including the ones your IDE, live-reload tool or Dockerfile run. To set it once, export `GOFLAGS=-tags=telemetry_duckdb`.
+- **CGO and a C compiler.** DuckDB is linked as a native library. CGO is on by default when a compiler is installed: the Xcode command line tools on macOS, `gcc` on Linux. If you turned it off, set `CGO_ENABLED=1`.
+- **A supported platform.** The driver ships prebuilt libraries for macOS (Intel and Apple silicon), Linux with glibc (amd64 and arm64) and Windows (amd64). Alpine and other musl-based images are not supported, so use a Debian or Ubuntu base for a dev container.
+
+There is nothing extra to `go get`: the DuckDB driver is already a dependency of the backend module. Expect the binary to grow by about 50 MB.
+
+If CGO is off, the build stops with an error like this:
+
+```
+build constraints exclude all Go files in .../github.com/duckdb/duckdb-go-bindings/lib/[email protected]
+```
+
+Install a C compiler, make sure `CGO_ENABLED` is not `0`, and build again. Cross-compiling needs a C toolchain for the target, so the simplest route is to build on the platform you run on.
+
+### Settings
+
+The embedded server reads these from your process environment. Set them before `Run` is called.
+
+| Variable | Default | Description |
+| --- | --- | --- |
+| `DUCKDB_MEMORY_LIMIT` | *(unset, about 80% of the RAM DuckDB sees)* | Memory budget for the engine, for example `1GB`. A ceiling, not a reservation. |
+| `DUCKDB_THREADS` | *(unset, all cores)* | Caps the engine's thread count, for example `2`. |
+| `DUCKDB_CHECKPOINT_THRESHOLD` | *(unset, 16MB)* | WAL size that triggers a checkpoint, for example `256MB`. Rarely needs changing in development. |
+| `DUCKDB_RETENTION_DAYS` | `30` | Days of telemetry to keep. A cleanup pass runs at startup and then every hour. Set to `0` to keep everything. |
+
+On a laptop the defaults are fine. Inside a memory-capped dev container DuckDB sees the host's RAM, not the container limit, so set `DUCKDB_MEMORY_LIMIT` to about half the container's memory. A value DuckDB cannot parse makes `Run` panic at startup with a message that names the option.
+
+<Callout type="info">
+ Backend v2.0.3 and earlier ignore these four variables in embedded mode and always use the defaults.
+</Callout>
+
## Configuration Options
-All options are passed to `tracewaybackend.Run()`. When any options are provided, embedded mode is activated automatically (pure SQLite).
+All options are passed to `tracewaybackend.Run()`. When any options are provided, embedded mode is activated automatically.
| Option | Description | Default |
| --------------------------------------------------- | -------------------------------------------------------------------------------------------------- | ------------------------- |
| `WithPort(port)` | HTTP port for the Traceway server | `8082` |
| `WithServerURL(url)` | Base URL override (used in connection string output) | `http://localhost:{port}` |
-| `WithSQLitePath(path)` | SQLite database file path | `:memory:` (in-memory) |
+| `WithSQLitePath(path)` | Path of the main SQLite database file. The telemetry file is created next to it | `:memory:` (in-memory) |
| `WithDefaultUser(email, password)` | Create a default admin user on startup | None |
| `WithDefaultProject(name, framework, token)` | Create a default project with a fixed token (can be called multiple times) | None |
| `DisableLogging()` | Silence the embedded server's startup and request logs so they don't mix with your app's output | Logging enabled |
@@ -220,7 +275,7 @@ go tracewaybackend.Run(
## Data Persistence
-By default, the database is **in-memory**, which means all data is lost when your application restarts. This is fine for quick debugging sessions, but if you want data to survive restarts, set a file path:
+By default, both databases are **in-memory**, which means all data is lost when your application restarts. This is fine for quick debugging sessions, but if you want data to survive restarts, set a file path:
Add `WithSQLitePath` to the options you already pass. It does not replace them. Keep the `go` keyword too, otherwise `Run` blocks and your own server never starts.
@@ -239,21 +294,36 @@ go tracewaybackend.Run(
| File | Contents |
| --- | --- |
-| `traceway.db` | Relational data: users, organizations, projects, settings |
-| `traceway_telemetry.db` | Telemetry: traces, endpoints, issues, logs, metrics |
+| `traceway.db` | SQLite. Relational data: users, organizations, projects, settings |
+| `traceway_telemetry.duckdb` | DuckDB. Telemetry: traces, endpoints, issues, logs, metrics |
+
+Both write sidecar files: `traceway.db-wal` and `traceway.db-shm` for SQLite, `traceway_telemetry.duckdb.wal` for DuckDB. Back up or ignore all of them, not just the file you named. To copy the DuckDB file safely, stop your app first and take the `.wal` file with it.
-SQLite also writes `-wal` and `-shm` sidecar files for each database. Back up or ignore all of them, not just the file you named. The server also creates a `storage/` directory in the process working directory for session recordings and synthetics screenshots. That directory appears in the default in-memory mode too.
+The server also creates a `storage/` directory in the process working directory for session recordings and synthetics screenshots. That directory appears in the default in-memory mode too.
+
+Only one process can open the DuckDB file at a time. A second copy of your app pointed at the same path panics at startup with `Could not set lock on file`.
Add all of it to `.gitignore`:
```text
traceway.db*
-traceway_telemetry.db*
+traceway_telemetry.duckdb*
storage/
```
Seeding is idempotent: if the user from `WithDefaultUser` already exists, it's skipped.
+## Builds without the tag
+
+A plain `go build` still works. It stores telemetry in SQLite instead of DuckDB, in `traceway_telemetry.db` with `-wal` and `-shm` sidecar files. SQLite telemetry is being phased out, so use it only where you cannot enable CGO.
+
+Two things to know while you are on it:
+
+- **Memory.** A file-backed SQLite telemetry database keeps a page cache inside your process, and its default can grow to several gigabytes. Set the `SQLITE_CACHE_SIZE_MB` environment variable, for example `16`, before your app starts. [Memory sizing](/server/sqlite#memory-sizing) explains the numbers. The in-memory default is not affected.
+- **Switching to DuckDB.** Add the tag and rebuild. Users, projects and tokens carry over, because the main database is the same file. Telemetry does not: the DuckDB build starts a fresh `traceway_telemetry.duckdb`, and the old `traceway_telemetry.db` files can be deleted.
+
+See [Build Tags](/learn/build-tags) for the full list of storage combinations.
+
## Development-Only Guard
A common pattern is guarding embedded mode behind an environment check so it only runs in development:
diff --git a/docs/pages/learn/hardware-requirements.mdx b/docs/pages/learn/hardware-requirements.mdx
index 76756428..82118c46 100644
--- a/docs/pages/learn/hardware-requirements.mdx
+++ b/docs/pages/learn/hardware-requirements.mdx
@@ -6,7 +6,7 @@ How much CPU, RAM, and disk Traceway needs depends almost entirely on **which st
There are three shapes:
-- **SQLite mode** (single binary, zero external dependencies). The whole stack is one process. This is the lightweight option.
+- **SQLite mode** (single binary, zero external dependencies). The whole stack is one process. This is the lightweight option, as long as `SQLITE_CACHE_SIZE_MB` is set to fit the host. For a single container we recommend DuckDB mode.
- **DuckDB mode** (`telemetry_duckdb` build tag, CGO required). Still a single container with zero external dependencies, but the telemetry store is DuckDB's columnar engine, which holds 10 to 100 times more queryable rows than SQLite on the same box. The catch: DuckDB sizes itself from host RAM, so memory-capped containers must set `DUCKDB_MEMORY_LIMIT`.
- **ClickHouse + PostgreSQL mode** (`transactional_pg telemetry_ch` build tags). ClickHouse is built to use a lot of RAM for caches and merges, so this is the heavier option that scales much further. It is also the only mode that supports running more than one backend instance; see [Running more than one instance](/server/minimal#running-more-than-one-instance).
@@ -14,7 +14,7 @@ There are three shapes:
| Deployment | Minimum to start | Comfortable | Notes |
|------------|------------------|-------------|-------|
-| **SQLite** (single binary) | 1 vCPU, 512 MB RAM | 2 vCPU, 1 to 2 GB RAM | The backend process idles at ~30 MB RSS. Memory grows with the SQLite page cache and source map cache, both capped. |
+| **SQLite** (single binary) | 1 vCPU, 512 MB RAM, `SQLITE_CACHE_SIZE_MB` set | 2 vCPU, 1 to 2 GB RAM, `SQLITE_CACHE_SIZE_MB` set | The backend process idles at ~30 MB RSS. Memory grows with the SQLite page cache and source map cache. The page cache default suits a 16 GB host and can take up to 4 GB, so set `SQLITE_CACHE_SIZE_MB` on anything smaller or the backend gets OOM-killed. |
| **DuckDB** (single container) | 2 vCPU, 2 GB RAM, `DUCKDB_MEMORY_LIMIT` set | 2 to 4 vCPU, 4 to 8 GB RAM | DuckDB auto-tunes to ~80% of what it thinks is host RAM. In a memory-capped container it reads the host, not the cgroup, so set `DUCKDB_MEMORY_LIMIT` explicitly or the backend gets OOM-killed under ingest. |
| **ClickHouse + PostgreSQL** | 2 vCPU, 4 GB RAM | 4 vCPU, 8 GB+ RAM | ClickHouse is the floor here, not Traceway. It claims up to 80% of host RAM by design. |
@@ -34,19 +34,25 @@ So the intuition that "it's about 100 MB" is, if anything, conservative for an i
What consumes memory under real traffic:
-1. **The SQLite page cache** (SQLite mode only), capped at ~512 MB.
+1. **The SQLite page cache** (SQLite mode only). `SQLITE_CACHE_SIZE_MB` sizes it per connection, there are up to 4 connections, and each megabyte costs about two. The default of 512 allows up to 4 GB.
2. **The source map cache**, capped at 500 MB in memory by default, and only filled if you upload source maps.
3. **Per-request conversion work**: decoding OTLP/JSON payloads and hashing, proportional to your ingest rate. This is CPU, not retained memory.
-None of these are committed up front. A box with no source maps and light ingest stays near the idle floor.
+None of these are committed up front. The page cache is the one that fills on its own: the retention pass at startup reads through the telemetry file, so one connection's cache is full within seconds once the file is larger than the cache.
## SQLite mode (the small one)
The [SQLite deployment](/server/sqlite) is one Alpine container, one process, two SQLite files, and a folder for blobs. No Postgres, no ClickHouse.
-A **1 vCPU / 512 MB** box is enough to stand it up and run a small project. **2 vCPU / 1 to 2 GB** is comfortable for a real workload and leaves headroom for the caches.
+A **1 vCPU / 512 MB** box is enough to stand it up and run a small project. **2 vCPU / 1 to 2 GB** is comfortable for a real workload. Both need `SQLITE_CACHE_SIZE_MB` set.
-The telemetry database is tuned for append-heavy writes and will grow its page cache up to ~512 MB under sustained ingest (`cache_size(-524288)`), with a 1 GB memory-mapped read window on top. On a small box you do not need to touch this. The cache is a ceiling, not a reservation, and SQLite only grows it as the working set demands.
+The telemetry database keeps a page cache per connection, sized by `SQLITE_CACHE_SIZE_MB` (default 512). Up to 4 connections are open and the driver uses about two megabytes of memory per megabyte of cache, so the caches can take up to 8 times the value: 4 GB at the default. Each cache fills as its connection reads or writes and is then kept, so the default is only safe on a host with 16 GB or more. On a small box, set the value to the container's memory in MB divided by 32: `16` for 512 MB, `32` for 1 GB, `64` for 2 GB. The [SQLite page](/server/sqlite#memory-sizing) has the full table.
+
+There is also a 1 GB memory-mapped read window. It is backed by the database file, so the kernel can reclaim it under pressure and it does not need to fit in the budget.
+
+<Callout type="warning">
+ For a single container we recommend [DuckDB mode](#duckdb-mode-the-small-one-that-reads-big) below. SQLite telemetry is fine for trying Traceway out or for a small instance, but we are not planning major work on it.
+</Callout>
We benchmark the single-binary SQLite build on dedicated Hetzner CCX instances, from **CCX13 (2 vCPU / 8 GB)** up to **CCX43 (16 vCPU / 64 GB)**. Even the small tiers sustain tens of thousands of OTLP data points per second over OTLP/HTTP before write latency starts to climb, which is far more headroom than most self-hosted projects need.
@@ -61,7 +67,7 @@ We benchmark the single-binary SQLite build on dedicated Hetzner CCX instances,
### Disk
-SQLite stores telemetry on local disk, so disk is the resource to watch, not RAM. Size it for your retention window. Telemetry rows are pruned by a background worker (`SQLITE_RETENTION_DAYS`, default 30); see [Data retention on the SQLite page](/server/sqlite#configuration). Blob storage (source maps, session recordings, AI traces) lives under the same `/data` volume unless you offload it to S3.
+SQLite stores telemetry on local disk, so disk is the other resource to watch. Size it for your retention window. Telemetry rows are pruned by a background worker (`SQLITE_RETENTION_DAYS`, default 30); see [Data retention on the SQLite page](/server/sqlite#configuration). Blob storage (source maps, session recordings, AI traces) lives under the same `/data` volume unless you offload it to S3.
## DuckDB mode (the small one that reads big)
@@ -100,11 +106,11 @@ If you are sizing a small box, these are the knobs that matter. Every one has a
|--------|---------|-----------|-----------------|
| Source map cache (in memory) | 500 MB cap, 200 resolvers | `SOURCEMAP_CACHE_MAX_BYTES_MB`, `SOURCEMAP_CACHE_MAX_ENTRIES` | JS/TS projects that upload source maps. On a 1 to 2 GB box, set the cap to `100`. |
| Source map cache (disk tier) | off | `SOURCEMAP_CACHE_TYPE=disk`, `SOURCEMAP_DISK_CACHE_MAX_MB` (2048) | Many active maps, or multiple backend instances. Moves `.tw` resolvers to memory-mapped files. |
-| SQLite page cache | ~512 MB cap | `cache_size` pragma | SQLite mode under sustained ingest. A ceiling, grown on demand. |
+| SQLite page cache | 512 MB per connection, up to 4 connections, about twice that in real memory | `SQLITE_CACHE_SIZE_MB` | SQLite mode. Set it on any host under 16 GB, to the container's memory in MB divided by 32. See [Memory sizing](/server/sqlite#memory-sizing). |
| DuckDB memory budget | ~80% of host RAM | `DUCKDB_MEMORY_LIMIT`, `DUCKDB_THREADS` | DuckDB mode. Mandatory in memory-capped containers - DuckDB sizes itself from the host, not the cgroup. |
| Session-recording uploader | 32 workers, 2048 queue | `SESSION_RECORDING_UPLOAD_WORKERS`, `SESSION_RECORDING_UPLOAD_QUEUE_SIZE` | Browser/mobile session replay. Set workers to `0` to disable entirely. |
-On a memory-constrained host, the single most effective change is lowering `SOURCEMAP_CACHE_MAX_BYTES_MB`. The source map cache is the largest tunable consumer, because resolving a minified stack frame means holding a built resolver in memory.
+On a memory-constrained host in SQLite mode, set `SQLITE_CACHE_SIZE_MB` first. At its default the page cache is the largest consumer, and it is the one that gets a small container OOM-killed. After that, the most effective change is lowering `SOURCEMAP_CACHE_MAX_BYTES_MB`, because resolving a minified stack frame means holding a built resolver in memory.
<Callout type="info">
Source maps are the heaviest single thing the backend keeps in memory, and the
@@ -116,8 +122,8 @@ On a memory-constrained host, the single most effective change is lowering `SOUR
## Rules of thumb
-- **"I want the cheapest thing that runs."** SQLite mode, 1 vCPU / 512 MB. It idles around 30 MB and the caches are capped.
-- **"Small production app, some JS source maps."** SQLite mode, 2 vCPU / 1 to 2 GB, source map cap at 100 to 250 MB.
+- **"I want the cheapest thing that runs."** SQLite mode, 1 vCPU / 512 MB, with `SQLITE_CACHE_SIZE_MB=16`. It idles around 30 MB.
+- **"Small production app, some JS source maps."** SQLite mode, 2 vCPU / 1 to 2 GB, with `SQLITE_CACHE_SIZE_MB` at 32 to 64 and the source map cap at 100 to 250 MB.
- **"Single container, but the dashboards got slow as data grew."** DuckDB mode, 2 to 4 vCPU / 4 to 8 GB, with `DUCKDB_MEMORY_LIMIT` set to roughly half the container's RAM.
- **"High ingest, long retention, want it to scale."** ClickHouse + PostgreSQL, 4 vCPU / 8 GB+, and size ClickHouse first.
- **"It's getting slow."** Don't guess. Turn on [self-monitoring](/server/capacity-monitoring) and let the metrics tell you whether to scale ClickHouse, scale the backend, or buffer ingestion.
diff --git a/docs/pages/server/duckdb.mdx b/docs/pages/server/duckdb.mdx
index be4236a2..afc59b08 100644
--- a/docs/pages/server/duckdb.mdx
+++ b/docs/pages/server/duckdb.mdx
@@ -4,7 +4,7 @@ import { Callout } from "nextra/components";
The same single-container shape as the [SQLite deployment](/server/sqlite), but with the telemetry store swapped for DuckDB's columnar engine. The main database (users, organizations, projects, notification rules) stays SQLite; endpoints, spans, exceptions, metrics, and logs go into a DuckDB file next to it. No external Postgres or ClickHouse required.
-Pick this over the plain SQLite image when your dashboards slow down as data grows: on the same box, DuckDB keeps the dashboard responsive at 10 to 100 times more stored telemetry rows, and columnar compression makes the telemetry file several times smaller for the same data. See [Hardware Requirements](/learn/hardware-requirements#duckdb-mode-the-small-one-that-reads-big) for sizing guidance and benchmark numbers.
+This is the single-container deployment we recommend. On the same box, DuckDB keeps the dashboard responsive at 10 to 100 times more stored telemetry rows than the plain SQLite image, and columnar compression makes the telemetry file several times smaller for the same data. See [Hardware Requirements](/learn/hardware-requirements#duckdb-mode-the-small-one-that-reads-big) for sizing guidance and benchmark numbers.
<Callout type="info">
Like the SQLite image, this one runs as a single instance: both database files live under `/data`, and a second container cannot share them or learn about the first one's writes. Running more than one backend instance needs the PostgreSQL + ClickHouse build; see [Running more than one instance](/server/minimal#running-more-than-one-instance).
@@ -51,7 +51,7 @@ Identical to the SQLite image: blobs (source maps, session recordings, AI traces
## Configuration
-All [SQLite deployment variables](/server/sqlite#configuration) apply unchanged (`JWT_SECRET`, `APP_BASE_URL`, `SQLITE_PATH`, `STORAGE_*`, `S3_*`, `PORTS`, `LOG_RECORDS_MAX_ROWS`, `SESSION_RECORDING_RETENTION_DAYS`). The retention worker prunes the DuckDB telemetry tables hourly and issues a `CHECKPOINT` afterwards so deleted rows are actually reclaimed from disk. It keeps 30 days by default, configurable with `DUCKDB_RETENTION_DAYS` (when that is unset, a configured `SQLITE_RETENTION_DAYS` still applies as a fallback).
+All [SQLite deployment variables](/server/sqlite#configuration) apply unchanged (`JWT_SECRET`, `APP_BASE_URL`, `SQLITE_PATH`, `STORAGE_*`, `S3_*`, `PORTS`, `LOG_RECORDS_MAX_ROWS`, `SESSION_RECORDING_RETENTION_DAYS`). The one exception is `SQLITE_CACHE_SIZE_MB`, which sizes the SQLite telemetry store and has no effect here. The retention worker prunes the DuckDB telemetry tables hourly and issues a `CHECKPOINT` afterwards so deleted rows are actually reclaimed from disk. It keeps 30 days by default, configurable with `DUCKDB_RETENTION_DAYS` (when that is unset, a configured `SQLITE_RETENTION_DAYS` still applies as a fallback).
On top of those, the DuckDB build has these knobs:
diff --git a/docs/pages/server/local-setup.mdx b/docs/pages/server/local-setup.mdx
index 0d8831b8..6107f2a7 100644
--- a/docs/pages/server/local-setup.mdx
+++ b/docs/pages/server/local-setup.mdx
@@ -81,6 +81,7 @@ POSTGRES_SSLMODE=disable
| `OAUTH_SESSION_SECRET` | Optional. Cookie signing secret for the OAuth round-trip. Falls back to `JWT_SECRET` when unset. |
| `SESSION_RECORDING_RETENTION_DAYS` | Optional. Days to keep session recording files on disk under `STORAGE_PATH/recordings/`. Defaults to `30`. Worker runs hourly and on startup. `0` disables; no effect when `STORAGE_TYPE=s3`. |
| `SQLITE_RETENTION_DAYS` | Optional. Days to keep SQLite telemetry rows. Defaults to `30`. `0` disables. Only applies when running with `DB_TYPE=sqlite`. |
+| `SQLITE_CACHE_SIZE_MB` | Optional. Page cache of the SQLite telemetry database in megabytes, per connection. Defaults to `512`. Up to 4 connections are open and each megabyte costs about two, so the caches can take up to 8 times the value. Lower it on a memory-constrained machine; see [Memory sizing](/server/sqlite#memory-sizing). Only applies to SQLite telemetry (the default build with `DB_TYPE=sqlite`). |
| `DUCKDB_RETENTION_DAYS` | Optional. Days to keep telemetry rows on the DuckDB backend (`-tags telemetry_duckdb` build). Defaults to `30`, falling back to `SQLITE_RETENTION_DAYS` when unset. `0` disables. Pruning runs hourly, so expired rows can linger between runs. |
| `LOG_RECORDS_MAX_ROWS` | Optional. Caps the log table via a cleanup task that runs every minute and keeps only the newest N rows. Not a hard limit: ingesting more than N rows within a minute exceeds the cap until the next run. Unset or `0` disables. Only applies when running with `DB_TYPE=sqlite` (SQLite or DuckDB telemetry). |
diff --git a/docs/pages/server/sqlite.mdx b/docs/pages/server/sqlite.mdx
index b7da7d3e..de5f4dc8 100644
--- a/docs/pages/server/sqlite.mdx
+++ b/docs/pages/server/sqlite.mdx
@@ -4,7 +4,9 @@ import { Callout } from "nextra/components";
Smallest possible Traceway deployment: a single Alpine container that uses SQLite for both the main database and telemetry, with the frontend embedded into the Go binary. No external Postgres or ClickHouse required. Blob storage (source maps, session recordings, AI traces) is written to a local folder by default and can optionally be backed by S3.
-If your dashboards slow down as telemetry grows, the [DuckDB deployment](/server/duckdb) keeps this exact single-container setup but swaps the telemetry store for a columnar engine with far more read headroom.
+<Callout type="warning">
+ **SQLite telemetry is not the recommended setup.** For a single-container deployment we recommend the [DuckDB image](/server/duckdb). It has the same shape as this one: one container, the same `/data` volume, and the main database stays SQLite. Only telemetry moves, to a columnar engine that keeps dashboards fast as data grows. SQLite telemetry still works and is fine for trying Traceway out or for a small instance, but we are not planning major work on it. If you run it on a memory-capped host, read [Memory sizing](#memory-sizing) first.
+</Callout>
All persistent state (both SQLite database files and blob storage) lives under `/data`. Bind-mount that single folder to keep your data across restarts.
@@ -31,6 +33,8 @@ Both SQLite files (`traceway.db`, `traceway_telemetry.db`) and uploaded blobs (u
After starting, open `http://localhost/register` to create your first account.
+If the host or container has less than 16 GB of memory, also set `SQLITE_CACHE_SIZE_MB`. The default can use several gigabytes; see [Memory sizing](#memory-sizing).
+
## Build from Source
Only needed if you want to run a custom or modified build:
@@ -74,6 +78,7 @@ If `STORAGE_TYPE` is unset (or set to `local`), blobs are written to `/data/stor
| `INGEST_MAX_CONCURRENT` | *(2 x CPU cores, min 4)* | Telemetry ingest requests processed at once (`/api/report`, `/api/profiles/ingest`, `/api/otel/*`). Excess requests wait up to `INGEST_ADMISSION_WAIT_SECONDS` (default 5) and then get `503` with `Retry-After`; past a bounded waiting room (four times the capacity, at least 16) they are turned away immediately. Each admitted request can hold a decoded body, so on a memory-capped container size this from the memory limit rather than the CPU count. |
| `UPLOAD_MAX_CONCURRENT` | `4` | Source map and symbol uploads processed at once. Each holds a whole file in memory while it is parsed. Excess uploads wait 30 seconds and then get `503`. Separate from the ingest pool so a CI burst cannot starve telemetry. |
| `SQLITE_PATH` | `/data/traceway.db` | Main SQLite file. Telemetry DB is derived as `<path>_telemetry.db` |
+| `SQLITE_CACHE_SIZE_MB` | `512` | Page cache of the telemetry database in megabytes, per connection. Up to 4 connections are open and each megabyte costs about two, so the caches can take up to 8 times this value: 4 GB at the default. Set it on any memory-capped host, for example `32` for a 1 GB container. A value that is not a positive number falls back to the default. See [Memory sizing](#memory-sizing). |
| `STORAGE_TYPE` | `local` | Blob storage backend. `local` or `s3` |
| `STORAGE_PATH` | `/data/storage` | Folder for local blob storage. Ignored when `STORAGE_TYPE=s3` |
| `S3_BUCKET` | *(unset)* | Required when `STORAGE_TYPE=s3` |
@@ -87,6 +92,57 @@ If `STORAGE_TYPE` is unset (or set to `local`), blobs are written to `/data/stor
| `LOG_RECORDS_MAX_ROWS` | *(unset)* | Optional cap on the number of log rows, independent of `SQLITE_RETENTION_DAYS`. This is not a hard limit: a cleanup task runs every minute and deletes the oldest logs until only the newest N rows remain. Between runs the table can grow past N, and if you ingest more than N rows in a single minute the table (and disk usage) will exceed the cap until the next cleanup. Size it with headroom for your peak ingest rate. Unset or `0` disables the cap. |
| `SESSION_RECORDING_RETENTION_DAYS` | `30` | Days to keep session recording files on disk under `STORAGE_PATH/recordings/`. Files older than the TTL are deleted hourly and on startup; empty subdirectories are removed. No effect when `STORAGE_TYPE=s3`. Set to `0` to disable. |
+## Memory sizing
+
+On a memory-capped host, set `SQLITE_CACHE_SIZE_MB`. The default suits a large dedicated machine, and it will get a small container killed for running out of memory.
+
+The variable sizes SQLite's page cache for the telemetry database. Three things decide what it really costs:
+
+- **It is per connection.** Every connection to the telemetry database keeps its own cache, and Traceway opens up to 4 of them.
+- **It costs about twice its size.** The SQLite driver stores each 4 KB page in an 8 KB slot, so a 32 MB cache takes about 64 MB of memory.
+- **It is kept once filled.** A cache grows as its connection reads or writes pages, and it stays allocated afterwards. Ingest, the hourly retention pass and dashboard queries over recent data all fill it.
+
+So the most the caches can take is:
+
+```
+4 connections x 2 x SQLITE_CACHE_SIZE_MB
+```
+
+With the default of `512` that is 4 GB. One connection alone reaches about 1 GB within seconds of startup once the telemetry file is larger than about 500 MB, because the retention pass reads through it.
+
+A good starting point is the container's memory in MB divided by 32. That keeps the caches within a quarter of the container:
+
+| Container memory | `SQLITE_CACHE_SIZE_MB` | Most the caches can take |
+|------------------|------------------------|--------------------------|
+| 512 MB | `16` | 128 MB |
+| 1 GB | `32` | 256 MB |
+| 2 GB | `64` | 512 MB |
+| 4 GB | `128` | 1 GB |
+| 8 GB | `256` | 2 GB |
+| 16 GB or more | `512` (default) | 4 GB |
+
+```bash
+docker run -d --name traceway \
+ --memory=1g \
+ -p 80:80 \
+ -v "$(pwd)/traceway-data:/data" \
+ -e JWT_SECRET="your-jwt-secret-min-32-characters-long" \
+ -e SQLITE_CACHE_SIZE_MB="32" \
+ ghcr.io/tracewayapp/traceway:sqlite
+```
+
+The rest of the container is not spare. Dashboard queries need about 150 MB of working memory on top of the caches, the source map cache has its own cap (`SOURCEMAP_CACHE_MAX_BYTES_MB`, 500 MB by default), and the operating system uses what is left to cache the database file. That file cache serves every read the SQLite cache does not hold, so a smaller `SQLITE_CACHE_SIZE_MB` changes where pages are cached, not what queries return.
+
+512 MB is only enough while the telemetry file is small. Once it grows to several GB, give the container at least 1 GB, or the first dashboard opened after a restart can take minutes.
+
+The server logs the value it is using at startup:
+
+```
+SQLite telemetry database opened at /data/traceway_telemetry.db (page cache: 32 MB per connection, up to 4 connections)
+```
+
+The sign of a cache that is too large is a container that restarts with exit code 137 (`OOMKilled`) when a dashboard opens or during heavy ingest, with nothing in the application log before the restart.
+
## SSO (optional)
Adds **Continue with Google** / **Continue with GitHub** buttons to the login and register pages. See the [SSO guide](/learn/sso) for the full provider setup walkthrough.
diff --git a/helm/traceway/templates/configmap.yaml b/helm/traceway/templates/configmap.yaml
index 5ea7cdd3..4cd5f83e 100644
--- a/helm/traceway/templates/configmap.yaml
+++ b/helm/traceway/templates/configmap.yaml
@@ -14,6 +14,9 @@ data:
DB_TYPE: "sqlite"
SQLITE_PATH: {{ .Values.sqlite.path | quote }}
SQLITE_RETENTION_DAYS: {{ .Values.sqlite.retentionDays | quote }}
+ {{- if .Values.sqlite.cacheSizeMB }}
+ SQLITE_CACHE_SIZE_MB: {{ .Values.sqlite.cacheSizeMB | quote }}
+ {{- end }}
{{- end }}
{{- if eq .Values.variant "minimal" }}
diff --git a/helm/traceway/values.yaml b/helm/traceway/values.yaml
index 68345509..1efe403d 100644
--- a/helm/traceway/values.yaml
+++ b/helm/traceway/values.yaml
@@ -121,6 +121,10 @@ postgres:
sqlite:
path: "/data/traceway.db"
retentionDays: "30"
+ # Telemetry page cache in MB, per connection. The caches can take up to 8x
+ # this value, so with a memory limit set it to the limit in MB divided by 32
+ # (32 for 1Gi). Empty keeps the backend default of 512.
+ cacheSizeMB: ""
# Session recording storage
storage: