The only tool you need to know what is happening and how to fix it.
CLAUDE.md - Traceway Project
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
Project Overview
Traceway is an error tracking and monitoring platform consisting of:
- Frontend: SvelteKit 2 dashboard application with Svelte 5
- Backend: Go/Gin API server with ClickHouse database
- CLI: Go/Cobra command-line client for the backend HTTP API (
/cli), built twice from one source: read-onlytraceway, andtraceway-write(-tags write, own config dir and OAuth client, write commands incmd/traceway/edition_write.go). Skills:skills/traceway(read),skills/traceway-learn(the product guide, one file per area fromknowledge/learn/, also served astraceway://knowledge/learn/*) andplugins/tw-write/skills/traceway-write(write, separate plugin), all generated fromcli/pkg/mcpserver/knowledgebyjust gen-skills. The skill and MCP evals live incli/test/evals/(own Go module, see its README), never underskills/: anything in a skill directory ships to users and to the agent under test
Code Style
- No pointless comments: Do not add comments that simply describe what the code does. The code should be self-explanatory. Only add comments when explaining non-obvious "why" decisions.
- No
py-4in dialog form content: Do not addpy-4on the content wrapper insideAlertDialogorDialogcomponents — it creates too much blank space between the form and the action buttons. - Dialog button labels & toasts: For form dialogs, use descriptive button labels with icons instead of generic "Create"/"Update". The
{Entity}is always a platform entity, capitalized (Project, Widget, Dashboard, Channel, Rule, Token, Invitation, ...). Create actions:<Plus icon> New {Entity}withvariant="success". Update actions:<Check icon> Update {Entity}with the default (primary) variant. Delete/revoke/remove confirm buttons:<Trash2 icon> Delete {Entity}(orRevoke {Entity}/Remove {Entity}) withvariant="destructive". After success, showtoast.success('Successfully created the {Entity}', { position: 'top-center' })for creates and'Successfully updated the {Entity}'for updates. The button should only bedisabledduring the loading state — never disable it to enforce validation; let the backend return 422 and show the error in the dialog instead.
Quick Start
Development Commands
| Component | Command | Description |
|---|---|---|
| Frontend | cd frontend && npm run dev |
Dev server (port 5173) |
| Frontend | npm run build |
Production build |
| Frontend | npm run check |
TypeScript checking |
| Frontend | npm run lint |
prettier --check . && eslint . (CI gate; npm run format fixes the prettier half) |
| Backend | cd backend && go run ./cmd/traceway |
API server (port 8082) |
| Backend | cd backend && govulncheck ./... |
Vulnerability scan (default tags only); CI also scans the other storage build-tag combos (.github/workflows/backend-vulncheck.yml) |
| CLI | cd cli && just build |
Builds bin/traceway |
| CLI | cd cli && just test |
Runs unit tests |
| CLI | cd cli && just check |
Lint + test + vulncheck + skill drift + contract tests (pre-commit gate; CI enforces it for cli/) |
| CLI | cd cli && just smoke-test |
Live E2E (needs TRACEWAY_SMOKE_* env vars) |
| CLI | cd cli && just eval-harness |
Eval harness tests plus every case's oracle trajectory, fake model only ($0) |
| CLI | cd cli && just eval-plan --preset smoke --max-cost-usd 30 |
Eval matrix and live-price cost estimate ($0) |
| CLI | cd cli && just eval-dispatch -f preset=smoke -f max_cost_usd=30 |
Starts a paid eval run in the CLI Evals workflow; it still waits for approval at the evals environment gate |
| CLI | cd cli && just eval-local ... |
Paid local eval run, for harness development only (yes typed on a terminal, refused without one; required --max-cost-usd) |
| Any | nix develop (or direnv via .envrc) |
Dev shell with the Go and Node versions read from backend/go.mod and .nvmrc, plus just, golangci-lint, govulncheck (flake.nix; shells default, backend, frontend, oxc). Put JWT_SECRET in the gitignored .envrc.local |
| Any | ./scripts/check-node-pins.sh |
Asserts every Dockerfile's ARG NODE_VERSION default matches .nvmrc; run by release-traceway.yml before it builds |
Node's version lives in .nvmrc (a bare major, currently 26) and nothing else derives it independently: every setup-node step uses node-version-file: .nvmrc, flake.nix reads it for the dev shells, and release-traceway.yml passes it to all five image builds as --build-arg NODE_VERSION. The Dockerfiles each carry an ARG NODE_VERSION=<major> default because a bare docker build cannot read .nvmrc; that default is the one value that can drift, which is what scripts/check-node-pins.sh exists to catch (it also fails on a literal node:<major> tag reintroduced in a FROM). To move Node, edit .nvmrc and the ARG defaults, then run the script. frontend/package.json's engines.node is deliberately left as a floor (>=22) rather than a pin — it is what npm enforces on consumers, and node-version-file pointed at it resolves a range to the newest release, which is the drift f95424cc was fixing. testing/devtesting-nestjs/Dockerfile is out of scope: it pins the runtime of a sample third-party app, not anything Traceway builds.
Set JWT_SECRET (min 32 characters) before running the backend, including the All-in-One image. Former public signing keys are rejected. The All-in-One shell script leaves .env loading to the backend so Docker-provided values win. Embedded Run(opts...) never reads JWT_SECRET, since it runs inside the host app's process where that name usually belongs to the host: it takes WithJWTSecret, else TRACEWAY_JWT_SECRET, else a random key per start; set a stable secret to preserve dashboard sessions across embedded restarts. The All-in-One image's supervisord sends the backend's output to the container's stdout/stderr, so docker logs shows a startup rejection. SQLITE_PATH sets the database location, defaulting to ./traceway.db in the working directory.
CI on pull requests is opt-in. backend.yml, backend-vulncheck.yml, cli.yml, cli-lint.yml, cli-contract.yml, eval-guard.yml, frontend.yml and plugin-scanner.yml listen for pull_request: types: [labeled] and every job is gated on github.event.label.name == 'ci', so a PR runs nothing until a maintainer applies the ci label (only collaborators can), and applying any other label runs nothing either. Each application validates the PR as it is at that moment: to validate later pushes, remove and re-add the label. The same workflows still run on pushes to main that touch their paths (backend/**, cli/**, frontend/**, skills/traceway/** for cli.yml, .nvmrc for frontend.yml, .github/workflows/**, .github/actions/**, cli/justfile and cli/test/evals/** for eval-guard.yml, their own workflow file; plugin-scanner.yml runs on every push to main because it scans the whole repo), on their daily schedules (the two vulnchecks), and from the Actions tab via workflow_dispatch. cli-evals.yml (the skill and MCP evals) is the one exception to this pattern: its only trigger is workflow_dispatch, a person starts every run by hand, the paid shard job waits for a required reviewer on the evals environment (the only place OPENROUTER_API_KEY lives, never a repository or organization secret; GitHub creates a missing environment without protection, so the preflight refuses a paid run unless evals exists with a required reviewer), and max_cost_usd is a required input with no default. It must never get a push, pull request, label, schedule, workflow_run, workflow_call or repository_dispatch trigger, no other workflow or composite action may call, dispatch (by file, name or just eval-dispatch) or use its environment, and no just check/test/default recipe may run it; cli/test/evals/internal/guard/triggers_test.go fails if any of that changes. cli/test/evals is its own Go module, which cli.yml's go test ./... never reaches, so eval-guard.yml runs that test. twevals run itself refuses every CI context but a dispatched, approved cli-evals.yml job, and locally it needs yes typed on a terminal. Validate edits to it (or Dependabot bumps of its actions) with dry_run: true, which costs nothing but skips the paid and report jobs, so it does not exercise download-artifact or the shard matrix.
Action versions are watched by .github/dependabot.yml (the github-actions ecosystem only; npm and Go modules are deliberately not watched there). It opens a weekly PR per group — actions/* in one, every third-party action in the other — and deliberately does not auto-apply ci, so a bot PR passes the same collaborator gate as any other. Label the first-party group to validate it; golangci/golangci-lint-action is reachable the same way via cli-lint.yml. The rest of the third-party group lands on release-*.yml, benchmark-*.yml and traceway-autofix.yml, which no label reaches — and workflow_dispatch is not a dry run on the release path: dispatching release-docs.yml or release-website.yml deploys to Cloudflare for real, and the other release-*.yml workflows publish real assets, so dispatching one from an unreviewed bot branch ships it. Validate those by reading the action's release notes instead, and dispatch only where the target is disposable. traceway-autofix.yml is the in-repo caller of the reusable autofix.yml (composite actions .github/actions/autofix-prepare and autofix-publish around the agent step; docs/pages/learn/auto-fix.mdx documents the contract); its workflow_dispatch takes an issue_number and runs the whole loop from the dispatched branch, which is the one pre-merge validation path for the claude-code-action pin — the most privileged action here — but it runs the agent for real, so dispatch it only at a scratch issue. Note too that labelling a PR which touches no gated workflow yields zero checks, which reads like a pass but is silence. Every updatable pin is a bare major tag, so Dependabot only ever proposes major bumps here. The exception is plugin-scanner.yml, which pins both actions to commit SHAs because the HOL scanner scores unpinned actions down; Dependabot bumps those SHAs along with their version comments. It uses the scanner config in .plugin-scanner.toml, whose ignore_paths cover fixtures, demos and build output plus three reviewed production files (the read-only token mask in both project.repository.go copies and KindPassword in cli/internal/state/state.go). The HOL registry's own scan ignores that file and still fails on syntax-highlighter strings in the committed backend/static/frontend bundle; that is accepted.
Tech Stack
- Frontend: SvelteKit 2.49, Svelte 5.45, Tailwind CSS v4, shadcn-svelte, Vite 7
- Backend: Go 1.26, Gin 1.11, ClickHouse, PostgreSQL (in
backend/go.modthegoline is the floor importers inherit and thetoolchaindirective is what builds it — keep the two distinct) - CLI: Go 1.26, Cobra 1.10, separate Go module (
github.com/tracewayapp/traceway/cli); justfile entrypoints, covered by the rootflake.nixdev shell - Client SDK: Go 1.25, Gin middleware support
lit Library (SQL mapper)
- Import:
github.com/tracewayapp/lit/v2(currently v2.0.5) - Purpose: Lightweight generic CRUD operations on top of
database/sql. Supports PostgreSQL, MySQL, SQLite, and DuckDB drivers (lit.PostgreSQL,lit.MySQL,lit.SQLite,lit.DuckDB). Traceway registers models againstdb.Driver(SQLite or PostgreSQL depending on build tags); the DuckDB telemetry backend deliberately keepslit.SQLitefor reads since DuckDB accepts?placeholders. - Docs for AI use: the lit repo ships a skill at
skills/lit/SKILL.mdplusllms.txtwith the full API contract and pitfall checklist.
Model Registration (required before use)
All models are registered centrally in models/models.go via models.Init(driver lit.Driver), called at boot with db.Driver:
func Init(driver lit.Driver) {
lit.RegisterModel[User](driver)
lit.RegisterModel[Project](driver)
// ...all models registered here
}
Repository-local result models (e.g., aggregate structs only used in one repo) can use file-level init() instead. Irregular table names use lit.RegisterModelWithNaming with a struct embedding lit.DefaultDbNamingStrategy (see escalationPolicyNaming in models.go).
Naming Conventions
- Fields: CamelCase → snake_case (
FirstName→first_name) - Consecutive uppercase: stay together (
HTTPCode→http_code) - Tables: pluralize + snake_case (
User→users) - Override via struct tag:
lit:"custom_name"
Core CRUD Operations
All lit functions take a lit.Executor as the first argument. Both *sql.Tx and *sql.DB qualify: main-DB repositories pass the *sql.Tx from the transaction middleware, telemetry repositories pass db.TelemetryDB directly.
| Function | Description |
|---|---|
lit.Insert[T](tx, &entity) |
Insert, returns auto-generated int ID |
lit.InsertUuid[T](tx, &entity) |
Insert with auto-generated UUID |
lit.InsertExistingUuid[T](tx, &entity) |
Insert with pre-set UUID |
lit.Select[T](tx, query, args...) |
Retrieve multiple records (returns []*T) |
lit.SelectSingle[T](tx, query, args...) |
Retrieve one record (returns *T) |
lit.Update[T](tx, &entity, "id = $1", id) |
Update (auto-prepends WHERE) |
lit.UpdateNative(tx, "UPDATE table SET col = $1 WHERE ...", args...) |
Raw SQL update for partial/single-field changes |
lit.Delete(tx, "DELETE FROM table WHERE id = $1", id) |
Delete records (any exec statement) |
lit.SelectNamed[T] / lit.SelectSingleNamed[T] / lit.UpdateNamed[T] |
:name placeholder variants taking a lit.P{...} map; render per-driver, used for dialect-neutral queries |
lit.DeleteNamed(driver, tx, query, lit.P{...}) |
Named delete/exec. Gotcha: takes the driver as the FIRST argument |
lit.ParseNamedQuery(driver, query, lit.P{...}) |
Renders :name to positional; escape hatch for QueryContext/RowsAffected via database/sql |
Transaction Helper (pgdb.ExecuteTransaction)
All PostgreSQL operations should use ExecuteTransaction for automatic commit/rollback:
// ExecuteTransaction[T] wraps a function in a transaction
// - Commits on success, rolls back on error or panic
// - Returns (T, error) directly - no pointer wrapping
project, err := pgdb.ExecuteTransaction(func(tx *sql.Tx) (*models.Project, error) {
// All repository calls receive the transaction
return transactional.ProjectRepository.FindById(tx, id)
})
Transactional Middleware (middleware.Transactional)
For auth flows and routes requiring transaction context throughout the request lifecycle, use the Transactional middleware:
// In routes.go - wrap routes that need transaction context
api.POST("/register", middleware.Transactional, authController.Register)
api.POST("/login", middleware.Transactional, authController.Login)
// In controller - retrieve transaction from Gin context
func (c *AuthController) Register(ctx *gin.Context) {
tx := db.GetTx(ctx) // Get transaction from context (db package, not middleware)
// Use tx for all repository calls
user, err := transactional.UserRepository.FindByEmail(tx, email)
if err != nil {
ctx.JSON(500, gin.H{"error": err.Error()})
return // Transaction auto-rolls back on non-success status
}
ctx.JSON(201, user) // Transaction auto-commits on 200/201/303
}
Auto-commit/rollback behavior:
- Commits on status codes: 200, 201, 303
- Rolls back on all other status codes or panics
Body buffering: Transactional reads the request body into memory (cap maxTransactionalBodyBytes, 8MB) before it calls db.DB.Begin(), answering 413/408/400 without a transaction when that fails; the handler's later bind is served from memory. This is what keeps a slow-drip body from holding the single SQLite main-DB connection, and it covers every transactional route without each one remembering a special middleware. Routes that want a tighter cap put middleware.BufferAuthBody (64KB) in front of Transactional, which then skips the body it finds already buffered; the anonymous auth endpoints do this. The self-transacting OAuth routes (/auth/device/*, /auth/token, /auth/logout) use BufferAuthBody on its own for the same cap. UseAppAuth caps every dashboard route's body at the same 8MB with MaxBytesReader, so the non-transactional telemetry reads are bounded too. Every plain JSON bind site in the controllers answers through middleware.RejectBindError(c, err, fallback), which maps MaxBytesError to 413 and the body guard's timeout to 408 with fixed messages and otherwise returns 400 with the fallback (the ingest routes use middleware.RejectIngestBindError, the same mapping except that the timeout becomes 503 + Retry-After, because OTLP exporters retry 503 but treat 408 as permanent; the Traceway SDKs re-queue a failed batch whatever the status); the guard itself swaps the raw deadline error (a *net.OpError naming the listener address) for middleware.ErrBodyTimedOut before it leaves Read, so no handler can echo the address.
Preference: For CRUD controller methods, always prefer using middleware.Transactional in the route + db.GetTx(ctx) in the controller over pgdb.ExecuteTransaction. The middleware approach keeps controllers flat, avoids nested closures, and follows the established pattern. (The tx getter lives in the db package: db.GetTx(ctx), not middleware.GetTx.)
Repository Pattern
Repositories accept *sql.Tx to participate in transactions:
func (p *projectRepository) FindById(tx *sql.Tx, id uuid.UUID) (*models.Project, error) {
return lit.SelectSingle[models.Project](
tx,
"SELECT id, name, token, framework, created_at FROM projects WHERE id = $1",
id,
)
}
PostgreSQL Specifics
- Uses
$1, $2, $3placeholders (not?) - Tables must have an
idcolumn - Always pass
*sql.TxfromExecuteTransactionto lit functions
Common Pitfalls
Always initialize all struct fields with defaults:
When using lit.Insert, all struct fields are included in the INSERT statement, overriding database defaults. Always set fields like CreatedAt explicitly:
// CORRECT - set CreatedAt explicitly
user := &models.User{
Email: email,
Name: name,
CreatedAt: time.Now().UTC(),
}
// WRONG - CreatedAt remains zero value (0001-01-01)
user := &models.User{
Email: email,
Name: name,
}
lit.Update WHERE clause:
The lit.Update function automatically includes WHERE in the generated SQL. Do not add WHERE yourself:
// CORRECT - just the condition
lit.Update(tx, &user, "id = $1", user.Id)
// WRONG - results in "WHERE WHERE id = $1"
lit.Update(tx, &user, "WHERE id = $1", user.Id)
Custom Result Models for Aggregates
For queries that return aggregated or computed values (not direct table rows), create a custom result model:
// Define a result model for the query output
type CountResult struct {
Count int `lit:"count"`
}
// Register in models.Init() (or file-level init() if repo-local)
lit.RegisterModel[CountResult](lit.PostgreSQL)
// Use in repository
func (r *userRepository) CountByOrganization(tx *sql.Tx, orgID uuid.UUID) (int, error) {
result, err := lit.SelectSingle[CountResult](
tx,
"SELECT COUNT(*) as count FROM users WHERE organization_id = $1",
orgID,
)
if err != nil {
return 0, err
}
if result == nil {
return 0, nil
}
return result.Count, nil
}
Handling "Not Found" Cases
IMPORTANT: When using lit/PostgreSQL, do NOT check for sql.ErrNoRows. The lit library returns nil when no record is found, not an error. Always check for nil instead:
// CORRECT - check for nil
user, err := transactional.UserRepository.FindByEmail(tx, email)
if err != nil {
return nil, err // actual database error
}
if user == nil {
// record not found - handle accordingly
return nil, errors.New("user not found")
}
// WRONG - do not use sql.ErrNoRows with lit
user, err := transactional.UserRepository.FindByEmail(tx, email)
if err == sql.ErrNoRows { // This won't work with lit!
// ...
}
Handling "Not Found" Cases (ClickHouse)
IMPORTANT: ClickHouse queries behave differently from lit/PostgreSQL. ClickHouse returns sql.ErrNoRows when no record is found. Always use errors.Is() to check:
// CORRECT - ClickHouse returns sql.ErrNoRows
exception, err := exceptionRepo.GetByHash(projectID, hash)
if errors.Is(err, sql.ErrNoRows) {
// record not found - handle accordingly
return nil, errors.New("exception not found")
}
if err != nil {
return nil, err // actual database error
}
// Summary of error handling:
// - lit/PostgreSQL: check `if result == nil` (no error returned)
// - ClickHouse: check `errors.Is(err, sql.ErrNoRows)`
Environment Variables (Backend)
JWT_SECRET=<min 32 char secret for JWT signing> # required in the standalone binary and every server image, including All-in-One. Missing, short and former public keys exit 1 with an actionable message. Docker's environment takes precedence over .env. Embedded Run(opts...) ignores JWT_SECRET (it belongs to the host app there): it uses WithJWTSecret, else TRACEWAY_JWT_SECRET (an explicitly empty value fails validation), else a random key per start, so set a stable value to preserve sessions across restarts. Retired public keys are matched by SHA-256 digest in jwt.service.go so no secret literal sits in scanned code.
DB_TYPE= # sqlite | postgres. Defaults to sqlite in every build EXCEPT -tags transactional_pg, where an empty value still selects PostgreSQL. Without that tag SQLite is the only supported main DB (the migration runner applies SQLite-dialect migrations unconditionally), so the default is set once in config.defaultDBType rather than at each DBType call site.
PORTS=80,8082 # comma-separated HTTP listen ports. Every port is bound up front; a port that fails to bind (typically :80 as a non-root user) logs a warning (and reports via CaptureException when monitoring is configured), and startup only fails if NO port binds. Order is not significant: a listener that later stops serving takes the process down whichever position it holds.
SQLITE_CACHE_SIZE_MB=512 # PRAGMA cache_size of the SQLite TELEMETRY DB in MB, per connection (the main DB keeps SQLite's 2MB default; no effect on DuckDB or ClickHouse telemetry). The telemetry pool opens up to 4 connections (`telemetryMaxOpenConns`, db_sqlite_open.go) and modernc's allocator rounds each ~4.4KB page buffer up to an 8KB slot, so resident memory is about 2x the value per connection: worst case 8x the value, 4GB at the default. A cache fills from ingest, from the retention DELETE scan at startup and every hour, and from reads of pages still in the WAL (write transactions and WAL pages bypass the 1GB mmap window), and it is held for as long as the pooled connection lives. At the default a 1GiB container is OOM-killed on a 24h dashboard or under sustained ingest (issue #396). Rule of thumb: container MB / 32 (16 for 512MB, 32 for 1GB, 64 for 2GB). A non-positive or unparsable value falls back to 512 (`config.SizeMB`). The effective value is logged at startup, and the embedded `Run(opts...)` path honors the env var through `applyEnvOverrides`. Operator docs: `docs/pages/server/sqlite.mdx#memory-sizing`.
SQLITE_PATH= # main DB path, defaults to ./traceway.db in the working directory. The telemetry DB derives from it (_telemetry.db, or _telemetry.duckdb under -tags telemetry_duckdb).
APP_BASE_URL= # public origin of this server (e.g. https://traceway.example.com). Used as the OAuth issuer / device verification URL and SSO redirect base, and to absolutize the deep links in notifications. If unset, the device-auth + well-known endpoints derive it per-request from the Host / X-Forwarded-* headers; set it explicitly behind a proxy that doesn't forward Host.
TRUSTED_PROXIES= # comma-separated IPs/CIDRs whose X-Forwarded-For/X-Real-IP gin trusts for c.ClientIP() (every per-IP rate limiter keys on it, and `/api/report` stores it on every session as `client.ip`). Unset = loopback only, so no peer can spoof its IP. Private ranges are deliberately NOT trusted by default: on a Docker/k8s network every peer has a private address, the clients included when the port is published directly, so trusting RFC1918 would let any of them forge XFF past every per-IP limiter. Set it to the CIDRs of whatever sits in front (a proxy container's compose network, the ingress pod range, the CDN ranges); otherwise every visitor shares one rate-limit bucket keyed on the proxy. The value REPLACES the default, so it must name every hop in the chain, not just the outermost (a CDN-only list leaves a private ingress untrusted and disables XFF entirely); `*` trusts every peer. A malformed entry exits 1 at startup with a FATAL message, before the database is opened (`trustedProxyNets`, cmd/run.go). The first request carrying X-Forwarded-For/X-Real-IP from a peer outside the list logs a one-time warning naming that peer (`warnOnUntrustedForwardedFor`, cmd/run.go), which is how a forgotten proxy hop shows up. Parsing lives in `Cfg.TrustedProxyList` (config.go), covered by config_test.go.
TRUSTED_PROXY_HEADER= # optional header taken verbatim as the client IP on every request (gin TrustedPlatform: X-Real-IP behind ingress-nginx, CF-Connecting-IP behind Cloudflare); it wins over TRUSTED_PROXIES and turns the untrusted-forwarder warning off. Only safe when every request provably passes through a proxy that overwrites the header.
UPLOAD_MAX_CONCURRENT=4 # concurrent /api/sourcemaps/upload + /api/symbols/upload requests; excess waits 30s then gets 503. Separate from INGEST_MAX_CONCURRENT so a CI upload burst cannot starve telemetry ingest. Each upload holds a whole file in memory while parsing, so this bounds peak upload memory at roughly concurrency x largest file. It is a count-based gate, not a memory budget: on a memory-capped container size it from the limit rather than trusting the default. Same gate and bounded waiting room as ingest, its own instance. The source-map warm-up that follows an upload (`services.GenerateTWArtifacts`) runs outside this gate on its own 2-worker pool with a 64-job queue: the stale `.tw` artifact is deleted and the cache invalidated inline, the rebuild is queued, identical pending jobs are coalesced, and a full queue drops the warm-up (rate-limited `CaptureException`) because lookups build a missing artifact on demand anyway.
REPORT_MAX_BODY_MB=64 # cap on the DECOMPRESSED /api/report and /api/profiles/ingest body (middleware.UseGzip bounds both the raw and gunzipped stream, so gzip cannot raise it). Deliberately far above the 10MB OTLP cap: session-replay frames carry rrweb segments and are a different size class, and the shipped browser SDKs re-queue a rejected batch forever, so a reachable cap is a permanent wedge rather than shed load. Overruns answer 413.
CLICKHOUSE_SERVER=localhost:9000
CLICKHOUSE_DATABASE=traceway
CLICKHOUSE_USERNAME=default
CLICKHOUSE_PASSWORD=
CLICKHOUSE_TLS=false
POSTGRES_HOST=localhost
POSTGRES_PORT=5432
POSTGRES_DATABASE=traceway
POSTGRES_USERNAME=traceway
POSTGRES_PASSWORD=
POSTGRES_SSLMODE=disable
# DuckDB telemetry backend (only with -tags telemetry_duckdb build; see "DuckDB Telemetry Backend" below). The embedded Run(opts...) path honors these three and DUCKDB_RETENTION_DAYS through applyEnvOverrides.
DUCKDB_MEMORY_LIMIT= # e.g. 4GB. Unset = DuckDB auto-tunes (~80% RAM). Set explicitly in memory-capped containers to avoid OOM.
DUCKDB_THREADS= # e.g. 4. Unset = DuckDB auto-tunes (= cores). Cap in constrained/shared environments.
DUCKDB_CHECKPOINT_THRESHOLD= # e.g. 256MB. Unset = DuckDB default (16MB). Raise under sustained ingest to reduce WAL checkpoint stalls; costs a larger WAL and longer restart replay.
# Ingest admission gate (all telemetry ingest endpoints: /api/report, /api/profiles/ingest, /api/otel/*)
INGEST_MAX_CONCURRENT= # max concurrently processed ingest requests. Unset = 2×CPU cores, min 4. Bounds ingest memory so overload sheds load with 503s instead of the process being OOM-killed (on DuckDB an OOM death is followed by a minutes-long WAL-replay stall on restart). The gate (`newAdmissionGate` in `middleware/ingest_admission.go`, a buffered-channel semaphore) also bounds waiters at 4×capacity (min 16); beyond that a request gets an immediate 503 instead of parking a goroutine and a timer for the wait window.
INGEST_ADMISSION_WAIT_SECONDS=5 # how long a request may wait for a slot before the 503 + Retry-After; 0 = reject immediately when saturated
# Email
EMAIL_PREVIEW_ENABLED=false # "true" registers GET /api/email-preview[/:template], rendering every email template with sample data for design review. Off by default; never enable in production.
# Notifications
NOTIFICATION_POLL_SECONDS=60 # polled rule evaluation interval; minimum 5, invalid values fall back to 60
ONCALL_POLL_SECONDS=30 # on-call escalation worker interval; minimum 5, invalid values fall back to 30. Kept separate from NOTIFICATION_POLL_SECONDS so raising rule-evaluation intervals never delays paging. A buffered Wake() channel makes freshly opened pages notify L1 near-instantly regardless of this interval.
OUTBOX_POLL_SECONDS=15 # notification outbox drain interval; minimum 5, invalid values fall back to 15. The outbox (backend/app/outbox, notification_outbox table in the main DB) is the persist-then-send layer for ALL notifications: rule dispatch and the escalator only enqueue (with an adapter-config snapshot) inside their transactions; the drain worker sends with retries (backoff 1m/5m/15m/60m, 5 attempts, then terminal failed + CaptureException). Crash-safe at-least-once: stale 'sending' rows are reclaimed after 5 min, cancelled rows can never resurrect (guarded status transitions), ack/resolve cancels queued page deliveries via outbox.CancelByKey. Cooldown and event-rule dedup record at enqueue commit (the durable promise), and fired_notifications is written at the terminal outcome. /api/health/deep exposes an `outbox` block; `traceway.outbox.*` metrics are emitted when monitoring is on; terminal rows are pruned daily (sent/cancelled 7d, failed 30d).
SLO_POLL_SECONDS=60 # SLO worker interval (backend/app/slo): recomputes the last 10 minutes of every enabled SLO into the telemetry `slo_buckets` table (every 15 minutes, in a slot derived from the SLO id and decided against its persisted `computed_at` (`passDue`, `slo/schedule.go`) so passes are spread over ticks and a restart or shard move neither skips nor bunches them, the other kinds also recompute the last 75 minutes, for long requests and retried exports, and session and task SLOs their exception slack plus 25 minutes, since a crash or failure lands after the bucket of its start; once a day session SLOs recompute the last 72 hours through the gap fill, for native crashes uploaded on the next launch), computes `SLO_COMPUTE_WORKERS` (default 4, max 64) SLOs at a time with a per-SLO `SLO_COMPUTE_TIMEOUT_SECONDS` (default 30; a timed-out SLO keeps its old snapshot and resumes from `last_bucket_at` on the next tick), works through the SLOs in groups of 200 so memory stays bounded and each group's buckets go out in one batched insert stamped at write time (ClickHouse makes a part of every insert); the window sum is incremental (`slo/snapshot_cache.go`): the part of the window before an anchor 7h back (73h for session SLOs) is cached in memory and only loses the buckets that slide out, the recent part is read every tick, and the whole window is re-read when the cache is missing, the revision or `computed_at` differs from what this process wrote (an edit, or another instance computed it), a bucket was written before the anchor, or once an hour in the SLO's slot; an SLO with no `last_bucket_at` (new, or reset by a definition change) has its buckets deleted by the worker before anything is written, never by the controller, so a delete cannot land after the new buckets; backfills older history in 6h chunks (24h for metric SLIs so ClickHouse reads the 1m rollups) under one 5s budget shared by every SLO of the tick (start rotated per tick), records an outage longer than the 24h catch-up as a `gap_from`/`gap_to` range it fills under the same budget, writes the snapshot columns on `slos` only while the row is still at the `revision` it loaded (an edit during a tick makes it drop its result; the reset of a definition change clears the buckets on the next tick), on Postgres splits the SLOs over 64 shards (`id % 64`) leased per instance through `slo_workers` (heartbeat) and `slo_shard_leases` (pg/0157, pg/0158; `slo/shards.go`): each tick an instance renews its leases, releases any above its fair share (`ceil(64 / live instances)`) and claims free or expired ones, renews again between groups, leases last max(5 min, 3x the poll) and a stopped instance's shards move once it ages out, shard 0's owner runs the daily prune; SQLite owns all shards, counts metric slices without samples as bad only from `sli_since` (set on create and on every definition change), rounds slo_burn_rate windows up to whole slices and clears it when either window drops below the threshold, slo_budget_consumed reads the snapshot's `budget_remaining` instead of re-summing the window, rule state moves through `NotificationRuleStateRepository.Transition` (a conditional update, so two instances never both notify), both slo_* rules hold their state while the SLO has no `last_bucket_at` (a definition change being recomputed) so a reset never sends a false recovery, a recovery notice does not record a cooldown fire, prunes buckets older than 120 days daily, and reports the `slo` block of `/api/health/deep` (totals incl. timeouts, `runningForSec`, owned shards and instances, the last tick's phase timings with its slowest SLO, and `freshness` read from the main DB: enabled, stale > 5 min, max staleness, backfill pending), which `monitoring.StartSLOReporter` emits every 30s as `traceway.slo.*` metrics when self-monitoring is on (documented in `docs/pages/server/capacity-monitoring.mdx`); minimum 5, invalid values fall back to 60. `slo.Wake()` runs a tick right after an SLO is created or changed. Buckets are deliberately NOT in the telemetry retention list; slo_* notification rules read them (never raw data) and keep their firing state in `notification_rule_states` so they can send recovery notices across restarts.
SLO_COMPUTE_WORKERS=4 # SLOs computed at once per instance (max 64)
SLO_COMPUTE_TIMEOUT_SECONDS=30 # per-SLO budget for its raw queries and window sums in one tick; a timeout is counted (traceway.slo.timeouts.delta), captured as an exception naming the SLO, and retried next tick
# Synthetics (synthetic uptime monitoring: backend/app/synthetics, /synthetics frontend route)
SYNTHETICS_POLL_SECONDS=15 # scheduler tick for due checks; minimum 5, invalid values fall back to 15. The scheduler enqueues due checks into the check_runs queue (main DB, outbox-style guarded claims, advisory lock 824737004) and records expired queued runs as `missed` in telemetry — a probe is never executed late. In-process executors claim http/tcp runs always, browser runs only when mode=embedded.
SYNTHETICS_BROWSER_MODE=off # off | embedded | remote. Browser checks are real @playwright/test specs executed by spawning Node against a harness dir (no npm/npx at runtime; allowlisted env so user scripts never see server secrets). `embedded` requires the :browser image (Dockerfile.browser, DuckDB base + Node + Chromium) and fails fast at startup otherwise; `remote` queues browser runs for traceway-runner binaries that long-poll /api/runners/poll authenticating with SYNTHETICS_RUNNER_SECRET. Hard-blocked in cloud mode (startup panic + 422 at check creation).
SYNTHETICS_BROWSER_SANDBOX=auto # auto | bwrap | off. Isolation for the Playwright subprocess in embedded mode and in traceway-runner. The env allowlist alone does not contain a spec: node runs as the same uid as its parent, so unconfined it can read the parent's /proc/<pid>/environ (the runner secret), overwrite node_modules in the shared harness to backdoor later runs, and read a sibling run's directory. `bwrap` gives each run its own pid/ipc/uts/user namespace with a fresh /proc, a read-only harness bind with only that run's dir writable, an empty tmpfs over the .runs container so sibling run dirs are hidden (not exposed by the wholesale harness bind), and no /etc beyond DNS/CA/locale (so a secret in the unit file or EnvironmentFile is unreachable); the network namespace stays shared since probes must reach the internet. `auto` uses bubblewrap when the host can and degrades to off with a warning; explicit `bwrap` fails fast at startup instead. Resolved once via browserexec.ResolveSandbox, which probes by running `node --version` through the real bind set.
SYNTHETICS_RUNNER_SECRET= # shared bearer credential for the operator's runner fleet; required for remote mode (fail-fast at startup). Runners are deployment infrastructure, NOT tenant entities: no dashboard CRUD, no per-runner tokens. A runner self-registers a liveness row in synthetic_runners under its X-Traceway-Runner-Name on first poll (upsert throttled to 1/min via an in-memory cache; claim identity is "runner:<name>"). Rotation = change the secret + restart runners. Runner-side env: TRACEWAY_URL, TRACEWAY_RUNNER_SECRET, TRACEWAY_RUNNER_NAME (default hostname), RUNNER_WORKERS (default 2, max 16), and optional TRACEWAY_RUNNER_MONITORING (<project_token>@<url>/api/report) which self-instruments the runner with the Traceway Go SDK: default server metrics, a "browser_run" task trace per executed run (tags check_id/run_id/status, server_name = runner name), and CaptureException on infra failures (start/report errors, per-job panics recovered without killing the worker).
HEALTH_DEEP_TOKEN= # operator bearer secret gating GET /api/health/deep (its payload is instance-wide: cross-tenant queue depth, runner fleet, storage engine stats). Unset = endpoint disabled with a 401 pointing here.
SYNTHETICS_HTTP_CONCURRENCY=8 # concurrent in-process http/tcp probes
SYNTHETICS_BROWSER_CONCURRENCY=2 # concurrent Chromium instances in embedded mode (~300-500MB each)
SYNTHETICS_ALLOW_PRIVATE_TARGETS=true # "false" rejects checks that resolve to private/LAN addresses, validated at save AND at dial time for both http and tcp probes (netguard.GuardedDialContext dials the vetted IP, DNS-rebinding safe). Forced false in cloud mode regardless of env (tenants must never probe the platform's network). Default allow elsewhere: probing LAN services is a core self-hosted use case. When the guard is off, http probes honor HTTP(S)_PROXY; when on, the proxy is ignored so the guard vets the real target. Check creation also runs controllers.CheckLimitHook (nil = unlimited; cloud caps monitors per org plan).
SYNTHETICS_PLAYWRIGHT_DIR=/opt/traceway-playwright # Playwright harness dir (node_modules with pinned @playwright/test; docker/playwright/package.json pins the version)
SYNTHETICS_SCREENSHOT_RETENTION_DAYS=30 # browser failure artifacts under STORAGE_PATH/synthetics/ (.png screenshots AND .log Playwright output tails — failed browser runs store both, referenced by screenshot_key/output_key on check_results, served via GET /api/synthetics/screenshot|output?key= with project prefix checks); local storage only, 0 disables the cleanup worker
# Retention (see "Data Retention" section below)
SQLITE_RETENTION_DAYS=30 # 0 to disable; only applies in SQLite mode
DUCKDB_RETENTION_DAYS=30 # telemetry TTL on the DuckDB backend; when set it wins over SQLITE_RETENTION_DAYS there, when unset SQLITE_RETENTION_DAYS applies as fallback; 0 to disable
LOG_RECORDS_MAX_ROWS= # optional cap on log_records rows; unset/0 = disabled. NOT a hard limit: a cleanup worker trims to the newest N rows once per minute, so ingest above N rows/minute overshoots the cap between runs. Only applies in SQLite mode (SQLite or DuckDB telemetry)
SESSION_RECORDING_RETENTION_DAYS=30 # 0 to disable; only applies when STORAGE_TYPE=local
PROFILE_ARCHIVE_RAW=false # native pprof ingest only: write the original pprof bytes to object storage as a lossless archive
PROFILE_RETENTION_DAYS=30 # 0 to disable; on-disk archive TTL, only with PROFILE_ARCHIVE_RAW + STORAGE_TYPE=local
# V2 trace tables move-over (see "Trace Storage (V2 tables)" below)
V2_MOVE_OVER= # "true" copies the history of the tables V2 replaced into the V2 tables, in the background, newest day first. Runs at startup and one hour after each pass; resumable (progress in the transactional DB v2_move_over_days, sqlite/0077 + pg/0143), so it can stay set. Finish upgrading old writers before enabling it: completed source-table/date pairs are skipped. With several backend instances set it on exactly one. On SQLite and DuckDB it stops at the telemetry retention window, since older days would be pruned at once.
V2_MOVE_OVER_OLDEST= # optional YYYY-MM-DD; days before it stay in the old tables. An unparsable value logs and does not start the move.
V2_MOVE_OVER_WORKERS=1 # source days copied concurrently inside the one enabled instance (max 16; invalid/non-positive = 1). Each day keeps table order; days are scheduled newest first. Increase to 4 for parallel backfill when the telemetry DB has spare capacity. Progress rows and logs update after successful batches roughly every 10s; slow queries/inserts can delay updates. Errors cancel and join all workers before the next hourly pass.
# Session recording uploads (see "Session Recording Uploader" section below)
SESSION_RECORDING_UPLOAD_WORKERS=32 # 0 to disable uploads entirely
SESSION_RECORDING_UPLOAD_QUEUE_SIZE=2048
# Source map symbolicator
SYMBOLICATOR_PARSER=goja # goja (default) or oxc (requires -tags oxc build, see scripts/build-oxc-shim.sh)
SOURCEMAP_CACHE_TYPE=memory # memory (default) or disk (mmap-backed .tw cache)
SOURCEMAP_DISK_CACHE_PATH=./twcache # only used when SOURCEMAP_CACHE_TYPE=disk
SOURCEMAP_DISK_CACHE_MAX_MB=2048 # capacity-based LRU eviction of local .tw files
Architecture Overview
Data Flow
Go Application → [traceway SDK] → GZIP POST /api/report → Backend → ClickHouse
↓
Dashboard ← [SvelteKit Frontend] ← JSON API ← Gin Controllers
Authentication
Two-tier system:
- Client Auth: Project bearer tokens (SDK telemetry via
Authorization: Bearer <project_token>) - App Auth: JWT-based user authentication (dashboard via
Authorization: Bearer <jwt_token>)
App Auth credentials. UseAppAuth accepts three credential shapes on the Authorization: Bearer header:
- Dashboard JWT — issued by
/api/login/ SSO (7-day expiry). - Personal access token (PAT) — opaque
twp_-prefixed token; looked up by SHA-256 hash inpersonal_access_tokens, resolves to its user. Created/listed/revoked from the account page (/api/personal-access-tokens*). Non-expiring or with an optional TTL;last_used_atis touched (throttled to 1/min, off the request path). - Device-flow access token — a short-lived (15-min) JWT minted by the CLI's OAuth device flow.
Agent scopes and the agent gate. Every credential except a dashboard session is an agent credential with a scope, traceway:read or traceway:write (models.ScopeRead/ScopeWrite). PATs store it (personal_access_tokens.scope; read tokens are twp_, write tokens tww_ and must expire within 90 days), refresh tokens and authorization codes store it, and access JWTs carry it as the scope claim (plus cid, the OAuth client). Device logins take their scope from the client id (knownDeviceClients: traceway-cli and traceway-mcp read, traceway-cli-write write); the authorization-code grant takes it from the request's scope param (models.ParseRequestedScope, default read); a refresh keeps the family's scope; POST /api/login with a scope field returns a token set instead of a session (token is a 15-minute scoped access JWT, plus refreshToken/expiresIn in a refresh family with client id traceway-cli or traceway-cli-write, so POST /api/auth/logout revokes it; the CLI's password login always sends a scope and stores the result like a device login). An unscoped JWT that lives one hour or less predates scopes and is held to read. UseAppAuth sends agent credentials through authorizeAgentRequest (middleware/agent_gate.go): the route must be in the allowlist agentRoutes (middleware/agent_routes.go, keyed METHOD /api/path by c.FullPath(), default deny, so new dashboard routes are never agent-reachable by accident; TestAgentRoutesAreRegistered keeps entries pointing at real routes); a write route needs the write scope (else 403 with a scope="traceway:write" challenge) and the owning organization's opt-in organizations.agent_write_enabled (Settings > Agent Access, PUT /api/organizations/:organizationId/agent-access, else 403), resolved from ?projectId= together with membership, which is why every allowlisted write is project-scoped. Accepted and opt-in-refused agent writes are recorded in agent_write_events (main DB, pruned after 90 days by the auth-tokens worker, listed at GET /api/organizations/:organizationId/agent-activity). The role checks (RequireWriteAccess etc.) still apply after the gate. Dashboard sessions skip the gate entirely. Docs: docs/pages/learn/agent-write-access.mdx.
CLI / OAuth device flow (backend/app/services/authserver/, controllers device_auth.controller.go / wellknown.controller.go / pat.controller.go): RFC 8628 device authorization grant plus rotating refresh tokens. traceway login (default) → POST /api/auth/device/authorize (client_id allowlisted, per-IP rate-limited, opportunistically prunes expired rows) → user approves at /device → the CLI polls POST /api/auth/device/token (grant device_code; /api/auth/token is an equivalent alias) which issues a 15-min access token + 90-day rotating refresh token (family-tracked in refresh_tokens). Refresh (grant_type=refresh_token) rotates the token atomically and revokes the whole family on genuine reuse; within a 30s grace window a benign concurrent retry is answered with the same rotated token set (from an in-memory rotation cache) instead of invalid_grant. POST /api/auth/logout revokes a family server-side. Tokens are stored SHA-256-hashed. The grant endpoints self-manage their transactions via db.ExecuteTransaction (not middleware.Transactional) because OAuth returns 400 for normal flow control (authorization_pending, reuse-revoke) and those side effects must still commit. /.well-known/oauth-authorization-server and /.well-known/oauth-protected-resource are served at the origin root (registered in cmd/run.go, not the /api group) per RFC 8414 / 9728. Beyond the device grant, the server implements the authorization-code + PKCE grant (S256 only) with RFC 7591 dynamic client registration for MCP clients: POST /api/oauth/register (rate-limited, open registration of public clients; https/custom-scheme redirect URIs anywhere, plain http only on loopback with port-flexible matching per RFC 8252) -> the client sends the user to /oauth/authorize (an SPA consent page like /device) -> the page calls GET /api/oauth/client + POST /api/oauth/approve|deny (approve mints a single-use 5-min twa_ code bound to user/client/redirect/challenge; the redirect target is validated server-side) -> the client exchanges it at POST /api/auth/token (grant authorization_code; the code is consumed even on a failed exchange, wrong verifier/client/redirect are all invalid_grant). RFC 8707 resource params are validated against the issuer origin (invalid_target). Expired codes are pruned by the auth-tokens retention worker and opportunistically at approve time.
Frontend (/frontend)
Framework & Build
- Framework: SvelteKit 2 with Svelte 5 runes API
- Styling: Tailwind CSS v4 with shadcn-svelte components
- Build: Vite 7, static adapter with SPA fallback
- SSR: Disabled - pure client-side SPA (
ssr = falsein+layout.ts)
Project Structure
frontend/
├── src/
│ ├── routes/ # SvelteKit pages
│ ├── lib/
│ │ ├── api.ts # API client with auth
│ │ ├── state/ # Svelte 5 state management
│ │ ├── components/
│ │ │ ├── ui/ # shadcn-svelte base components
│ │ │ └── traceway/ # Custom Traceway components
│ │ └── utils/ # Helpers (formatting, sorting)
│ └── app.css # Tailwind + global styles
├── static/ # Static assets
└── svelte.config.js # SvelteKit config
State Management (Svelte 5 Runes)
Runes Pattern
// Use $state() for reactive state
let data = $state<Type>(initial)
// Use $derived() for computed values
let computed = $derived(expression)
// Use $effect() for side effects
$effect(() => { /* reactive code */ })
State Files
| File | Purpose | Persistence |
|---|---|---|
src/lib/state/auth.svelte.ts |
Token auth, login/logout | localStorage |
src/lib/state/projects.svelte.ts |
Multi-project management | localStorage |
src/lib/state/theme.svelte.ts |
Dark/light mode toggle | localStorage |
src/lib/state/timezone.svelte.ts |
UTC/local timezone toggle | localStorage |
Singleton Pattern
State files export class instances as singletons:
// src/lib/state/auth.svelte.ts
class AuthState {
token = $state<string | null>(null)
isAuthenticated = $derived(!!this.token)
constructor() {
// Load from localStorage on init
this.token = localStorage.getItem('token')
}
}
export const authState = new AuthState()
API Client (src/lib/api.ts)
The API client automatically:
- Includes
Authorization: Bearer <token>header - Adds
projectIdas a query parameter to all requests - Handles 401 responses by logging out and redirecting to
/login
// Usage
const data = await api.post<ResponseType>('/endpoint', { body })
const data = await api.get<ResponseType>('/endpoint')
Component Patterns
Table System
Tables use shadcn-svelte base components with custom Traceway wrappers:
| Component | Location | Purpose |
|---|---|---|
Table, TableHeader, etc. |
src/lib/components/ui/table/ |
Base shadcn components |
TracewayTableHeader |
src/lib/components/traceway/traceway-table-header.svelte |
Adds sorting + tooltips |
TableEmptyState |
src/lib/components/traceway/table-empty-state.svelte |
Empty state display |
PaginationFooter |
src/lib/components/traceway/pagination-footer.svelte |
Pagination controls |
Sorting Storage
Table sorting persists to localStorage using a consistent pattern via src/lib/utils/sort-storage.ts:
// Types
type SortState = { field: string; direction: 'asc' | 'desc' }
// Key format: traceway_sort_{pageKey}
// Example: traceway_sort_issues, traceway_sort_endpoints
// In +page.svelte - load initial state
let sortState = $state<SortState>(getSortState('issues', { field: 'last_seen', direction: 'desc' }))
// After sort change
function onSortClick(field: string) {
sortState = handleSortClick(field, sortState.field, sortState.direction, 'desc')
setSortState('issues', sortState) // Persist to localStorage
}
Available functions (src/lib/utils/sort-storage.ts):
| Function | Description |
|---|---|
getSortState(pageKey, defaultState) |
Load sort state from localStorage |
setSortState(pageKey, state) |
Save sort state to localStorage |
handleSortClick(field, currentField, currentDirection, defaultDirection) |
Toggle sort direction, returns new SortState |
TracewayTableHeader Component
<TracewayTableHeader
label="Last Seen"
column="last_seen"
tooltip="When this issue was last reported"
{orderBy}
onclick={() => handleSortClick('last_seen')}
/>
URL State Management
Time range and filters persist in URL query params via src/lib/utils/url-params.ts:
// Available presets: 30m, 60m, 3h, 6h, 12h, 24h, 3d, 7d, 1M, 3M
// Parse time range from URL (in +page.svelte)
const timeRange = parseTimeRangeFromUrl(timezoneState.timezone, '24h')
// Get resolved Date objects for API calls
const { from, to } = getResolvedTimeRange(timeRange, timezoneState.timezone)
// Update URL with new time range (preserves other params)
updateUrl({ preset: '7d' })
updateUrl({ from: customFrom, to: customTo }) // Custom range
Available functions (src/lib/utils/url-params.ts):
| Function | Description |
|---|---|
parseTimeRangeFromUrl(timezone, defaultPreset) |
Parse TimeRangeParams from current URL |
getResolvedTimeRange(params, timezone) |
Convert params to { from: Date, to: Date } |
updateUrl(params, options?) |
Update URL query params, optionally replace history |
Navigation Utilities
Helper functions for preserving URL params during navigation (src/lib/utils/navigation.ts):
// Add sticky params (like time range) to href for <a> tags
const href = addStickyParamsToHref('/issues/abc123', 'preset', 'from', 'to')
// Result: "/issues/abc123?preset=24h" (if preset=24h is in current URL)
// Create click handler for table rows that preserves params
const handleClick = createRowClickHandler('/issues/abc123', 'preset', 'from', 'to')
Available functions:
| Function | Description |
|---|---|
addStickyParamsToHref(href, ...stickyParams) |
Returns href with specified params from current URL |
createRowClickHandler(href, ...stickyParams) |
Returns click handler that navigates with sticky params |
Routes
/ Dashboard (protected) - overview metrics
/login Login page (public)
/register Registration page (public)
/issues Issues list with filtering/sorting
/issues/[hash] Exception details view
/issues/[hash]/events Exception events timeline
/endpoints Endpoint analytics with P50/P95/P99
/endpoints/[endpoint] Single endpoint details
/tasks Background tasks list
/tasks/[task] Single task details
/spans Span explorer: every OTel span of the project, filtered by name, service, kind, status, duration, trace ID and attribute values (time range defaults to 60m)
/spans/[traceId] Whole trace waterfall across services and across every project of the organization (?at= is required, ?span= highlights one span), with the current project's correlated logs on backend projects
/dashboards Dashboards page (tabs of org dashboards; /metrics redirects here)
/monitors Monitors (synthetic checks): single sidebar item, TabsRow tabs Monitors | Status Pages | Post-Mortems (?tab=, Status Pages admin-only, Post-Mortems for all members); status pages tab has branding (description, logo upload, custom domain) plus a link to the per-page incidents page (which owns the Record Incident dialog); old /monitors/status-pages redirects to the tab, /monitors/runners to /monitors (runners have no UI — operator infra), bare /monitors/post-mortems to the tab
/monitors/status-pages/[pageId]/incidents Paginated incident management for one status page (timeline updates, titles, manual resolve/delete, post-mortem links); reads for members, mutations admin-gated in the UI
/monitors/[checkId] Monitor detail (latency chart, uptime bars, runs w/ result filter, incidents w/ titles + post-mortem links)
/monitors/post-mortems/[id] Post-mortem editor/viewer (@milkdown/crepe WYSIWYG with a Rich text/Markdown raw toggle, tag pills + Add-tags dialog, incident link pill + searchable picker, Activity sheet; write gating follows the effective project role). Project-scoped: opening a doc under a different project 404s. Creation is a title-only dialog (new-post-mortem-dialog.svelte, openable from the tab, monitor detail incidents, and the status page incidents page with an optional incident pre-link) that POSTs immediately and navigates here; /monitors/post-mortems/new redirects to the tab
/slos SLOs (service level objectives, `backend/app/slo`): one row per objective with kind badge, SLI summary, target, window, current SLI, error-budget bar, 1h burn rate and status; sort key `slos`; sidebar badge = breached count; frontend and mobile projects see only the `session` kind (crash-free sessions, `sessions` joined to `exceptions_v2.session_id`), backend projects the other five
/slos/[id] SLO detail: SLI and error-budget charts over a range clamped to the window (default = the whole window), burn-rate tiles, a coverage note while history backfills, a "View bad events" deep link per kind, and the alert rules that reference the SLO
/status/[slug] Public status page (light standalone design, no auth, raw fetch, listed in isPublicPath)
/connection SDK integration guide
UI Components
Location: src/lib/components/ui/*
Uses shadcn-svelte registry with bits-ui primitives. Key components:
button,card,table,badge,tooltipselect,input,checkboxsheet(slide-out panels),dialog(modals)
Backend (/backend)
Architecture
- Framework: Gin Gonic HTTP framework
- Database: ClickHouse (columnar OLAP for telemetry), PostgreSQL (relational for projects)
- Port: 8082
- Pattern: Repository pattern with singleton controllers
Project Structure
backend/
├── traceway.go # package tracewaybackend - embeddable API (Run, WithPort, ...)
├── cmd/
│ ├── run.go # Run() implementation: DB init, routes, server start
│ ├── traceway/main.go # Binary entry point (package main)
│ └── traceway-runner/ # Synthetics remote runner binary
├── app/
│ ├── controllers/
│ │ ├── routes.go # Route registration
│ │ ├── dashboard.go # Dashboard metrics
│ │ ├── auth.go # Login handler
│ │ ├── projects.go # Project CRUD
│ │ └── clientcontrollers/
│ │ └── report.go # Telemetry ingestion (/api/report)
│ ├── repositories/ # ClickHouse queries
│ │ ├── transactions.go
│ │ ├── exceptions.go
│ │ ├── metrics.go
│ │ └── projects.go
│ ├── models/ # Data structures
│ ├── middleware/
│ │ ├── auth.go # Token validation
│ │ └── gzip.go # Request decompression
│ ├── cache/ # In-memory project cache; on Postgres, replicas refresh each other through LISTEN/NOTIFY (project_cache_changed, sent by the project repository inside the write transaction)
│ ├── pgdb/ # PostgreSQL connection manager
│ └── migrations/
│ ├── ch/ # ClickHouse migrations
│ └── pg/ # PostgreSQL migrations
Middleware Chain Composition
| Route Type | Middleware Chain |
|---|---|
| Read-only telemetry | UseAppAuth, RequireProjectAccess |
| Write telemetry | UseAppAuth, RequireProjectAccess, RequireWriteAccess |
| PostgreSQL CRUD (read) | UseAppAuth, RequireProjectAccess, Transactional |
| PostgreSQL CRUD (write) | UseAppAuth, RequireProjectAccess, RequireWriteAccess, Transactional |
| Admin org management | UseAppAuth, RequireAdminAccess, Transactional |
| Public (auth/invitations) | RateLimitPerIP, BufferAuthBody, Transactional (limiter first so a throttled request never reads a body or opens a transaction) |
| Client SDK ingestion | CORSReport, UseClientAuth, UseGzip |
| Synthetic runner API | UseRunnerAuth only (shared SYNTHETICS_RUNNER_SECRET bearer + self-registration by X-Traceway-Runner-Name; no Transactional — poll holds the request up to 25s) |
| Public status page | RateLimitPerIP only |
Global middleware (registered in cmd/run.go ahead of the routes): middleware.GuardBodyReads (every request body gets the progress guard, 30s idle / 10 min total / 1 KB/s floor after 20s, answering through middleware.BodyReadError as 408; routes with their own budget call GuardBodyRead again, which re-arms the same guard rather than wrapping twice, so uploads get 30s/30 min, /api/report 20s/10 min and OTLP 20s/2 min. The guard clears the socket read deadline the moment the body is complete: Go's background connection read would otherwise time out and cancel the request context of any handler that outlives the idle window. The http.Server ReadTimeout is a 1h backstop that only applies while a body is still arriving (the guard lifts it for bodiless requests too, so a long-lived handler such as the MCP GET /mcp stream is not cancelled after an hour), ReadHeaderTimeout is 15s), configureClientIP (gin's trusted proxies from Cfg.TrustedProxyList, plus TrustedPlatform from TRUSTED_PROXY_HEADER; a malformed list panics), warnOnUntrustedForwardedFor (list mode only: one-time log when a forwarding header arrives from a peer outside TRUSTED_PROXIES), and middleware.SecurityHeaders (X-Frame-Options: DENY, CSP frame-ancestors 'none'; base-uri 'self'; form-action 'self', Referrer-Policy, nosniff). The two frame headers are skipped for any /status/* or /api/status/* path so public status pages stay embeddable in an iframe. A custom-domain status host is exempt as a whole: the SPA handler serves nothing but that status page there, so it drops the two headers itself (see the status domains row below).
API Endpoints
Client SDK Ingestion
| Method | Endpoint | Auth | Purpose |
|---|---|---|---|
| POST | /api/report |
Client | Telemetry ingestion (gzipped) |
| POST | /api/otel/v1/traces |
Client | OTLP/HTTP trace ingestion |
| POST | /api/otel/v1/metrics |
Client | OTLP/HTTP metric ingestion |
| POST | /api/otel/v1/logs |
Client | OTLP/HTTP log ingestion |
| POST | /api/otel/v1development/profiles |
Client | OTLP/HTTP profile ingestion (development signal) |
Auth & Registration
| Method | Endpoint | Auth | Purpose |
|---|---|---|---|
| POST | /api/login |
None | Dashboard authentication |
| POST | /api/register |
None | New user registration |
| GET | /api/has-organizations |
None | Check if any orgs exist (self-hosted only) |
| POST | /api/forgot-password |
None | Request password reset email |
| GET | /api/password-reset/:token |
None | Validate reset token |
| POST | /api/password-reset/:token |
None | Reset password with token |
CLI Device Auth & OAuth (see the Authentication section above)
| Method | Endpoint | Auth | Purpose |
|---|---|---|---|
| POST | /api/auth/device/authorize |
None | Start device flow; returns device/user code + verification URL |
| POST | /api/auth/device/token |
None | Poll for the token (grant device_code); also accepts refresh_token. Shares a 60/min per-IP limiter with /api/auth/token; over it a device_code grant answers the RFC 8628 400 slow_down (RateLimitOAuthTokenPerIP peeks grant_type from the buffered body), so the shipped CLI backs off instead of aborting the login, while every other grant gets a plain 429 + Retry-After because slow_down is undefined for them and OAuth clients treat an unknown 400 as terminal. Body capped at 64KB by BufferAuthBody like the other anonymous auth routes |
| POST | /api/auth/token |
None | Token endpoint: device_code, refresh_token or authorization_code grant (JSON or form-encoded). Same shared limiter, grant-aware rejection and 64KB body cap as the device token route |
| POST | /api/auth/logout |
None | Revoke the presented refresh token's family (idempotent) |
| GET | /api/device |
App | Look up a user code for the approval screen |
| POST | /api/device/approve |
App | Approve a pending device authorization (tokens carry only the approving user's own role, so no write guard) |
| POST | /api/device/deny |
App | Deny a pending device authorization |
| POST | /api/oauth/register |
None | RFC 7591 dynamic client registration (rate-limited) |
| GET | /api/oauth/client |
App | Resolve a client_id to its display name (consent page) |
| POST | /api/oauth/approve |
App | Approve an authorization request; mints the code, returns the validated redirect |
| POST | /api/oauth/deny |
App | Deny an authorization request; returns the error redirect |
| GET | /.well-known/oauth-authorization-server |
None | RFC 8414 metadata (served at origin root, not /api) |
| GET | /.well-known/oauth-protected-resource |
None | RFC 9728 metadata (served at origin root, not /api) |
| GET/POST/DELETE | /mcp |
Bearer | Streamable HTTP MCP server (origin root), read tools only for every credential; 401s carry a WWW-Authenticate resource-metadata challenge |
| GET/POST/DELETE | /mcp/write |
Bearer (write scope) | Same server with mcpserver.Config.AllowWrite: adds the mutating tools and the operations playbook. Tokens without traceway:write get 403 with a scope challenge, and so do dashboard sessions (BearerIdentity.HasAgentWriteScope), which would otherwise skip the agent gate's opt-in and audit log; resource metadata at /.well-known/oauth-protected-resource/mcp/write |
Personal Access Tokens
| Method | Endpoint | Auth | Purpose |
|---|---|---|---|
| POST | /api/personal-access-tokens |
App | Create a PAT (returns the token once). scope is read (default, twp_) or write (tww_, expiresInDays 1-90 required) |
| GET | /api/personal-access-tokens |
App | List the current user's active PATs |
| DELETE | /api/personal-access-tokens/:id |
App | Revoke a PAT |
Projects
| Method | Endpoint | Auth | Purpose |
|---|---|---|---|
| GET | /api/projects |
App | List projects |
| POST | /api/projects |
App | Create project (optional organizationId body field targets another org; the handler checks the caller's org role in the target org and 403s for non-members/readonly; creation is org-scoped, so per-project overrides neither grant nor deny it) |
| POST | /api/projects/source-map-token |
App+Write | Generate source map upload token |
Dashboard
| Method | Endpoint | Auth | Purpose |
|---|---|---|---|
| GET | /api/dashboard |
App | Dashboard metrics |
| GET | /api/dashboard/overview |
App | Recent issues + top endpoints |
| POST | /api/stats |
App | Homepage stats |
Metrics
| Method | Endpoint | Auth | Purpose |
|---|---|---|---|
| GET | /api/metrics/application |
App | Application metrics |
| GET | /api/metrics/stats |
App | Stats metrics |
| GET | /api/metrics/server |
App | Server metrics |
| POST | /api/metrics/query |
App | Custom metric queries. Aggregations avg/min/max/sum/count/last plus rate, which differences every series (full tag set = identity) against its previous sample, drops resets, and sums the deltas per bucket as a per-second rate; the reported unit gains /s (By/s), a counter of seconds becomes the ratio 1, and on ClickHouse rate follows selectTable like every other aggregation: past 6h it reads the rollups through maxMerge(max_val) (a counter only grows, so a rollup bucket's max is its last sample), never metric_points_1d, with the chart bucket clamped to the rollup step and the lookback widened by it (rateSource). The traceway-otel-agent template relies on it (plus state filters, groupBy: server_name, and the per-source complement flag that the widget renderer applies as 1 − value); backfill.RunOtelAgentDashboardSources upgrades installed copies whose widgets still carry the original plain-average source. Bounded in metric_query.controller.go: 256KB body, max 50 queries and 20 tag filters each, range at most 731 days and to >= from (422), buckets widened so no series exceeds 2000 points (the effective intervalMinutes is returned at the top level of the response), at most 200 groups per query (the repositories return one extra so the result carries truncatedGroups: true), and a 30s deadline for the whole request that answers 504. The discovery endpoints validate their from/to the same way |
| GET | /api/metrics/discover |
App | Discover available metrics |
| GET | /api/metrics/discover/tags |
App | Discover metric tags |
| GET | /api/metrics/discover/instances |
App | Distinct server_name values in a range (dashboard instance filter) |
| PUT | /api/metrics/registry |
App+Write | Update metric registry entry |
Dashboards & Templates
Dashboards are org-owned JSON documents ({schemaVersion, widgets: [{id, title, widgetType, config}]}, widget ids are server-generated w_xxxxxxxx strings, array order = display order) applied to projects via project_dashboards. Dashboard mutations require org role above readonly (checked in-handler); project-scoped routes (list/star/reorder/populate) use the standard middleware chains; apply/unapply also check the effective role of each affected project. The old per-project widget-group tables are converted once at startup by backfill.RunDashboards (advisory-locked on PG) and retained for rollback.
| Method | Endpoint | Auth | Purpose |
|---|---|---|---|
| GET | /api/dashboards |
App | Dashboards applied to the project (tab order) |
| GET | /api/dashboards/library |
App | All dashboards across the user's orgs with applied project ids |
| POST | /api/dashboards |
App | Create in org (auto-applies to current project unless applyToProjectIds given) |
| GET | /api/dashboards/:id |
App | Meta + widgets (+ per-project isStarred, appliedProjectIds) |
| PUT | /api/dashboards/:id |
App | Update name/description and/or full definition (the as-code path) |
| DELETE | /api/dashboards/:id |
App | Delete everywhere (assignments + stars cascade) |
| PUT | /api/dashboards/:id/apply |
App | Set the full project assignment list |
| DELETE | /api/dashboards/:id/apply/:projectId |
App | Unassign from one project |
| POST | /api/dashboards/:id/copy |
App | Copy (also cross-org) with optional apply |
| PUT | /api/dashboards/reorder |
App+Write | Tab order for a project (explicit id order) |
| POST | /api/dashboards/:id/widgets |
App | Add widget |
| PUT | /api/dashboards/:id/widgets/reorder |
App | Reorder widgets (explicit id order) |
| PUT | /api/dashboards/:id/widgets/:wid |
App | Update widget |
| DELETE | /api/dashboards/:id/widgets/:wid |
App | Delete widget (+ its stars) |
| PUT | /api/dashboards/:id/widgets/:wid/star |
App+Write | Star/unstar for the project homepage |
| GET | /api/dashboards/starred |
App | Starred widgets with homepage layout |
| PUT | /api/starred-widgets/reorder |
App+Write | Reorder homepage starred widgets ({ids} = starred row ids) |
| PUT | /api/starred-widgets/:id |
App+Write | Update homepage layout (colSpan/size) |
| GET | /api/dashboards/:id/export |
App | Export one dashboard as JSON |
| GET | /api/dashboards/export?organizationId= |
App | Export the org bundle |
| POST | /api/dashboards/import |
App | Import doc/bundle (mode: create|upsert, upsert matches by name) |
| POST | /api/dashboards/import/grafana |
App | Convert a Grafana export (best-effort, returns warnings[]) |
| GET | /api/dashboard-templates |
App | Marketplace list/search (search, category params) |
| POST | /api/dashboard-templates/:key/install |
App | Copy a template into the org and apply |
| POST | /api/dashboards/populate-defaults |
App+Write | Install the framework-default template set for an empty project |
| GET | /api/metrics/discover/org |
App | Metric names across all org projects (command palette) |
Templates are DB rows seeded by migrations (traceway-otel-agent for the OTel host agent, golang for Go SDK apps, traceway-clickhouse/traceway-duckdb for the telemetry stores of a monitored Traceway instance; SQLite emits no store-specific metrics so it has no template); cloud can insert more rows without a release. The OTLP metric ingest allowlists per-resource identity and grouping tags (host.name, host.id, os.type, cloud.region, container.name, k8s.cluster.name, k8s.pod.name, k8s.node.name, postgresql.database.name, ...) in otelcontrollers/metric_converter.go for infrastructure views and custom widgets.
Endpoints
| Method | Endpoint | Auth | Purpose |
|---|---|---|---|
| POST | /api/endpoints |
App | List all endpoints |
| POST | /api/endpoints/grouped |
App | Endpoint aggregates (P50/P95/P99) |
| POST | /api/endpoints/endpoint |
App | Single endpoint details |
| POST | /api/endpoints/chart |
App | Stacked chart data |
| GET | /api/endpoints/slow |
App | Get slow endpoint threshold |
| POST | /api/endpoints/slow |
App+Write | Set slow endpoint threshold |
| POST | /api/endpoints/:endpointId |
App | Endpoint detail view |
Tasks
| Method | Endpoint | Auth | Purpose |
|---|---|---|---|
| POST | /api/tasks |
App | List all tasks |
| POST | /api/tasks/grouped |
App | Grouped by task name |
| POST | /api/tasks/task |
App | Single task details |
| POST | /api/tasks/:taskId |
App | Task detail view |
Exceptions
| Method | Endpoint | Auth | Purpose |
|---|---|---|---|
| POST | /api/exception-stack-traces |
App | Grouped exception list |
| POST | /api/exception-stack-traces/archive |
App+Write | Archive exceptions |
| POST | /api/exception-stack-traces/unarchive |
App+Write | Unarchive exceptions |
| POST | /api/exception-stack-traces/by-id/:exceptionId |
App | Single exception by ID |
| POST | /api/exception-stack-traces/:hash |
App | Exception by hash |
AI Traces & Conversations
One ai_traces_v2 row per LLM call (any OTLP span with gen_ai.* attributes). Each row carries a conversation_id resolved at ingest (gen_ai.conversation.id -> session.id span/resource attr -> trace id), tool_call_count/tool_names parsed from the completion payload (OpenAI tool_calls, Anthropic tool_use, OTel output messages, gen_ai.tool.* fallback), and flagged/flagged_terms from an ingest-time word-boundary scan of prompt+completion against per-project selected built-in language packs (projects.ai_flagged_languages, default ["en"]; packs live in backend/app/services/contentflag/terms/*.txt — en/de/es/fr/it/pt/sr; empty array = custom terms only) plus per-project custom terms (projects.ai_flagged_terms); both edited in the project settings AI tab and cached via the project cache. All three project-creation paths (Create, CreateWithOrganization, cmd/seed.go) must set AiFlaggedLanguages explicitly since lit inserts every struct field. tool_names/flagged_terms are stored comma-separated (values sanitized at ingest); the content matcher lives in backend/app/services/contentflag/. Conversation analytics exclude rows with an empty conversation_id; user analytics additionally require a non-empty user_id.
| Method | Endpoint | Auth | Purpose |
|---|---|---|---|
| POST | /api/ai-traces/grouped |
App | AI traces grouped by trace name |
| POST | /api/ai-traces/trace |
App | Calls for one trace name (?traceName=) |
| POST | /api/ai-traces/:traceId |
App | Single call detail + conversation blob |
| POST | /api/ai-conversations/grouped |
App | Conversations (GROUP BY conversation_id): turns, cost, tokens, tools, models, flagged; filters userId/model/toolName/flaggedOnly/search (search matches conversation id, user, model, tool names, and flagged terms; row-level filters are semi-joins on conversation_id so a match on any turn returns the whole conversation's aggregates); response also carries thresholds (range-wide P95 cost/turns for outlier highlighting) and facets (models, tools) |
| POST | /api/ai-conversations/conversation |
App | All turns of one conversation (id in body) ordered by recorded_at, each with its stored input/output payload (capped at 200 turns of payloads), plus stats |
| POST | /api/ai-users/grouped |
App | Per-user conversation analytics: conversation count, total calls, avg/min/median turns, avg cost per conversation, total cost, flagged conversation count |
Frontend routes: /ai-traces (tabs: Traces, Conversations, Users), /ai-traces/conversations/[conversationId] (chat timeline with tool calls rendered). Trace names conversations and users are shadowed by these static routes. Notification rule types ai_trace_cost (per call), ai_conversation_cost (24h cumulative per conversation), and ai_flagged_content (flagged term match, optional term filter) are event-driven; remember both notification_rule.repository.go copies list event rule types explicitly.
Organization Management
| Method | Endpoint | Auth | Purpose |
|---|---|---|---|
| POST | /api/organizations |
App | Create an organization owned by the caller; the recovery path for a user removed from their last organization (self-hosted: 422 once one exists; cloud: OrganizationLimitHook). Runs PostRegistrationHooks like register and SSO finish-setup |
| GET | /api/organizations/:orgId/settings |
Admin | Get org settings |
| PUT | /api/organizations/:orgId/settings |
Admin | Update org settings |
| PUT | /api/organizations/:orgId/agent-access |
Admin | {agentWriteEnabled}: allow agent write credentials to write in this org (settings GET returns the flag) |
| GET | /api/organizations/:orgId/agent-activity |
Admin | Paginated agent write log (who, credential, project, route, status) |
| GET | /api/organizations/:orgId/members |
Admin | List members |
| PUT | /api/organizations/:orgId/members/:userId |
Admin | Update member role |
| DELETE | /api/organizations/:orgId/members/:userId |
Admin | Remove member |
| GET | /api/organizations/:orgId/members/:userId/project-roles |
Admin | List member's per-project role overrides |
| PUT | /api/organizations/:orgId/members/:userId/project-roles/:projectId |
Admin | Set/clear a per-project role override |
Organization Overview (frontend routes /organization = Servers, /organization/issues, /organization/monitors, /organization/on-call, /organization/projects)
Org-wide read views that fan out over every project of the org, in organization_overview.controller.go. RequireOrganizationAccess (any org role) gates them; the servers/issues/monitors handlers deliberately skip Transactional because they run per-project telemetry queries and must not hold the single-connection SQLite main DB across them (they open their own short db.ExecuteTransaction for the project list first). The pure main-DB pages endpoint keeps Transactional. The frontend renders the organization routes in the same sidebar shell as project pages, with the sidebar switched to organization items (Servers, Issues, Monitors, On-Call, Projects under an "Organization" label; badges come from /overview/counts). The selected organization is the shared organizationContext in frontend/src/lib/state/organization-context.svelte.ts (the organizationId query param, else the first membership), and src/routes/organization/+layout.svelte owns the redirects: a single-project org into that project, legacy ?tab= links to the new paths, and frontend-only orgs from Servers/Monitors to Issues. Login auto-lands on /organization when the first org has more than one project (frontend/src/lib/utils/landing.ts). A single-project organization never shows the view: /organization redirects into that project and the switcher renders the org as a plain label, since the project's own pages already cover everything it would aggregate. When none of the org's projects is a backend project (isBackendFramework in projects.svelte.ts, i.e. every project is a browser or mobile framework), the Servers and Monitors sidebar items are hidden and /organization redirects to /organization/issues.
| Method | Endpoint | Auth | Purpose |
|---|---|---|---|
| GET | /api/organizations/:organizationId/overview/servers |
Member | One row per server_name per project over the last 30 min: CPU/memory/disk/network + a CPU sparkline, plus host/os/cloud/k8s metadata lifted from metric tags. Reads OTel hostmetrics (system.cpu.utilization state=idle inverted, system.memory.utilization state=used, system.filesystem.utilization state=used, system.network.io by direction) and falls back to the legacy SDK names (cpu.used_pcnt, mem.used/mem.total). A project that fails to read sets partial: true instead of failing the request |
| POST | /api/organizations/:organizationId/overview/issues |
Member | Paginated issues across all projects. Body mirrors /api/exception-stack-traces (fromDate, toDate, orderBy, search, searchType, pagination); every project contributes its top page*pageSize groups in the requested order and the page is a slice of their merged order, so it equals what one combined project would return. That per-project fetch is capped at 1000 rows (orgOverviewMaxIssueFetch): a deeper page answers 422 and totalPages is clamped so the footer never offers one; FindGrouped orders ties by exception_hash on every backend so the merge is stable across pages. Returns {data, pagination, partial} with 24h hourly trends attached; a project that fails to read sets partial: true like the servers endpoint |
| POST | /api/organizations/:organizationId/overview/pages |
Member | Paginated on-call pages across all projects: body {status, search, fromDate, toDate, pagination} (status active/open/acknowledged/resolved, search over subject + rule name, range on created_at); returns {data, pagination, openPagesCount} |
| GET | /api/organizations/:organizationId/overview/counts |
Member | openPagesCount + downMonitorsCount for the organization sidebar badges |
| POST | /api/organizations/:organizationId/overview/incidents |
Member | Paginated incidents across monitors and status pages: body {search, fromDate, toDate, pagination}; an incident matches when it overlaps the range (started_at <= to and unresolved or resolved_at >= from), search covers title, monitor, status page, and error message |
| GET | /api/organizations/:organizationId/overview/monitors |
Member | Every synthetic check in the org with 30-day aggregates; a project whose results fail to read sets partial: true like the servers endpoint |
Instance identity is the server_name tag, which comes from the OTLP service.name resource attribute; grouping by Kubernetes cluster keys off the k8s.cluster.name tag and the group selector only offers it when at least one instance carries one. Clicking a row opens /dashboards?projectId=&server=&preset=30m (plus dashboard= when the project has the traceway-otel-agent template applied); server scopes every widget query via scopeTagFilters in widget-grid/widget-renderer. Cluster-wide instrumentation manifests live in examples/kubernetes/, documented at docs/pages/learn/kubernetes.mdx.
Invitations
| Method | Endpoint | Auth | Purpose |
|---|---|---|---|
| POST | /api/organizations/:orgId/invitations |
Admin | Send invitation |
| GET | /api/organizations/:orgId/invitations |
Admin | List invitations |
| DELETE | /api/organizations/:orgId/invitations/:id |
Admin | Revoke invitation |
| GET | /api/invitations/:token |
None | Get invitation info |
| POST | /api/invitations/:token/accept |
None | Accept (new user) |
| POST | /api/invitations/:token/accept-existing |
App | Accept (existing user) |
On-Call (teams, schedules, escalation policies, pages)
Org-scoped entities in the main DB. Teams (teams/team_members) own projects one-to-one (project_teams, unique on project_id). Schedules (oncall_schedules) store PagerDuty-style calendar layers as a JSON definition document (rotations daily/weekly/custom, handoff time/day, time-of-day and day-of-week restrictions); one-off overrides are normalized rows (oncall_overrides). The pure resolution engine lives in backend/app/oncall/ (ResolveRange/ResolveAt, tz-aware calendar math, later layer wins, overrides trump all). Both apply the same stacking, so a schedule puts exactly one person on call at any instant: ResolveAt is ResolveRange over a single instant, and paging a whole schedule stack (waking the person an override was meant to relieve) is the bug it exists to prevent. Escalation policies (escalation_policies) hold JSON steps (targets schedule/user/team/channel + delayMinutes, repeatCount); rules page on-call via the escalation notification channel type (config {"policyId"}), special-cased in notifications/dispatch.go through RegisterPageOpener (wired in cmd/run.go). A fired rule opens a pages row (dedup key ruleId|dedupToken with a partial unique index while unresolved; rules without a dedup token dedup at rule level; refires bump event_count and never reset the escalation clock). The escalator worker (oncall/escalator.go, ONCALL_POLL_SECONDS) claims due pages in a transaction (pg advisory lock 824737002 for multi-instance), inserts page_notifications rows, resolves targets to users, delivers via each user's user_contact_methods (email/slack/pushover/telegram adapter configs; account-email fallback always), then escalates level by level until ack/resolve or exhaustion. RequireOrganizationAccess middleware (any org role) gates member-level reads; mutations are org-admin; page acknowledge deliberately requires no write access.
| Method | Endpoint | Auth | Purpose |
|---|---|---|---|
| GET/POST | /api/organizations/:organizationId/teams |
Member / Admin | List (with members+projects) / create |
| PUT/DELETE | /api/organizations/:organizationId/teams/:teamId |
Admin | Update / delete |
| PUT | .../teams/:teamId/members, .../teams/:teamId/projects |
Admin | Replace ordered members / owned projects |
| GET/POST | /api/organizations/:organizationId/schedules |
Member / Admin | List / create |
| GET/PUT/DELETE | .../schedules/:scheduleId |
Member / Admin / Admin | Detail+overrides / whole-document update / delete |
| GET | .../schedules/:scheduleId/timeline?from=&to= |
Member | Rendered per-layer + final shifts (max 62 days) |
| POST/DELETE | .../schedules/:scheduleId/overrides(/:overrideId) |
Member | Create (any member, max 30d) / delete (creator, covered user, or admin) |
| GET | /api/organizations/:organizationId/oncall/now |
Member | Overview: per team/schedule current + next on-call |
| GET | /api/oncall/current?projectId= |
App | Owning team + current on-call for a project (issue page) |
| GET | /api/escalation-policies |
App | Policies of the project's org (channel dialog picker) |
| GET/POST | /api/organizations/:organizationId/escalation-policies |
Member / Admin | List / create |
| PUT/DELETE | .../escalation-policies/:id |
Admin | Update / delete (422 while referenced by a channel) |
| POST | /api/pages |
App | List (POST-body: status open/acknowledged/resolved/active + pagination) |
| GET | /api/pages/:id |
App | Detail + delivery log |
| POST | /api/pages/:id/acknowledge |
App (no write gate) | open -> acknowledged, stops escalation; 409 if not open |
| POST | /api/pages/:id/resolve |
App | open/acknowledged -> resolved; 409 if already resolved |
| GET | /api/pages/open-count |
App | Sidebar badge count |
| GET/POST | /api/contact-methods |
App | Own contact methods (self-scoped). Types: email, slack, pushover, telegram, sms. SMS requires Twilio (TWILIO_ACCOUNT_SID + TWILIO_AUTH_TOKEN + one of TWILIO_FROM_NUMBER / TWILIO_MESSAGING_SERVICE_SID); without them SMS is not offered at all — the list response carries smsEnabled: false so the type picker hides it; create, re-point, resend-code and test answer 422; the escalator drops existing sms methods before its "no methods left" check so those users fall back to the account email; and the adapter errors instead of reporting a delivery nobody received. Disabling and deleting a leftover sms method stay available. Creating/re-pointing an sms method starts code verification; unverified numbers are never paged |
| PUT/DELETE | /api/contact-methods/:id |
App | Update (incl. enabled toggle) / delete |
| POST | /api/contact-methods/:id/test |
App | Send a canned test through one method (422 for unverified sms) |
| POST | /api/contact-methods/:id/verify |
App (rate-limited) | Confirm the 6-digit SMS code (hashed at rest, 10-min expiry, 5-attempt cap). Deliberately not under middleware.Transactional: a wrong code answers 422, which would roll the consumed attempt back, so the handler manages its own transactions (nesting one under the middleware would also deadlock the single-connection SQLite main DB) |
| POST | /api/contact-methods/:id/resend-code |
App (rate-limited) | Re-issue the verification code |
| GET/PUT | /api/user-notification-rules |
App | Per-user notification-rule chains {high: [{contactMethodId, delayMinutes}], low: [...]} (PagerDuty-style: the page's urgency picks the chain; steps are enqueued at claim time as scheduled outbox deliveries and cancelled on ack; no chain = all enabled+verified methods immediately). Escalation policies carry urgency: auto|high|low in their definition (auto: critical -> high); pages store the resolved urgency |
| GET/POST | /api/ack/:token |
None (rate-limited) | Tokenized no-login acknowledge: per-delivery twk_ tokens (SHA-256-hashed on page_notifications), GET = read-only summary (scanner-safe), POST = idempotent ack recorded as acknowledged_via='link' attributed to the delivery's recipient; 404 after resolve. Frontend page: /ack/[token] |
Monitors (user-facing name for synthetic uptime checks; engine in backend/app/synthetics, frontend under /monitors, see the SYNTHETICS_* env block)
Checks (synthetic_checks, main DB) are http/tcp/browser probes with per-check interval, timeout, and a consecutive-failure threshold (flap damping). Runs flow through the check_runs queue (outbox-style guarded claims, terminal rows deleted, expired queued runs recorded as missed); results are telemetry (check_results, all three backends, pruned by the SQLite retention worker / 90d CH TTL). State transitions open/resolve check_incidents and feed the event-driven notification rule type check_down (recovery auto-resolves the page a rule-scoped dedup key opened via oncall.AutoResolveByDedupKey; recovery never dispatches to escalation channels). The recovery notice carries Recovered on NotificationMessage: dispatch leaves its info severity alone instead of applying the rule's, and the Slack adapter colours it green. The notify hook runs post-commit with no ambient tx (SQLite single-connection). Remember: notification_rule.repository.go (both copies) lists event rule types explicitly, and check_down is one of them.
Incidents, incident updates & post-mortems. check_incidents covers both auto and manual incidents: auto rows carry check_id+project_id (nullable now), hand-recorded ones carry only status_page_id (org admins record them from a status page for outages monitors missed; they never affect uptime numbers). All incidents can be given a public title and statuspage.io-style timeline updates (incident_updates: status investigating/identified/monitoring/update/resolved + message; auto open/resolve seeds empty-message investigating/resolved updates in the same tx in synthetics/result.go, and the probe error_message stays internal, never in the public payload). A resolved update closes a manual incident; auto lifecycle stays owned by ProcessOutcome (manual resolve/time-edit/delete of auto incidents answer 422). The public /api/status/:slug payload gained pastIncidents (90d union of the page's checks' auto incidents + its manual ones, titles defaulting to " is down", nested updates, cap 30, no ids/error messages); incident mutations invalidate every status-page cache slug of the org. Post-mortems (post_mortems, project-scoped: project_id NOT-NULL-in-practice with ON DELETE CASCADE, backfilled by migration from the linked incident's project else the org's oldest project) are internal markdown documents, never public: optional 1:1 incident link (incident_id unique where set, ON DELETE SET NULL so documents survive monitor/incident deletion; linkable incidents are the project's own autos plus the org's manual status-page incidents — cross-project autos answer 422), JSON-array tags (models.StringSlice), LIKE search over title/content/tags, and an append-only edit log (post_mortem_events: action created/updated + JSON changes field list, recorded only when something actually changed; served as the Activity panel). Routes are project-scoped (/api/post-mortems*, standard RequireProjectAccess/RequireWriteAccess chains, org resolved from the project for incident validation); incident mutations are org-admin since they are public content. Frontend: Post-Mortems tab on /monitors (all members; creation is a title-only dialog that creates the document immediately and opens the editor page), full-page WYSIWYG editor at /monitors/post-mortems/[id] (@milkdown/crepe, dynamically imported, app font stack overrides the theme's serif), paginated incident management page at /monitors/status-pages/[pageId]/incidents (which owns the Record Incident dialog).
| Method | Endpoint | Auth | Purpose |
|---|---|---|---|
| GET/POST | /api/synthetics/checks |
App / App+Write | List / create checks (422 validation incl. browser-mode + cloud gates) |
| GET/PUT/DELETE | /api/synthetics/checks/:id |
App / +Write / +Write | Detail+incidents / update (type immutable; pausing drops queued runs) / delete (auto-resolves the check's open pages) |
| POST | /api/synthetics/checks/:id/run |
App+Write | Run now (422 if a run is already queued; OnCommit wake) |
| POST | /api/synthetics/overview |
App | Checks + per-range aggregates (uptime, avg latency) |
| POST | /api/synthetics/checks/:id/results |
App | Paginated run history (telemetry read, no Transactional; optional status filter up/down/missed + fromDate/toDate) |
| POST | /api/synthetics/checks/:id/series |
App | Bucketed uptime/latency series (telemetry read) |
| GET | /api/synthetics/screenshot?key= |
App | Streams a failure screenshot; key prefix-checked against the project |
| GET | /api/synthetics/output?key= |
App | Streams a failed browser run's stored Playwright output (.log keys only, same prefix check); "logs" link in the run history UI |
| GET | /api/synthetics/open-count |
App | Down-check count for the sidebar badge |
| POST | /api/runners/poll |
Runner (shared secret) | Instance-wide long-poll claim of browser runs (25s hold on the wake channel, NO Transactional, excluded from tracewaygin self-monitoring); self-registers the runner's liveness row |
| POST | /api/runners/results/:runId |
Runner (shared secret) | Report an outcome (MaxBytesReader 6MB incl. screenshotBase64; idempotent: lost/missing claim = 200 no-op; claims keyed "runner:") |
| GET/POST/PUT/DELETE | /api/organizations/:organizationId/status-pages(/:id) |
Member / Admin | Status page CRUD (slug [a-z0-9-]{3,60} unique, checkIds validated against the org; branding fields description, customDomain unique hostname) |
| POST | /api/organizations/:organizationId/status-pages/:id/logo |
Admin | Upload a PNG/JPEG logo (raw body, 1MB cap, sniffed content type; SVG rejected as scriptable) stored via storage.Store under statuspages/<id>/logo |
| GET | /api/status/:slug/logo |
None (rate-limited) | Streams a public page's logo |
| GET | / on a custom domain |
None | Status domains (CNAMEd vanity hosts, TLS at the operator's proxy). The SPA handler in cmd/run.go looks the Host up with controllers.StatusPageForHost (public pages only, never the APP_BASE_URL host) and, on a match, serves index.html with a traceway-status-slug meta tag plus the page's title and description. The frontend reads it in $lib/status-domain.ts and the root layout renders only StatusPageView at /, whatever the auth state. Every other page path on that host 302s to /, so the dashboard never loads on a customer's domain; static assets and /api still work. Saving a custom domain equal to APP_BASE_URL's host or the admin's current host answers 422 |
| GET | /api/status/:slug |
None (rate-limited) | Public status payload: per-check status + 90 daily uptime buckets + incidents + pastIncidents (date-grouped titles with timeline updates); cached ~30s per slug; private/unknown slugs are both 404; latency values and internal error messages stripped. Frontend page: /status/[slug] (in isPublicPath) |
| GET | /api/organizations/:organizationId/incidents |
Member | Org incidents (90d, joined check/status-page names, updatesCount, postMortemId) |
| GET | /api/organizations/:organizationId/incidents/:incidentId/updates |
Member | One incident + its timeline updates |
| GET | /api/organizations/:organizationId/status-pages/:id/incidents |
Member | Paginated incident history of one status page (auto incidents of its checks + its manual ones, no time window; standard pagination envelope + statusPage meta). Frontend page: /monitors/status-pages/[pageId]/incidents |
| POST | /api/organizations/:organizationId/status-pages/:id/incidents |
Admin | Record a manual incident on a status page (title required, optional first update message, backdatable, optional resolvedAt) |
| PUT/DELETE | /api/organizations/:organizationId/incidents/:incidentId |
Admin | Edit title (any incident) / times (manual only, empty resolvedAt reopens; 422 for auto) / delete (manual only, 422 for auto) |
| POST/DELETE | /api/organizations/:organizationId/incidents/:incidentId/updates(/:updateId) |
Admin | Post / delete a public timeline update (a resolved update also closes an unresolved manual incident) |
| GET/POST | /api/post-mortems |
App / App+Write | Paginated list for the project (query params search/page/pageSize + repeatable tag params ANDed together) / create in the project (422 on duplicate or cross-project incident link) |
| GET/PUT/DELETE | /api/post-mortems/:id |
App / +Write / +Write | Full document with author names (contentMd) / update (records a post_mortem_events diff row; no-op saves change nothing; an optional expectedUpdatedAt answers 409 when the document changed since that read; the row is locked first (LockForUpdate) so concurrent Postgres updates cannot both pass the check. The CLI and MCP send the revision the caller's edit is based on, required when they replace the body (--expected-updated-at / expected_updated_at), and fall back to their own fresh read for title, tag and incident-only edits) / delete |
| GET | /api/post-mortems/:id/activity |
App | Edit log for the Activity panel (who created/edited what, joined user names) |
SLOs (service level objectives; worker backend/app/slo, controller slo.controller.go, telemetry table slo_buckets, main-DB tables slos and notification_rule_states; docs docs/pages/learn/slos.mdx)
| Method | Endpoint | Auth | Purpose |
|---|---|---|---|
| GET/POST | /api/slos |
App / App+Write | List (snapshot columns included, disabled ones too) / create. sliKind is endpoint, span, task, monitor, metric or session with a per-kind sliConfig (validated and canonicalized in validateSLIDefinition), targetPercent in (0,100), windowDays 7, 30 or 90, bucketMinutes only for metric (1, 5, 15 or 60; other kinds are forced to 1). Create and update call OnCommit(slo.Wake) |
| GET/PUT/DELETE | /api/slos/:id |
App / +Write / +Write | {slo, rules} (the project rules of type slo_* whose config names it) / update (a change of kind, config or slice length resets the snapshot, and the worker deletes the old buckets on its next tick; target and window changes keep the history) / delete (auto-resolves pages of referencing rules through the slo:<id> dedup token, deletes those rules, deletes the buckets post-commit) |
| POST | /api/slos/:id/series |
App | {buckets: [{bucket, good, total}]} re-bucketed to intervalMinutes for {fromDate, toDate} (max 91 days); telemetry read, no Transactional, scoped by project through the bucket key |
| POST | /api/slos/preview |
App | {good, total, sli, from, to} of a definition over the last 24h straight from raw telemetry (the sheet's Preview button) |
| GET | /api/slos/breached-count |
App | Sidebar badge |
Logs
| Method | Endpoint | Auth | Purpose |
|---|---|---|---|
| POST | /api/logs |
App | List/search logs (filters: severity, service, trace, body) |
Source Maps
| Method | Endpoint | Auth | Purpose |
|---|---|---|---|
| POST | /api/sourcemaps/upload |
SourceMap | Upload source map file |
Data Ingestion Flow (/api/report)
- Gzip Middleware: Decompresses request body (SDK sends gzipped data)
- Auth Middleware: Validates
Authorization: Bearer <project_token> - Parse Frame: JSON decode into
models.Frame(transactions, exceptions, metrics) - Batch Insert: Repository methods insert batches into ClickHouse
// backend/app/controllers/clientcontrollers/report.go
func (c *ReportController) Report(ctx *gin.Context) {
var frame models.Frame
ctx.BindJSON(&frame)
// Insert each data type
transactionRepo.BatchInsert(frame.Transactions)
exceptionRepo.BatchInsert(frame.Exceptions)
metricRepo.BatchInsert(frame.Metrics)
}
Database Schema
Tables (ClickHouse)
| Table | Purpose | Partitioning |
|---|---|---|
transactions |
HTTP request metadata | Monthly (toYYYYMM(timestamp)) |
spans_v2, endpoints_v2, tasks_v2, ai_traces_v2, exceptions_v2 |
Traces, see "Trace Storage (V2 tables)" below | Daily (toYYYYMMDD(recorded_at)) |
spans, endpoints, tasks, ai_traces, exception_stack_traces |
The tables V2 replaced. History only, read by the move-over | Monthly |
metric_records |
Time-series system metrics | Monthly |
archived_exceptions |
Archived/resolved exceptions | None |
log_records |
OTel logs with 3 attribute maps (resource/scope/log) | Daily (toDate(timestamp)) |
Data retention/TTLs are documented separately under Data Retention below.
Tables (PostgreSQL)
| Table | Purpose |
|---|---|
users |
User accounts with email/password |
organizations |
Multi-tenant organizations |
organization_users |
Junction table linking users to organizations with roles |
projects |
Project config + tokens, linked to organizations |
project_user_roles |
Per-project role overrides (user/readonly) for org members |
invitations |
Team invitations with token, role, expiry |
source_maps |
Uploaded source map files (project, version, storage key) |
metric_registry |
Custom metric definitions (type, unit, description) |
dashboards |
Org-owned dashboards (name, JSONB definition with widgets, template provenance) |
project_dashboards |
Which projects show a dashboard, and tab order |
dashboard_templates |
Marketplace templates (key, category, definition), seeded by migrations |
starred_dashboard_widgets |
Homepage layout per project (dashboard id + widget id, position, col_span, size) |
synthetic_checks |
Synthetic check config + current state (status, consecutive_failures, next_run_at) |
check_runs |
Synthetic run queue (queued/claimed only; terminal rows deleted, telemetry is the record) |
check_incidents |
Incident spans: auto (nullable check_id+project_id) and manual (nullable status_page_id), plus public title (feeds status pages) |
incident_updates |
Public timeline updates on incidents (status + message, seeded on auto open/resolve) |
post_mortems |
Internal markdown post-mortems (project-scoped via project_id ON DELETE CASCADE, tags JSON array, optional unique incident link with ON DELETE SET NULL) |
post_mortem_events |
Append-only post-mortem edit log (action, JSON changes list, user, ON DELETE CASCADE) |
synthetic_runners |
Liveness registry for self-registered runner fleet (unique name, version, first/last_seen_at; no credentials, no org scoping) |
status_pages |
Public uptime pages (org-scoped slug + selected check ids + branding: description, logo_key, unique custom_domain) |
widget_groups / widget_group_widgets / starred_widgets |
Legacy pre-dashboards tables, retained read-only for rollback until a follow-up drop |
ClickHouse vs PostgreSQL Decision Guide
- PostgreSQL: Relational/config data needing ACID, frequent updates, JOINs, low volume (users, organizations, projects, invitations, widgets, source maps, metric registry)
- ClickHouse: High-volume append-only telemetry with time-series aggregations, batch inserts only (transactions, exceptions, metric points, spans, tasks, sessions)
- Rule of thumb: "Will this data be updated after creation?" → PostgreSQL. "Is this immutable, time-stamped, high-volume data queried with aggregations?" → ClickHouse.
SQLite Dual-Database Architecture (self-hosted mode)
In SQLite mode (DB_TYPE=sqlite), the backend uses two separate SQLite databases mirroring the PostgreSQL/ClickHouse split:
| Database | Variable | File | Purpose | Transactions |
|---|---|---|---|---|
| Main DB | db.DB |
traceway.db |
PostgreSQL replacement — relational/config data | Yes (middleware.Transactional, db.ExecuteTransaction) |
| Telemetry DB | db.TelemetryDB |
traceway_telemetry.db |
ClickHouse replacement — append-only telemetry | No — direct inserts without transactions |
Both embedded builds (SQLite and DuckDB telemetry) are single-instance: the databases are local files, and the in-memory project cache only learns about another process's writes through the Postgres LISTEN/NOTIFY listener, which exists only under transactional_pg. Running more than one backend instance is supported only on the transactional_pg telemetry_ch build (docs: /server/minimal#running-more-than-one-instance).
Main DB tables (db.DB — transactional, uses lit with *sql.Tx):
users,organizations,organization_users,projects,invitationssource_maps,metric_registry,dashboards,project_dashboards,dashboard_templates,starred_dashboard_widgets(plus the legacywidget_groups/widget_group_widgets/starred_widgets)notification_channels,notification_rulessynthetic_checks,check_runs,check_incidents,incident_updates,post_mortems,post_mortem_events,synthetic_runners,status_pages
Telemetry DB tables (db.TelemetryDB — non-transactional, uses lit with db.TelemetryDB directly):
endpoints_v2,tasks_v2,ai_traces_v2,exceptions_v2,spans_v2,metric_pointssession_recordings,archived_exceptions,slow_endpoints,fired_notifications,check_resultsendpoints,tasks,ai_traces,exception_stack_traces,spans: the tables V2 replaced, kept as history for the move-over
How to access each database in repository code:
// Main DB (PostgreSQL replacement) — use transactions via middleware or ExecuteTransaction
// Repositories receive *sql.Tx from middleware.GetTx(ctx) or db.ExecuteTransaction
user, err := lit.SelectSingleNamed[models.User](tx, "SELECT ... FROM users WHERE ...", lit.P{...})
// Telemetry DB (ClickHouse replacement) — use db.TelemetryDB directly, no transactions
results, err := lit.SelectNamed[endpointRow](db.TelemetryDB, "SELECT ... FROM endpoints WHERE ...", lit.P{...})
// Telemetry inserts — no transaction wrapping
for _, item := range items {
row := modelToRow(item)
lit.InsertExistingUuid(db.TelemetryDB, &row)
}
Migrations are split into two directories:
backend/app/migrations/sqlite/— runs ondb.DB(main)backend/app/migrations/sqlite_telemetry/— runs ondb.TelemetryDB(telemetry)
SQLite-specific type helpers (backend/app/repositories/telemetry/sqlitetypes/):
SQLiteTime— implementssql.Scanner/driver.Valuerfortime.Time↔ SQLite TEXTSQLiteJSONMap— implementssql.Scanner/driver.Valuerformap[string]string↔ SQLite JSON TEXT- Row types (e.g.,
endpointRow,taskRow) wrap domain models with these types for lit compatibility
DuckDB Telemetry Backend (self-hosted, opt-in)
Built with -tags telemetry_duckdb (CGO_ENABLED=1 required), this is an alternative telemetry store for the same DB_TYPE=sqlite deployment: the main DB stays SQLite (db.DB, relational/config), while the telemetry DB becomes DuckDB (db.TelemetryDB, columnar). It exists because DuckDB's columnar engine is dramatically faster on the analytics/aggregation reads the dashboard issues — at 10M rows it clears read-probe thresholds that SQLite times out on. Backends are selected on two build-tag axes: telemetry_ch / telemetry_duckdb / (none = SQLite telemetry) for the telemetry store and transactional_pg / (none = SQLite main) for the relational store. Only three combinations are supported — (no tags) dual SQLite, telemetry_duckdb, and transactional_pg telemetry_ch — enforced by compile-time guard files in backend/app/db/ (stale pgch/duckdb/oltp_* tags also fail with a rename message). Repositories are organized on the same two axes: telemetry repositories live in per-backend packages backend/app/repositories/telemetry/{clickhouse,sqlite,duckdb}/ and transactional (relational) repositories in backend/app/repositories/transactional/{pg,sqlite}/, each re-exported as singletons through tag-guarded facade files at the axis package root (telemetry/telemetry_ch.go etc., transactional/transactional_pg.go etc.). Consumers import the facade packages — telemetry.SpanRepository, transactional.UserRepository — never a backend package directly. Helpers shared by all telemetry backends are in telemetry/shared/, the SQLite scan/value types shared by the sqlite+duckdb backends are in telemetry/sqlitetypes/, and helpers/types shared by the transactional backends (auth-token hashing/time formats, facade-crossing structs) are in transactional/shared/. The transactional/pg and transactional/sqlite implementations are intentionally kept dialect-neutral (lit :name queries rendered per db.Driver), enforced byte-for-byte by transactional/parity_test.go. Running Postgres requires the transactional_pg build: the default build's migration runner applies SQLite-dialect migrations unconditionally, so DB_TYPE=postgres without the tag is not a supported combination.
- Driver:
github.com/duckdb/duckdb-go/v2(the official driver; marcboeker/go-duckdb is deprecated). Bundles prebuilt static libs for glibc only — not musl/Alpine, so the image uses Debian (Dockerfile.duckdb). - Opened in
backend/app/db/db_telemetry_duckdb.go: telemetry path is the SQLite path with.dbswapped for_telemetry.duckdb. By default DuckDB auto-tunes to the host;DUCKDB_MEMORY_LIMIT/DUCKDB_THREADS/DUCKDB_CHECKPOINT_THRESHOLD(passed through as DSN config options) let operators cap memory/threads so a memory-capped container doesn't read the host's RAM and OOM-kill the backend, and raise the WAL checkpoint threshold (default 16MB) so sustained Appender ingest isn't stalled by frequent checkpoints.preserve_insertion_order=falseis always set — telemetry reads all have explicit ORDER BY, and dropping the guarantee lets DuckDB parallelize bulk loads and large scans with less memory. The read pool is bounded (SetMaxOpenConns(duckDBMaxReadConns)) since each DuckDB connection can use all threads + its own query memory; Appender writes use their ownDuckDBConnector.Connect()connections and bypass that cap. Exposesdb.DuckDBConnector(needed for the Appender). - Writes use the Appender API, not
INSERT(duckdb.NewAppenderFromConn(conn, "", table)→AppendRow(...)→Close()flushes). Upserts still go throughExecContextwithON CONFLICT. The Appender rejects typed*stringfor nullable VARCHAR — usenullableString()inbackend/app/repositories/telemetry/duckdb/helpers.go(returns untypednilor the dereferenced value). - Write-path observability: a row the Appender rejects is dropped rather than failing the whole frame (the SQLite backend 500s instead), so a poison row cannot wedge the SDK's retry loop. Every drop increments a per-table counter (
db.RecordTelemetryRowDropped) and fires a rate-limited (1/min per table)traceway.CaptureException; Appender flush/connect failures still propagate to the request (500, SDK retries) and increment an insert-failure counter.GET /api/health/deep(operator endpoint: requires the HEALTH_DEEP_TOKEN bearer secret, unset disables it; all telemetry backends) exposestelemetryBackend,droppedRowsper table,droppedRowsTotal,insertFailures,ingestRejected(requests turned away by the ingest admission gate), and on DuckDB anengineobject (db/WAL file bytes,duckdb_memory()usage, read-pool in-use/wait stats) alongside its existing ClickHouse fields; it 503s only when the configured telemetry backend is ClickHouse and CH is unreachable (the embedded backends answer 200 withchReachable:false). The benchmark loadgen polls it before/after every ramp step and fails any step whose drop delta is nonzero; read-probe fills record cumulativedroppedRowsper fill level. WhenMONITORING_TRACEWAY_URLis set,monitoring.StartTelemetryDBReporteralso emitstraceway.duckdb.*metrics every 10s:rows_dropped.delta,insert_failures.delta,db_size_mb,wal_size_mb,memory_used_mb,read_pool.in_use,read_pool.wait_count.delta,read_pool.wait_ms.delta. The hourly retention worker issues aCHECKPOINTafter its deletes so retention actually reclaims disk (DuckDB otherwise defers reclamation to the WAL checkpoint threshold). litplaceholders:db.Driverstayslit.SQLite, which emits?— DuckDB accepts these, so no separate driver was needed for reads.- Migrations:
backend/app/migrations/duckdb_telemetry/(mirrorssqlite_telemetry/table-for-table; integer columns areBIGINT, JSON isVARCHAR, no secondary indexes since it's columnar, with one exception:0005_create_v2_tablesaddsidx_spans_v2_trace_keyonspans_v2(trace_key)for the span graph lookup. While that index existsALTER TABLE spans_v2 ADD COLUMNstill works, butDROP COLUMNandALTER COLUMN ... TYPEfail with a dependency error, so a migration that needs either must drop the index first). - Dialect gotchas vs SQLite (the read queries differ): native
quantile_cont(col, p)for P50/P95/P99 instead of fetch-and-sort;strftime('%s',col)→epoch(col); time bucketing viatime_bucket(to_seconds(N), col, TIMESTAMP '1970-01-01')— the explicit epoch origin is required because DuckDB anchors sub-day buckets at 2000-01-03 by default, which would misalign chart buckets against the SQLite backend's epoch-floored buckets for any interval that doesn't evenly divide a day;json_extract→json_extract_string;json_each→LATERAL unnest(json_keys(x)); strict GROUP BY needsANY_VALUE/arg_max;SUMreturns HUGEINT (CAST to BIGINT);CAST(.. AS REAL)→CAST(.. AS DOUBLE). - Tests:
backend/app/repositories/telemetry/testhelper_duckdb_test.go(taggedtelemetry_duckdb) providessetupTestDBso the entire existing telemetry test suite runs against an in-memory DuckDB.
Trace Storage (V2 tables)
A trace is a trace and a span is a span. Five tables hold traces on every backend: spans_v2, endpoints_v2, tasks_v2, ai_traces_v2, exceptions_v2 (migrations ch/0085-0089, one CREATE TABLE each; sqlite_telemetry/0025_create_v2_tables; duckdb_telemetry/0005_create_v2_tables). They sit next to the tables they replaced (spans, endpoints, tasks, ai_traces, exception_stack_traces), which nothing reads or writes any more except the move-over and retention. migrations/v2_tables_test.go holds the V2 migrations to CREATE TABLE/CREATE INDEX on _v2 tables only.
- Ids. Every row carries the ids its span arrived with, as lowercase hex:
trace_id(32 characters),span_id(16 for OTel, 32 for a native run or span),parent_span_id(''for a root). Custom linked-trace identity is retired. Startup cleanup migrations removelinked_trace_idand its indexes from all four V2 entity/exception tables; OTLP span links remain intact. Entities and exceptions keep a UUIDidfor URLs and notifications. For OTel rows it isotelOccurrenceID=uuid.NewSHA1(projectId, traceId + spanId), so a retried export produces the same id. Nothing about identity lives in the attributes JSON, and ingest adds notraceway.otel.*keys. - Identity schema cleanup. Forward migrations
ch/0091–0099,sqlite_telemetry/0026,duckdb_telemetry/0006–0007replacesessions.distributed_trace_idwith canonical non-null stringtrace_id(empty = no association), remove V2linked_trace_idcolumns/indexes, and drop the unusedprofiles.distributed_trace_idcolumn/index. Profiles keep their existingtrace_id/span_id. ClickHouse synchronously materializes session IDs before dropping the source column. SQLite/DuckDB commit each migration and its version atomically; DuckDB adds the session non-null constraint in the next transaction. Stop old backends before the first upgraded start and restore a pre-upgrade database for rollback: old session/profile SQL is incompatible. The five old trace tables remain for the move-over. - Ingest (
otelcontrollers/trace_converter.go) validates and indexes spans once per resource, then returns canonical spans, entity projections, exceptions and the invalid-ID count together. Each span is promoted when its own kind and attributes make it an entry point, and each exception event becomes anexceptions_v2row with the span's trace id and span id. Canonical span attributes are rendered directly from the OTLP payload at storage time. Trace and log JSON IDs are normalized at the decoder boundary (otelcontrollers/otlp_json.go); converters consume decoded bytes without a UUID/base64 repair roundtrip. No owner resolution, no caches, no reads.trace_typeon an exception is stored only when that span itself was promoted (native rows say it themselves). Every entity is recorded at its span's start, OTel tasks included. Spans are inserted before the rows promoted from them, and a transient storage error answers 503 +Retry-After(abortIngestStorage,telemetry.IsTransientStorageError) so OTLP exporters retry. Retries are append-only; the span reader deduplicates per span. - Entry points. An HTTP SERVER span becomes an endpoint unless a local span above it already is one (
isEntryPoint), which is what keeps a SERVER span nested under another (Next.js under the HTTP instrumentation) from counting as a second request. This is the one check that looks beyond the span, and only inside itsResourceSpansblock. The walk up the parent chain stops at a remote parent per the OTLP span flags (0x100has-is-remote,0x200is-remote) and at a CLIENT or PRODUCER span. A missing parent has an unknown kind even when flags mark it local, so it cannot suppress an HTTP entry point. Separately exported nested HTTP spans can create extra projections. - Native
/api/reportrows go to the same tables (clientmodels.ClientTrace.RootSpan,ClientSpan.ToSpan): a run is stored as the root span of a trace,span_id= the run's id,trace_id= the run's id. Its spans hang under it, and a native exception sits on the run's span. The report wire format is unchanged: an exception'straceIdnames its owning run,distributedTraceIdis ignored on runs, exceptions and sessions until the JS SDK supports canonical trace propagation, and native run/span IDs remain UUIDs. spans_v2keeps the lossless payload of accepted OTLP spans (resource_pb/scope_pb/span_pbon ClickHouse,otlpon SQLite and DuckDB;shared.AssembleOtelPayloadjoins them back) next to the readable columns, built once byOtelSpanRow.NestedValues:span_attributesis a string map (aMap(LowCardinality(String), String)on ClickHouse, a JSON object elsewhere) filled byshared.StringAttributesthe way the OpenTelemetry Collector's ClickHouse exporter does it, andresource,scope,events,linksare JSON text. A span that did not arrive as OTLP gets a payload synthesized from its available fields (NewOtelSpanRow), including string attributes and service/scope metadata. Native UUID span IDs remain in the API; their exported span/parent IDs use a stable eight-byte hash mapping with original IDs carried intraceway.native.*attributes. Native synthesis cannot recover metadata never sent by the SDK. On ClickHouserecorded_atisDateTime64(6)on endpoint/task/exception tables andDateTime64(3)on AI traces (the waterfall's origin is the entity'srecordedAt) and a plain partition time onspans_v2.- Reads. A waterfall is
spans_v2by project and trace, the subtree under the entity'sspan_id:telemetry.SpanRepository.FindGraph(ctx, shared.NewSpanLookup(projectId, traceId, spanId, recordedAt)), answered byshared.FindOtelGraphs(topology first, attributes second). A limit never fails the page: the result carriesspanGraphStatus(complete/partial/unavailableplus reasons) and spans may haveattributesOmitted. The subtree includes spans that are entities of their own, such as a CONSUMER span started inside the request. - Distributed trace card (
controllers/distributed_trace.controller.go,POST /api/distributed-traces/:traceId, hex or dashed).FindByTraceIdson the four entity and exception repositories matches onlytrace_id IN (...)across projects the user can open. Every returned node belongs to the requested trace; W3C propagation makes browser and backend share that identity. Nodes are nested by one read of the trace's topology, without attributes (SpanRepository.FindTraceParents): each node walks up until it meets another node's span (parentEntitySpanId). An exception is attached to the nearest node above its span, else it is a node of its own. - Exceptions. A detail page reads
exceptions_v2by trace and keeps those on its own span or a span of its subtree (findOccurrenceExceptions).telemetry.FindExceptionOwnerdoes the walk the other way, from the exception's span up to the nearest endpoint, task or AI trace of the same project. The exception responses return it asrelatedEntity, and notification text uses it (resolveExceptionOwner). A failed task, for the task failure rule, is a task with an exception on its own span. - Every span read is bound in time. A lookup reads
recorded_atwithin 24 hours either side of the entity it starts from (shared.OtelLookupWindow,shared.TraceWindowBounds; entity lookups by trace id use 48 hours), which on ClickHouse prunes the read to a few daily partitions. A lookup that names no time is anchored on now. The cost is deliberate: a span recorded more than 24 hours away from its entity is not shown on that entity's page. Tests that store spans under a fixed timestamp must pass that timestamp as the lookup time, never the wall clock. - Orphans. A span whose parent never arrived has no place in a subtree. The entity promoted from the trace's root span (the one with no parent) shows such spans of its own project and trace, so they are visible exactly once, and the waterfall marks them "Parent span is unavailable". Once the parent arrives they nest normally.
- Span explorer (
controllers/span_explorer.controller.go, frontend/spans).POST /api/spans/search(App auth + project access) pages the spans of a project in a required time range of at most 31 days. Body:fromDate,toDate,orderBy(start_time desc|asc,duration desc|asc),serviceName,name(case-insensitive contains),traceId(32 hex),kind(0-5),status(0-2),minDurationMs,maxDurationMs,attributeFilters(at most 10 exact{key, value}matches onspan_attributes, value required since a ClickHouse Map answers''for a missing key),pagination. The response addsservices, the busiest 100 service names in the range. Validation problems and a search that hits its 15s read limit both answer 422, which the page shows as a warning callout, not an error. The query builder isshared.OtelSearchQuerieswith one small dialect struct per backend.GET /api/spans/traces/:traceId?at=<RFC 3339>returns the whole trace (shared.FindOtelTrace) and 404s when nothing is recorded within 24 hours ofat. It reads every project of the current project's organization, not only the current one (organizationProjectsin the controller, one lookup per project, one query): services of one trace usually report to different projects, and organization membership is what grants read access to each. Other organizations of the same user stay out. The response names the projects that hold spans (projects), each span keeps its ownprojectId, the page links spans across projects (buildSpanTree(..., acrossProjects)), and the popover loads attributes with the span's own project id. Search stays per project. Unlike the detail pages it sends no attributes: each span carries onlydbStatement, a 240 character preview ofdb.query.text(ordb.statement) that the row label needs, and the page fetches one span's attributes fromGET /api/spans/traces/:traceId/spans/:spanId/attributes?at=when its popover opens. One query serves both cases of the row cap (FindTraceOutlineon each backend, ordered byshared.OtelImportanceOrder): underMaxOtelGraphRowsevery span comes back, over it theLIMITkeeps errors first, then entry points (no parent, SERVER or CONSUMER), then the slowest, and the status ispartialwith reasonsrow_limitandmost_important. Ordering by duration also tends to keep the path above a span, since a parent usually outlasts its children. Responses are not compressed by the backend: 20,000 spans are about 13 MB of JSON, about 1.1 MB once a proxy gzips them. The sidebar hides Spans for browser and mobile projects (hiddenForFrontendinapp-sidebar.svelte); the route itself still works for links into a trace. GET /api/otel/spans/:traceId/:spanId?at=<RFC 3339>(App auth + project access) returns one stored span (16 or 32 hex span id) as an OTLPExportTraceServiceRequestprotobuf.atis required (400 without it) for the same 24 hour rule.- Frontend: large traces. A trace over 500 spans opens with every node that has more than 50 children folded (
initiallyCollapsedinutils/span-tree.ts); smaller traces open fully expanded. A folded row shows a pill with the number of hidden spans and an error marker when one of them failed, and the header has Expand all and Collapse all. The waterfall keeps what was clicked as the difference from that default, so there is no effect to reset. Pages must hold span payloads in$state.raw: a deep$stateproxy over 20,000 spans took four minutes to read in the browser, against 0.4 seconds raw. The payload is only ever replaced, never edited. - Frontend.
span-waterfall.sveltenames the service on each row and colours bars by service once a trace crosses services (one colour per row otherwise), marks error spans and missing parents, renders rows 500 at a time so a 20,000 span trace cannot stall the page, caps the name column to half the visible width on narrow screens, and links to the whole trace in the explorer. The span popover shows service, kind, status and scope (utils/span-fields.ts). Span payloads are held in$state.raw. The tree is keyedtraceId:spanId(utils/span-tree.ts), and the issue pages build their related entity from the backend'srelatedEntity. GET /api/health/deepalso reportsspanPartitionTimeFallbacks: spans whose source start was missing or unrepresentable, so their partition time is the ingest time (shared.OtelStorageTimes).- Move-over (
repositories/telemetry/move_over.go, started byretention.startMoveOverwhenV2_MOVE_OVER=true). Copies the old tables into the V2 tables inside the backend (an embedded DuckDB file cannot be opened by a second process), UTC days scheduled newest first with configurable concurrency (V2_MOVE_OVER_WORKERS, default 1, cap 16), each day keeping table order, recorded in the transactional DBv2_move_over_days(migrationssqlite/0077,pg/0143) so it stops and resumes. The scheduler runs at startup and one hour after each pass ends, including failures; passes do not overlap.startedis committed before telemetry writes,doneafter all batches succeed, with no main-DB transaction held during the copy.moved_rowscounts scanned source rows in the current attempt, including already-copied rows skipped during recovery; it resets on resume. Progress and logs update after successful batches roughly every 10s (a slow query or insert can delay this). A failed worker cancels the pass, and all workers are joined before return; separate backend instances are still not coordinated. Finish upgrading old writers first: completed table/date pairs are not revisited. A resumed day skips rows already written. It reads throughLegacyRepository(one per backend, the old ids parked in the V2 model fields, seeshared/legacy.go) and writes through the V2 repositories, so the id mapping lives in one place: the trace id comes from thetraceway.otel.trace_idattribute the old ingest kept, else the olddistributed_trace_id, else the row id; an old zero padded span UUID becomes its last 16 hex characters; spans and exceptions find their old owner by id, nearest in time, and take its trace; each entity also gets its own span row. Slices are halved when a read comes back full, so no day is sorted, and only a single overfull second is paged by stable full-row order and offset. Operator notes:docs/pages/server/upgrade-v2-tables.mdx.
ClickHouse ch/0090_add_span_version adds a materialized SHA-256 fingerprint over length-prefixed protobuf components. All span readers rank by duration then this fingerprint, so topology never reads payload columns for new parts. Old V2 parts calculate it on read until merged or explicitly materialized; no rewrite is scheduled by migration. The column costs 32 uncompressed bytes per physical row.
V2 review corrections
Legacy history paging uses stable full-row ordering and offsets only within overfull one-second slices; source tables must be immutable after cutover. Recovery checks project, occurrence ID, trace/span IDs and destination-precision timestamps. Rejected canonical rows fail the native/migration facade, so a day cannot be marked complete after silent row rejection. An old preview's incorrectly completed day can be set back to started for checked recovery; see docs/pages/server/upgrade-v2-tables.mdx.
Span search, topology, attributes and export choose one retry version (longest duration, then the stored SHA-256 span_version on ClickHouse or stored protobuf bytes on SQLite/DuckDB), with UTF-8 byte budgets applied in SQL. Native report spans are written before projections. OTLP storage returns rejected span identities so their endpoint/task/AI/exception projections and conversation uploads are omitted while reporting partial success. HTTP entry-point classification suppresses only an observed local HTTP ancestor, never an absent parent's assumed kind. Fallback AI conversation IDs retain the historical dashed UUID spelling. Log search accepts traceId and optional spanId, scoped to the selected project and time window. Retired distributedTraceId and excludeTraceId selectors return 400. The trace log panel has one paginated list with stale-response protection.
Data Retention
Retention is handled in several different ways depending on the deployment.
0. Main-DB auth prune — retention.Start worker (backend/app/retention/auth_tokens.go + oauth_sessions.go, sharing startDBPruneWorker in prune_worker.go). Runs in all modes (Postgres and SQLite) against db.DB: once at startup, then every 24h, deleting expired/consumed auth rows. auth_tokens prunes expired device_authorizations (also pruned opportunistically on every /api/auth/device/authorize call, so the daily worker is a backstop there), refresh_tokens that are expired, revoked, or used more than 30 days ago (used rows are kept a month for replay detection, then dropped to bound per-user growth), plus revoked-or-expired personal_access_tokens (PAT expiry was otherwise only enforced lazily at read time). oauth_sessions prunes expired SSO login sessions. No env var — always on. (Refresh-token families and PATs also have explicit revoke paths: POST /api/auth/logout and the account PAT UI.)
1. ClickHouse — TTL clauses on the table itself. Only a few tables have a TTL; everything else is kept indefinitely (operators can drop monthly partitions manually if needed).
| Table | Retention | Source migration |
|---|---|---|
metric_points (raw) |
7 days | 0034_add_ttl_metric_points.up.sql |
metric_points_1m (1-min rollup) |
30 days | 0035_add_ttl_metric_points_1m.up.sql |
metric_points_1h (1-hour rollup) |
1 year | 0036_add_ttl_metric_points_1h.up.sql |
log_records |
90 days | 0075_increase_ttl_log_records.up.sql (raised from 30d set in 0045_create_log_records.up.sql) |
profiling_samples (the bulk) |
30 days | 0068_add_ttl_profiling_samples.up.sql |
profiles (slim metadata) |
30 days | 0069_add_ttl_profiles.up.sql |
profiling_stacks (dedup table) |
30 days | 0070_add_ttl_profiling_stacks.up.sql |
check_results (synthetic check probes) |
90 days | 0083_add_ttl_check_results.up.sql |
All other CH tables (the five V2 trace tables and the five they replaced, transactions, sessions, session_recordings, fired_notifications, archived_exceptions, slow_endpoints, etc.) |
No TTL, retained indefinitely | none |
The three profiling tables share a 30-day TTL keyed on each table's time column (start_time / recorded_at / last_seen). profiling_stacks is a ReplacingMergeTree(last_seen) dedup table, so last_seen is bumped on every re-ingest that references a stack — a stack only ages out once it has gone unreferenced for the full window, which is exactly when its samples have also expired, so no sample is ever left pointing at a dropped stack.
2. SQLite — retention.Start worker (backend/app/retention/sqlite.go). In SQLite mode, neither db.DB nor db.TelemetryDB has any built-in expiry, so a background worker fires once at startup and then every hour and runs a DELETE FROM <table> WHERE <time_column> < cutoff against each telemetry table.
| Variable | Default | Notes |
|---|---|---|
SQLITE_RETENTION_DAYS |
30 |
TTL in days. Set to 0 to disable the worker entirely. Has no effect outside SQLite mode. |
DUCKDB_RETENTION_DAYS |
30 |
Same worker on the DuckDB telemetry backend. When set it takes precedence there; when unset the worker falls back to SQLITE_RETENTION_DAYS (selection in retention.telemetryRetentionConfig). |
Tables it prunes (and the column used):
| Database | Table | Time column |
|---|---|---|
Telemetry (db.TelemetryDB) |
endpoints_v2, tasks_v2, exceptions_v2, spans_v2, ai_traces_v2, the five tables they replaced (endpoints, tasks, exception_stack_traces, spans, ai_traces), metric_points, session_recordings |
recorded_at |
| Telemetry | log_records |
timestamp |
| Telemetry | sessions |
started_at |
| Telemetry | fired_notifications |
fired_at |
| Telemetry | profiling_samples |
start_time |
| Telemetry | profiles |
recorded_at |
| Telemetry | profiling_stacks |
last_seen |
| Telemetry | check_results |
recorded_at |
archived_exceptions (per-hash flags) and slow_endpoints (per-endpoint config) are intentionally skipped — they are not time-series data. profiling_stacks is pruned (unlike those two) because it holds no user intent — it is a regenerable dedup table whose last_seen tracks the most recent referencing sample, so deleting expired stacks is safe.
2c. Synthetics failure screenshots — retention.Start worker (backend/app/retention/synthetics.go). Browser-check failure screenshots written to local disk under <STORAGE_PATH>/synthetics/ are aged out hourly by mtime, mirroring the session-recording split: the check_results rows referencing them are pruned separately (item 2 / CH TTL) and deliberately not coupled. SYNTHETICS_SCREENSHOT_RETENTION_DAYS (default 30, 0 disables); no-op unless STORAGE_TYPE=local.
2b. Log row cap — retention.Start worker (backend/app/retention/log_cap.go). SQLite mode only (SQLite or DuckDB telemetry), off by default. When LOG_RECORDS_MAX_ROWS is set to a positive N, a worker runs once at startup and then every minute and deletes log_records rows strictly older than the Nth-newest row's timestamp (single portable DELETE with an ORDER BY timestamp DESC LIMIT 1 OFFSET N-1 subquery; NULL boundary = no-op under the cap, boundary ties are kept). The cap is best-effort, not a hard limit: nothing throttles ingest, so between passes the table can exceed N, and sustained ingest above N rows/minute keeps it above the cap permanently — document it to users as a cleanup task with a 1-minute window, sized with headroom for peak log volume. Bounds log disk usage independently of SQLITE_RETENTION_DAYS; disk reclamation still happens via the hourly retention pass / DuckDB WAL checkpointing.
3. On-disk session recordings — retention.Start worker (backend/app/retention/recordings.go). Session recordings written to local disk (STORAGE_TYPE=local) accumulate under <STORAGE_PATH>/recordings/. A second worker walks that directory once at startup and then every hour and removes files whose mtime is older than the TTL, then prunes any directories left empty. The worker is a no-op when STORAGE_TYPE=s3.
| Variable | Default | Notes |
|---|---|---|
SESSION_RECORDING_RETENTION_DAYS |
30 |
TTL in days. Set to 0 to disable the worker. Only runs when STORAGE_TYPE=local (default). |
The DB rows in session_recordings are pruned by the SQLite retention worker (above) or by ClickHouse TTL — they are intentionally not coupled to the disk cleanup. Controllers that read recordings already log a non-fatal traceway.CaptureException when a referenced file is missing.
4. On-disk raw profile archives — retention.Start worker (backend/app/retention/profiles.go). When PROFILE_ARCHIVE_RAW is enabled, the native pprof ingest path (/profiles/ingest) writes each upload's original pprof bytes to <STORAGE_PATH>/profiles/<projectId>/<yyyymmdd>/<id>.pprof (recorded on Profile.StorageKey) as a lossless archive for download / re-ingest / PGO — off the read path. The OTLP endpoint does not archive (its rows carry an empty StorageKey). A worker walks that directory once at startup and then every hour and removes files whose mtime is older than the TTL, then prunes empty directories. It reuses the same generic age-based cleanup as the recordings worker (runDirAgeCleanup / isSafeStorageSubdir) and is a no-op unless PROFILE_ARCHIVE_RAW is on and STORAGE_TYPE=local.
| Variable | Default | Notes |
|---|---|---|
PROFILE_ARCHIVE_RAW |
false |
Master switch for the raw archive. When off, no blob is written and the disk worker does nothing. |
PROFILE_RETENTION_DAYS |
30 |
TTL in days for the on-disk archive. Set to 0 to disable the disk worker. Only runs when PROFILE_ARCHIVE_RAW is on and STORAGE_TYPE=local. |
The profiles DB rows (and their storage_key) are pruned by the SQLite retention worker / ClickHouse TTL above — not coupled to this disk cleanup, mirroring the session-recording split.
5. Main-DB outbox prune — retention.Start worker (backend/app/retention/outbox.go, same startDBPruneWorker scaffolding as item 0). Runs in all modes against db.DB: once at startup, then every 24h, deleting terminal notification_outbox rows — sent/cancelled older than 7 days, failed older than 30 days. pending and sending rows are never pruned, so nothing undelivered is dropped. No env var — always on. The durable record of what was notified lives in fired_notifications and page_notifications, which this does not touch. Note that pages and page_notifications themselves are currently retained indefinitely.
Session Recording Uploader
Session recording segments arriving on /api/report are not uploaded inline. The handler enqueues each segment onto a bounded worker pool (backend/app/recordings/uploader.go, started from cmd/run.go next to retention.Start). Workers drain the queue and write the body via storage.Store.Write (S3 or local disk); successful writes are handed to a single batcher goroutine that calls SessionRecordingRepository.InsertAsync once per ~1000 rows or every 2 s, whichever comes first — single-row inserts are an anti-pattern for ClickHouse. Enqueue is non-blocking: when the queue is full the segment is dropped (newest-first) so a burst of /api/report traffic cannot spawn unbounded goroutines or saturate S3.
| Variable | Default | Notes |
|---|---|---|
SESSION_RECORDING_UPLOAD_WORKERS |
32 |
Concurrent uploaders. Set to 0 to drop every segment (uploads disabled). |
SESSION_RECORDING_UPLOAD_QUEUE_SIZE |
2048 |
Max queued segments. Overflow is dropped. |
Observability (emitted every 10s via traceway.CaptureMetric):
| Metric | Type | Meaning |
|---|---|---|
traceway.recordings.queue_depth |
gauge | Segments currently waiting in the channel. |
traceway.recordings.in_flight |
gauge | Workers mid-upload. |
traceway.recordings.uploaded |
counter | Successful S3/local + DB writes since startup. |
traceway.recordings.dropped |
counter | Segments dropped on overflow since startup. |
traceway.recordings.failed |
counter | Upload or DB-insert errors since startup. |
Sustained drops also fire a rate-limited (1/min) traceway.CaptureException so overload is visible in the issues feed without flooding it.
Session end is derived on read. The browser SDK's closing payload is unreliable (a keepalive body over 64 KB is rejected on unload, and frozen mobile tabs never fire pagehide), so the stored sessions.ended_at is only a hint. Each segment row carries its client-side ended_at (ch/0100, sqlite_telemetry/0027, duckdb_telemetry/0008), and every session read joins the latest segment activity and applies shared.ResolveSessionEnd: an explicit end is kept only when it lands within SessionIdleTimeout (15 min, the SDK's inactivity timeout) after the last activity, otherwise the session ends at its last activity; it is still live while a segment arrived (server time) within the timeout. Every session is capped at shared.SessionMaxSpan (65 min: the SDK's 60 min cap plus slack) because @tracewayapp/frontend <= 1.2.0 kept uploading segments under an ended session id for the tab's whole lifetime (18h "sessions"); the end and duration are clamped to it (sessions without segments too), and when segments run past it the session ends at the last segment inside it (span_activity, computed per backend against the session start; the usual 15 min rule then honours the SDK's own close), not at the cap, because an idle tab uploads nothing until it wakes hours later (sleep/wake, or 1.2.0's zombie uploads) and resumes under the same id; Session.CutoffAt carries the resulting cut-off, a session whose segments run past it is never live (compared on the client clock, so skew between browser and server cannot end a live session), and the session detail and GET /api/sessions/:id/recording drop exceptions and segments past CutoffAt, anchored on the stored session start rather than the startedAt query param (shared.SegmentsWithinSession). HasRecording is set when any segment exists, and the frontend labels an old session with neither as "No recording" (utils/session-duration.ts). Sorting the list by duration uses a SQL copy of the same rule in each backend (sessionDurationSortKey); TestSessionRepository_EndFromRecordingActivity holds the SQL order to the Go rule. GET /api/sessions/:id/recording?startedAt= bounds the segment lookup to the session's window, and segments are read from storage 16 at a time.
Users, Organizations & Projects
- Users to organizations is many-to-many via
organization_users, one role per membership. A user joins additional organizations through invitations (POST /api/invitations/:token/accept-existingfor existing accounts) and, in cloud mode, registration. Login/register/LoginBundlereturn every membership asOrganizations[]with roles; the frontend keeps them inauthState.organizationsand groups the navbar project selector by organization when there is more than one. - Projects belong to exactly one organization (
projects.organization_id). Org membership grants read access to all of the org's projects. - Per-project role overrides live in
project_user_roles(project_id, user_id, role)with roleuserorreadonly. Overrides only apply to members whose org role isuserorreadonly; owners and admins always have full access to every org project. Override rows are kept when a member's org role changes (they are inert for owner/admin), and are deleted when the member is removed from the org or the project is deleted. Managed from Settings > Team Members (expand a member row) viaGET/PUT /api/organizations/:orgId/members/:userId/project-roles(/:projectId);PUTwithrole: "default"clears the override and is accepted for any member regardless of org role (so stale rows on members promoted to admin can still be cleaned up); settinguser/readonlyis rejected with 422 for owners and admins. - Effective project role (
ProjectRepository.GetEffectiveRole): the org role ifowner/admin, otherwise the override if present, otherwise the org role./api/projectsreturns it asroleon each project and maskstoken/sourceMapTokenwhen it resolves toreadonly; the frontend derives write gating from it (isProjectReadonlyinprojects.svelte.ts).
Organization Roles
| Role | Description |
|---|---|
owner |
Full access, can manage organization |
admin |
Full access to projects |
user |
Standard access to projects (can be overridden per project to readonly) |
readonly |
Read-only access, cannot create projects or archive exceptions (can be overridden per project to user) |
Middleware enforcement: RequireProjectAccess checks org membership (read access, unaffected by overrides); RequireWriteAccess blocks writes when the effective project role is readonly; RequireAdminAccess requires org role owner/admin for the :organizationId route param.
Key Columns - transactions
project_id UUID,
timestamp DateTime64(3),
trace_id String,
endpoint String, -- normalized: "GET /api/users"
duration_ms Float64,
status_code UInt16,
app_version String,
server_name String,
-- Indexes: bloom_filter(trace_id), tokenbf_v1(endpoint)
Key Columns - exception_stack_traces
project_id UUID,
timestamp DateTime64(3),
hash String, -- normalized hash for grouping
type String, -- error type (e.g., "RuntimeError")
value String, -- error message
stacktrace String, -- full stack trace
tags Map(String, String), -- contextual tags from scope
Database Migrations
CRITICAL RULES:
migrations/ch/andmigrations/pg/files must contain exactly ONE SQL statement. Both run throughgolang-migrate, and the ClickHouse driver is constructed withMultiStatementEnabled: false(migrations_telemetry_ch.go), so a second statement fails.pg/follows the same rule for symmetry.migrations/sqlite/,migrations/sqlite_telemetry/andmigrations/duckdb_telemetry/run throughrunMigrationsOninmigrations.go, which splits on semicolons outside string literals (splitStatements, covered bysplit_test.go). A file there may hold aCREATE TABLEplus its indexes — that is the existing convention, e.g.sqlite/0043_create_pages.up.sql.- Only create
.up.sqlfiles (no down migrations) - Use sequential numbering:
NNNN_description.up.sql
Example - Adding two columns requires TWO files:
backend/app/migrations/ch/
├── 0013_add_app_version_to_transactions.up.sql
│ └── ALTER TABLE transactions ADD COLUMN app_version String DEFAULT ''
├── 0014_add_server_name_to_transactions.up.sql
│ └── ALTER TABLE transactions ADD COLUMN server_name String DEFAULT ''
Exception Hash Normalization
The backend normalizes stack traces before hashing to group identical errors despite different runtime values. This happens in backend/app/controllers/clientcontrollers/client.controller.go.
Normalization Steps (ComputeExceptionHash, applied in this order, and skipped entirely when isMessage is true):
- Strip the message from
Caused by:lines, keeping the class name (causedByRe) - Strip the message from the error line, keeping the error type (
errorMessageRe) - Collapse JS SDK function-name lines (ending in
(), directly above a 4-space-indentedfile:line:collocation line) to<fn>so resolved function names never affect grouping; anchoring on the location line keeps Go traces (tab-indented file lines) untouched (jsFuncLineRe) - Remove URL origins such as
https://cdn.example.com(urlOriginRe) - Remove absolute file paths, keeping
filename:line(absolutePathRe) - Drop the column from any frame whose line number is 2 or higher (
laterLineColRe) - Remove
@v1.2.3module version suffixes (versionRe) - Replace hexadecimal addresses with
<hex>(hexRe) - Replace UUIDs with
<uuid>(uuidRe) - Replace runs of 5 or more digits with
<id>, unless preceded by a colon or another digit (largeNumberRe) - Replace email addresses with
<email>(emailRe) - Replace IP addresses, with an optional port, with
<ip>(ipRe) - Replace Go goroutine ids with
goroutine <n>(goroutineRe) - Drop line numbers from Java, Kotlin and Scala frames (
javaLineNumRe) - Collapse
... 12 moreto... more(javaEllipsisRe) - Collapse runs of spaces and tabs, then runs of newlines (
spacesRe,newlinesRe) - Trim, hash with SHA-256, truncate to 16 hex chars
Not normalized: timestamps and ANSI color codes. Two otherwise identical traces that differ only in an embedded timestamp, or only in \x1b[31m escapes, produce two separate Issues. Add a regex to the block in client.controller.go if that ever matters.
Note: the column is kept only on line 1 (step 6). Minified bundles put everything on line 1, so there the column is the only frame disambiguator when no source map matched, while on real source lines the column is noise. Function names are excluded (step 3) because they are derived from the location and change as symbolication improves.
Result: Same logical error gets same hash, even if:
- Error message contains different user IDs
- Stack trace has different memory addresses
- File paths differ between environments
Repository Patterns
Singleton Pattern
Repositories are exported as package-level singletons, re-exported per storage axis through the facade packages app/repositories/transactional and app/repositories/telemetry:
// backend/app/repositories/transactional/sqlite/user.repository.go (and the pg/ twin)
var UserRepository = userRepository{}
// backend/app/repositories/transactional/transactional_sqlite.go (tag-guarded facade)
var UserRepository = sqliterepo.UserRepository
// Usage in controllers
user, err := transactional.UserRepository.FindByEmail(tx, email)
spans, err := telemetry.SpanRepository.FindByTraceId(projectId, traceId)
Batch Insert (ClickHouse)
func (r *TransactionRepository) BatchInsert(txns []models.Transaction) error {
batch, _ := r.db.PrepareBatch(ctx, "INSERT INTO transactions ...")
for _, txn := range txns {
batch.Append(txn.ProjectID, txn.Timestamp, ...)
}
return batch.Send()
}
Aggregation with Quantiles
// P50, P95, P99 percentiles
query := `
SELECT
endpoint,
count() as count,
quantile(0.5)(duration_ms) as p50,
quantile(0.95)(duration_ms) as p95,
quantile(0.99)(duration_ms) as p99
FROM transactions
WHERE project_id = ? AND timestamp BETWEEN ? AND ?
GROUP BY endpoint
ORDER BY count DESC
`
Error Handling Pattern
IMPORTANT: When handling errors in controllers, always use c.AbortWithError with traceway.NewStackTraceErrorf instead of c.JSON with a generic error message. This ensures proper error tracking with stack traces.
// CORRECT - Use AbortWithError with descriptive reason
projectId, err := middleware.GetProjectId(c)
if err != nil {
c.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("RequireProjectAccess middleware must be applied: %w", err))
return
}
// WRONG - Do not use c.JSON for internal server errors
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
Key points:
- The reason should describe the actual cause (e.g., "RequireProjectAccess middleware must be applied")
- Use
%wto wrap the original error for proper error chaining - This pattern applies to all 500 Internal Server Error responses
- For client errors (400, 404),
c.JSONwith an error message is acceptable
Non-stopping errors: For errors that should not abort the request (e.g., optional feature failed to load), use traceway.CaptureException to report them instead of log.Printf:
// CORRECT - Report non-stopping errors via traceway
if err != nil {
traceway.CaptureException(fmt.Errorf("failed to read session recording (key=%s): %w", key, err))
}
// WRONG - Do not use log.Printf for errors
if err != nil {
log.Printf("Failed to read session recording (key=%s): %v", key, err)
}
Validation error conventions:
400 Bad Request: Malformed requests, missing required params, type errors422 Unprocessable Entity: Business validation in form dialogs (name too long, duplicate name, required field empty). Returnc.JSON(422, gin.H{"error": "descriptive message"}). The frontendapi.tsextracts 422 error messages — dialogs catch and display them inline.
Summary:
- Stopping errors (abort the request):
c.AbortWithError(status, traceway.NewStackTraceErrorf("reason: %w", err)) - Non-stopping errors (continue serving):
traceway.CaptureException(fmt.Errorf("reason: %w", err)) - Validation errors (user-facing):
c.JSON(422, gin.H{"error": "message"})for form validation - Always wrap errors with
traceway.NewStackTraceErrorforfmt.Errorfusing%w— never discard the original error
Emails
Every email Traceway sends is a hardcoded template in backend/app/services/emailtemplates/ (embedded with go:embed, parsed once at init). The copy of an email lives in its own .gohtml file; the payload is the only thing it interpolates. There is no shared layout document, so changing one email cannot reshape the others, and every file is a complete HTML document styled like the rest.
Templates: new_error, error_regression, check_down, check_recovered, alert (the threshold email every rate/latency/apdex/throughput/task/impact/cost rule sends), ai_flagged, page (on-call escalation), test (both test buttons), invitation, password_reset.
- One send path —
services.SendEmail(ctx, services.Email{...})inbackend/app/services/email.service.gois the only way mail leaves the process: it renders the template, wraps it asmultipart/alternative(plaintext part first, quoted-printable), and talks SMTP. WithSMTP_ENABLEDoff it logs the plaintext instead.services.RenderEmailis the render-only half, used by the preview endpoint. services.Emailcarries the chrome every template shares (Title,Badge/BadgeColorviaEmailColor*,URL,Footer,LogoURLdefaulting to{APP_BASE_URL}/traceway-mark.png) plusTemplateandData, the typed payload reached as{{.Data.X}}.Textis the plaintext alternative and must stand on its own.- Notification payloads live on
models.NotificationMessage.Email(models.NotificationEmail: a template name plus one typed struct —EmailException,EmailCheck,EmailAlert,EmailFlagged,EmailPage,EmailTest). The builders innotifications/messages.goattach it;services.NotificationEmailturns a message into theEmailat send time (severity chip, absolute link, "why you got this" footer). It rides through the notification outbox as JSON, so the shape is a wire format: add fields, never repurpose them. Only the email adapter reads it — Slack/Telegram/SMS/webhook andfired_notificationsstill seeBodyalone, soBodymust always stay complete. - Adding an email: add
emailtemplates/<name>.gohtml, add its payload struct, and callSendEmailwithTemplate: "<name>". For a new rule type that only needs a sentence and a link, reusealertby fillingmodels.EmailAlert. - Preview:
EMAIL_PREVIEW_ENABLED=trueregistersGET /api/email-preview(index) andGET /api/email-preview/:template, which renders any template with sample data (?format=textshows the plaintext alternative). Off by default, so it is never a route in production. Sample data lives incontrollers/email_preview.controller.go.
Native /api/report Protocol
There is no Traceway Go SDK. It was retired, and every backend, Go included, instruments with
OpenTelemetry and exports over OTLP/HTTP. The docs carry no Go SDK pages, and docs/public/_redirects
301s the old /client/sdk and /client/*-middleware URLs to /client/otel. Do not reintroduce them.
The native protocol is deprecated. Traceway is OpenTelemetry first, and the public docs do not
describe /api/report as a way to send data: there is no protocol page (/protocol 301s to
/client/otel) and docs/pages/learn/data-flow.mdx is written around OTLP. Do not document it again.
The URL still shows up in the docs in one role only, as part of the connection string the browser and
mobile SDKs are configured with.
The endpoint itself is still served, because those SDKs (@tracewayapp/*, the iOS and Android
libraries, Flutter) speak it and the backend uses it to report its own telemetry. This section is the
engineering reference for keeping it working. Its rows land in the same V2 tables as OTLP: a run is
stored as the root span of a trace with its spans under it (see "Trace Storage (V2 tables)").
Data Format (Frame)
Clients send data as gzipped JSON. The wire shape is ReportRequest in backend/app/controllers/clientcontrollers/client.controller.go wrapping CollectionFrame from backend/app/models/clientmodels/:
{
"appVersion": "1.2.3",
"serverName": "myapp-host-1",
"collectionFrames": [
{
"traces": [
{
"id": "5b8e1a2f-3c4d-4e5f-8a9b-0c1d2e3f4a5b",
"endpoint": "GET /api/users",
"duration": 45200000,
"statusCode": 200,
"recordedAt": "2024-01-15T10:30:00Z",
"isTask": false,
"attributes": {},
"spans": []
}
],
"stackTraces": [
{
"stackTrace": "RuntimeError: connection refused\n at ...",
"recordedAt": "2024-01-15T10:30:00Z",
"attributes": {"user_id": "123"},
"isMessage": false,
"isTask": false
}
],
"metrics": [
{
"name": "cpu.used_pcnt",
"value": 45.2,
"recordedAt": "2024-01-15T10:30:00Z",
"tags": {}
}
]
}
]
}
Notes:
- Timestamps are
recordedAt(RFC 3339), nevertimestamp.durationis a Gotime.Duration, i.e. integer nanoseconds (45200000 = 45.2ms). - Endpoints vs tasks share the
tracesarray, split byisTask.CollectionFramealso carriessessionRecordingsandsessions. - Each entry in a trace's
spansmay carry an optionalparentSpanId(UUID) andattributes(string map). A span that names no parent hangs under the run's own span. The ids are stored as 32 hex characters. - Unknown fields are silently ignored: a payload in the wrong shape (e.g. top-level
metrics) still returns 200 but inserts nothing. When hand-crafting test payloads, confirm ingestion landed viaPOST /api/metrics/query(or the relevant list endpoint) instead of trusting the status code.
Common Patterns
Adding a New API Endpoint
Add model in
backend/app/models/type NewEntity struct { ID uuid.UUID `json:"id"` Name string `json:"name"` CreatedAt time.Time `json:"created_at"` }Add repository in
backend/app/repositories/func (r *NewEntityRepository) GetAll(projectID uuid.UUID) ([]models.NewEntity, error) { // ClickHouse query }Add controller in
backend/app/controllers/func (c *NewEntityController) List(ctx *gin.Context) { entities, err := repo.GetAll(projectID) ctx.JSON(200, entities) }Register route in
backend/app/controllers/routes.goapi.GET("/new-entities", newEntityController.List)Add frontend API call in
frontend/src/lib/api.tsor directly in page
POST body convention: List/search endpoints use POST with a JSON body containing filters and pagination:
type ListRequest struct {
ProjectId string `json:"projectId"`
FromDate string `json:"fromDate"`
ToDate string `json:"toDate"`
OrderBy string `json:"orderBy"`
Search string `json:"search"`
Pagination PaginationParams `json:"pagination"`
}
Paginated response: Use PaginatedResponse[T] from routes.go for all paginated endpoints:
type PaginatedResponse[T any] struct {
Data []T `json:"data"`
Pagination Pagination `json:"pagination"`
}
type Pagination struct {
Page int `json:"page"`
PageSize int `json:"pageSize"`
Total int64 `json:"total"`
TotalPages int64 `json:"totalPages"`
}
type PaginationParams struct {
Page int `json:"page" binding:"min=1"`
PageSize int `json:"pageSize" binding:"min=1,max=100"`
}
Adding a New Frontend Page
Create route folder:
frontend/src/routes/new-page/Add page component:
+page.sveltewith the standard loading pattern:<script lang="ts"> import { onMount } from 'svelte' import { api } from '$lib/api' import { projectsState } from '$lib/state/projects.svelte' import { ErrorDisplay } from '$lib/components/ui/error-display' import { getErrorMessage, getErrorStatus } from '$lib/utils/errors' let data = $state<DataType[]>([]) let loading = $state(true) let error = $state('') let notFound = $state(false) async function loadData() { loading = true error = '' try { const response = await api.post<ResponseType>('/endpoint', payload, { projectId: projectsState.currentProjectId ?? undefined }) data = response.data || [] } catch (e) { if (getErrorStatus(e) === 404) { notFound = true } else { error = getErrorMessage(e) || 'Failed to load data' } } finally { loading = false } } onMount(() => { loadData() }) </script> {#if loading} <LoadingCircle size="xlg" /> {:else if notFound} <ErrorDisplay status={404} title="Not Found" description="..." onRetry={() => loadData()} /> {:else if error} <ErrorDisplay status={400} title="Error" description={error} onRetry={() => loadData()} /> {:else} <!-- Content --> {/if}Add data loading (optional):
+page.tsfor URL paramsexport const load = async ({ params }) => { return { param: params.id } }Add navigation in
src/lib/components/app-sidebar.svelte
Adding a New Metric to Dashboard
Ensure SDK captures metric (or add to
traceway.gometrics collection)Add repository query to each telemetry backend's
metric_point.repository.gounderbackend/app/repositories/telemetry/{clickhouse,sqlite,duckdb}/func (r *metricPointRepository) GetNewMetric(ctx context.Context, projectId uuid.UUID, from, to time.Time) ([]models.TimeSeriesPoint, error) { // Query metric_points with the backend's dialect }Add to dashboard controller in
backend/app/controllers/dashboard.goFrontend auto-renders from API response (metrics dashboard uses dynamic rendering)
Adding a Database Column
Create migration file (remember: ONE statement per file!)
backend/app/migrations/ch/0015_add_new_column.up.sqlALTER TABLE transactions ADD COLUMN new_column String DEFAULT ''Update model in
backend/app/models/Update repository queries to include new column
Run migrations: Backend runs migrations automatically on startup
Adding Table Sorting to a Page
Import and add state:
import { getSortState, setSortState, handleSortClick } from '$lib/utils/sort-storage' import type { SortState } from '$lib/utils/sort-storage' let sortState = $state<SortState>(getSortState('page-key', { field: 'default_column', direction: 'desc' }))Add sort handler:
function onSortClick(field: string) { sortState = handleSortClick(field, sortState.field, sortState.direction, 'desc') setSortState('page-key', sortState) }Use TracewayTableHeader:
<TracewayTableHeader label="Column" column="column_name" orderBy={`${sortState.field} ${sortState.direction}`} onclick={() => onSortClick('column_name')} />Pass to API call - convert to backend format:
"column asc"or"column desc":const orderBy = `${sortState.field} ${sortState.direction}`
Adding a New Framework
Backends do not get a new framework value. OpenTelemetry is the single backend integration path, and opentelemetry is the only backend option in the project-creation picker (plus the preselected default). A new backend language or web framework is a documentation and Connection-page change, never a new project framework:
- Connection page targets —
frontend/src/lib/utils/otel-setup.ts: add the framework to the matchingOTEL_TARGETSlanguage entry (or add a new language target) and its setup steps - Docs guide —
docs/pages/client/otel/<framework>/: create_meta.jsonandindex.mdx, then list it indocs/pages/client/otel/_meta.jsonand the "Next Steps" block ofdocs/pages/client/otel/index.mdx - Docs OTel language table —
docs/pages/client/otel/index.mdx: add a row - Docs framework picker — nothing to do.
docs/components/FrameworkPicker.jsxcarries exactly one backend card, OpenTelemetry, pointing at/client/otel, so it mirrors the dashboard's project-creation picker. Do not add a per-framework backend card; the guide is discovered from/client/otelinstead. - Combobox search keywords —
frontend/src/lib/components/framework-combobox.svelte: append the name to the OpenTelemetry entry'skeywordsso searching for it still finds the right option - README — add to the supported-frameworks table if it warrants a row
Only a browser or mobile framework needs a real new framework value, because those use platform SDKs with their own Connection-page content:
- Backend —
backend/app/controllers/project.controller.go: add tovalidFrameworksand update the validation error message - Frontend state —
frontend/src/lib/state/projects.svelte.ts: add to theFrameworkunion,FRAMEWORK_LABELS, andMOBILE_FRAMEWORKS/FRONTEND_FRAMEWORKS - Frontend combobox —
frontend/src/lib/components/framework-combobox.svelte: add an entry to theBrowserorMobilegroup withkeywords - Frontend icon —
frontend/src/lib/components/framework-icon.svelte: add the icon mapping - Framework code —
frontend/src/lib/utils/framework-code.ts: add install command, integration snippet, label, code language, and testing routes - Connection page —
frontend/src/routes/connection/+page.svelte: add highlight language mapping and install description - Dashboard page —
frontend/src/routes/+page.svelte: add highlight language mapping - Docs —
docs/pages/client/<framework>/,docs/pages/client/_meta.json,SDK_OPTIONS+FOLDER_SDKindocs/components/SdkContext.jsx,SDK_VISIBILITYindocs/theme.config.jsx,SDK_QUICK_STARTindocs/components/SdkSelector.jsx, and aFrameworkPicker.jsxcard
Values removed from the picker (gin, fiber, chi, fasthttp, stdlib, custom, nextjs, nestjs, express, remix, hono, cloudflare, symfony, laravel, django) stay valid in validFrameworks and in FRAMEWORK_LABELS — existing projects still carry them and must keep rendering.