Skip to content

The only tool you need to know what is happening and how to fix it.

168 KB
Raw Blame

CLAUDE.md - Traceway Project

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

Traceway is an error tracking and monitoring platform consisting of:

  • Frontend: SvelteKit 2 dashboard application with Svelte 5
  • Backend: Go/Gin API server with ClickHouse database
  • CLI: Go/Cobra command-line client for the backend HTTP API (/cli), built twice from one source: read-only traceway, and traceway-write (-tags write, own config dir and OAuth client, write commands in cmd/traceway/edition_write.go). Skills: skills/traceway (read), skills/traceway-learn (the product guide, one file per area from knowledge/learn/, also served as traceway://knowledge/learn/*) and plugins/tw-write/skills/traceway-write (write, separate plugin), all generated from cli/pkg/mcpserver/knowledge by just gen-skills. The skill and MCP evals live in cli/test/evals/ (own Go module, see its README), never under skills/: anything in a skill directory ships to users and to the agent under test

Code Style

  • No pointless comments: Do not add comments that simply describe what the code does. The code should be self-explanatory. Only add comments when explaining non-obvious "why" decisions.
  • No py-4 in dialog form content: Do not add py-4 on the content wrapper inside AlertDialog or Dialog components — it creates too much blank space between the form and the action buttons.
  • Dialog button labels & toasts: For form dialogs, use descriptive button labels with icons instead of generic "Create"/"Update". The {Entity} is always a platform entity, capitalized (Project, Widget, Dashboard, Channel, Rule, Token, Invitation, ...). Create actions: <Plus icon> New {Entity} with variant="success". Update actions: <Check icon> Update {Entity} with the default (primary) variant. Delete/revoke/remove confirm buttons: <Trash2 icon> Delete {Entity} (or Revoke {Entity} / Remove {Entity}) with variant="destructive". After success, show toast.success('Successfully created the {Entity}', { position: 'top-center' }) for creates and 'Successfully updated the {Entity}' for updates. The button should only be disabled during the loading state — never disable it to enforce validation; let the backend return 422 and show the error in the dialog instead.

Quick Start

Development Commands

Component Command Description
Frontend cd frontend && npm run dev Dev server (port 5173)
Frontend npm run build Production build
Frontend npm run check TypeScript checking
Frontend npm run lint prettier --check . && eslint . (CI gate; npm run format fixes the prettier half)
Backend cd backend && go run ./cmd/traceway API server (port 8082)
Backend cd backend && govulncheck ./... Vulnerability scan (default tags only); CI also scans the other storage build-tag combos (.github/workflows/backend-vulncheck.yml)
CLI cd cli && just build Builds bin/traceway
CLI cd cli && just test Runs unit tests
CLI cd cli && just check Lint + test + vulncheck + skill drift + contract tests (pre-commit gate; CI enforces it for cli/)
CLI cd cli && just smoke-test Live E2E (needs TRACEWAY_SMOKE_* env vars)
CLI cd cli && just eval-harness Eval harness tests plus every case's oracle trajectory, fake model only ($0)
CLI cd cli && just eval-plan --preset smoke --max-cost-usd 30 Eval matrix and live-price cost estimate ($0)
CLI cd cli && just eval-dispatch -f preset=smoke -f max_cost_usd=30 Starts a paid eval run in the CLI Evals workflow; it still waits for approval at the evals environment gate
CLI cd cli && just eval-local ... Paid local eval run, for harness development only (yes typed on a terminal, refused without one; required --max-cost-usd)
Any nix develop (or direnv via .envrc) Dev shell with the Go and Node versions read from backend/go.mod and .nvmrc, plus just, golangci-lint, govulncheck (flake.nix; shells default, backend, frontend, oxc). Put JWT_SECRET in the gitignored .envrc.local
Any ./scripts/check-node-pins.sh Asserts every Dockerfile's ARG NODE_VERSION default matches .nvmrc; run by release-traceway.yml before it builds

Node's version lives in .nvmrc (a bare major, currently 26) and nothing else derives it independently: every setup-node step uses node-version-file: .nvmrc, flake.nix reads it for the dev shells, and release-traceway.yml passes it to all five image builds as --build-arg NODE_VERSION. The Dockerfiles each carry an ARG NODE_VERSION=<major> default because a bare docker build cannot read .nvmrc; that default is the one value that can drift, which is what scripts/check-node-pins.sh exists to catch (it also fails on a literal node:<major> tag reintroduced in a FROM). To move Node, edit .nvmrc and the ARG defaults, then run the script. frontend/package.json's engines.node is deliberately left as a floor (>=22) rather than a pin — it is what npm enforces on consumers, and node-version-file pointed at it resolves a range to the newest release, which is the drift f95424cc was fixing. testing/devtesting-nestjs/Dockerfile is out of scope: it pins the runtime of a sample third-party app, not anything Traceway builds.

Set JWT_SECRET (min 32 characters) before running the backend, including the All-in-One image. Former public signing keys are rejected. The All-in-One shell script leaves .env loading to the backend so Docker-provided values win. Embedded Run(opts...) never reads JWT_SECRET, since it runs inside the host app's process where that name usually belongs to the host: it takes WithJWTSecret, else TRACEWAY_JWT_SECRET, else a random key per start; set a stable secret to preserve dashboard sessions across embedded restarts. The All-in-One image's supervisord sends the backend's output to the container's stdout/stderr, so docker logs shows a startup rejection. SQLITE_PATH sets the database location, defaulting to ./traceway.db in the working directory.

CI on pull requests is opt-in. backend.yml, backend-vulncheck.yml, cli.yml, cli-lint.yml, cli-contract.yml, eval-guard.yml, frontend.yml and plugin-scanner.yml listen for pull_request: types: [labeled] and every job is gated on github.event.label.name == 'ci', so a PR runs nothing until a maintainer applies the ci label (only collaborators can), and applying any other label runs nothing either. Each application validates the PR as it is at that moment: to validate later pushes, remove and re-add the label. The same workflows still run on pushes to main that touch their paths (backend/**, cli/**, frontend/**, skills/traceway/** for cli.yml, .nvmrc for frontend.yml, .github/workflows/**, .github/actions/**, cli/justfile and cli/test/evals/** for eval-guard.yml, their own workflow file; plugin-scanner.yml runs on every push to main because it scans the whole repo), on their daily schedules (the two vulnchecks), and from the Actions tab via workflow_dispatch. cli-evals.yml (the skill and MCP evals) is the one exception to this pattern: its only trigger is workflow_dispatch, a person starts every run by hand, the paid shard job waits for a required reviewer on the evals environment (the only place OPENROUTER_API_KEY lives, never a repository or organization secret; GitHub creates a missing environment without protection, so the preflight refuses a paid run unless evals exists with a required reviewer), and max_cost_usd is a required input with no default. It must never get a push, pull request, label, schedule, workflow_run, workflow_call or repository_dispatch trigger, no other workflow or composite action may call, dispatch (by file, name or just eval-dispatch) or use its environment, and no just check/test/default recipe may run it; cli/test/evals/internal/guard/triggers_test.go fails if any of that changes. cli/test/evals is its own Go module, which cli.yml's go test ./... never reaches, so eval-guard.yml runs that test. twevals run itself refuses every CI context but a dispatched, approved cli-evals.yml job, and locally it needs yes typed on a terminal. Validate edits to it (or Dependabot bumps of its actions) with dry_run: true, which costs nothing but skips the paid and report jobs, so it does not exercise download-artifact or the shard matrix.

Action versions are watched by .github/dependabot.yml (the github-actions ecosystem only; npm and Go modules are deliberately not watched there). It opens a weekly PR per group — actions/* in one, every third-party action in the other — and deliberately does not auto-apply ci, so a bot PR passes the same collaborator gate as any other. Label the first-party group to validate it; golangci/golangci-lint-action is reachable the same way via cli-lint.yml. The rest of the third-party group lands on release-*.yml, benchmark-*.yml and traceway-autofix.yml, which no label reaches — and workflow_dispatch is not a dry run on the release path: dispatching release-docs.yml or release-website.yml deploys to Cloudflare for real, and the other release-*.yml workflows publish real assets, so dispatching one from an unreviewed bot branch ships it. Validate those by reading the action's release notes instead, and dispatch only where the target is disposable. traceway-autofix.yml is the in-repo caller of the reusable autofix.yml (composite actions .github/actions/autofix-prepare and autofix-publish around the agent step; docs/pages/learn/auto-fix.mdx documents the contract); its workflow_dispatch takes an issue_number and runs the whole loop from the dispatched branch, which is the one pre-merge validation path for the claude-code-action pin — the most privileged action here — but it runs the agent for real, so dispatch it only at a scratch issue. Note too that labelling a PR which touches no gated workflow yields zero checks, which reads like a pass but is silence. Every updatable pin is a bare major tag, so Dependabot only ever proposes major bumps here. The exception is plugin-scanner.yml, which pins both actions to commit SHAs because the HOL scanner scores unpinned actions down; Dependabot bumps those SHAs along with their version comments. It uses the scanner config in .plugin-scanner.toml, whose ignore_paths cover fixtures, demos and build output plus three reviewed production files (the read-only token mask in both project.repository.go copies and KindPassword in cli/internal/state/state.go). The HOL registry's own scan ignores that file and still fails on syntax-highlighter strings in the committed backend/static/frontend bundle; that is accepted.

Tech Stack

  • Frontend: SvelteKit 2.49, Svelte 5.45, Tailwind CSS v4, shadcn-svelte, Vite 7
  • Backend: Go 1.26, Gin 1.11, ClickHouse, PostgreSQL (in backend/go.mod the go line is the floor importers inherit and the toolchain directive is what builds it — keep the two distinct)
  • CLI: Go 1.26, Cobra 1.10, separate Go module (github.com/tracewayapp/traceway/cli); justfile entrypoints, covered by the root flake.nix dev shell
  • Client SDK: Go 1.25, Gin middleware support

lit Library (SQL mapper)

  • Import: github.com/tracewayapp/lit/v2 (currently v2.0.5)
  • Purpose: Lightweight generic CRUD operations on top of database/sql. Supports PostgreSQL, MySQL, SQLite, and DuckDB drivers (lit.PostgreSQL, lit.MySQL, lit.SQLite, lit.DuckDB). Traceway registers models against db.Driver (SQLite or PostgreSQL depending on build tags); the DuckDB telemetry backend deliberately keeps lit.SQLite for reads since DuckDB accepts ? placeholders.
  • Docs for AI use: the lit repo ships a skill at skills/lit/SKILL.md plus llms.txt with the full API contract and pitfall checklist.

Model Registration (required before use)

All models are registered centrally in models/models.go via models.Init(driver lit.Driver), called at boot with db.Driver:

func Init(driver lit.Driver) {
    lit.RegisterModel[User](driver)
    lit.RegisterModel[Project](driver)
    // ...all models registered here
}

Repository-local result models (e.g., aggregate structs only used in one repo) can use file-level init() instead. Irregular table names use lit.RegisterModelWithNaming with a struct embedding lit.DefaultDbNamingStrategy (see escalationPolicyNaming in models.go).

Naming Conventions

  • Fields: CamelCase → snake_case (FirstName → first_name)
  • Consecutive uppercase: stay together (HTTPCode → http_code)
  • Tables: pluralize + snake_case (User → users)
  • Override via struct tag: lit:"custom_name"

Core CRUD Operations

All lit functions take a lit.Executor as the first argument. Both *sql.Tx and *sql.DB qualify: main-DB repositories pass the *sql.Tx from the transaction middleware, telemetry repositories pass db.TelemetryDB directly.

Function Description
lit.Insert[T](tx, &entity) Insert, returns auto-generated int ID
lit.InsertUuid[T](tx, &entity) Insert with auto-generated UUID
lit.InsertExistingUuid[T](tx, &entity) Insert with pre-set UUID
lit.Select[T](tx, query, args...) Retrieve multiple records (returns []*T)
lit.SelectSingle[T](tx, query, args...) Retrieve one record (returns *T)
lit.Update[T](tx, &entity, "id = $1", id) Update (auto-prepends WHERE)
lit.UpdateNative(tx, "UPDATE table SET col = $1 WHERE ...", args...) Raw SQL update for partial/single-field changes
lit.Delete(tx, "DELETE FROM table WHERE id = $1", id) Delete records (any exec statement)
lit.SelectNamed[T] / lit.SelectSingleNamed[T] / lit.UpdateNamed[T] :name placeholder variants taking a lit.P{...} map; render per-driver, used for dialect-neutral queries
lit.DeleteNamed(driver, tx, query, lit.P{...}) Named delete/exec. Gotcha: takes the driver as the FIRST argument
lit.ParseNamedQuery(driver, query, lit.P{...}) Renders :name to positional; escape hatch for QueryContext/RowsAffected via database/sql

Transaction Helper (pgdb.ExecuteTransaction)

All PostgreSQL operations should use ExecuteTransaction for automatic commit/rollback:

// ExecuteTransaction[T] wraps a function in a transaction
// - Commits on success, rolls back on error or panic
// - Returns (T, error) directly - no pointer wrapping

project, err := pgdb.ExecuteTransaction(func(tx *sql.Tx) (*models.Project, error) {
    // All repository calls receive the transaction
    return transactional.ProjectRepository.FindById(tx, id)
})

Transactional Middleware (middleware.Transactional)

For auth flows and routes requiring transaction context throughout the request lifecycle, use the Transactional middleware:

// In routes.go - wrap routes that need transaction context
api.POST("/register", middleware.Transactional, authController.Register)
api.POST("/login", middleware.Transactional, authController.Login)

// In controller - retrieve transaction from Gin context
func (c *AuthController) Register(ctx *gin.Context) {
    tx := db.GetTx(ctx)  // Get transaction from context (db package, not middleware)

    // Use tx for all repository calls
    user, err := transactional.UserRepository.FindByEmail(tx, email)
    if err != nil {
        ctx.JSON(500, gin.H{"error": err.Error()})
        return  // Transaction auto-rolls back on non-success status
    }

    ctx.JSON(201, user)  // Transaction auto-commits on 200/201/303
}

Auto-commit/rollback behavior:

  • Commits on status codes: 200, 201, 303
  • Rolls back on all other status codes or panics

Body buffering: Transactional reads the request body into memory (cap maxTransactionalBodyBytes, 8MB) before it calls db.DB.Begin(), answering 413/408/400 without a transaction when that fails; the handler's later bind is served from memory. This is what keeps a slow-drip body from holding the single SQLite main-DB connection, and it covers every transactional route without each one remembering a special middleware. Routes that want a tighter cap put middleware.BufferAuthBody (64KB) in front of Transactional, which then skips the body it finds already buffered; the anonymous auth endpoints do this. The self-transacting OAuth routes (/auth/device/*, /auth/token, /auth/logout) use BufferAuthBody on its own for the same cap. UseAppAuth caps every dashboard route's body at the same 8MB with MaxBytesReader, so the non-transactional telemetry reads are bounded too. Every plain JSON bind site in the controllers answers through middleware.RejectBindError(c, err, fallback), which maps MaxBytesError to 413 and the body guard's timeout to 408 with fixed messages and otherwise returns 400 with the fallback (the ingest routes use middleware.RejectIngestBindError, the same mapping except that the timeout becomes 503 + Retry-After, because OTLP exporters retry 503 but treat 408 as permanent; the Traceway SDKs re-queue a failed batch whatever the status); the guard itself swaps the raw deadline error (a *net.OpError naming the listener address) for middleware.ErrBodyTimedOut before it leaves Read, so no handler can echo the address.

Preference: For CRUD controller methods, always prefer using middleware.Transactional in the route + db.GetTx(ctx) in the controller over pgdb.ExecuteTransaction. The middleware approach keeps controllers flat, avoids nested closures, and follows the established pattern. (The tx getter lives in the db package: db.GetTx(ctx), not middleware.GetTx.)

Repository Pattern

Repositories accept *sql.Tx to participate in transactions:

func (p *projectRepository) FindById(tx *sql.Tx, id uuid.UUID) (*models.Project, error) {
    return lit.SelectSingle[models.Project](
        tx,
        "SELECT id, name, token, framework, created_at FROM projects WHERE id = $1",
        id,
    )
}

PostgreSQL Specifics

  • Uses $1, $2, $3 placeholders (not ?)
  • Tables must have an id column
  • Always pass *sql.Tx from ExecuteTransaction to lit functions

Common Pitfalls

Always initialize all struct fields with defaults: When using lit.Insert, all struct fields are included in the INSERT statement, overriding database defaults. Always set fields like CreatedAt explicitly:

// CORRECT - set CreatedAt explicitly
user := &models.User{
    Email:     email,
    Name:      name,
    CreatedAt: time.Now().UTC(),
}

// WRONG - CreatedAt remains zero value (0001-01-01)
user := &models.User{
    Email: email,
    Name:  name,
}

lit.Update WHERE clause: The lit.Update function automatically includes WHERE in the generated SQL. Do not add WHERE yourself:

// CORRECT - just the condition
lit.Update(tx, &user, "id = $1", user.Id)

// WRONG - results in "WHERE WHERE id = $1"
lit.Update(tx, &user, "WHERE id = $1", user.Id)

Custom Result Models for Aggregates

For queries that return aggregated or computed values (not direct table rows), create a custom result model:

// Define a result model for the query output
type CountResult struct {
    Count int `lit:"count"`
}

// Register in models.Init() (or file-level init() if repo-local)
lit.RegisterModel[CountResult](lit.PostgreSQL)

// Use in repository
func (r *userRepository) CountByOrganization(tx *sql.Tx, orgID uuid.UUID) (int, error) {
    result, err := lit.SelectSingle[CountResult](
        tx,
        "SELECT COUNT(*) as count FROM users WHERE organization_id = $1",
        orgID,
    )
    if err != nil {
        return 0, err
    }
    if result == nil {
        return 0, nil
    }
    return result.Count, nil
}

Handling "Not Found" Cases

IMPORTANT: When using lit/PostgreSQL, do NOT check for sql.ErrNoRows. The lit library returns nil when no record is found, not an error. Always check for nil instead:

// CORRECT - check for nil
user, err := transactional.UserRepository.FindByEmail(tx, email)
if err != nil {
    return nil, err  // actual database error
}
if user == nil {
    // record not found - handle accordingly
    return nil, errors.New("user not found")
}

// WRONG - do not use sql.ErrNoRows with lit
user, err := transactional.UserRepository.FindByEmail(tx, email)
if err == sql.ErrNoRows {  // This won't work with lit!
    // ...
}

Handling "Not Found" Cases (ClickHouse)

IMPORTANT: ClickHouse queries behave differently from lit/PostgreSQL. ClickHouse returns sql.ErrNoRows when no record is found. Always use errors.Is() to check:

// CORRECT - ClickHouse returns sql.ErrNoRows
exception, err := exceptionRepo.GetByHash(projectID, hash)
if errors.Is(err, sql.ErrNoRows) {
    // record not found - handle accordingly
    return nil, errors.New("exception not found")
}
if err != nil {
    return nil, err  // actual database error
}

// Summary of error handling:
// - lit/PostgreSQL: check `if result == nil` (no error returned)
// - ClickHouse: check `errors.Is(err, sql.ErrNoRows)`

Environment Variables (Backend)

JWT_SECRET=<min 32 char secret for JWT signing>   # required in the standalone binary and every server image, including All-in-One. Missing, short and former public keys exit 1 with an actionable message. Docker's environment takes precedence over .env. Embedded Run(opts...) ignores JWT_SECRET (it belongs to the host app there): it uses WithJWTSecret, else TRACEWAY_JWT_SECRET (an explicitly empty value fails validation), else a random key per start, so set a stable value to preserve sessions across restarts. Retired public keys are matched by SHA-256 digest in jwt.service.go so no secret literal sits in scanned code.
DB_TYPE=                              # sqlite | postgres. Defaults to sqlite in every build EXCEPT -tags transactional_pg, where an empty value still selects PostgreSQL. Without that tag SQLite is the only supported main DB (the migration runner applies SQLite-dialect migrations unconditionally), so the default is set once in config.defaultDBType rather than at each DBType call site.
PORTS=80,8082                         # comma-separated HTTP listen ports. Every port is bound up front; a port that fails to bind (typically :80 as a non-root user) logs a warning (and reports via CaptureException when monitoring is configured), and startup only fails if NO port binds. Order is not significant: a listener that later stops serving takes the process down whichever position it holds.
SQLITE_CACHE_SIZE_MB=512              # PRAGMA cache_size of the SQLite TELEMETRY DB in MB, per connection (the main DB keeps SQLite's 2MB default; no effect on DuckDB or ClickHouse telemetry). The telemetry pool opens up to 4 connections (`telemetryMaxOpenConns`, db_sqlite_open.go) and modernc's allocator rounds each ~4.4KB page buffer up to an 8KB slot, so resident memory is about 2x the value per connection: worst case 8x the value, 4GB at the default. A cache fills from ingest, from the retention DELETE scan at startup and every hour, and from reads of pages still in the WAL (write transactions and WAL pages bypass the 1GB mmap window), and it is held for as long as the pooled connection lives. At the default a 1GiB container is OOM-killed on a 24h dashboard or under sustained ingest (issue #396). Rule of thumb: container MB / 32 (16 for 512MB, 32 for 1GB, 64 for 2GB). A non-positive or unparsable value falls back to 512 (`config.SizeMB`). The effective value is logged at startup, and the embedded `Run(opts...)` path honors the env var through `applyEnvOverrides`. Operator docs: `docs/pages/server/sqlite.mdx#memory-sizing`.
SQLITE_PATH=                          # main DB path, defaults to ./traceway.db in the working directory. The telemetry DB derives from it (_telemetry.db, or _telemetry.duckdb under -tags telemetry_duckdb).
APP_BASE_URL=                         # public origin of this server (e.g. https://traceway.example.com). Used as the OAuth issuer / device verification URL and SSO redirect base, and to absolutize the deep links in notifications. If unset, the device-auth + well-known endpoints derive it per-request from the Host / X-Forwarded-* headers; set it explicitly behind a proxy that doesn't forward Host.
TRUSTED_PROXIES=                      # comma-separated IPs/CIDRs whose X-Forwarded-For/X-Real-IP gin trusts for c.ClientIP() (every per-IP rate limiter keys on it, and `/api/report` stores it on every session as `client.ip`). Unset = loopback only, so no peer can spoof its IP. Private ranges are deliberately NOT trusted by default: on a Docker/k8s network every peer has a private address, the clients included when the port is published directly, so trusting RFC1918 would let any of them forge XFF past every per-IP limiter. Set it to the CIDRs of whatever sits in front (a proxy container's compose network, the ingress pod range, the CDN ranges); otherwise every visitor shares one rate-limit bucket keyed on the proxy. The value REPLACES the default, so it must name every hop in the chain, not just the outermost (a CDN-only list leaves a private ingress untrusted and disables XFF entirely); `*` trusts every peer. A malformed entry exits 1 at startup with a FATAL message, before the database is opened (`trustedProxyNets`, cmd/run.go). The first request carrying X-Forwarded-For/X-Real-IP from a peer outside the list logs a one-time warning naming that peer (`warnOnUntrustedForwardedFor`, cmd/run.go), which is how a forgotten proxy hop shows up. Parsing lives in `Cfg.TrustedProxyList` (config.go), covered by config_test.go.
TRUSTED_PROXY_HEADER=                 # optional header taken verbatim as the client IP on every request (gin TrustedPlatform: X-Real-IP behind ingress-nginx, CF-Connecting-IP behind Cloudflare); it wins over TRUSTED_PROXIES and turns the untrusted-forwarder warning off. Only safe when every request provably passes through a proxy that overwrites the header.
UPLOAD_MAX_CONCURRENT=4               # concurrent /api/sourcemaps/upload + /api/symbols/upload requests; excess waits 30s then gets 503. Separate from INGEST_MAX_CONCURRENT so a CI upload burst cannot starve telemetry ingest. Each upload holds a whole file in memory while parsing, so this bounds peak upload memory at roughly concurrency x largest file. It is a count-based gate, not a memory budget: on a memory-capped container size it from the limit rather than trusting the default. Same gate and bounded waiting room as ingest, its own instance. The source-map warm-up that follows an upload (`services.GenerateTWArtifacts`) runs outside this gate on its own 2-worker pool with a 64-job queue: the stale `.tw` artifact is deleted and the cache invalidated inline, the rebuild is queued, identical pending jobs are coalesced, and a full queue drops the warm-up (rate-limited `CaptureException`) because lookups build a missing artifact on demand anyway.
REPORT_MAX_BODY_MB=64                 # cap on the DECOMPRESSED /api/report and /api/profiles/ingest body (middleware.UseGzip bounds both the raw and gunzipped stream, so gzip cannot raise it). Deliberately far above the 10MB OTLP cap: session-replay frames carry rrweb segments and are a different size class, and the shipped browser SDKs re-queue a rejected batch forever, so a reachable cap is a permanent wedge rather than shed load. Overruns answer 413.
CLICKHOUSE_SERVER=localhost:9000
CLICKHOUSE_DATABASE=traceway
CLICKHOUSE_USERNAME=default
CLICKHOUSE_PASSWORD=
CLICKHOUSE_TLS=false
POSTGRES_HOST=localhost
POSTGRES_PORT=5432
POSTGRES_DATABASE=traceway
POSTGRES_USERNAME=traceway
POSTGRES_PASSWORD=
POSTGRES_SSLMODE=disable

# DuckDB telemetry backend (only with -tags telemetry_duckdb build; see "DuckDB Telemetry Backend" below). The embedded Run(opts...) path honors these three and DUCKDB_RETENTION_DAYS through applyEnvOverrides.
DUCKDB_MEMORY_LIMIT=                  # e.g. 4GB. Unset = DuckDB auto-tunes (~80% RAM). Set explicitly in memory-capped containers to avoid OOM.
DUCKDB_THREADS=                       # e.g. 4. Unset = DuckDB auto-tunes (= cores). Cap in constrained/shared environments.
DUCKDB_CHECKPOINT_THRESHOLD=          # e.g. 256MB. Unset = DuckDB default (16MB). Raise under sustained ingest to reduce WAL checkpoint stalls; costs a larger WAL and longer restart replay.

# Ingest admission gate (all telemetry ingest endpoints: /api/report, /api/profiles/ingest, /api/otel/*)
INGEST_MAX_CONCURRENT=                # max concurrently processed ingest requests. Unset = 2×CPU cores, min 4. Bounds ingest memory so overload sheds load with 503s instead of the process being OOM-killed (on DuckDB an OOM death is followed by a minutes-long WAL-replay stall on restart). The gate (`newAdmissionGate` in `middleware/ingest_admission.go`, a buffered-channel semaphore) also bounds waiters at 4×capacity (min 16); beyond that a request gets an immediate 503 instead of parking a goroutine and a timer for the wait window.
INGEST_ADMISSION_WAIT_SECONDS=5       # how long a request may wait for a slot before the 503 + Retry-After; 0 = reject immediately when saturated

# Email
EMAIL_PREVIEW_ENABLED=false           # "true" registers GET /api/email-preview[/:template], rendering every email template with sample data for design review. Off by default; never enable in production.

# Notifications
NOTIFICATION_POLL_SECONDS=60          # polled rule evaluation interval; minimum 5, invalid values fall back to 60
ONCALL_POLL_SECONDS=30                # on-call escalation worker interval; minimum 5, invalid values fall back to 30. Kept separate from NOTIFICATION_POLL_SECONDS so raising rule-evaluation intervals never delays paging. A buffered Wake() channel makes freshly opened pages notify L1 near-instantly regardless of this interval.
OUTBOX_POLL_SECONDS=15                # notification outbox drain interval; minimum 5, invalid values fall back to 15. The outbox (backend/app/outbox, notification_outbox table in the main DB) is the persist-then-send layer for ALL notifications: rule dispatch and the escalator only enqueue (with an adapter-config snapshot) inside their transactions; the drain worker sends with retries (backoff 1m/5m/15m/60m, 5 attempts, then terminal failed + CaptureException). Crash-safe at-least-once: stale 'sending' rows are reclaimed after 5 min, cancelled rows can never resurrect (guarded status transitions), ack/resolve cancels queued page deliveries via outbox.CancelByKey. Cooldown and event-rule dedup record at enqueue commit (the durable promise), and fired_notifications is written at the terminal outcome. /api/health/deep exposes an `outbox` block; `traceway.outbox.*` metrics are emitted when monitoring is on; terminal rows are pruned daily (sent/cancelled 7d, failed 30d).
SLO_POLL_SECONDS=60                   # SLO worker interval (backend/app/slo): recomputes the last 10 minutes of every enabled SLO into the telemetry `slo_buckets` table (every 15 minutes, in a slot derived from the SLO id and decided against its persisted `computed_at` (`passDue`, `slo/schedule.go`) so passes are spread over ticks and a restart or shard move neither skips nor bunches them, the other kinds also recompute the last 75 minutes, for long requests and retried exports, and session and task SLOs their exception slack plus 25 minutes, since a crash or failure lands after the bucket of its start; once a day session SLOs recompute the last 72 hours through the gap fill, for native crashes uploaded on the next launch), computes `SLO_COMPUTE_WORKERS` (default 4, max 64) SLOs at a time with a per-SLO `SLO_COMPUTE_TIMEOUT_SECONDS` (default 30; a timed-out SLO keeps its old snapshot and resumes from `last_bucket_at` on the next tick), works through the SLOs in groups of 200 so memory stays bounded and each group's buckets go out in one batched insert stamped at write time (ClickHouse makes a part of every insert); the window sum is incremental (`slo/snapshot_cache.go`): the part of the window before an anchor 7h back (73h for session SLOs) is cached in memory and only loses the buckets that slide out, the recent part is read every tick, and the whole window is re-read when the cache is missing, the revision or `computed_at` differs from what this process wrote (an edit, or another instance computed it), a bucket was written before the anchor, or once an hour in the SLO's slot; an SLO with no `last_bucket_at` (new, or reset by a definition change) has its buckets deleted by the worker before anything is written, never by the controller, so a delete cannot land after the new buckets; backfills older history in 6h chunks (24h for metric SLIs so ClickHouse reads the 1m rollups) under one 5s budget shared by every SLO of the tick (start rotated per tick), records an outage longer than the 24h catch-up as a `gap_from`/`gap_to` range it fills under the same budget, writes the snapshot columns on `slos` only while the row is still at the `revision` it loaded (an edit during a tick makes it drop its result; the reset of a definition change clears the buckets on the next tick), on Postgres splits the SLOs over 64 shards (`id % 64`) leased per instance through `slo_workers` (heartbeat) and `slo_shard_leases` (pg/0157, pg/0158; `slo/shards.go`): each tick an instance renews its leases, releases any above its fair share (`ceil(64 / live instances)`) and claims free or expired ones, renews again between groups, leases last max(5 min, 3x the poll) and a stopped instance's shards move once it ages out, shard 0's owner runs the daily prune; SQLite owns all shards, counts metric slices without samples as bad only from `sli_since` (set on create and on every definition change), rounds slo_burn_rate windows up to whole slices and clears it when either window drops below the threshold, slo_budget_consumed reads the snapshot's `budget_remaining` instead of re-summing the window, rule state moves through `NotificationRuleStateRepository.Transition` (a conditional update, so two instances never both notify), both slo_* rules hold their state while the SLO has no `last_bucket_at` (a definition change being recomputed) so a reset never sends a false recovery, a recovery notice does not record a cooldown fire, prunes buckets older than 120 days daily, and reports the `slo` block of `/api/health/deep` (totals incl. timeouts, `runningForSec`, owned shards and instances, the last tick's phase timings with its slowest SLO, and `freshness` read from the main DB: enabled, stale > 5 min, max staleness, backfill pending), which `monitoring.StartSLOReporter` emits every 30s as `traceway.slo.*` metrics when self-monitoring is on (documented in `docs/pages/server/capacity-monitoring.mdx`); minimum 5, invalid values fall back to 60. `slo.Wake()` runs a tick right after an SLO is created or changed. Buckets are deliberately NOT in the telemetry retention list; slo_* notification rules read them (never raw data) and keep their firing state in `notification_rule_states` so they can send recovery notices across restarts.
SLO_COMPUTE_WORKERS=4                 # SLOs computed at once per instance (max 64)
SLO_COMPUTE_TIMEOUT_SECONDS=30        # per-SLO budget for its raw queries and window sums in one tick; a timeout is counted (traceway.slo.timeouts.delta), captured as an exception naming the SLO, and retried next tick

# Synthetics (synthetic uptime monitoring: backend/app/synthetics, /synthetics frontend route)
SYNTHETICS_POLL_SECONDS=15            # scheduler tick for due checks; minimum 5, invalid values fall back to 15. The scheduler enqueues due checks into the check_runs queue (main DB, outbox-style guarded claims, advisory lock 824737004) and records expired queued runs as `missed` in telemetry — a probe is never executed late. In-process executors claim http/tcp runs always, browser runs only when mode=embedded.
SYNTHETICS_BROWSER_MODE=off           # off | embedded | remote. Browser checks are real @playwright/test specs executed by spawning Node against a harness dir (no npm/npx at runtime; allowlisted env so user scripts never see server secrets). `embedded` requires the :browser image (Dockerfile.browser, DuckDB base + Node + Chromium) and fails fast at startup otherwise; `remote` queues browser runs for traceway-runner binaries that long-poll /api/runners/poll authenticating with SYNTHETICS_RUNNER_SECRET. Hard-blocked in cloud mode (startup panic + 422 at check creation).
SYNTHETICS_BROWSER_SANDBOX=auto       # auto | bwrap | off. Isolation for the Playwright subprocess in embedded mode and in traceway-runner. The env allowlist alone does not contain a spec: node runs as the same uid as its parent, so unconfined it can read the parent's /proc/<pid>/environ (the runner secret), overwrite node_modules in the shared harness to backdoor later runs, and read a sibling run's directory. `bwrap` gives each run its own pid/ipc/uts/user namespace with a fresh /proc, a read-only harness bind with only that run's dir writable, an empty tmpfs over the .runs container so sibling run dirs are hidden (not exposed by the wholesale harness bind), and no /etc beyond DNS/CA/locale (so a secret in the unit file or EnvironmentFile is unreachable); the network namespace stays shared since probes must reach the internet. `auto` uses bubblewrap when the host can and degrades to off with a warning; explicit `bwrap` fails fast at startup instead. Resolved once via browserexec.ResolveSandbox, which probes by running `node --version` through the real bind set.
SYNTHETICS_RUNNER_SECRET=             # shared bearer credential for the operator's runner fleet; required for remote mode (fail-fast at startup). Runners are deployment infrastructure, NOT tenant entities: no dashboard CRUD, no per-runner tokens. A runner self-registers a liveness row in synthetic_runners under its X-Traceway-Runner-Name on first poll (upsert throttled to 1/min via an in-memory cache; claim identity is "runner:<name>"). Rotation = change the secret + restart runners. Runner-side env: TRACEWAY_URL, TRACEWAY_RUNNER_SECRET, TRACEWAY_RUNNER_NAME (default hostname), RUNNER_WORKERS (default 2, max 16), and optional TRACEWAY_RUNNER_MONITORING (<project_token>@<url>/api/report) which self-instruments the runner with the Traceway Go SDK: default server metrics, a "browser_run" task trace per executed run (tags check_id/run_id/status, server_name = runner name), and CaptureException on infra failures (start/report errors, per-job panics recovered without killing the worker).
HEALTH_DEEP_TOKEN=                    # operator bearer secret gating GET /api/health/deep (its payload is instance-wide: cross-tenant queue depth, runner fleet, storage engine stats). Unset = endpoint disabled with a 401 pointing here.
SYNTHETICS_HTTP_CONCURRENCY=8         # concurrent in-process http/tcp probes
SYNTHETICS_BROWSER_CONCURRENCY=2      # concurrent Chromium instances in embedded mode (~300-500MB each)
SYNTHETICS_ALLOW_PRIVATE_TARGETS=true # "false" rejects checks that resolve to private/LAN addresses, validated at save AND at dial time for both http and tcp probes (netguard.GuardedDialContext dials the vetted IP, DNS-rebinding safe). Forced false in cloud mode regardless of env (tenants must never probe the platform's network). Default allow elsewhere: probing LAN services is a core self-hosted use case. When the guard is off, http probes honor HTTP(S)_PROXY; when on, the proxy is ignored so the guard vets the real target. Check creation also runs controllers.CheckLimitHook (nil = unlimited; cloud caps monitors per org plan).
SYNTHETICS_PLAYWRIGHT_DIR=/opt/traceway-playwright   # Playwright harness dir (node_modules with pinned @playwright/test; docker/playwright/package.json pins the version)
SYNTHETICS_SCREENSHOT_RETENTION_DAYS=30 # browser failure artifacts under STORAGE_PATH/synthetics/ (.png screenshots AND .log Playwright output tails — failed browser runs store both, referenced by screenshot_key/output_key on check_results, served via GET /api/synthetics/screenshot|output?key= with project prefix checks); local storage only, 0 disables the cleanup worker

# Retention (see "Data Retention" section below)
SQLITE_RETENTION_DAYS=30              # 0 to disable; only applies in SQLite mode
DUCKDB_RETENTION_DAYS=30              # telemetry TTL on the DuckDB backend; when set it wins over SQLITE_RETENTION_DAYS there, when unset SQLITE_RETENTION_DAYS applies as fallback; 0 to disable
LOG_RECORDS_MAX_ROWS=                 # optional cap on log_records rows; unset/0 = disabled. NOT a hard limit: a cleanup worker trims to the newest N rows once per minute, so ingest above N rows/minute overshoots the cap between runs. Only applies in SQLite mode (SQLite or DuckDB telemetry)
SESSION_RECORDING_RETENTION_DAYS=30   # 0 to disable; only applies when STORAGE_TYPE=local
PROFILE_ARCHIVE_RAW=false             # native pprof ingest only: write the original pprof bytes to object storage as a lossless archive
PROFILE_RETENTION_DAYS=30             # 0 to disable; on-disk archive TTL, only with PROFILE_ARCHIVE_RAW + STORAGE_TYPE=local

# V2 trace tables move-over (see "Trace Storage (V2 tables)" below)
V2_MOVE_OVER=                         # "true" copies the history of the tables V2 replaced into the V2 tables, in the background, newest day first. Runs at startup and one hour after each pass; resumable (progress in the transactional DB v2_move_over_days, sqlite/0077 + pg/0143), so it can stay set. Finish upgrading old writers before enabling it: completed source-table/date pairs are skipped. With several backend instances set it on exactly one. On SQLite and DuckDB it stops at the telemetry retention window, since older days would be pruned at once.
V2_MOVE_OVER_OLDEST=                  # optional YYYY-MM-DD; days before it stay in the old tables. An unparsable value logs and does not start the move.
V2_MOVE_OVER_WORKERS=1                # source days copied concurrently inside the one enabled instance (max 16; invalid/non-positive = 1). Each day keeps table order; days are scheduled newest first. Increase to 4 for parallel backfill when the telemetry DB has spare capacity. Progress rows and logs update after successful batches roughly every 10s; slow queries/inserts can delay updates. Errors cancel and join all workers before the next hourly pass.

# Session recording uploads (see "Session Recording Uploader" section below)
SESSION_RECORDING_UPLOAD_WORKERS=32   # 0 to disable uploads entirely
SESSION_RECORDING_UPLOAD_QUEUE_SIZE=2048

# Source map symbolicator
SYMBOLICATOR_PARSER=goja              # goja (default) or oxc (requires -tags oxc build, see scripts/build-oxc-shim.sh)
SOURCEMAP_CACHE_TYPE=memory           # memory (default) or disk (mmap-backed .tw cache)
SOURCEMAP_DISK_CACHE_PATH=./twcache   # only used when SOURCEMAP_CACHE_TYPE=disk
SOURCEMAP_DISK_CACHE_MAX_MB=2048      # capacity-based LRU eviction of local .tw files

Architecture Overview

Data Flow

Go Application → [traceway SDK] → GZIP POST /api/report → Backend → ClickHouse
                                                              ↓
Dashboard ← [SvelteKit Frontend] ← JSON API ← Gin Controllers

Authentication

Two-tier system:

  1. Client Auth: Project bearer tokens (SDK telemetry via Authorization: Bearer <project_token>)
  2. App Auth: JWT-based user authentication (dashboard via Authorization: Bearer <jwt_token>)

App Auth credentials. UseAppAuth accepts three credential shapes on the Authorization: Bearer header:

  • Dashboard JWT — issued by /api/login / SSO (7-day expiry).
  • Personal access token (PAT) — opaque twp_-prefixed token; looked up by SHA-256 hash in personal_access_tokens, resolves to its user. Created/listed/revoked from the account page (/api/personal-access-tokens*). Non-expiring or with an optional TTL; last_used_at is touched (throttled to 1/min, off the request path).
  • Device-flow access token — a short-lived (15-min) JWT minted by the CLI's OAuth device flow.

Agent scopes and the agent gate. Every credential except a dashboard session is an agent credential with a scope, traceway:read or traceway:write (models.ScopeRead/ScopeWrite). PATs store it (personal_access_tokens.scope; read tokens are twp_, write tokens tww_ and must expire within 90 days), refresh tokens and authorization codes store it, and access JWTs carry it as the scope claim (plus cid, the OAuth client). Device logins take their scope from the client id (knownDeviceClients: traceway-cli and traceway-mcp read, traceway-cli-write write); the authorization-code grant takes it from the request's scope param (models.ParseRequestedScope, default read); a refresh keeps the family's scope; POST /api/login with a scope field returns a token set instead of a session (token is a 15-minute scoped access JWT, plus refreshToken/expiresIn in a refresh family with client id traceway-cli or traceway-cli-write, so POST /api/auth/logout revokes it; the CLI's password login always sends a scope and stores the result like a device login). An unscoped JWT that lives one hour or less predates scopes and is held to read. UseAppAuth sends agent credentials through authorizeAgentRequest (middleware/agent_gate.go): the route must be in the allowlist agentRoutes (middleware/agent_routes.go, keyed METHOD /api/path by c.FullPath(), default deny, so new dashboard routes are never agent-reachable by accident; TestAgentRoutesAreRegistered keeps entries pointing at real routes); a write route needs the write scope (else 403 with a scope="traceway:write" challenge) and the owning organization's opt-in organizations.agent_write_enabled (Settings > Agent Access, PUT /api/organizations/:organizationId/agent-access, else 403), resolved from ?projectId= together with membership, which is why every allowlisted write is project-scoped. Accepted and opt-in-refused agent writes are recorded in agent_write_events (main DB, pruned after 90 days by the auth-tokens worker, listed at GET /api/organizations/:organizationId/agent-activity). The role checks (RequireWriteAccess etc.) still apply after the gate. Dashboard sessions skip the gate entirely. Docs: docs/pages/learn/agent-write-access.mdx.

CLI / OAuth device flow (backend/app/services/authserver/, controllers device_auth.controller.go / wellknown.controller.go / pat.controller.go): RFC 8628 device authorization grant plus rotating refresh tokens. traceway login (default) → POST /api/auth/device/authorize (client_id allowlisted, per-IP rate-limited, opportunistically prunes expired rows) → user approves at /device → the CLI polls POST /api/auth/device/token (grant device_code; /api/auth/token is an equivalent alias) which issues a 15-min access token + 90-day rotating refresh token (family-tracked in refresh_tokens). Refresh (grant_type=refresh_token) rotates the token atomically and revokes the whole family on genuine reuse; within a 30s grace window a benign concurrent retry is answered with the same rotated token set (from an in-memory rotation cache) instead of invalid_grant. POST /api/auth/logout revokes a family server-side. Tokens are stored SHA-256-hashed. The grant endpoints self-manage their transactions via db.ExecuteTransaction (not middleware.Transactional) because OAuth returns 400 for normal flow control (authorization_pending, reuse-revoke) and those side effects must still commit. /.well-known/oauth-authorization-server and /.well-known/oauth-protected-resource are served at the origin root (registered in cmd/run.go, not the /api group) per RFC 8414 / 9728. Beyond the device grant, the server implements the authorization-code + PKCE grant (S256 only) with RFC 7591 dynamic client registration for MCP clients: POST /api/oauth/register (rate-limited, open registration of public clients; https/custom-scheme redirect URIs anywhere, plain http only on loopback with port-flexible matching per RFC 8252) -> the client sends the user to /oauth/authorize (an SPA consent page like /device) -> the page calls GET /api/oauth/client + POST /api/oauth/approve|deny (approve mints a single-use 5-min twa_ code bound to user/client/redirect/challenge; the redirect target is validated server-side) -> the client exchanges it at POST /api/auth/token (grant authorization_code; the code is consumed even on a failed exchange, wrong verifier/client/redirect are all invalid_grant). RFC 8707 resource params are validated against the issuer origin (invalid_target). Expired codes are pruned by the auth-tokens retention worker and opportunistically at approve time.


Frontend (/frontend)

Framework & Build

  • Framework: SvelteKit 2 with Svelte 5 runes API
  • Styling: Tailwind CSS v4 with shadcn-svelte components
  • Build: Vite 7, static adapter with SPA fallback
  • SSR: Disabled - pure client-side SPA (ssr = false in +layout.ts)

Project Structure

frontend/
├── src/
│   ├── routes/              # SvelteKit pages
│   ├── lib/
│   │   ├── api.ts           # API client with auth
│   │   ├── state/           # Svelte 5 state management
│   │   ├── components/
│   │   │   ├── ui/          # shadcn-svelte base components
│   │   │   └── traceway/    # Custom Traceway components
│   │   └── utils/           # Helpers (formatting, sorting)
│   └── app.css              # Tailwind + global styles
├── static/                  # Static assets
└── svelte.config.js         # SvelteKit config

State Management (Svelte 5 Runes)

Runes Pattern

// Use $state() for reactive state
let data = $state<Type>(initial)

// Use $derived() for computed values
let computed = $derived(expression)

// Use $effect() for side effects
$effect(() => { /* reactive code */ })

State Files

File Purpose Persistence
src/lib/state/auth.svelte.ts Token auth, login/logout localStorage
src/lib/state/projects.svelte.ts Multi-project management localStorage
src/lib/state/theme.svelte.ts Dark/light mode toggle localStorage
src/lib/state/timezone.svelte.ts UTC/local timezone toggle localStorage

Singleton Pattern

State files export class instances as singletons:

// src/lib/state/auth.svelte.ts
class AuthState {
    token = $state<string | null>(null)
    isAuthenticated = $derived(!!this.token)

    constructor() {
        // Load from localStorage on init
        this.token = localStorage.getItem('token')
    }
}
export const authState = new AuthState()

API Client (src/lib/api.ts)

The API client automatically:

  • Includes Authorization: Bearer <token> header
  • Adds projectId as a query parameter to all requests
  • Handles 401 responses by logging out and redirecting to /login
// Usage
const data = await api.post<ResponseType>('/endpoint', { body })
const data = await api.get<ResponseType>('/endpoint')

Component Patterns

Table System

Tables use shadcn-svelte base components with custom Traceway wrappers:

Component Location Purpose
Table, TableHeader, etc. src/lib/components/ui/table/ Base shadcn components
TracewayTableHeader src/lib/components/traceway/traceway-table-header.svelte Adds sorting + tooltips
TableEmptyState src/lib/components/traceway/table-empty-state.svelte Empty state display
PaginationFooter src/lib/components/traceway/pagination-footer.svelte Pagination controls

Sorting Storage

Table sorting persists to localStorage using a consistent pattern via src/lib/utils/sort-storage.ts:

// Types
type SortState = { field: string; direction: 'asc' | 'desc' }

// Key format: traceway_sort_{pageKey}
// Example: traceway_sort_issues, traceway_sort_endpoints

// In +page.svelte - load initial state
let sortState = $state<SortState>(getSortState('issues', { field: 'last_seen', direction: 'desc' }))

// After sort change
function onSortClick(field: string) {
    sortState = handleSortClick(field, sortState.field, sortState.direction, 'desc')
    setSortState('issues', sortState)  // Persist to localStorage
}

Available functions (src/lib/utils/sort-storage.ts):

Function Description
getSortState(pageKey, defaultState) Load sort state from localStorage
setSortState(pageKey, state) Save sort state to localStorage
handleSortClick(field, currentField, currentDirection, defaultDirection) Toggle sort direction, returns new SortState

TracewayTableHeader Component

<TracewayTableHeader
    label="Last Seen"
    column="last_seen"
    tooltip="When this issue was last reported"
    {orderBy}
    onclick={() => handleSortClick('last_seen')}
/>

URL State Management

Time range and filters persist in URL query params via src/lib/utils/url-params.ts:

// Available presets: 30m, 60m, 3h, 6h, 12h, 24h, 3d, 7d, 1M, 3M

// Parse time range from URL (in +page.svelte)
const timeRange = parseTimeRangeFromUrl(timezoneState.timezone, '24h')

// Get resolved Date objects for API calls
const { from, to } = getResolvedTimeRange(timeRange, timezoneState.timezone)

// Update URL with new time range (preserves other params)
updateUrl({ preset: '7d' })
updateUrl({ from: customFrom, to: customTo })  // Custom range

Available functions (src/lib/utils/url-params.ts):

Function Description
parseTimeRangeFromUrl(timezone, defaultPreset) Parse TimeRangeParams from current URL
getResolvedTimeRange(params, timezone) Convert params to { from: Date, to: Date }
updateUrl(params, options?) Update URL query params, optionally replace history

Navigation Utilities

Helper functions for preserving URL params during navigation (src/lib/utils/navigation.ts):

// Add sticky params (like time range) to href for <a> tags
const href = addStickyParamsToHref('/issues/abc123', 'preset', 'from', 'to')
// Result: "/issues/abc123?preset=24h" (if preset=24h is in current URL)

// Create click handler for table rows that preserves params
const handleClick = createRowClickHandler('/issues/abc123', 'preset', 'from', 'to')

Available functions:

Function Description
addStickyParamsToHref(href, ...stickyParams) Returns href with specified params from current URL
createRowClickHandler(href, ...stickyParams) Returns click handler that navigates with sticky params

Routes

/                           Dashboard (protected) - overview metrics
/login                      Login page (public)
/register                   Registration page (public)
/issues                     Issues list with filtering/sorting
/issues/[hash]              Exception details view
/issues/[hash]/events       Exception events timeline
/endpoints                  Endpoint analytics with P50/P95/P99
/endpoints/[endpoint]       Single endpoint details
/tasks                      Background tasks list
/tasks/[task]               Single task details
/spans                      Span explorer: every OTel span of the project, filtered by name, service, kind, status, duration, trace ID and attribute values (time range defaults to 60m)
/spans/[traceId]            Whole trace waterfall across services and across every project of the organization (?at= is required, ?span= highlights one span), with the current project's correlated logs on backend projects
/dashboards                 Dashboards page (tabs of org dashboards; /metrics redirects here)
/monitors                   Monitors (synthetic checks): single sidebar item, TabsRow tabs Monitors | Status Pages | Post-Mortems (?tab=, Status Pages admin-only, Post-Mortems for all members); status pages tab has branding (description, logo upload, custom domain) plus a link to the per-page incidents page (which owns the Record Incident dialog); old /monitors/status-pages redirects to the tab, /monitors/runners to /monitors (runners have no UI — operator infra), bare /monitors/post-mortems to the tab
/monitors/status-pages/[pageId]/incidents  Paginated incident management for one status page (timeline updates, titles, manual resolve/delete, post-mortem links); reads for members, mutations admin-gated in the UI
/monitors/[checkId]         Monitor detail (latency chart, uptime bars, runs w/ result filter, incidents w/ titles + post-mortem links)
/monitors/post-mortems/[id] Post-mortem editor/viewer (@milkdown/crepe WYSIWYG with a Rich text/Markdown raw toggle, tag pills + Add-tags dialog, incident link pill + searchable picker, Activity sheet; write gating follows the effective project role). Project-scoped: opening a doc under a different project 404s. Creation is a title-only dialog (new-post-mortem-dialog.svelte, openable from the tab, monitor detail incidents, and the status page incidents page with an optional incident pre-link) that POSTs immediately and navigates here; /monitors/post-mortems/new redirects to the tab
/slos                       SLOs (service level objectives, `backend/app/slo`): one row per objective with kind badge, SLI summary, target, window, current SLI, error-budget bar, 1h burn rate and status; sort key `slos`; sidebar badge = breached count; frontend and mobile projects see only the `session` kind (crash-free sessions, `sessions` joined to `exceptions_v2.session_id`), backend projects the other five
/slos/[id]                  SLO detail: SLI and error-budget charts over a range clamped to the window (default = the whole window), burn-rate tiles, a coverage note while history backfills, a "View bad events" deep link per kind, and the alert rules that reference the SLO
/status/[slug]              Public status page (light standalone design, no auth, raw fetch, listed in isPublicPath)
/connection                 SDK integration guide

UI Components

Location: src/lib/components/ui/* Uses shadcn-svelte registry with bits-ui primitives. Key components:

  • button, card, table, badge, tooltip
  • select, input, checkbox
  • sheet (slide-out panels), dialog (modals)

Backend (/backend)

Architecture

  • Framework: Gin Gonic HTTP framework
  • Database: ClickHouse (columnar OLAP for telemetry), PostgreSQL (relational for projects)
  • Port: 8082
  • Pattern: Repository pattern with singleton controllers

Project Structure

backend/
├── traceway.go                 # package tracewaybackend - embeddable API (Run, WithPort, ...)
├── cmd/
│   ├── run.go                  # Run() implementation: DB init, routes, server start
│   ├── traceway/main.go        # Binary entry point (package main)
│   └── traceway-runner/        # Synthetics remote runner binary
├── app/
│   ├── controllers/
│   │   ├── routes.go           # Route registration
│   │   ├── dashboard.go        # Dashboard metrics
│   │   ├── auth.go             # Login handler
│   │   ├── projects.go         # Project CRUD
│   │   └── clientcontrollers/
│   │       └── report.go       # Telemetry ingestion (/api/report)
│   ├── repositories/           # ClickHouse queries
│   │   ├── transactions.go
│   │   ├── exceptions.go
│   │   ├── metrics.go
│   │   └── projects.go
│   ├── models/                 # Data structures
│   ├── middleware/
│   │   ├── auth.go             # Token validation
│   │   └── gzip.go             # Request decompression
│   ├── cache/                  # In-memory project cache; on Postgres, replicas refresh each other through LISTEN/NOTIFY (project_cache_changed, sent by the project repository inside the write transaction)
│   ├── pgdb/                   # PostgreSQL connection manager
│   └── migrations/
│       ├── ch/                 # ClickHouse migrations
│       └── pg/                 # PostgreSQL migrations

Middleware Chain Composition

Route Type Middleware Chain
Read-only telemetry UseAppAuth, RequireProjectAccess
Write telemetry UseAppAuth, RequireProjectAccess, RequireWriteAccess
PostgreSQL CRUD (read) UseAppAuth, RequireProjectAccess, Transactional
PostgreSQL CRUD (write) UseAppAuth, RequireProjectAccess, RequireWriteAccess, Transactional
Admin org management UseAppAuth, RequireAdminAccess, Transactional
Public (auth/invitations) RateLimitPerIP, BufferAuthBody, Transactional (limiter first so a throttled request never reads a body or opens a transaction)
Client SDK ingestion CORSReport, UseClientAuth, UseGzip
Synthetic runner API UseRunnerAuth only (shared SYNTHETICS_RUNNER_SECRET bearer + self-registration by X-Traceway-Runner-Name; no Transactional — poll holds the request up to 25s)
Public status page RateLimitPerIP only

Global middleware (registered in cmd/run.go ahead of the routes): middleware.GuardBodyReads (every request body gets the progress guard, 30s idle / 10 min total / 1 KB/s floor after 20s, answering through middleware.BodyReadError as 408; routes with their own budget call GuardBodyRead again, which re-arms the same guard rather than wrapping twice, so uploads get 30s/30 min, /api/report 20s/10 min and OTLP 20s/2 min. The guard clears the socket read deadline the moment the body is complete: Go's background connection read would otherwise time out and cancel the request context of any handler that outlives the idle window. The http.Server ReadTimeout is a 1h backstop that only applies while a body is still arriving (the guard lifts it for bodiless requests too, so a long-lived handler such as the MCP GET /mcp stream is not cancelled after an hour), ReadHeaderTimeout is 15s), configureClientIP (gin's trusted proxies from Cfg.TrustedProxyList, plus TrustedPlatform from TRUSTED_PROXY_HEADER; a malformed list panics), warnOnUntrustedForwardedFor (list mode only: one-time log when a forwarding header arrives from a peer outside TRUSTED_PROXIES), and middleware.SecurityHeaders (X-Frame-Options: DENY, CSP frame-ancestors 'none'; base-uri 'self'; form-action 'self', Referrer-Policy, nosniff). The two frame headers are skipped for any /status/* or /api/status/* path so public status pages stay embeddable in an iframe. A custom-domain status host is exempt as a whole: the SPA handler serves nothing but that status page there, so it drops the two headers itself (see the status domains row below).

API Endpoints

Client SDK Ingestion

Method Endpoint Auth Purpose
POST /api/report Client Telemetry ingestion (gzipped)
POST /api/otel/v1/traces Client OTLP/HTTP trace ingestion
POST /api/otel/v1/metrics Client OTLP/HTTP metric ingestion
POST /api/otel/v1/logs Client OTLP/HTTP log ingestion
POST /api/otel/v1development/profiles Client OTLP/HTTP profile ingestion (development signal)

Auth & Registration

Method Endpoint Auth Purpose
POST /api/login None Dashboard authentication
POST /api/register None New user registration
GET /api/has-organizations None Check if any orgs exist (self-hosted only)
POST /api/forgot-password None Request password reset email
GET /api/password-reset/:token None Validate reset token
POST /api/password-reset/:token None Reset password with token

CLI Device Auth & OAuth (see the Authentication section above)

Method Endpoint Auth Purpose
POST /api/auth/device/authorize None Start device flow; returns device/user code + verification URL
POST /api/auth/device/token None Poll for the token (grant device_code); also accepts refresh_token. Shares a 60/min per-IP limiter with /api/auth/token; over it a device_code grant answers the RFC 8628 400 slow_down (RateLimitOAuthTokenPerIP peeks grant_type from the buffered body), so the shipped CLI backs off instead of aborting the login, while every other grant gets a plain 429 + Retry-After because slow_down is undefined for them and OAuth clients treat an unknown 400 as terminal. Body capped at 64KB by BufferAuthBody like the other anonymous auth routes
POST /api/auth/token None Token endpoint: device_code, refresh_token or authorization_code grant (JSON or form-encoded). Same shared limiter, grant-aware rejection and 64KB body cap as the device token route
POST /api/auth/logout None Revoke the presented refresh token's family (idempotent)
GET /api/device App Look up a user code for the approval screen
POST /api/device/approve App Approve a pending device authorization (tokens carry only the approving user's own role, so no write guard)
POST /api/device/deny App Deny a pending device authorization
POST /api/oauth/register None RFC 7591 dynamic client registration (rate-limited)
GET /api/oauth/client App Resolve a client_id to its display name (consent page)
POST /api/oauth/approve App Approve an authorization request; mints the code, returns the validated redirect
POST /api/oauth/deny App Deny an authorization request; returns the error redirect
GET /.well-known/oauth-authorization-server None RFC 8414 metadata (served at origin root, not /api)
GET /.well-known/oauth-protected-resource None RFC 9728 metadata (served at origin root, not /api)
GET/POST/DELETE /mcp Bearer Streamable HTTP MCP server (origin root), read tools only for every credential; 401s carry a WWW-Authenticate resource-metadata challenge
GET/POST/DELETE /mcp/write Bearer (write scope) Same server with mcpserver.Config.AllowWrite: adds the mutating tools and the operations playbook. Tokens without traceway:write get 403 with a scope challenge, and so do dashboard sessions (BearerIdentity.HasAgentWriteScope), which would otherwise skip the agent gate's opt-in and audit log; resource metadata at /.well-known/oauth-protected-resource/mcp/write

Personal Access Tokens

Method Endpoint Auth Purpose
POST /api/personal-access-tokens App Create a PAT (returns the token once). scope is read (default, twp_) or write (tww_, expiresInDays 1-90 required)
GET /api/personal-access-tokens App List the current user's active PATs
DELETE /api/personal-access-tokens/:id App Revoke a PAT

Projects

Method Endpoint Auth Purpose
GET /api/projects App List projects
POST /api/projects App Create project (optional organizationId body field targets another org; the handler checks the caller's org role in the target org and 403s for non-members/readonly; creation is org-scoped, so per-project overrides neither grant nor deny it)
POST /api/projects/source-map-token App+Write Generate source map upload token

Dashboard

Method Endpoint Auth Purpose
GET /api/dashboard App Dashboard metrics
GET /api/dashboard/overview App Recent issues + top endpoints
POST /api/stats App Homepage stats

Metrics

Method Endpoint Auth Purpose
GET /api/metrics/application App Application metrics
GET /api/metrics/stats App Stats metrics
GET /api/metrics/server App Server metrics
POST /api/metrics/query App Custom metric queries. Aggregations avg/min/max/sum/count/last plus rate, which differences every series (full tag set = identity) against its previous sample, drops resets, and sums the deltas per bucket as a per-second rate; the reported unit gains /s (By/s), a counter of seconds becomes the ratio 1, and on ClickHouse rate follows selectTable like every other aggregation: past 6h it reads the rollups through maxMerge(max_val) (a counter only grows, so a rollup bucket's max is its last sample), never metric_points_1d, with the chart bucket clamped to the rollup step and the lookback widened by it (rateSource). The traceway-otel-agent template relies on it (plus state filters, groupBy: server_name, and the per-source complement flag that the widget renderer applies as 1 − value); backfill.RunOtelAgentDashboardSources upgrades installed copies whose widgets still carry the original plain-average source. Bounded in metric_query.controller.go: 256KB body, max 50 queries and 20 tag filters each, range at most 731 days and to >= from (422), buckets widened so no series exceeds 2000 points (the effective intervalMinutes is returned at the top level of the response), at most 200 groups per query (the repositories return one extra so the result carries truncatedGroups: true), and a 30s deadline for the whole request that answers 504. The discovery endpoints validate their from/to the same way
GET /api/metrics/discover App Discover available metrics
GET /api/metrics/discover/tags App Discover metric tags
GET /api/metrics/discover/instances App Distinct server_name values in a range (dashboard instance filter)
PUT /api/metrics/registry App+Write Update metric registry entry

Dashboards & Templates

Dashboards are org-owned JSON documents ({schemaVersion, widgets: [{id, title, widgetType, config}]}, widget ids are server-generated w_xxxxxxxx strings, array order = display order) applied to projects via project_dashboards. Dashboard mutations require org role above readonly (checked in-handler); project-scoped routes (list/star/reorder/populate) use the standard middleware chains; apply/unapply also check the effective role of each affected project. The old per-project widget-group tables are converted once at startup by backfill.RunDashboards (advisory-locked on PG) and retained for rollback.

Method Endpoint Auth Purpose
GET /api/dashboards App Dashboards applied to the project (tab order)
GET /api/dashboards/library App All dashboards across the user's orgs with applied project ids
POST /api/dashboards App Create in org (auto-applies to current project unless applyToProjectIds given)
GET /api/dashboards/:id App Meta + widgets (+ per-project isStarred, appliedProjectIds)
PUT /api/dashboards/:id App Update name/description and/or full definition (the as-code path)
DELETE /api/dashboards/:id App Delete everywhere (assignments + stars cascade)
PUT /api/dashboards/:id/apply App Set the full project assignment list
DELETE /api/dashboards/:id/apply/:projectId App Unassign from one project
POST /api/dashboards/:id/copy App Copy (also cross-org) with optional apply
PUT /api/dashboards/reorder App+Write Tab order for a project (explicit id order)
POST /api/dashboards/:id/widgets App Add widget
PUT /api/dashboards/:id/widgets/reorder App Reorder widgets (explicit id order)
PUT /api/dashboards/:id/widgets/:wid App Update widget
DELETE /api/dashboards/:id/widgets/:wid App Delete widget (+ its stars)
PUT /api/dashboards/:id/widgets/:wid/star App+Write Star/unstar for the project homepage
GET /api/dashboards/starred App Starred widgets with homepage layout
PUT /api/starred-widgets/reorder App+Write Reorder homepage starred widgets ({ids} = starred row ids)
PUT /api/starred-widgets/:id App+Write Update homepage layout (colSpan/size)
GET /api/dashboards/:id/export App Export one dashboard as JSON
GET /api/dashboards/export?organizationId= App Export the org bundle
POST /api/dashboards/import App Import doc/bundle (mode: create|upsert, upsert matches by name)
POST /api/dashboards/import/grafana App Convert a Grafana export (best-effort, returns warnings[])
GET /api/dashboard-templates App Marketplace list/search (search, category params)
POST /api/dashboard-templates/:key/install App Copy a template into the org and apply
POST /api/dashboards/populate-defaults App+Write Install the framework-default template set for an empty project
GET /api/metrics/discover/org App Metric names across all org projects (command palette)

Templates are DB rows seeded by migrations (traceway-otel-agent for the OTel host agent, golang for Go SDK apps, traceway-clickhouse/traceway-duckdb for the telemetry stores of a monitored Traceway instance; SQLite emits no store-specific metrics so it has no template); cloud can insert more rows without a release. The OTLP metric ingest allowlists per-resource identity and grouping tags (host.name, host.id, os.type, cloud.region, container.name, k8s.cluster.name, k8s.pod.name, k8s.node.name, postgresql.database.name, ...) in otelcontrollers/metric_converter.go for infrastructure views and custom widgets.

Endpoints

Method Endpoint Auth Purpose
POST /api/endpoints App List all endpoints
POST /api/endpoints/grouped App Endpoint aggregates (P50/P95/P99)
POST /api/endpoints/endpoint App Single endpoint details
POST /api/endpoints/chart App Stacked chart data
GET /api/endpoints/slow App Get slow endpoint threshold
POST /api/endpoints/slow App+Write Set slow endpoint threshold
POST /api/endpoints/:endpointId App Endpoint detail view

Tasks

Method Endpoint Auth Purpose
POST /api/tasks App List all tasks
POST /api/tasks/grouped App Grouped by task name
POST /api/tasks/task App Single task details
POST /api/tasks/:taskId App Task detail view

Exceptions

Method Endpoint Auth Purpose
POST /api/exception-stack-traces App Grouped exception list
POST /api/exception-stack-traces/archive App+Write Archive exceptions
POST /api/exception-stack-traces/unarchive App+Write Unarchive exceptions
POST /api/exception-stack-traces/by-id/:exceptionId App Single exception by ID
POST /api/exception-stack-traces/:hash App Exception by hash

AI Traces & Conversations

One ai_traces_v2 row per LLM call (any OTLP span with gen_ai.* attributes). Each row carries a conversation_id resolved at ingest (gen_ai.conversation.id -> session.id span/resource attr -> trace id), tool_call_count/tool_names parsed from the completion payload (OpenAI tool_calls, Anthropic tool_use, OTel output messages, gen_ai.tool.* fallback), and flagged/flagged_terms from an ingest-time word-boundary scan of prompt+completion against per-project selected built-in language packs (projects.ai_flagged_languages, default ["en"]; packs live in backend/app/services/contentflag/terms/*.txt — en/de/es/fr/it/pt/sr; empty array = custom terms only) plus per-project custom terms (projects.ai_flagged_terms); both edited in the project settings AI tab and cached via the project cache. All three project-creation paths (Create, CreateWithOrganization, cmd/seed.go) must set AiFlaggedLanguages explicitly since lit inserts every struct field. tool_names/flagged_terms are stored comma-separated (values sanitized at ingest); the content matcher lives in backend/app/services/contentflag/. Conversation analytics exclude rows with an empty conversation_id; user analytics additionally require a non-empty user_id.

Method Endpoint Auth Purpose
POST /api/ai-traces/grouped App AI traces grouped by trace name
POST /api/ai-traces/trace App Calls for one trace name (?traceName=)
POST /api/ai-traces/:traceId App Single call detail + conversation blob
POST /api/ai-conversations/grouped App Conversations (GROUP BY conversation_id): turns, cost, tokens, tools, models, flagged; filters userId/model/toolName/flaggedOnly/search (search matches conversation id, user, model, tool names, and flagged terms; row-level filters are semi-joins on conversation_id so a match on any turn returns the whole conversation's aggregates); response also carries thresholds (range-wide P95 cost/turns for outlier highlighting) and facets (models, tools)
POST /api/ai-conversations/conversation App All turns of one conversation (id in body) ordered by recorded_at, each with its stored input/output payload (capped at 200 turns of payloads), plus stats
POST /api/ai-users/grouped App Per-user conversation analytics: conversation count, total calls, avg/min/median turns, avg cost per conversation, total cost, flagged conversation count

Frontend routes: /ai-traces (tabs: Traces, Conversations, Users), /ai-traces/conversations/[conversationId] (chat timeline with tool calls rendered). Trace names conversations and users are shadowed by these static routes. Notification rule types ai_trace_cost (per call), ai_conversation_cost (24h cumulative per conversation), and ai_flagged_content (flagged term match, optional term filter) are event-driven; remember both notification_rule.repository.go copies list event rule types explicitly.

Organization Management

Method Endpoint Auth Purpose
POST /api/organizations App Create an organization owned by the caller; the recovery path for a user removed from their last organization (self-hosted: 422 once one exists; cloud: OrganizationLimitHook). Runs PostRegistrationHooks like register and SSO finish-setup
GET /api/organizations/:orgId/settings Admin Get org settings
PUT /api/organizations/:orgId/settings Admin Update org settings
PUT /api/organizations/:orgId/agent-access Admin {agentWriteEnabled}: allow agent write credentials to write in this org (settings GET returns the flag)
GET /api/organizations/:orgId/agent-activity Admin Paginated agent write log (who, credential, project, route, status)
GET /api/organizations/:orgId/members Admin List members
PUT /api/organizations/:orgId/members/:userId Admin Update member role
DELETE /api/organizations/:orgId/members/:userId Admin Remove member
GET /api/organizations/:orgId/members/:userId/project-roles Admin List member's per-project role overrides
PUT /api/organizations/:orgId/members/:userId/project-roles/:projectId Admin Set/clear a per-project role override

Organization Overview (frontend routes /organization = Servers, /organization/issues, /organization/monitors, /organization/on-call, /organization/projects)

Org-wide read views that fan out over every project of the org, in organization_overview.controller.go. RequireOrganizationAccess (any org role) gates them; the servers/issues/monitors handlers deliberately skip Transactional because they run per-project telemetry queries and must not hold the single-connection SQLite main DB across them (they open their own short db.ExecuteTransaction for the project list first). The pure main-DB pages endpoint keeps Transactional. The frontend renders the organization routes in the same sidebar shell as project pages, with the sidebar switched to organization items (Servers, Issues, Monitors, On-Call, Projects under an "Organization" label; badges come from /overview/counts). The selected organization is the shared organizationContext in frontend/src/lib/state/organization-context.svelte.ts (the organizationId query param, else the first membership), and src/routes/organization/+layout.svelte owns the redirects: a single-project org into that project, legacy ?tab= links to the new paths, and frontend-only orgs from Servers/Monitors to Issues. Login auto-lands on /organization when the first org has more than one project (frontend/src/lib/utils/landing.ts). A single-project organization never shows the view: /organization redirects into that project and the switcher renders the org as a plain label, since the project's own pages already cover everything it would aggregate. When none of the org's projects is a backend project (isBackendFramework in projects.svelte.ts, i.e. every project is a browser or mobile framework), the Servers and Monitors sidebar items are hidden and /organization redirects to /organization/issues.

Method Endpoint Auth Purpose
GET /api/organizations/:organizationId/overview/servers Member One row per server_name per project over the last 30 min: CPU/memory/disk/network + a CPU sparkline, plus host/os/cloud/k8s metadata lifted from metric tags. Reads OTel hostmetrics (system.cpu.utilization state=idle inverted, system.memory.utilization state=used, system.filesystem.utilization state=used, system.network.io by direction) and falls back to the legacy SDK names (cpu.used_pcnt, mem.used/mem.total). A project that fails to read sets partial: true instead of failing the request
POST /api/organizations/:organizationId/overview/issues Member Paginated issues across all projects. Body mirrors /api/exception-stack-traces (fromDate, toDate, orderBy, search, searchType, pagination); every project contributes its top page*pageSize groups in the requested order and the page is a slice of their merged order, so it equals what one combined project would return. That per-project fetch is capped at 1000 rows (orgOverviewMaxIssueFetch): a deeper page answers 422 and totalPages is clamped so the footer never offers one; FindGrouped orders ties by exception_hash on every backend so the merge is stable across pages. Returns {data, pagination, partial} with 24h hourly trends attached; a project that fails to read sets partial: true like the servers endpoint
POST /api/organizations/:organizationId/overview/pages Member Paginated on-call pages across all projects: body {status, search, fromDate, toDate, pagination} (status active/open/acknowledged/resolved, search over subject + rule name, range on created_at); returns {data, pagination, openPagesCount}
GET /api/organizations/:organizationId/overview/counts Member openPagesCount + downMonitorsCount for the organization sidebar badges
POST /api/organizations/:organizationId/overview/incidents Member Paginated incidents across monitors and status pages: body {search, fromDate, toDate, pagination}; an incident matches when it overlaps the range (started_at <= to and unresolved or resolved_at >= from), search covers title, monitor, status page, and error message
GET /api/organizations/:organizationId/overview/monitors Member Every synthetic check in the org with 30-day aggregates; a project whose results fail to read sets partial: true like the servers endpoint

Instance identity is the server_name tag, which comes from the OTLP service.name resource attribute; grouping by Kubernetes cluster keys off the k8s.cluster.name tag and the group selector only offers it when at least one instance carries one. Clicking a row opens /dashboards?projectId=&server=&preset=30m (plus dashboard= when the project has the traceway-otel-agent template applied); server scopes every widget query via scopeTagFilters in widget-grid/widget-renderer. Cluster-wide instrumentation manifests live in examples/kubernetes/, documented at docs/pages/learn/kubernetes.mdx.

Invitations

Method Endpoint Auth Purpose
POST /api/organizations/:orgId/invitations Admin Send invitation
GET /api/organizations/:orgId/invitations Admin List invitations
DELETE /api/organizations/:orgId/invitations/:id Admin Revoke invitation
GET /api/invitations/:token None Get invitation info
POST /api/invitations/:token/accept None Accept (new user)
POST /api/invitations/:token/accept-existing App Accept (existing user)

On-Call (teams, schedules, escalation policies, pages)

Org-scoped entities in the main DB. Teams (teams/team_members) own projects one-to-one (project_teams, unique on project_id). Schedules (oncall_schedules) store PagerDuty-style calendar layers as a JSON definition document (rotations daily/weekly/custom, handoff time/day, time-of-day and day-of-week restrictions); one-off overrides are normalized rows (oncall_overrides). The pure resolution engine lives in backend/app/oncall/ (ResolveRange/ResolveAt, tz-aware calendar math, later layer wins, overrides trump all). Both apply the same stacking, so a schedule puts exactly one person on call at any instant: ResolveAt is ResolveRange over a single instant, and paging a whole schedule stack (waking the person an override was meant to relieve) is the bug it exists to prevent. Escalation policies (escalation_policies) hold JSON steps (targets schedule/user/team/channel + delayMinutes, repeatCount); rules page on-call via the escalation notification channel type (config {"policyId"}), special-cased in notifications/dispatch.go through RegisterPageOpener (wired in cmd/run.go). A fired rule opens a pages row (dedup key ruleId|dedupToken with a partial unique index while unresolved; rules without a dedup token dedup at rule level; refires bump event_count and never reset the escalation clock). The escalator worker (oncall/escalator.go, ONCALL_POLL_SECONDS) claims due pages in a transaction (pg advisory lock 824737002 for multi-instance), inserts page_notifications rows, resolves targets to users, delivers via each user's user_contact_methods (email/slack/pushover/telegram adapter configs; account-email fallback always), then escalates level by level until ack/resolve or exhaustion. RequireOrganizationAccess middleware (any org role) gates member-level reads; mutations are org-admin; page acknowledge deliberately requires no write access.

Method Endpoint Auth Purpose
GET/POST /api/organizations/:organizationId/teams Member / Admin List (with members+projects) / create
PUT/DELETE /api/organizations/:organizationId/teams/:teamId Admin Update / delete
PUT .../teams/:teamId/members, .../teams/:teamId/projects Admin Replace ordered members / owned projects
GET/POST /api/organizations/:organizationId/schedules Member / Admin List / create
GET/PUT/DELETE .../schedules/:scheduleId Member / Admin / Admin Detail+overrides / whole-document update / delete
GET .../schedules/:scheduleId/timeline?from=&to= Member Rendered per-layer + final shifts (max 62 days)
POST/DELETE .../schedules/:scheduleId/overrides(/:overrideId) Member Create (any member, max 30d) / delete (creator, covered user, or admin)
GET /api/organizations/:organizationId/oncall/now Member Overview: per team/schedule current + next on-call
GET /api/oncall/current?projectId= App Owning team + current on-call for a project (issue page)
GET /api/escalation-policies App Policies of the project's org (channel dialog picker)
GET/POST /api/organizations/:organizationId/escalation-policies Member / Admin List / create
PUT/DELETE .../escalation-policies/:id Admin Update / delete (422 while referenced by a channel)
POST /api/pages App List (POST-body: status open/acknowledged/resolved/active + pagination)
GET /api/pages/:id App Detail + delivery log
POST /api/pages/:id/acknowledge App (no write gate) open -> acknowledged, stops escalation; 409 if not open
POST /api/pages/:id/resolve App open/acknowledged -> resolved; 409 if already resolved
GET /api/pages/open-count App Sidebar badge count
GET/POST /api/contact-methods App Own contact methods (self-scoped). Types: email, slack, pushover, telegram, sms. SMS requires Twilio (TWILIO_ACCOUNT_SID + TWILIO_AUTH_TOKEN + one of TWILIO_FROM_NUMBER / TWILIO_MESSAGING_SERVICE_SID); without them SMS is not offered at all — the list response carries smsEnabled: false so the type picker hides it; create, re-point, resend-code and test answer 422; the escalator drops existing sms methods before its "no methods left" check so those users fall back to the account email; and the adapter errors instead of reporting a delivery nobody received. Disabling and deleting a leftover sms method stay available. Creating/re-pointing an sms method starts code verification; unverified numbers are never paged
PUT/DELETE /api/contact-methods/:id App Update (incl. enabled toggle) / delete
POST /api/contact-methods/:id/test App Send a canned test through one method (422 for unverified sms)
POST /api/contact-methods/:id/verify App (rate-limited) Confirm the 6-digit SMS code (hashed at rest, 10-min expiry, 5-attempt cap). Deliberately not under middleware.Transactional: a wrong code answers 422, which would roll the consumed attempt back, so the handler manages its own transactions (nesting one under the middleware would also deadlock the single-connection SQLite main DB)
POST /api/contact-methods/:id/resend-code App (rate-limited) Re-issue the verification code
GET/PUT /api/user-notification-rules App Per-user notification-rule chains {high: [{contactMethodId, delayMinutes}], low: [...]} (PagerDuty-style: the page's urgency picks the chain; steps are enqueued at claim time as scheduled outbox deliveries and cancelled on ack; no chain = all enabled+verified methods immediately). Escalation policies carry urgency: auto|high|low in their definition (auto: critical -> high); pages store the resolved urgency
GET/POST /api/ack/:token None (rate-limited) Tokenized no-login acknowledge: per-delivery twk_ tokens (SHA-256-hashed on page_notifications), GET = read-only summary (scanner-safe), POST = idempotent ack recorded as acknowledged_via='link' attributed to the delivery's recipient; 404 after resolve. Frontend page: /ack/[token]

Monitors (user-facing name for synthetic uptime checks; engine in backend/app/synthetics, frontend under /monitors, see the SYNTHETICS_* env block)

Checks (synthetic_checks, main DB) are http/tcp/browser probes with per-check interval, timeout, and a consecutive-failure threshold (flap damping). Runs flow through the check_runs queue (outbox-style guarded claims, terminal rows deleted, expired queued runs recorded as missed); results are telemetry (check_results, all three backends, pruned by the SQLite retention worker / 90d CH TTL). State transitions open/resolve check_incidents and feed the event-driven notification rule type check_down (recovery auto-resolves the page a rule-scoped dedup key opened via oncall.AutoResolveByDedupKey; recovery never dispatches to escalation channels). The recovery notice carries Recovered on NotificationMessage: dispatch leaves its info severity alone instead of applying the rule's, and the Slack adapter colours it green. The notify hook runs post-commit with no ambient tx (SQLite single-connection). Remember: notification_rule.repository.go (both copies) lists event rule types explicitly, and check_down is one of them.

Incidents, incident updates & post-mortems. check_incidents covers both auto and manual incidents: auto rows carry check_id+project_id (nullable now), hand-recorded ones carry only status_page_id (org admins record them from a status page for outages monitors missed; they never affect uptime numbers). All incidents can be given a public title and statuspage.io-style timeline updates (incident_updates: status investigating/identified/monitoring/update/resolved + message; auto open/resolve seeds empty-message investigating/resolved updates in the same tx in synthetics/result.go, and the probe error_message stays internal, never in the public payload). A resolved update closes a manual incident; auto lifecycle stays owned by ProcessOutcome (manual resolve/time-edit/delete of auto incidents answer 422). The public /api/status/:slug payload gained pastIncidents (90d union of the page's checks' auto incidents + its manual ones, titles defaulting to " is down", nested updates, cap 30, no ids/error messages); incident mutations invalidate every status-page cache slug of the org. Post-mortems (post_mortems, project-scoped: project_id NOT-NULL-in-practice with ON DELETE CASCADE, backfilled by migration from the linked incident's project else the org's oldest project) are internal markdown documents, never public: optional 1:1 incident link (incident_id unique where set, ON DELETE SET NULL so documents survive monitor/incident deletion; linkable incidents are the project's own autos plus the org's manual status-page incidents — cross-project autos answer 422), JSON-array tags (models.StringSlice), LIKE search over title/content/tags, and an append-only edit log (post_mortem_events: action created/updated + JSON changes field list, recorded only when something actually changed; served as the Activity panel). Routes are project-scoped (/api/post-mortems*, standard RequireProjectAccess/RequireWriteAccess chains, org resolved from the project for incident validation); incident mutations are org-admin since they are public content. Frontend: Post-Mortems tab on /monitors (all members; creation is a title-only dialog that creates the document immediately and opens the editor page), full-page WYSIWYG editor at /monitors/post-mortems/[id] (@milkdown/crepe, dynamically imported, app font stack overrides the theme's serif), paginated incident management page at /monitors/status-pages/[pageId]/incidents (which owns the Record Incident dialog).

Method Endpoint Auth Purpose
GET/POST /api/synthetics/checks App / App+Write List / create checks (422 validation incl. browser-mode + cloud gates)
GET/PUT/DELETE /api/synthetics/checks/:id App / +Write / +Write Detail+incidents / update (type immutable; pausing drops queued runs) / delete (auto-resolves the check's open pages)
POST /api/synthetics/checks/:id/run App+Write Run now (422 if a run is already queued; OnCommit wake)
POST /api/synthetics/overview App Checks + per-range aggregates (uptime, avg latency)
POST /api/synthetics/checks/:id/results App Paginated run history (telemetry read, no Transactional; optional status filter up/down/missed + fromDate/toDate)
POST /api/synthetics/checks/:id/series App Bucketed uptime/latency series (telemetry read)
GET /api/synthetics/screenshot?key= App Streams a failure screenshot; key prefix-checked against the project
GET /api/synthetics/output?key= App Streams a failed browser run's stored Playwright output (.log keys only, same prefix check); "logs" link in the run history UI
GET /api/synthetics/open-count App Down-check count for the sidebar badge
POST /api/runners/poll Runner (shared secret) Instance-wide long-poll claim of browser runs (25s hold on the wake channel, NO Transactional, excluded from tracewaygin self-monitoring); self-registers the runner's liveness row
POST /api/runners/results/:runId Runner (shared secret) Report an outcome (MaxBytesReader 6MB incl. screenshotBase64; idempotent: lost/missing claim = 200 no-op; claims keyed "runner:")
GET/POST/PUT/DELETE /api/organizations/:organizationId/status-pages(/:id) Member / Admin Status page CRUD (slug [a-z0-9-]{3,60} unique, checkIds validated against the org; branding fields description, customDomain unique hostname)
POST /api/organizations/:organizationId/status-pages/:id/logo Admin Upload a PNG/JPEG logo (raw body, 1MB cap, sniffed content type; SVG rejected as scriptable) stored via storage.Store under statuspages/<id>/logo
GET /api/status/:slug/logo None (rate-limited) Streams a public page's logo
GET / on a custom domain None Status domains (CNAMEd vanity hosts, TLS at the operator's proxy). The SPA handler in cmd/run.go looks the Host up with controllers.StatusPageForHost (public pages only, never the APP_BASE_URL host) and, on a match, serves index.html with a traceway-status-slug meta tag plus the page's title and description. The frontend reads it in $lib/status-domain.ts and the root layout renders only StatusPageView at /, whatever the auth state. Every other page path on that host 302s to /, so the dashboard never loads on a customer's domain; static assets and /api still work. Saving a custom domain equal to APP_BASE_URL's host or the admin's current host answers 422
GET /api/status/:slug None (rate-limited) Public status payload: per-check status + 90 daily uptime buckets + incidents + pastIncidents (date-grouped titles with timeline updates); cached ~30s per slug; private/unknown slugs are both 404; latency values and internal error messages stripped. Frontend page: /status/[slug] (in isPublicPath)
GET /api/organizations/:organizationId/incidents Member Org incidents (90d, joined check/status-page names, updatesCount, postMortemId)
GET /api/organizations/:organizationId/incidents/:incidentId/updates Member One incident + its timeline updates
GET /api/organizations/:organizationId/status-pages/:id/incidents Member Paginated incident history of one status page (auto incidents of its checks + its manual ones, no time window; standard pagination envelope + statusPage meta). Frontend page: /monitors/status-pages/[pageId]/incidents
POST /api/organizations/:organizationId/status-pages/:id/incidents Admin Record a manual incident on a status page (title required, optional first update message, backdatable, optional resolvedAt)
PUT/DELETE /api/organizations/:organizationId/incidents/:incidentId Admin Edit title (any incident) / times (manual only, empty resolvedAt reopens; 422 for auto) / delete (manual only, 422 for auto)
POST/DELETE /api/organizations/:organizationId/incidents/:incidentId/updates(/:updateId) Admin Post / delete a public timeline update (a resolved update also closes an unresolved manual incident)
GET/POST /api/post-mortems App / App+Write Paginated list for the project (query params search/page/pageSize + repeatable tag params ANDed together) / create in the project (422 on duplicate or cross-project incident link)
GET/PUT/DELETE /api/post-mortems/:id App / +Write / +Write Full document with author names (contentMd) / update (records a post_mortem_events diff row; no-op saves change nothing; an optional expectedUpdatedAt answers 409 when the document changed since that read; the row is locked first (LockForUpdate) so concurrent Postgres updates cannot both pass the check. The CLI and MCP send the revision the caller's edit is based on, required when they replace the body (--expected-updated-at / expected_updated_at), and fall back to their own fresh read for title, tag and incident-only edits) / delete
GET /api/post-mortems/:id/activity App Edit log for the Activity panel (who created/edited what, joined user names)

SLOs (service level objectives; worker backend/app/slo, controller slo.controller.go, telemetry table slo_buckets, main-DB tables slos and notification_rule_states; docs docs/pages/learn/slos.mdx)

Method Endpoint Auth Purpose
GET/POST /api/slos App / App+Write List (snapshot columns included, disabled ones too) / create. sliKind is endpoint, span, task, monitor, metric or session with a per-kind sliConfig (validated and canonicalized in validateSLIDefinition), targetPercent in (0,100), windowDays 7, 30 or 90, bucketMinutes only for metric (1, 5, 15 or 60; other kinds are forced to 1). Create and update call OnCommit(slo.Wake)
GET/PUT/DELETE /api/slos/:id App / +Write / +Write {slo, rules} (the project rules of type slo_* whose config names it) / update (a change of kind, config or slice length resets the snapshot, and the worker deletes the old buckets on its next tick; target and window changes keep the history) / delete (auto-resolves pages of referencing rules through the slo:<id> dedup token, deletes those rules, deletes the buckets post-commit)
POST /api/slos/:id/series App {buckets: [{bucket, good, total}]} re-bucketed to intervalMinutes for {fromDate, toDate} (max 91 days); telemetry read, no Transactional, scoped by project through the bucket key
POST /api/slos/preview App {good, total, sli, from, to} of a definition over the last 24h straight from raw telemetry (the sheet's Preview button)
GET /api/slos/breached-count App Sidebar badge

Logs

Method Endpoint Auth Purpose
POST /api/logs App List/search logs (filters: severity, service, trace, body)

Source Maps

Method Endpoint Auth Purpose
POST /api/sourcemaps/upload SourceMap Upload source map file

Data Ingestion Flow (/api/report)

  1. Gzip Middleware: Decompresses request body (SDK sends gzipped data)
  2. Auth Middleware: Validates Authorization: Bearer <project_token>
  3. Parse Frame: JSON decode into models.Frame (transactions, exceptions, metrics)
  4. Batch Insert: Repository methods insert batches into ClickHouse
// backend/app/controllers/clientcontrollers/report.go
func (c *ReportController) Report(ctx *gin.Context) {
    var frame models.Frame
    ctx.BindJSON(&frame)

    // Insert each data type
    transactionRepo.BatchInsert(frame.Transactions)
    exceptionRepo.BatchInsert(frame.Exceptions)
    metricRepo.BatchInsert(frame.Metrics)
}

Database Schema

Tables (ClickHouse)

Table Purpose Partitioning
transactions HTTP request metadata Monthly (toYYYYMM(timestamp))
spans_v2, endpoints_v2, tasks_v2, ai_traces_v2, exceptions_v2 Traces, see "Trace Storage (V2 tables)" below Daily (toYYYYMMDD(recorded_at))
spans, endpoints, tasks, ai_traces, exception_stack_traces The tables V2 replaced. History only, read by the move-over Monthly
metric_records Time-series system metrics Monthly
archived_exceptions Archived/resolved exceptions None
log_records OTel logs with 3 attribute maps (resource/scope/log) Daily (toDate(timestamp))

Data retention/TTLs are documented separately under Data Retention below.

Tables (PostgreSQL)

Table Purpose
users User accounts with email/password
organizations Multi-tenant organizations
organization_users Junction table linking users to organizations with roles
projects Project config + tokens, linked to organizations
project_user_roles Per-project role overrides (user/readonly) for org members
invitations Team invitations with token, role, expiry
source_maps Uploaded source map files (project, version, storage key)
metric_registry Custom metric definitions (type, unit, description)
dashboards Org-owned dashboards (name, JSONB definition with widgets, template provenance)
project_dashboards Which projects show a dashboard, and tab order
dashboard_templates Marketplace templates (key, category, definition), seeded by migrations
starred_dashboard_widgets Homepage layout per project (dashboard id + widget id, position, col_span, size)
synthetic_checks Synthetic check config + current state (status, consecutive_failures, next_run_at)
check_runs Synthetic run queue (queued/claimed only; terminal rows deleted, telemetry is the record)
check_incidents Incident spans: auto (nullable check_id+project_id) and manual (nullable status_page_id), plus public title (feeds status pages)
incident_updates Public timeline updates on incidents (status + message, seeded on auto open/resolve)
post_mortems Internal markdown post-mortems (project-scoped via project_id ON DELETE CASCADE, tags JSON array, optional unique incident link with ON DELETE SET NULL)
post_mortem_events Append-only post-mortem edit log (action, JSON changes list, user, ON DELETE CASCADE)
synthetic_runners Liveness registry for self-registered runner fleet (unique name, version, first/last_seen_at; no credentials, no org scoping)
status_pages Public uptime pages (org-scoped slug + selected check ids + branding: description, logo_key, unique custom_domain)
widget_groups / widget_group_widgets / starred_widgets Legacy pre-dashboards tables, retained read-only for rollback until a follow-up drop

ClickHouse vs PostgreSQL Decision Guide

  • PostgreSQL: Relational/config data needing ACID, frequent updates, JOINs, low volume (users, organizations, projects, invitations, widgets, source maps, metric registry)
  • ClickHouse: High-volume append-only telemetry with time-series aggregations, batch inserts only (transactions, exceptions, metric points, spans, tasks, sessions)
  • Rule of thumb: "Will this data be updated after creation?" → PostgreSQL. "Is this immutable, time-stamped, high-volume data queried with aggregations?" → ClickHouse.

SQLite Dual-Database Architecture (self-hosted mode)

In SQLite mode (DB_TYPE=sqlite), the backend uses two separate SQLite databases mirroring the PostgreSQL/ClickHouse split:

Database Variable File Purpose Transactions
Main DB db.DB traceway.db PostgreSQL replacement — relational/config data Yes (middleware.Transactional, db.ExecuteTransaction)
Telemetry DB db.TelemetryDB traceway_telemetry.db ClickHouse replacement — append-only telemetry No — direct inserts without transactions

Both embedded builds (SQLite and DuckDB telemetry) are single-instance: the databases are local files, and the in-memory project cache only learns about another process's writes through the Postgres LISTEN/NOTIFY listener, which exists only under transactional_pg. Running more than one backend instance is supported only on the transactional_pg telemetry_ch build (docs: /server/minimal#running-more-than-one-instance).

Main DB tables (db.DB — transactional, uses lit with *sql.Tx):

  • users, organizations, organization_users, projects, invitations
  • source_maps, metric_registry, dashboards, project_dashboards, dashboard_templates, starred_dashboard_widgets (plus the legacy widget_groups/widget_group_widgets/starred_widgets)
  • notification_channels, notification_rules
  • synthetic_checks, check_runs, check_incidents, incident_updates, post_mortems, post_mortem_events, synthetic_runners, status_pages

Telemetry DB tables (db.TelemetryDB — non-transactional, uses lit with db.TelemetryDB directly):

  • endpoints_v2, tasks_v2, ai_traces_v2, exceptions_v2, spans_v2, metric_points
  • session_recordings, archived_exceptions, slow_endpoints, fired_notifications, check_results
  • endpoints, tasks, ai_traces, exception_stack_traces, spans: the tables V2 replaced, kept as history for the move-over

How to access each database in repository code:

// Main DB (PostgreSQL replacement) — use transactions via middleware or ExecuteTransaction
// Repositories receive *sql.Tx from middleware.GetTx(ctx) or db.ExecuteTransaction
user, err := lit.SelectSingleNamed[models.User](tx, "SELECT ... FROM users WHERE ...", lit.P{...})

// Telemetry DB (ClickHouse replacement) — use db.TelemetryDB directly, no transactions
results, err := lit.SelectNamed[endpointRow](db.TelemetryDB, "SELECT ... FROM endpoints WHERE ...", lit.P{...})

// Telemetry inserts — no transaction wrapping
for _, item := range items {
    row := modelToRow(item)
    lit.InsertExistingUuid(db.TelemetryDB, &row)
}

Migrations are split into two directories:

  • backend/app/migrations/sqlite/ — runs on db.DB (main)
  • backend/app/migrations/sqlite_telemetry/ — runs on db.TelemetryDB (telemetry)

SQLite-specific type helpers (backend/app/repositories/telemetry/sqlitetypes/):

  • SQLiteTime — implements sql.Scanner/driver.Valuer for time.Time ↔ SQLite TEXT
  • SQLiteJSONMap — implements sql.Scanner/driver.Valuer for map[string]string ↔ SQLite JSON TEXT
  • Row types (e.g., endpointRow, taskRow) wrap domain models with these types for lit compatibility

DuckDB Telemetry Backend (self-hosted, opt-in)

Built with -tags telemetry_duckdb (CGO_ENABLED=1 required), this is an alternative telemetry store for the same DB_TYPE=sqlite deployment: the main DB stays SQLite (db.DB, relational/config), while the telemetry DB becomes DuckDB (db.TelemetryDB, columnar). It exists because DuckDB's columnar engine is dramatically faster on the analytics/aggregation reads the dashboard issues — at 10M rows it clears read-probe thresholds that SQLite times out on. Backends are selected on two build-tag axes: telemetry_ch / telemetry_duckdb / (none = SQLite telemetry) for the telemetry store and transactional_pg / (none = SQLite main) for the relational store. Only three combinations are supported — (no tags) dual SQLite, telemetry_duckdb, and transactional_pg telemetry_ch — enforced by compile-time guard files in backend/app/db/ (stale pgch/duckdb/oltp_* tags also fail with a rename message). Repositories are organized on the same two axes: telemetry repositories live in per-backend packages backend/app/repositories/telemetry/{clickhouse,sqlite,duckdb}/ and transactional (relational) repositories in backend/app/repositories/transactional/{pg,sqlite}/, each re-exported as singletons through tag-guarded facade files at the axis package root (telemetry/telemetry_ch.go etc., transactional/transactional_pg.go etc.). Consumers import the facade packages — telemetry.SpanRepository, transactional.UserRepository — never a backend package directly. Helpers shared by all telemetry backends are in telemetry/shared/, the SQLite scan/value types shared by the sqlite+duckdb backends are in telemetry/sqlitetypes/, and helpers/types shared by the transactional backends (auth-token hashing/time formats, facade-crossing structs) are in transactional/shared/. The transactional/pg and transactional/sqlite implementations are intentionally kept dialect-neutral (lit :name queries rendered per db.Driver), enforced byte-for-byte by transactional/parity_test.go. Running Postgres requires the transactional_pg build: the default build's migration runner applies SQLite-dialect migrations unconditionally, so DB_TYPE=postgres without the tag is not a supported combination.

  • Driver: github.com/duckdb/duckdb-go/v2 (the official driver; marcboeker/go-duckdb is deprecated). Bundles prebuilt static libs for glibc only — not musl/Alpine, so the image uses Debian (Dockerfile.duckdb).
  • Opened in backend/app/db/db_telemetry_duckdb.go: telemetry path is the SQLite path with .db swapped for _telemetry.duckdb. By default DuckDB auto-tunes to the host; DUCKDB_MEMORY_LIMIT/DUCKDB_THREADS/DUCKDB_CHECKPOINT_THRESHOLD (passed through as DSN config options) let operators cap memory/threads so a memory-capped container doesn't read the host's RAM and OOM-kill the backend, and raise the WAL checkpoint threshold (default 16MB) so sustained Appender ingest isn't stalled by frequent checkpoints. preserve_insertion_order=false is always set — telemetry reads all have explicit ORDER BY, and dropping the guarantee lets DuckDB parallelize bulk loads and large scans with less memory. The read pool is bounded (SetMaxOpenConns(duckDBMaxReadConns)) since each DuckDB connection can use all threads + its own query memory; Appender writes use their own DuckDBConnector.Connect() connections and bypass that cap. Exposes db.DuckDBConnector (needed for the Appender).
  • Writes use the Appender API, not INSERT (duckdb.NewAppenderFromConn(conn, "", table) → AppendRow(...) → Close() flushes). Upserts still go through ExecContext with ON CONFLICT. The Appender rejects typed *string for nullable VARCHAR — use nullableString() in backend/app/repositories/telemetry/duckdb/helpers.go (returns untyped nil or the dereferenced value).
  • Write-path observability: a row the Appender rejects is dropped rather than failing the whole frame (the SQLite backend 500s instead), so a poison row cannot wedge the SDK's retry loop. Every drop increments a per-table counter (db.RecordTelemetryRowDropped) and fires a rate-limited (1/min per table) traceway.CaptureException; Appender flush/connect failures still propagate to the request (500, SDK retries) and increment an insert-failure counter. GET /api/health/deep (operator endpoint: requires the HEALTH_DEEP_TOKEN bearer secret, unset disables it; all telemetry backends) exposes telemetryBackend, droppedRows per table, droppedRowsTotal, insertFailures, ingestRejected (requests turned away by the ingest admission gate), and on DuckDB an engine object (db/WAL file bytes, duckdb_memory() usage, read-pool in-use/wait stats) alongside its existing ClickHouse fields; it 503s only when the configured telemetry backend is ClickHouse and CH is unreachable (the embedded backends answer 200 with chReachable:false). The benchmark loadgen polls it before/after every ramp step and fails any step whose drop delta is nonzero; read-probe fills record cumulative droppedRows per fill level. When MONITORING_TRACEWAY_URL is set, monitoring.StartTelemetryDBReporter also emits traceway.duckdb.* metrics every 10s: rows_dropped.delta, insert_failures.delta, db_size_mb, wal_size_mb, memory_used_mb, read_pool.in_use, read_pool.wait_count.delta, read_pool.wait_ms.delta. The hourly retention worker issues a CHECKPOINT after its deletes so retention actually reclaims disk (DuckDB otherwise defers reclamation to the WAL checkpoint threshold).
  • lit placeholders: db.Driver stays lit.SQLite, which emits ? — DuckDB accepts these, so no separate driver was needed for reads.
  • Migrations: backend/app/migrations/duckdb_telemetry/ (mirrors sqlite_telemetry/ table-for-table; integer columns are BIGINT, JSON is VARCHAR, no secondary indexes since it's columnar, with one exception: 0005_create_v2_tables adds idx_spans_v2_trace_key on spans_v2(trace_key) for the span graph lookup. While that index exists ALTER TABLE spans_v2 ADD COLUMN still works, but DROP COLUMN and ALTER COLUMN ... TYPE fail with a dependency error, so a migration that needs either must drop the index first).
  • Dialect gotchas vs SQLite (the read queries differ): native quantile_cont(col, p) for P50/P95/P99 instead of fetch-and-sort; strftime('%s',col)→epoch(col); time bucketing via time_bucket(to_seconds(N), col, TIMESTAMP '1970-01-01') — the explicit epoch origin is required because DuckDB anchors sub-day buckets at 2000-01-03 by default, which would misalign chart buckets against the SQLite backend's epoch-floored buckets for any interval that doesn't evenly divide a day; json_extract→json_extract_string; json_each→LATERAL unnest(json_keys(x)); strict GROUP BY needs ANY_VALUE/arg_max; SUM returns HUGEINT (CAST to BIGINT); CAST(.. AS REAL)→CAST(.. AS DOUBLE).
  • Tests: backend/app/repositories/telemetry/testhelper_duckdb_test.go (tagged telemetry_duckdb) provides setupTestDB so the entire existing telemetry test suite runs against an in-memory DuckDB.

Trace Storage (V2 tables)

A trace is a trace and a span is a span. Five tables hold traces on every backend: spans_v2, endpoints_v2, tasks_v2, ai_traces_v2, exceptions_v2 (migrations ch/0085-0089, one CREATE TABLE each; sqlite_telemetry/0025_create_v2_tables; duckdb_telemetry/0005_create_v2_tables). They sit next to the tables they replaced (spans, endpoints, tasks, ai_traces, exception_stack_traces), which nothing reads or writes any more except the move-over and retention. migrations/v2_tables_test.go holds the V2 migrations to CREATE TABLE/CREATE INDEX on _v2 tables only.

  • Ids. Every row carries the ids its span arrived with, as lowercase hex: trace_id (32 characters), span_id (16 for OTel, 32 for a native run or span), parent_span_id ('' for a root). Custom linked-trace identity is retired. Startup cleanup migrations remove linked_trace_id and its indexes from all four V2 entity/exception tables; OTLP span links remain intact. Entities and exceptions keep a UUID id for URLs and notifications. For OTel rows it is otelOccurrenceID = uuid.NewSHA1(projectId, traceId + spanId), so a retried export produces the same id. Nothing about identity lives in the attributes JSON, and ingest adds no traceway.otel.* keys.
  • Identity schema cleanup. Forward migrations ch/0091–0099, sqlite_telemetry/0026, duckdb_telemetry/0006–0007 replace sessions.distributed_trace_id with canonical non-null string trace_id (empty = no association), remove V2 linked_trace_id columns/indexes, and drop the unused profiles.distributed_trace_id column/index. Profiles keep their existing trace_id / span_id. ClickHouse synchronously materializes session IDs before dropping the source column. SQLite/DuckDB commit each migration and its version atomically; DuckDB adds the session non-null constraint in the next transaction. Stop old backends before the first upgraded start and restore a pre-upgrade database for rollback: old session/profile SQL is incompatible. The five old trace tables remain for the move-over.
  • Ingest (otelcontrollers/trace_converter.go) validates and indexes spans once per resource, then returns canonical spans, entity projections, exceptions and the invalid-ID count together. Each span is promoted when its own kind and attributes make it an entry point, and each exception event becomes an exceptions_v2 row with the span's trace id and span id. Canonical span attributes are rendered directly from the OTLP payload at storage time. Trace and log JSON IDs are normalized at the decoder boundary (otelcontrollers/otlp_json.go); converters consume decoded bytes without a UUID/base64 repair roundtrip. No owner resolution, no caches, no reads. trace_type on an exception is stored only when that span itself was promoted (native rows say it themselves). Every entity is recorded at its span's start, OTel tasks included. Spans are inserted before the rows promoted from them, and a transient storage error answers 503 + Retry-After (abortIngestStorage, telemetry.IsTransientStorageError) so OTLP exporters retry. Retries are append-only; the span reader deduplicates per span.
  • Entry points. An HTTP SERVER span becomes an endpoint unless a local span above it already is one (isEntryPoint), which is what keeps a SERVER span nested under another (Next.js under the HTTP instrumentation) from counting as a second request. This is the one check that looks beyond the span, and only inside its ResourceSpans block. The walk up the parent chain stops at a remote parent per the OTLP span flags (0x100 has-is-remote, 0x200 is-remote) and at a CLIENT or PRODUCER span. A missing parent has an unknown kind even when flags mark it local, so it cannot suppress an HTTP entry point. Separately exported nested HTTP spans can create extra projections.
  • Native /api/report rows go to the same tables (clientmodels.ClientTrace.RootSpan, ClientSpan.ToSpan): a run is stored as the root span of a trace, span_id = the run's id, trace_id = the run's id. Its spans hang under it, and a native exception sits on the run's span. The report wire format is unchanged: an exception's traceId names its owning run, distributedTraceId is ignored on runs, exceptions and sessions until the JS SDK supports canonical trace propagation, and native run/span IDs remain UUIDs.
  • spans_v2 keeps the lossless payload of accepted OTLP spans (resource_pb/scope_pb/span_pb on ClickHouse, otlp on SQLite and DuckDB; shared.AssembleOtelPayload joins them back) next to the readable columns, built once by OtelSpanRow.NestedValues: span_attributes is a string map (a Map(LowCardinality(String), String) on ClickHouse, a JSON object elsewhere) filled by shared.StringAttributes the way the OpenTelemetry Collector's ClickHouse exporter does it, and resource, scope, events, links are JSON text. A span that did not arrive as OTLP gets a payload synthesized from its available fields (NewOtelSpanRow), including string attributes and service/scope metadata. Native UUID span IDs remain in the API; their exported span/parent IDs use a stable eight-byte hash mapping with original IDs carried in traceway.native.* attributes. Native synthesis cannot recover metadata never sent by the SDK. On ClickHouse recorded_at is DateTime64(6) on endpoint/task/exception tables and DateTime64(3) on AI traces (the waterfall's origin is the entity's recordedAt) and a plain partition time on spans_v2.
  • Reads. A waterfall is spans_v2 by project and trace, the subtree under the entity's span_id: telemetry.SpanRepository.FindGraph(ctx, shared.NewSpanLookup(projectId, traceId, spanId, recordedAt)), answered by shared.FindOtelGraphs (topology first, attributes second). A limit never fails the page: the result carries spanGraphStatus (complete/partial/unavailable plus reasons) and spans may have attributesOmitted. The subtree includes spans that are entities of their own, such as a CONSUMER span started inside the request.
  • Distributed trace card (controllers/distributed_trace.controller.go, POST /api/distributed-traces/:traceId, hex or dashed). FindByTraceIds on the four entity and exception repositories matches only trace_id IN (...) across projects the user can open. Every returned node belongs to the requested trace; W3C propagation makes browser and backend share that identity. Nodes are nested by one read of the trace's topology, without attributes (SpanRepository.FindTraceParents): each node walks up until it meets another node's span (parentEntitySpanId). An exception is attached to the nearest node above its span, else it is a node of its own.
  • Exceptions. A detail page reads exceptions_v2 by trace and keeps those on its own span or a span of its subtree (findOccurrenceExceptions). telemetry.FindExceptionOwner does the walk the other way, from the exception's span up to the nearest endpoint, task or AI trace of the same project. The exception responses return it as relatedEntity, and notification text uses it (resolveExceptionOwner). A failed task, for the task failure rule, is a task with an exception on its own span.
  • Every span read is bound in time. A lookup reads recorded_at within 24 hours either side of the entity it starts from (shared.OtelLookupWindow, shared.TraceWindowBounds; entity lookups by trace id use 48 hours), which on ClickHouse prunes the read to a few daily partitions. A lookup that names no time is anchored on now. The cost is deliberate: a span recorded more than 24 hours away from its entity is not shown on that entity's page. Tests that store spans under a fixed timestamp must pass that timestamp as the lookup time, never the wall clock.
  • Orphans. A span whose parent never arrived has no place in a subtree. The entity promoted from the trace's root span (the one with no parent) shows such spans of its own project and trace, so they are visible exactly once, and the waterfall marks them "Parent span is unavailable". Once the parent arrives they nest normally.
  • Span explorer (controllers/span_explorer.controller.go, frontend /spans). POST /api/spans/search (App auth + project access) pages the spans of a project in a required time range of at most 31 days. Body: fromDate, toDate, orderBy (start_time desc|asc, duration desc|asc), serviceName, name (case-insensitive contains), traceId (32 hex), kind (0-5), status (0-2), minDurationMs, maxDurationMs, attributeFilters (at most 10 exact {key, value} matches on span_attributes, value required since a ClickHouse Map answers '' for a missing key), pagination. The response adds services, the busiest 100 service names in the range. Validation problems and a search that hits its 15s read limit both answer 422, which the page shows as a warning callout, not an error. The query builder is shared.OtelSearchQueries with one small dialect struct per backend. GET /api/spans/traces/:traceId?at=<RFC 3339> returns the whole trace (shared.FindOtelTrace) and 404s when nothing is recorded within 24 hours of at. It reads every project of the current project's organization, not only the current one (organizationProjects in the controller, one lookup per project, one query): services of one trace usually report to different projects, and organization membership is what grants read access to each. Other organizations of the same user stay out. The response names the projects that hold spans (projects), each span keeps its own projectId, the page links spans across projects (buildSpanTree(..., acrossProjects)), and the popover loads attributes with the span's own project id. Search stays per project. Unlike the detail pages it sends no attributes: each span carries only dbStatement, a 240 character preview of db.query.text (or db.statement) that the row label needs, and the page fetches one span's attributes from GET /api/spans/traces/:traceId/spans/:spanId/attributes?at= when its popover opens. One query serves both cases of the row cap (FindTraceOutline on each backend, ordered by shared.OtelImportanceOrder): under MaxOtelGraphRows every span comes back, over it the LIMIT keeps errors first, then entry points (no parent, SERVER or CONSUMER), then the slowest, and the status is partial with reasons row_limit and most_important. Ordering by duration also tends to keep the path above a span, since a parent usually outlasts its children. Responses are not compressed by the backend: 20,000 spans are about 13 MB of JSON, about 1.1 MB once a proxy gzips them. The sidebar hides Spans for browser and mobile projects (hiddenForFrontend in app-sidebar.svelte); the route itself still works for links into a trace.
  • GET /api/otel/spans/:traceId/:spanId?at=<RFC 3339> (App auth + project access) returns one stored span (16 or 32 hex span id) as an OTLP ExportTraceServiceRequest protobuf. at is required (400 without it) for the same 24 hour rule.
  • Frontend: large traces. A trace over 500 spans opens with every node that has more than 50 children folded (initiallyCollapsed in utils/span-tree.ts); smaller traces open fully expanded. A folded row shows a pill with the number of hidden spans and an error marker when one of them failed, and the header has Expand all and Collapse all. The waterfall keeps what was clicked as the difference from that default, so there is no effect to reset. Pages must hold span payloads in $state.raw: a deep $state proxy over 20,000 spans took four minutes to read in the browser, against 0.4 seconds raw. The payload is only ever replaced, never edited.
  • Frontend. span-waterfall.svelte names the service on each row and colours bars by service once a trace crosses services (one colour per row otherwise), marks error spans and missing parents, renders rows 500 at a time so a 20,000 span trace cannot stall the page, caps the name column to half the visible width on narrow screens, and links to the whole trace in the explorer. The span popover shows service, kind, status and scope (utils/span-fields.ts). Span payloads are held in $state.raw. The tree is keyed traceId:spanId (utils/span-tree.ts), and the issue pages build their related entity from the backend's relatedEntity.
  • GET /api/health/deep also reports spanPartitionTimeFallbacks: spans whose source start was missing or unrepresentable, so their partition time is the ingest time (shared.OtelStorageTimes).
  • Move-over (repositories/telemetry/move_over.go, started by retention.startMoveOver when V2_MOVE_OVER=true). Copies the old tables into the V2 tables inside the backend (an embedded DuckDB file cannot be opened by a second process), UTC days scheduled newest first with configurable concurrency (V2_MOVE_OVER_WORKERS, default 1, cap 16), each day keeping table order, recorded in the transactional DB v2_move_over_days (migrations sqlite/0077, pg/0143) so it stops and resumes. The scheduler runs at startup and one hour after each pass ends, including failures; passes do not overlap. started is committed before telemetry writes, done after all batches succeed, with no main-DB transaction held during the copy. moved_rows counts scanned source rows in the current attempt, including already-copied rows skipped during recovery; it resets on resume. Progress and logs update after successful batches roughly every 10s (a slow query or insert can delay this). A failed worker cancels the pass, and all workers are joined before return; separate backend instances are still not coordinated. Finish upgrading old writers first: completed table/date pairs are not revisited. A resumed day skips rows already written. It reads through LegacyRepository (one per backend, the old ids parked in the V2 model fields, see shared/legacy.go) and writes through the V2 repositories, so the id mapping lives in one place: the trace id comes from the traceway.otel.trace_id attribute the old ingest kept, else the old distributed_trace_id, else the row id; an old zero padded span UUID becomes its last 16 hex characters; spans and exceptions find their old owner by id, nearest in time, and take its trace; each entity also gets its own span row. Slices are halved when a read comes back full, so no day is sorted, and only a single overfull second is paged by stable full-row order and offset. Operator notes: docs/pages/server/upgrade-v2-tables.mdx.

ClickHouse ch/0090_add_span_version adds a materialized SHA-256 fingerprint over length-prefixed protobuf components. All span readers rank by duration then this fingerprint, so topology never reads payload columns for new parts. Old V2 parts calculate it on read until merged or explicitly materialized; no rewrite is scheduled by migration. The column costs 32 uncompressed bytes per physical row.

V2 review corrections

Legacy history paging uses stable full-row ordering and offsets only within overfull one-second slices; source tables must be immutable after cutover. Recovery checks project, occurrence ID, trace/span IDs and destination-precision timestamps. Rejected canonical rows fail the native/migration facade, so a day cannot be marked complete after silent row rejection. An old preview's incorrectly completed day can be set back to started for checked recovery; see docs/pages/server/upgrade-v2-tables.mdx.

Span search, topology, attributes and export choose one retry version (longest duration, then the stored SHA-256 span_version on ClickHouse or stored protobuf bytes on SQLite/DuckDB), with UTF-8 byte budgets applied in SQL. Native report spans are written before projections. OTLP storage returns rejected span identities so their endpoint/task/AI/exception projections and conversation uploads are omitted while reporting partial success. HTTP entry-point classification suppresses only an observed local HTTP ancestor, never an absent parent's assumed kind. Fallback AI conversation IDs retain the historical dashed UUID spelling. Log search accepts traceId and optional spanId, scoped to the selected project and time window. Retired distributedTraceId and excludeTraceId selectors return 400. The trace log panel has one paginated list with stale-response protection.

Data Retention

Retention is handled in several different ways depending on the deployment.

0. Main-DB auth prune — retention.Start worker (backend/app/retention/auth_tokens.go + oauth_sessions.go, sharing startDBPruneWorker in prune_worker.go). Runs in all modes (Postgres and SQLite) against db.DB: once at startup, then every 24h, deleting expired/consumed auth rows. auth_tokens prunes expired device_authorizations (also pruned opportunistically on every /api/auth/device/authorize call, so the daily worker is a backstop there), refresh_tokens that are expired, revoked, or used more than 30 days ago (used rows are kept a month for replay detection, then dropped to bound per-user growth), plus revoked-or-expired personal_access_tokens (PAT expiry was otherwise only enforced lazily at read time). oauth_sessions prunes expired SSO login sessions. No env var — always on. (Refresh-token families and PATs also have explicit revoke paths: POST /api/auth/logout and the account PAT UI.)

1. ClickHouse — TTL clauses on the table itself. Only a few tables have a TTL; everything else is kept indefinitely (operators can drop monthly partitions manually if needed).

Table Retention Source migration
metric_points (raw) 7 days 0034_add_ttl_metric_points.up.sql
metric_points_1m (1-min rollup) 30 days 0035_add_ttl_metric_points_1m.up.sql
metric_points_1h (1-hour rollup) 1 year 0036_add_ttl_metric_points_1h.up.sql
log_records 90 days 0075_increase_ttl_log_records.up.sql (raised from 30d set in 0045_create_log_records.up.sql)
profiling_samples (the bulk) 30 days 0068_add_ttl_profiling_samples.up.sql
profiles (slim metadata) 30 days 0069_add_ttl_profiles.up.sql
profiling_stacks (dedup table) 30 days 0070_add_ttl_profiling_stacks.up.sql
check_results (synthetic check probes) 90 days 0083_add_ttl_check_results.up.sql
All other CH tables (the five V2 trace tables and the five they replaced, transactions, sessions, session_recordings, fired_notifications, archived_exceptions, slow_endpoints, etc.) No TTL, retained indefinitely none

The three profiling tables share a 30-day TTL keyed on each table's time column (start_time / recorded_at / last_seen). profiling_stacks is a ReplacingMergeTree(last_seen) dedup table, so last_seen is bumped on every re-ingest that references a stack — a stack only ages out once it has gone unreferenced for the full window, which is exactly when its samples have also expired, so no sample is ever left pointing at a dropped stack.

2. SQLite — retention.Start worker (backend/app/retention/sqlite.go). In SQLite mode, neither db.DB nor db.TelemetryDB has any built-in expiry, so a background worker fires once at startup and then every hour and runs a DELETE FROM <table> WHERE <time_column> < cutoff against each telemetry table.

Variable Default Notes
SQLITE_RETENTION_DAYS 30 TTL in days. Set to 0 to disable the worker entirely. Has no effect outside SQLite mode.
DUCKDB_RETENTION_DAYS 30 Same worker on the DuckDB telemetry backend. When set it takes precedence there; when unset the worker falls back to SQLITE_RETENTION_DAYS (selection in retention.telemetryRetentionConfig).

Tables it prunes (and the column used):

Database Table Time column
Telemetry (db.TelemetryDB) endpoints_v2, tasks_v2, exceptions_v2, spans_v2, ai_traces_v2, the five tables they replaced (endpoints, tasks, exception_stack_traces, spans, ai_traces), metric_points, session_recordings recorded_at
Telemetry log_records timestamp
Telemetry sessions started_at
Telemetry fired_notifications fired_at
Telemetry profiling_samples start_time
Telemetry profiles recorded_at
Telemetry profiling_stacks last_seen
Telemetry check_results recorded_at

archived_exceptions (per-hash flags) and slow_endpoints (per-endpoint config) are intentionally skipped — they are not time-series data. profiling_stacks is pruned (unlike those two) because it holds no user intent — it is a regenerable dedup table whose last_seen tracks the most recent referencing sample, so deleting expired stacks is safe.

2c. Synthetics failure screenshots — retention.Start worker (backend/app/retention/synthetics.go). Browser-check failure screenshots written to local disk under <STORAGE_PATH>/synthetics/ are aged out hourly by mtime, mirroring the session-recording split: the check_results rows referencing them are pruned separately (item 2 / CH TTL) and deliberately not coupled. SYNTHETICS_SCREENSHOT_RETENTION_DAYS (default 30, 0 disables); no-op unless STORAGE_TYPE=local.

2b. Log row cap — retention.Start worker (backend/app/retention/log_cap.go). SQLite mode only (SQLite or DuckDB telemetry), off by default. When LOG_RECORDS_MAX_ROWS is set to a positive N, a worker runs once at startup and then every minute and deletes log_records rows strictly older than the Nth-newest row's timestamp (single portable DELETE with an ORDER BY timestamp DESC LIMIT 1 OFFSET N-1 subquery; NULL boundary = no-op under the cap, boundary ties are kept). The cap is best-effort, not a hard limit: nothing throttles ingest, so between passes the table can exceed N, and sustained ingest above N rows/minute keeps it above the cap permanently — document it to users as a cleanup task with a 1-minute window, sized with headroom for peak log volume. Bounds log disk usage independently of SQLITE_RETENTION_DAYS; disk reclamation still happens via the hourly retention pass / DuckDB WAL checkpointing.

3. On-disk session recordings — retention.Start worker (backend/app/retention/recordings.go). Session recordings written to local disk (STORAGE_TYPE=local) accumulate under <STORAGE_PATH>/recordings/. A second worker walks that directory once at startup and then every hour and removes files whose mtime is older than the TTL, then prunes any directories left empty. The worker is a no-op when STORAGE_TYPE=s3.

Variable Default Notes
SESSION_RECORDING_RETENTION_DAYS 30 TTL in days. Set to 0 to disable the worker. Only runs when STORAGE_TYPE=local (default).

The DB rows in session_recordings are pruned by the SQLite retention worker (above) or by ClickHouse TTL — they are intentionally not coupled to the disk cleanup. Controllers that read recordings already log a non-fatal traceway.CaptureException when a referenced file is missing.

4. On-disk raw profile archives — retention.Start worker (backend/app/retention/profiles.go). When PROFILE_ARCHIVE_RAW is enabled, the native pprof ingest path (/profiles/ingest) writes each upload's original pprof bytes to <STORAGE_PATH>/profiles/<projectId>/<yyyymmdd>/<id>.pprof (recorded on Profile.StorageKey) as a lossless archive for download / re-ingest / PGO — off the read path. The OTLP endpoint does not archive (its rows carry an empty StorageKey). A worker walks that directory once at startup and then every hour and removes files whose mtime is older than the TTL, then prunes empty directories. It reuses the same generic age-based cleanup as the recordings worker (runDirAgeCleanup / isSafeStorageSubdir) and is a no-op unless PROFILE_ARCHIVE_RAW is on and STORAGE_TYPE=local.

Variable Default Notes
PROFILE_ARCHIVE_RAW false Master switch for the raw archive. When off, no blob is written and the disk worker does nothing.
PROFILE_RETENTION_DAYS 30 TTL in days for the on-disk archive. Set to 0 to disable the disk worker. Only runs when PROFILE_ARCHIVE_RAW is on and STORAGE_TYPE=local.

The profiles DB rows (and their storage_key) are pruned by the SQLite retention worker / ClickHouse TTL above — not coupled to this disk cleanup, mirroring the session-recording split.

5. Main-DB outbox prune — retention.Start worker (backend/app/retention/outbox.go, same startDBPruneWorker scaffolding as item 0). Runs in all modes against db.DB: once at startup, then every 24h, deleting terminal notification_outbox rows — sent/cancelled older than 7 days, failed older than 30 days. pending and sending rows are never pruned, so nothing undelivered is dropped. No env var — always on. The durable record of what was notified lives in fired_notifications and page_notifications, which this does not touch. Note that pages and page_notifications themselves are currently retained indefinitely.

Session Recording Uploader

Session recording segments arriving on /api/report are not uploaded inline. The handler enqueues each segment onto a bounded worker pool (backend/app/recordings/uploader.go, started from cmd/run.go next to retention.Start). Workers drain the queue and write the body via storage.Store.Write (S3 or local disk); successful writes are handed to a single batcher goroutine that calls SessionRecordingRepository.InsertAsync once per ~1000 rows or every 2 s, whichever comes first — single-row inserts are an anti-pattern for ClickHouse. Enqueue is non-blocking: when the queue is full the segment is dropped (newest-first) so a burst of /api/report traffic cannot spawn unbounded goroutines or saturate S3.

Variable Default Notes
SESSION_RECORDING_UPLOAD_WORKERS 32 Concurrent uploaders. Set to 0 to drop every segment (uploads disabled).
SESSION_RECORDING_UPLOAD_QUEUE_SIZE 2048 Max queued segments. Overflow is dropped.

Observability (emitted every 10s via traceway.CaptureMetric):

Metric Type Meaning
traceway.recordings.queue_depth gauge Segments currently waiting in the channel.
traceway.recordings.in_flight gauge Workers mid-upload.
traceway.recordings.uploaded counter Successful S3/local + DB writes since startup.
traceway.recordings.dropped counter Segments dropped on overflow since startup.
traceway.recordings.failed counter Upload or DB-insert errors since startup.

Sustained drops also fire a rate-limited (1/min) traceway.CaptureException so overload is visible in the issues feed without flooding it.

Session end is derived on read. The browser SDK's closing payload is unreliable (a keepalive body over 64 KB is rejected on unload, and frozen mobile tabs never fire pagehide), so the stored sessions.ended_at is only a hint. Each segment row carries its client-side ended_at (ch/0100, sqlite_telemetry/0027, duckdb_telemetry/0008), and every session read joins the latest segment activity and applies shared.ResolveSessionEnd: an explicit end is kept only when it lands within SessionIdleTimeout (15 min, the SDK's inactivity timeout) after the last activity, otherwise the session ends at its last activity; it is still live while a segment arrived (server time) within the timeout. Every session is capped at shared.SessionMaxSpan (65 min: the SDK's 60 min cap plus slack) because @tracewayapp/frontend <= 1.2.0 kept uploading segments under an ended session id for the tab's whole lifetime (18h "sessions"); the end and duration are clamped to it (sessions without segments too), and when segments run past it the session ends at the last segment inside it (span_activity, computed per backend against the session start; the usual 15 min rule then honours the SDK's own close), not at the cap, because an idle tab uploads nothing until it wakes hours later (sleep/wake, or 1.2.0's zombie uploads) and resumes under the same id; Session.CutoffAt carries the resulting cut-off, a session whose segments run past it is never live (compared on the client clock, so skew between browser and server cannot end a live session), and the session detail and GET /api/sessions/:id/recording drop exceptions and segments past CutoffAt, anchored on the stored session start rather than the startedAt query param (shared.SegmentsWithinSession). HasRecording is set when any segment exists, and the frontend labels an old session with neither as "No recording" (utils/session-duration.ts). Sorting the list by duration uses a SQL copy of the same rule in each backend (sessionDurationSortKey); TestSessionRepository_EndFromRecordingActivity holds the SQL order to the Go rule. GET /api/sessions/:id/recording?startedAt= bounds the segment lookup to the session's window, and segments are read from storage 16 at a time.

Users, Organizations & Projects

  • Users to organizations is many-to-many via organization_users, one role per membership. A user joins additional organizations through invitations (POST /api/invitations/:token/accept-existing for existing accounts) and, in cloud mode, registration. Login/register/LoginBundle return every membership as Organizations[] with roles; the frontend keeps them in authState.organizations and groups the navbar project selector by organization when there is more than one.
  • Projects belong to exactly one organization (projects.organization_id). Org membership grants read access to all of the org's projects.
  • Per-project role overrides live in project_user_roles(project_id, user_id, role) with role user or readonly. Overrides only apply to members whose org role is user or readonly; owners and admins always have full access to every org project. Override rows are kept when a member's org role changes (they are inert for owner/admin), and are deleted when the member is removed from the org or the project is deleted. Managed from Settings > Team Members (expand a member row) via GET/PUT /api/organizations/:orgId/members/:userId/project-roles(/:projectId); PUT with role: "default" clears the override and is accepted for any member regardless of org role (so stale rows on members promoted to admin can still be cleaned up); setting user/readonly is rejected with 422 for owners and admins.
  • Effective project role (ProjectRepository.GetEffectiveRole): the org role if owner/admin, otherwise the override if present, otherwise the org role. /api/projects returns it as role on each project and masks token/sourceMapToken when it resolves to readonly; the frontend derives write gating from it (isProjectReadonly in projects.svelte.ts).

Organization Roles

Role Description
owner Full access, can manage organization
admin Full access to projects
user Standard access to projects (can be overridden per project to readonly)
readonly Read-only access, cannot create projects or archive exceptions (can be overridden per project to user)

Middleware enforcement: RequireProjectAccess checks org membership (read access, unaffected by overrides); RequireWriteAccess blocks writes when the effective project role is readonly; RequireAdminAccess requires org role owner/admin for the :organizationId route param.

Key Columns - transactions

project_id UUID,
timestamp DateTime64(3),
trace_id String,
endpoint String,           -- normalized: "GET /api/users"
duration_ms Float64,
status_code UInt16,
app_version String,
server_name String,
-- Indexes: bloom_filter(trace_id), tokenbf_v1(endpoint)

Key Columns - exception_stack_traces

project_id UUID,
timestamp DateTime64(3),
hash String,               -- normalized hash for grouping
type String,               -- error type (e.g., "RuntimeError")
value String,              -- error message
stacktrace String,         -- full stack trace
tags Map(String, String),  -- contextual tags from scope

Database Migrations

CRITICAL RULES:

  1. migrations/ch/ and migrations/pg/ files must contain exactly ONE SQL statement. Both run through golang-migrate, and the ClickHouse driver is constructed with MultiStatementEnabled: false (migrations_telemetry_ch.go), so a second statement fails. pg/ follows the same rule for symmetry.
  2. migrations/sqlite/, migrations/sqlite_telemetry/ and migrations/duckdb_telemetry/ run through runMigrationsOn in migrations.go, which splits on semicolons outside string literals (splitStatements, covered by split_test.go). A file there may hold a CREATE TABLE plus its indexes — that is the existing convention, e.g. sqlite/0043_create_pages.up.sql.
  3. Only create .up.sql files (no down migrations)
  4. Use sequential numbering: NNNN_description.up.sql

Example - Adding two columns requires TWO files:

backend/app/migrations/ch/
├── 0013_add_app_version_to_transactions.up.sql
│   └── ALTER TABLE transactions ADD COLUMN app_version String DEFAULT ''
├── 0014_add_server_name_to_transactions.up.sql
│   └── ALTER TABLE transactions ADD COLUMN server_name String DEFAULT ''

Exception Hash Normalization

The backend normalizes stack traces before hashing to group identical errors despite different runtime values. This happens in backend/app/controllers/clientcontrollers/client.controller.go.

Normalization Steps (ComputeExceptionHash, applied in this order, and skipped entirely when isMessage is true):

  1. Strip the message from Caused by: lines, keeping the class name (causedByRe)
  2. Strip the message from the error line, keeping the error type (errorMessageRe)
  3. Collapse JS SDK function-name lines (ending in (), directly above a 4-space-indented file:line:col location line) to <fn> so resolved function names never affect grouping; anchoring on the location line keeps Go traces (tab-indented file lines) untouched (jsFuncLineRe)
  4. Remove URL origins such as https://cdn.example.com (urlOriginRe)
  5. Remove absolute file paths, keeping filename:line (absolutePathRe)
  6. Drop the column from any frame whose line number is 2 or higher (laterLineColRe)
  7. Remove @v1.2.3 module version suffixes (versionRe)
  8. Replace hexadecimal addresses with <hex> (hexRe)
  9. Replace UUIDs with <uuid> (uuidRe)
  10. Replace runs of 5 or more digits with <id>, unless preceded by a colon or another digit (largeNumberRe)
  11. Replace email addresses with <email> (emailRe)
  12. Replace IP addresses, with an optional port, with <ip> (ipRe)
  13. Replace Go goroutine ids with goroutine <n> (goroutineRe)
  14. Drop line numbers from Java, Kotlin and Scala frames (javaLineNumRe)
  15. Collapse ... 12 more to ... more (javaEllipsisRe)
  16. Collapse runs of spaces and tabs, then runs of newlines (spacesRe, newlinesRe)
  17. Trim, hash with SHA-256, truncate to 16 hex chars

Not normalized: timestamps and ANSI color codes. Two otherwise identical traces that differ only in an embedded timestamp, or only in \x1b[31m escapes, produce two separate Issues. Add a regex to the block in client.controller.go if that ever matters.

Note: the column is kept only on line 1 (step 6). Minified bundles put everything on line 1, so there the column is the only frame disambiguator when no source map matched, while on real source lines the column is noise. Function names are excluded (step 3) because they are derived from the location and change as symbolication improves.

Result: Same logical error gets same hash, even if:

  • Error message contains different user IDs
  • Stack trace has different memory addresses
  • File paths differ between environments

Repository Patterns

Singleton Pattern

Repositories are exported as package-level singletons, re-exported per storage axis through the facade packages app/repositories/transactional and app/repositories/telemetry:

// backend/app/repositories/transactional/sqlite/user.repository.go (and the pg/ twin)
var UserRepository = userRepository{}

// backend/app/repositories/transactional/transactional_sqlite.go (tag-guarded facade)
var UserRepository = sqliterepo.UserRepository

// Usage in controllers
user, err := transactional.UserRepository.FindByEmail(tx, email)
spans, err := telemetry.SpanRepository.FindByTraceId(projectId, traceId)

Batch Insert (ClickHouse)

func (r *TransactionRepository) BatchInsert(txns []models.Transaction) error {
    batch, _ := r.db.PrepareBatch(ctx, "INSERT INTO transactions ...")
    for _, txn := range txns {
        batch.Append(txn.ProjectID, txn.Timestamp, ...)
    }
    return batch.Send()
}

Aggregation with Quantiles

// P50, P95, P99 percentiles
query := `
    SELECT
        endpoint,
        count() as count,
        quantile(0.5)(duration_ms) as p50,
        quantile(0.95)(duration_ms) as p95,
        quantile(0.99)(duration_ms) as p99
    FROM transactions
    WHERE project_id = ? AND timestamp BETWEEN ? AND ?
    GROUP BY endpoint
    ORDER BY count DESC
`

Error Handling Pattern

IMPORTANT: When handling errors in controllers, always use c.AbortWithError with traceway.NewStackTraceErrorf instead of c.JSON with a generic error message. This ensures proper error tracking with stack traces.

// CORRECT - Use AbortWithError with descriptive reason
projectId, err := middleware.GetProjectId(c)
if err != nil {
    c.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("RequireProjectAccess middleware must be applied: %w", err))
    return
}

// WRONG - Do not use c.JSON for internal server errors
if err != nil {
    c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
    return
}

Key points:

  • The reason should describe the actual cause (e.g., "RequireProjectAccess middleware must be applied")
  • Use %w to wrap the original error for proper error chaining
  • This pattern applies to all 500 Internal Server Error responses
  • For client errors (400, 404), c.JSON with an error message is acceptable

Non-stopping errors: For errors that should not abort the request (e.g., optional feature failed to load), use traceway.CaptureException to report them instead of log.Printf:

// CORRECT - Report non-stopping errors via traceway
if err != nil {
    traceway.CaptureException(fmt.Errorf("failed to read session recording (key=%s): %w", key, err))
}

// WRONG - Do not use log.Printf for errors
if err != nil {
    log.Printf("Failed to read session recording (key=%s): %v", key, err)
}

Validation error conventions:

  • 400 Bad Request: Malformed requests, missing required params, type errors
  • 422 Unprocessable Entity: Business validation in form dialogs (name too long, duplicate name, required field empty). Return c.JSON(422, gin.H{"error": "descriptive message"}). The frontend api.ts extracts 422 error messages — dialogs catch and display them inline.

Summary:

  • Stopping errors (abort the request): c.AbortWithError(status, traceway.NewStackTraceErrorf("reason: %w", err))
  • Non-stopping errors (continue serving): traceway.CaptureException(fmt.Errorf("reason: %w", err))
  • Validation errors (user-facing): c.JSON(422, gin.H{"error": "message"}) for form validation
  • Always wrap errors with traceway.NewStackTraceErrorf or fmt.Errorf using %w — never discard the original error

Emails

Every email Traceway sends is a hardcoded template in backend/app/services/emailtemplates/ (embedded with go:embed, parsed once at init). The copy of an email lives in its own .gohtml file; the payload is the only thing it interpolates. There is no shared layout document, so changing one email cannot reshape the others, and every file is a complete HTML document styled like the rest.

Templates: new_error, error_regression, check_down, check_recovered, alert (the threshold email every rate/latency/apdex/throughput/task/impact/cost rule sends), ai_flagged, page (on-call escalation), test (both test buttons), invitation, password_reset.

  • One send path — services.SendEmail(ctx, services.Email{...}) in backend/app/services/email.service.go is the only way mail leaves the process: it renders the template, wraps it as multipart/alternative (plaintext part first, quoted-printable), and talks SMTP. With SMTP_ENABLED off it logs the plaintext instead. services.RenderEmail is the render-only half, used by the preview endpoint.
  • services.Email carries the chrome every template shares (Title, Badge/BadgeColor via EmailColor*, URL, Footer, LogoURL defaulting to {APP_BASE_URL}/traceway-mark.png) plus Template and Data, the typed payload reached as {{.Data.X}}. Text is the plaintext alternative and must stand on its own.
  • Notification payloads live on models.NotificationMessage.Email (models.NotificationEmail: a template name plus one typed struct — EmailException, EmailCheck, EmailAlert, EmailFlagged, EmailPage, EmailTest). The builders in notifications/messages.go attach it; services.NotificationEmail turns a message into the Email at send time (severity chip, absolute link, "why you got this" footer). It rides through the notification outbox as JSON, so the shape is a wire format: add fields, never repurpose them. Only the email adapter reads it — Slack/Telegram/SMS/webhook and fired_notifications still see Body alone, so Body must always stay complete.
  • Adding an email: add emailtemplates/<name>.gohtml, add its payload struct, and call SendEmail with Template: "<name>". For a new rule type that only needs a sentence and a link, reuse alert by filling models.EmailAlert.
  • Preview: EMAIL_PREVIEW_ENABLED=true registers GET /api/email-preview (index) and GET /api/email-preview/:template, which renders any template with sample data (?format=text shows the plaintext alternative). Off by default, so it is never a route in production. Sample data lives in controllers/email_preview.controller.go.

Native /api/report Protocol

There is no Traceway Go SDK. It was retired, and every backend, Go included, instruments with OpenTelemetry and exports over OTLP/HTTP. The docs carry no Go SDK pages, and docs/public/_redirects 301s the old /client/sdk and /client/*-middleware URLs to /client/otel. Do not reintroduce them.

The native protocol is deprecated. Traceway is OpenTelemetry first, and the public docs do not describe /api/report as a way to send data: there is no protocol page (/protocol 301s to /client/otel) and docs/pages/learn/data-flow.mdx is written around OTLP. Do not document it again. The URL still shows up in the docs in one role only, as part of the connection string the browser and mobile SDKs are configured with.

The endpoint itself is still served, because those SDKs (@tracewayapp/*, the iOS and Android libraries, Flutter) speak it and the backend uses it to report its own telemetry. This section is the engineering reference for keeping it working. Its rows land in the same V2 tables as OTLP: a run is stored as the root span of a trace with its spans under it (see "Trace Storage (V2 tables)").

Data Format (Frame)

Clients send data as gzipped JSON. The wire shape is ReportRequest in backend/app/controllers/clientcontrollers/client.controller.go wrapping CollectionFrame from backend/app/models/clientmodels/:

{
  "appVersion": "1.2.3",
  "serverName": "myapp-host-1",
  "collectionFrames": [
    {
      "traces": [
        {
          "id": "5b8e1a2f-3c4d-4e5f-8a9b-0c1d2e3f4a5b",
          "endpoint": "GET /api/users",
          "duration": 45200000,
          "statusCode": 200,
          "recordedAt": "2024-01-15T10:30:00Z",
          "isTask": false,
          "attributes": {},
          "spans": []
        }
      ],
      "stackTraces": [
        {
          "stackTrace": "RuntimeError: connection refused\n  at ...",
          "recordedAt": "2024-01-15T10:30:00Z",
          "attributes": {"user_id": "123"},
          "isMessage": false,
          "isTask": false
        }
      ],
      "metrics": [
        {
          "name": "cpu.used_pcnt",
          "value": 45.2,
          "recordedAt": "2024-01-15T10:30:00Z",
          "tags": {}
        }
      ]
    }
  ]
}

Notes:

  • Timestamps are recordedAt (RFC 3339), never timestamp. duration is a Go time.Duration, i.e. integer nanoseconds (45200000 = 45.2ms).
  • Endpoints vs tasks share the traces array, split by isTask. CollectionFrame also carries sessionRecordings and sessions.
  • Each entry in a trace's spans may carry an optional parentSpanId (UUID) and attributes (string map). A span that names no parent hangs under the run's own span. The ids are stored as 32 hex characters.
  • Unknown fields are silently ignored: a payload in the wrong shape (e.g. top-level metrics) still returns 200 but inserts nothing. When hand-crafting test payloads, confirm ingestion landed via POST /api/metrics/query (or the relevant list endpoint) instead of trusting the status code.

Common Patterns

Adding a New API Endpoint

  1. Add model in backend/app/models/

    type NewEntity struct {
        ID        uuid.UUID `json:"id"`
        Name      string    `json:"name"`
        CreatedAt time.Time `json:"created_at"`
    }
    
  2. Add repository in backend/app/repositories/

    func (r *NewEntityRepository) GetAll(projectID uuid.UUID) ([]models.NewEntity, error) {
        // ClickHouse query
    }
    
  3. Add controller in backend/app/controllers/

    func (c *NewEntityController) List(ctx *gin.Context) {
        entities, err := repo.GetAll(projectID)
        ctx.JSON(200, entities)
    }
    
  4. Register route in backend/app/controllers/routes.go

    api.GET("/new-entities", newEntityController.List)
    
  5. Add frontend API call in frontend/src/lib/api.ts or directly in page

POST body convention: List/search endpoints use POST with a JSON body containing filters and pagination:

type ListRequest struct {
    ProjectId  string           `json:"projectId"`
    FromDate   string           `json:"fromDate"`
    ToDate     string           `json:"toDate"`
    OrderBy    string           `json:"orderBy"`
    Search     string           `json:"search"`
    Pagination PaginationParams `json:"pagination"`
}

Paginated response: Use PaginatedResponse[T] from routes.go for all paginated endpoints:

type PaginatedResponse[T any] struct {
    Data       []T        `json:"data"`
    Pagination Pagination `json:"pagination"`
}

type Pagination struct {
    Page       int   `json:"page"`
    PageSize   int   `json:"pageSize"`
    Total      int64 `json:"total"`
    TotalPages int64 `json:"totalPages"`
}

type PaginationParams struct {
    Page     int `json:"page" binding:"min=1"`
    PageSize int `json:"pageSize" binding:"min=1,max=100"`
}

Adding a New Frontend Page

  1. Create route folder: frontend/src/routes/new-page/

  2. Add page component: +page.svelte with the standard loading pattern:

    <script lang="ts">
      import { onMount } from 'svelte'
      import { api } from '$lib/api'
      import { projectsState } from '$lib/state/projects.svelte'
      import { ErrorDisplay } from '$lib/components/ui/error-display'
      import { getErrorMessage, getErrorStatus } from '$lib/utils/errors'
    
      let data = $state<DataType[]>([])
      let loading = $state(true)
      let error = $state('')
      let notFound = $state(false)
    
      async function loadData() {
        loading = true
        error = ''
        try {
          const response = await api.post<ResponseType>('/endpoint', payload, {
            projectId: projectsState.currentProjectId ?? undefined
          })
          data = response.data || []
        } catch (e) {
          if (getErrorStatus(e) === 404) {
            notFound = true
          } else {
            error = getErrorMessage(e) || 'Failed to load data'
          }
        } finally {
          loading = false
        }
      }
    
      onMount(() => { loadData() })
    </script>
    
    {#if loading}
      <LoadingCircle size="xlg" />
    {:else if notFound}
      <ErrorDisplay status={404} title="Not Found" description="..." onRetry={() => loadData()} />
    {:else if error}
      <ErrorDisplay status={400} title="Error" description={error} onRetry={() => loadData()} />
    {:else}
      <!-- Content -->
    {/if}
    
  3. Add data loading (optional): +page.ts for URL params

    export const load = async ({ params }) => {
        return { param: params.id }
    }
    
  4. Add navigation in src/lib/components/app-sidebar.svelte

Adding a New Metric to Dashboard

  1. Ensure SDK captures metric (or add to traceway.go metrics collection)

  2. Add repository query to each telemetry backend's metric_point.repository.go under backend/app/repositories/telemetry/{clickhouse,sqlite,duckdb}/

    func (r *metricPointRepository) GetNewMetric(ctx context.Context, projectId uuid.UUID, from, to time.Time) ([]models.TimeSeriesPoint, error) {
        // Query metric_points with the backend's dialect
    }
    
  3. Add to dashboard controller in backend/app/controllers/dashboard.go

  4. Frontend auto-renders from API response (metrics dashboard uses dynamic rendering)

Adding a Database Column

  1. Create migration file (remember: ONE statement per file!)

    backend/app/migrations/ch/0015_add_new_column.up.sql
    
    ALTER TABLE transactions ADD COLUMN new_column String DEFAULT ''
    
  2. Update model in backend/app/models/

  3. Update repository queries to include new column

  4. Run migrations: Backend runs migrations automatically on startup

Adding Table Sorting to a Page

  1. Import and add state:

    import { getSortState, setSortState, handleSortClick } from '$lib/utils/sort-storage'
    import type { SortState } from '$lib/utils/sort-storage'
    
    let sortState = $state<SortState>(getSortState('page-key', { field: 'default_column', direction: 'desc' }))
    
  2. Add sort handler:

    function onSortClick(field: string) {
        sortState = handleSortClick(field, sortState.field, sortState.direction, 'desc')
        setSortState('page-key', sortState)
    }
    
  3. Use TracewayTableHeader:

    <TracewayTableHeader
        label="Column"
        column="column_name"
        orderBy={`${sortState.field} ${sortState.direction}`}
        onclick={() => onSortClick('column_name')}
    />
    
  4. Pass to API call - convert to backend format: "column asc" or "column desc":

    const orderBy = `${sortState.field} ${sortState.direction}`
    

Adding a New Framework

Backends do not get a new framework value. OpenTelemetry is the single backend integration path, and opentelemetry is the only backend option in the project-creation picker (plus the preselected default). A new backend language or web framework is a documentation and Connection-page change, never a new project framework:

  1. Connection page targets — frontend/src/lib/utils/otel-setup.ts: add the framework to the matching OTEL_TARGETS language entry (or add a new language target) and its setup steps
  2. Docs guide — docs/pages/client/otel/<framework>/: create _meta.json and index.mdx, then list it in docs/pages/client/otel/_meta.json and the "Next Steps" block of docs/pages/client/otel/index.mdx
  3. Docs OTel language table — docs/pages/client/otel/index.mdx: add a row
  4. Docs framework picker — nothing to do. docs/components/FrameworkPicker.jsx carries exactly one backend card, OpenTelemetry, pointing at /client/otel, so it mirrors the dashboard's project-creation picker. Do not add a per-framework backend card; the guide is discovered from /client/otel instead.
  5. Combobox search keywords — frontend/src/lib/components/framework-combobox.svelte: append the name to the OpenTelemetry entry's keywords so searching for it still finds the right option
  6. README — add to the supported-frameworks table if it warrants a row

Only a browser or mobile framework needs a real new framework value, because those use platform SDKs with their own Connection-page content:

  1. Backend — backend/app/controllers/project.controller.go: add to validFrameworks and update the validation error message
  2. Frontend state — frontend/src/lib/state/projects.svelte.ts: add to the Framework union, FRAMEWORK_LABELS, and MOBILE_FRAMEWORKS/FRONTEND_FRAMEWORKS
  3. Frontend combobox — frontend/src/lib/components/framework-combobox.svelte: add an entry to the Browser or Mobile group with keywords
  4. Frontend icon — frontend/src/lib/components/framework-icon.svelte: add the icon mapping
  5. Framework code — frontend/src/lib/utils/framework-code.ts: add install command, integration snippet, label, code language, and testing routes
  6. Connection page — frontend/src/routes/connection/+page.svelte: add highlight language mapping and install description
  7. Dashboard page — frontend/src/routes/+page.svelte: add highlight language mapping
  8. Docs — docs/pages/client/<framework>/, docs/pages/client/_meta.json, SDK_OPTIONS + FOLDER_SDK in docs/components/SdkContext.jsx, SDK_VISIBILITY in docs/theme.config.jsx, SDK_QUICK_START in docs/components/SdkSelector.jsx, and a FrameworkPicker.jsx card

Values removed from the picker (gin, fiber, chi, fasthttp, stdlib, custom, nextjs, nestjs, express, remix, hono, cloudflare, symfony, laravel, django) stay valid in validFrameworks and in FRAMEWORK_LABELS — existing projects still carry them and must keep rendering.