The only tool you need to know what is happening and how to fix it.
Go alertingexception-trackerissue-trackinglogsmetricsmonitoringobservability-platformopen-telemetryotelsession-replaytelemetrytraces
cleanup + docs update + minor bug fixes
narrow-adventure3e9884549e39
diff --git a/.github/workflows/frontend.yml b/.github/workflows/frontend.ymlindex c529e40d..3ee24c65 100644--- a/.github/workflows/frontend.yml+++ b/.github/workflows/frontend.yml@@ -9,11 +9,13 @@ on:branches: [main]paths:- "frontend/**"+ - ".nvmrc"- ".github/workflows/frontend.yml"pull_request:types: [labeled]paths:- "frontend/**"+ - ".nvmrc"- ".github/workflows/frontend.yml"workflow_dispatch:@@ -37,13 +39,10 @@ jobs:- name: Checkoutuses: actions/checkout@v7- # engines.node in frontend/package.json is the single source of truth for- # the Node version -- flake.nix reads that same field to build the dev- # shell, so CI and `nix develop` cannot drift.- name: Setup Nodeuses: actions/setup-node@v7with:- node-version-file: frontend/package.json+ node-version-file: .nvmrccache: npmcache-dependency-path: frontend/package-lock.jsondiff --git a/.github/workflows/release-docs.yml b/.github/workflows/release-docs.ymlindex 6cf77d8a..9485a1ff 100644--- a/.github/workflows/release-docs.yml+++ b/.github/workflows/release-docs.yml@@ -35,10 +35,8 @@ jobs:with:apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }}accountId: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}- # wrangler-action v4 defaults to the Wrangler 4 CLI, a separate- # migration. Pinned so this change is only the runtime move.- # Unpinning is tracked in #331.- wranglerVersion: "3"+ # Wrangler 4 is a separate migration; exact so the action cannot resolve a newer 3.x at deploy time.+ wranglerVersion: "3.114.17"workingDirectory: docscommand: deploydiff --git a/.github/workflows/release-helm.yml b/.github/workflows/release-helm.ymlindex 2399e49b..480de350 100644--- a/.github/workflows/release-helm.yml+++ b/.github/workflows/release-helm.yml@@ -93,7 +93,6 @@ jobs:# one, so it is the single site that needs the version stated. Exact,# not a bare major: "3" is a range the action resolves to the newest# 3.x at deploy time, which is the drift this pin exists to stop.- # Tracked with the other two in #331.wranglerVersion: "3.114.17"command: pages deploy deploy --project-name traceway-charts --commit-dirty=truediff --git a/.github/workflows/release-website.yml b/.github/workflows/release-website.ymlindex 219406cb..3f5f2aa3 100644--- a/.github/workflows/release-website.yml+++ b/.github/workflows/release-website.yml@@ -35,11 +35,8 @@ jobs:with:apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }}accountId: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}- # wrangler-action v4 defaults to the Wrangler 4 CLI, a separate- # migration on a production deploy path. Pinned so this change is- # only the runtime move.- # Unpinning is tracked in #331.- wranglerVersion: "3"+ # Wrangler 4 is a separate migration; exact so the action cannot resolve a newer 3.x at deploy time.+ wranglerVersion: "3.114.17"workingDirectory: websitecommand: deploydiff --git a/CLAUDE.md b/CLAUDE.mdindex 5af41a58..48bb9258 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -41,7 +41,7 @@ Traceway is an error tracking and monitoring platform consisting of:Set `JWT_SECRET` (min 32 characters) before running the backend — it is the one variable with no default in the standalone binary (the all-in-one Docker image bakes a public one in via `backend/.env.docker`, which must be overridden in production). `SQLITE_PATH` sets the database location, defaulting to `./traceway.db` in the working directory.-CI on pull requests is opt-in. `backend.yml`, `backend-vulncheck.yml`, `cli.yml`, `cli-lint.yml` and `cli-contract.yml` listen for `pull_request: types: [labeled]` and every job is gated on `github.event.label.name == 'ci'`, so a PR runs nothing until a maintainer applies the `ci` label (only collaborators can), and applying any other label runs nothing either. Each application validates the PR as it is at that moment: to validate later pushes, remove and re-add the label. The same workflows still run on pushes to `main` that touch their paths (`backend/**`, `cli/**`, their own workflow file), on their daily schedules (the two vulnchecks), and from the Actions tab via `workflow_dispatch`.+CI on pull requests is opt-in. `backend.yml`, `backend-vulncheck.yml`, `cli.yml`, `cli-lint.yml`, `cli-contract.yml` and `frontend.yml` listen for `pull_request: types: [labeled]` and every job is gated on `github.event.label.name == 'ci'`, so a PR runs nothing until a maintainer applies the `ci` label (only collaborators can), and applying any other label runs nothing either. Each application validates the PR as it is at that moment: to validate later pushes, remove and re-add the label. The same workflows still run on pushes to `main` that touch their paths (`backend/**`, `cli/**`, `frontend/**`, `skills/traceway/**` for `cli.yml`, `.nvmrc` for `frontend.yml`, their own workflow file), on their daily schedules (the two vulnchecks), and from the Actions tab via `workflow_dispatch`.Action versions are watched by `.github/dependabot.yml` (the `github-actions` ecosystem only; npm and Go modules are deliberately not watched there). It opens a weekly PR per group — `actions/*` in one, every third-party action in the other — and deliberately does **not** auto-apply `ci`, so a bot PR passes the same collaborator gate as any other. Label the first-party group to validate it; `golangci/golangci-lint-action` is reachable the same way via `cli-lint.yml`. The rest of the third-party group lands on `release-*.yml`, `benchmark-*.yml` and `traceway-autofix.yml`, which no label reaches — and `workflow_dispatch` is **not** a dry run on the release path: dispatching `release-docs.yml` or `release-website.yml` deploys to Cloudflare for real, and the other `release-*.yml` workflows publish real assets, so dispatching one from an unreviewed bot branch ships it. Validate those by reading the action's release notes instead, and dispatch only where the target is disposable. `traceway-autofix.yml` has no dispatch at all, so its `claude-code-action` pin — the most privileged action here — has no pre-merge validation path. Note too that labelling a PR which touches no gated workflow yields zero checks, which reads like a pass but is silence. Every updatable pin is a bare major tag, so Dependabot only ever proposes major bumps here.@@ -581,7 +581,7 @@ backend/│ ├── middleware/│ │ ├── auth.go # Token validation│ │ └── gzip.go # Request decompression-│ ├── cache/ # In-memory project token cache+│ ├── cache/ # In-memory project cache; on Postgres, replicas refresh each other through LISTEN/NOTIFY (project_cache_changed, sent by the project repository inside the write transaction)│ ├── pgdb/ # PostgreSQL connection manager│ └── migrations/│ ├── ch/ # ClickHouse migrations@@ -757,6 +757,7 @@ Frontend routes: `/ai-traces` (tabs: Traces, Conversations, Users), `/ai-traces/**Organization Management**| Method | Endpoint | Auth | Purpose ||--------|----------|------|---------|+| POST | `/api/organizations` | App | Create an organization owned by the caller; the recovery path for a user removed from their last organization (self-hosted: 422 once one exists; cloud: `OrganizationLimitHook`). Runs `PostRegistrationHooks` like register and SSO finish-setup || GET | `/api/organizations/:orgId/settings` | Admin | Get org settings || PUT | `/api/organizations/:orgId/settings` | Admin | Update org settings || GET | `/api/organizations/:orgId/members` | Admin | List members |@@ -936,6 +937,8 @@ In SQLite mode (`DB_TYPE=sqlite`), the backend uses **two separate SQLite databa| **Main DB** | `db.DB` | `traceway.db` | PostgreSQL replacement — relational/config data | Yes (`middleware.Transactional`, `db.ExecuteTransaction`) || **Telemetry DB** | `db.TelemetryDB` | `traceway_telemetry.db` | ClickHouse replacement — append-only telemetry | No — direct inserts without transactions |+Both embedded builds (SQLite and DuckDB telemetry) are single-instance: the databases are local files, and the in-memory project cache only learns about another process's writes through the Postgres LISTEN/NOTIFY listener, which exists only under `transactional_pg`. Running more than one backend instance is supported only on the `transactional_pg telemetry_ch` build (docs: `/server/minimal#running-more-than-one-instance`).+**Main DB tables** (`db.DB` — transactional, uses lit with `*sql.Tx`):- `users`, `organizations`, `organization_users`, `projects`, `invitations`- `source_maps`, `metric_registry`, `dashboards`, `project_dashboards`, `dashboard_templates`, `starred_dashboard_widgets` (plus the legacy `widget_groups`/`widget_group_widgets`/`starred_widgets`)diff --git a/Dockerfile b/Dockerfileindex 4fcc1595..fb84e6ae 100644--- a/Dockerfile+++ b/Dockerfile@@ -7,10 +7,6 @@# ==============================================================================# Stage 1: Build Frontend# ==============================================================================-# NODE_VERSION must match .nvmrc, the repo-wide Node pin. release-traceway.yml-# passes it as a --build-arg read from that file, so a released image cannot be-# built on a different major from CI; the default below is what a bare-# `docker build` gets.ARG NODE_VERSION=26FROM node:${NODE_VERSION}-alpine AS frontend-builder@@ -105,8 +101,6 @@ VOLUME ["/var/lib/clickhouse", "/var/lib/postgresql/data"]EXPOSE 80 8082-# 300s, the Helm chart's default startupProbe budget: the server binds only after-# migrations run, and Docker ends the start period at the first healthy probe.HEALTHCHECK --interval=30s --timeout=10s --start-period=300s --retries=3 \CMD curl -f http://localhost/health || exit 1diff --git a/Dockerfile.browser b/Dockerfile.browserindex 0457e42b..6a61b5d6 100644--- a/Dockerfile.browser+++ b/Dockerfile.browser@@ -15,10 +15,6 @@# ==============================================================================# Stage 1: Build Frontend# ==============================================================================-# NODE_VERSION must match .nvmrc, the repo-wide Node pin. release-traceway.yml-# passes it as a --build-arg read from that file, so a released image cannot be-# built on a different major from CI; the default below is what a bare-# `docker build` gets.ARG NODE_VERSION=26FROM --platform=$BUILDPLATFORM node:${NODE_VERSION}-alpine AS frontend-builder@@ -112,8 +108,6 @@ ENV GIN_MODE=release \EXPOSE 80 8082-# 300s, the Helm chart's default startupProbe budget: the server binds only after-# migrations run, and Docker ends the start period at the first healthy probe.HEALTHCHECK --interval=30s --timeout=3s --start-period=300s --retries=3 \CMD wget -qO- http://localhost/health || exit 1diff --git a/Dockerfile.duckdb b/Dockerfile.duckdbindex 17a460e9..478f98fd 100644--- a/Dockerfile.duckdb+++ b/Dockerfile.duckdb@@ -11,10 +11,6 @@# ==============================================================================# Stage 1: Build Frontend# ==============================================================================-# NODE_VERSION must match .nvmrc, the repo-wide Node pin. release-traceway.yml-# passes it as a --build-arg read from that file, so a released image cannot be-# built on a different major from CI; the default below is what a bare-# `docker build` gets.ARG NODE_VERSION=26FROM --platform=$BUILDPLATFORM node:${NODE_VERSION}-alpine AS frontend-builder@@ -76,8 +72,6 @@ ENV GIN_MODE=release \EXPOSE 80 8082-# 300s, the Helm chart's default startupProbe budget: the server binds only after-# migrations run, and Docker ends the start period at the first healthy probe.HEALTHCHECK --interval=30s --timeout=3s --start-period=300s --retries=3 \CMD wget -qO- http://localhost/health || exit 1diff --git a/Dockerfile.minimal b/Dockerfile.minimalindex e6895dd1..42325882 100644--- a/Dockerfile.minimal+++ b/Dockerfile.minimal@@ -8,10 +8,6 @@# ==============================================================================# Stage 1: Build Frontend# ==============================================================================-# NODE_VERSION must match .nvmrc, the repo-wide Node pin. release-traceway.yml-# passes it as a --build-arg read from that file, so a released image cannot be-# built on a different major from CI; the default below is what a bare-# `docker build` gets.ARG NODE_VERSION=26FROM node:${NODE_VERSION}-alpine AS frontend-builder@@ -79,9 +75,6 @@ ENV GIN_MODE=release# 8082: Direct API accessEXPOSE 80 8082-# Health check using wget (alpine-native, no curl needed)-# 300s, the Helm chart's default startupProbe budget: the server binds only after-# migrations run, and Docker ends the start period at the first healthy probe.HEALTHCHECK --interval=30s --timeout=3s --start-period=300s --retries=3 \CMD wget -qO- http://localhost/health || exit 1diff --git a/Dockerfile.sqlite b/Dockerfile.sqliteindex 45f1fc1b..6911970a 100644--- a/Dockerfile.sqlite+++ b/Dockerfile.sqlite@@ -7,10 +7,6 @@# ==============================================================================# Stage 1: Build Frontend# ==============================================================================-# NODE_VERSION must match .nvmrc, the repo-wide Node pin. release-traceway.yml-# passes it as a --build-arg read from that file, so a released image cannot be-# built on a different major from CI; the default below is what a bare-# `docker build` gets.ARG NODE_VERSION=26FROM --platform=$BUILDPLATFORM node:${NODE_VERSION}-alpine AS frontend-builder@@ -78,8 +74,6 @@ ENV GIN_MODE=release \EXPOSE 80 8082-# 300s, the Helm chart's default startupProbe budget: the server binds only after-# migrations run, and Docker ends the start period at the first healthy probe.HEALTHCHECK --interval=30s --timeout=3s --start-period=300s --retries=3 \CMD wget -qO- http://localhost/health || exit 1diff --git a/backend/app/cache/project_cache_listener_pg.go b/backend/app/cache/project_cache_listener_pg.goindex cca62771..e270c179 100644--- a/backend/app/cache/project_cache_listener_pg.go+++ b/backend/app/cache/project_cache_listener_pg.go@@ -4,9 +4,10 @@ package cacheimport ("context"+ "fmt"- "github.com/tracewayapp/traceway/backend/app/config""github.com/tracewayapp/traceway/backend/app/db"+ traceway "go.tracewayapp.com")func (c *projectCache) startListener(ctx context.Context) error {@@ -22,15 +23,12 @@ func (c *projectCache) startListener(ctx context.Context) error {select {case <-ctx.Done():return- case notification, ok := <-listener.Notify:+ case _, ok := <-listener.Notify:if !ok {return}- if notification == nil {- continue- }if err := c.Refresh(ctx); err != nil {- config.Logf("project cache refresh failed after notification for project %s: %v", notification.Extra, err)+ traceway.CaptureException(fmt.Errorf("project cache refresh after notification failed: %w", err))}}}diff --git a/backend/app/controllers/ai_trace.controller.go b/backend/app/controllers/ai_trace.controller.goindex f583fae2..ae7465dc 100644--- a/backend/app/controllers/ai_trace.controller.go+++ b/backend/app/controllers/ai_trace.controller.go@@ -163,8 +163,6 @@ func (a aiTraceController) FindByTraceName(c *gin.Context) {return}- // Stats are best-effort: the trace list still renders without them, so a- // failure here is reported rather than aborting the request.stats, err := telemetry.AiTraceRepository.GetTraceNameStats(c, projectId, traceName, request.FromDate, request.ToDate)if err != nil {traceway.CaptureException(fmt.Errorf("failed to load ai trace stats (traceName=%s): %w", traceName, err))diff --git a/backend/app/controllers/member.controller.go b/backend/app/controllers/member.controller.goindex 962a787c..efba95b8 100644--- a/backend/app/controllers/member.controller.go+++ b/backend/app/controllers/member.controller.go@@ -50,6 +50,11 @@ func (c *memberController) UpdateRole(ctx *gin.Context) {return}+ if targetRole == currentUserRole {+ ctx.JSON(http.StatusForbidden, gin.H{"error": "Only owners can change another admin's role"})+ return+ }+if request.Role == "admin" && currentUserRole != "owner" {ctx.JSON(http.StatusForbidden, gin.H{"error": "Only owners can promote members to admin"})return@@ -85,11 +90,7 @@ func (c *memberController) RemoveMember(ctx *gin.Context) {return}- currentUserRole, err := transactional.OrganizationRepository.GetUserRole(tx, organizationId, currentUserId)- if err != nil {- ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("Failed to get user role: %w", err))- return- }+ currentUserRole := middleware.GetUserOrgRole(ctx)targetRole, err := transactional.OrganizationRepository.GetUserRole(tx, organizationId, targetUserId)if err != nil {ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("Failed to get user role: %w", err))@@ -106,7 +107,7 @@ func (c *memberController) RemoveMember(ctx *gin.Context) {}if targetRole == currentUserRole {- ctx.JSON(http.StatusForbidden, gin.H{"error": "Cannot remove the user with the same role"})+ ctx.JSON(http.StatusForbidden, gin.H{"error": "Cannot remove a member with the same role"})return}diff --git a/backend/app/controllers/member_role_test.go b/backend/app/controllers/member_role_test.gonew file mode 100644index 00000000..82a59c6e--- /dev/null+++ b/backend/app/controllers/member_role_test.go@@ -0,0 +1,97 @@+//go:build !telemetry_ch && !transactional_pg && !telemetry_duckdb++package controllers++import (+ "database/sql"+ "net/http/httptest"+ "strconv"+ "testing"++ "github.com/gin-gonic/gin"+ "github.com/tracewayapp/traceway/backend/app/db"+ "github.com/tracewayapp/traceway/backend/app/middleware"+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional"+)++func addOrgMember(t *testing.T, tx *sql.Tx, orgId int, email, role string) int {+ t.Helper()+ user, err := transactional.UserRepository.Create(tx, email, "Test User", "hashed-password")+ if err != nil {+ t.Fatalf("create user: %v", err)+ }+ if _, err := transactional.OrganizationRepository.AddUser(tx, orgId, user.Id, role); err != nil {+ t.Fatalf("add user to org: %v", err)+ }+ return user.Id+}++func memberRequest(t *testing.T, tx *sql.Tx, orgId, callerId int, callerRole, method string, targetId int, body string) (*gin.Context, *httptest.ResponseRecorder) {+ t.Helper()+ c, recorder := newControllerTestContext(t, tx, callerId, method, "/organizations/"+strconv.Itoa(orgId)+"/members/"+strconv.Itoa(targetId), body)+ c.Set(middleware.OrganizationIdContextKey, orgId)+ c.Set(middleware.UserOrgRoleContextKey, callerRole)+ c.Params = gin.Params{{Key: "organizationId", Value: strconv.Itoa(orgId)}, {Key: "userId", Value: strconv.Itoa(targetId)}}+ return c, recorder+}++func roleOf(t *testing.T, tx *sql.Tx, orgId, userId int) string {+ t.Helper()+ role, err := transactional.OrganizationRepository.GetUserRole(tx, orgId, userId)+ if err != nil {+ t.Fatalf("get role: %v", err)+ }+ return role+}++func TestAdminCannotDemoteThenRemoveAnotherAdmin(t *testing.T) {+ setupSetupControllerDB(t)+ tx, err := db.DB.Begin()+ if err != nil {+ t.Fatalf("begin: %v", err)+ }+ t.Cleanup(func() { tx.Rollback() })++ ownerId, orgId := createSetupTestAccount(t, tx, "[email protected]", "owner")+ adminA := addOrgMember(t, tx, orgId, "[email protected]", "admin")+ adminB := addOrgMember(t, tx, orgId, "[email protected]", "admin")+ member := addOrgMember(t, tx, orgId, "[email protected]", "user")++ c, rec := memberRequest(t, tx, orgId, adminA, "admin", "PUT", adminB, `{"role":"user"}`)+ MemberController.UpdateRole(c)+ if rec.Code != 403 {+ t.Fatalf("admin demoting another admin: expected 403, got %d %s", rec.Code, rec.Body.String())+ }+ if got := roleOf(t, tx, orgId, adminB); got != "admin" {+ t.Fatalf("admin B role changed to %q", got)+ }++ c, rec = memberRequest(t, tx, orgId, adminA, "admin", "DELETE", adminB, "")+ MemberController.RemoveMember(c)+ if rec.Code != 403 {+ t.Fatalf("admin removing another admin: expected 403, got %d %s", rec.Code, rec.Body.String())+ }+ if got := roleOf(t, tx, orgId, adminB); got != "admin" {+ t.Fatalf("admin B was removed, role now %q", got)+ }++ c, rec = memberRequest(t, tx, orgId, adminA, "admin", "PUT", member, `{"role":"readonly"}`)+ MemberController.UpdateRole(c)+ if rec.Code != 200 {+ t.Fatalf("admin demoting a user: expected 200, got %d %s", rec.Code, rec.Body.String())+ }++ c, rec = memberRequest(t, tx, orgId, ownerId, "owner", "PUT", adminB, `{"role":"user"}`)+ MemberController.UpdateRole(c)+ if rec.Code != 200 {+ t.Fatalf("owner demoting an admin: expected 200, got %d %s", rec.Code, rec.Body.String())+ }+ c, rec = memberRequest(t, tx, orgId, ownerId, "owner", "DELETE", adminA, "")+ MemberController.RemoveMember(c)+ if rec.Code != 200 {+ t.Fatalf("owner removing an admin: expected 200, got %d %s", rec.Code, rec.Body.String())+ }+ if got := roleOf(t, tx, orgId, adminA); got != "" {+ t.Fatalf("admin A still a member with role %q", got)+ }+}diff --git a/backend/app/controllers/organization.controller.go b/backend/app/controllers/organization.controller.goindex 34f3550b..8cc8fcf5 100644--- a/backend/app/controllers/organization.controller.go+++ b/backend/app/controllers/organization.controller.go@@ -96,7 +96,7 @@ func (c *organizationController) UpdateSettings(ctx *gin.Context) {var req UpdateSettingsRequestif err := ctx.ShouldBindJSON(&req); err != nil {- ctx.JSON(http.StatusBadRequest, gin.H{"error": "Invalid request"})+ middleware.RejectBindError(ctx, err, "Invalid request")return}@@ -122,41 +122,23 @@ type CreateOrganizationRequest struct {Timezone string `json:"timezone"`}-func validateOrganizationName(name string) string {- nameLen := utf8.RuneCountInString(name)- if nameLen < 1 || nameLen > 100 {- return "Organization name must be between 1 and 100 characters"- }- return ""-}--// Create lets an authenticated user start a new organization they own. Every-// other org-creating path (register, SSO finish-setup) is an account-creation-// path, so a user removed from their last organization otherwise has no way-// back into a usable account.func (c *organizationController) Create(ctx *gin.Context) {var request CreateOrganizationRequestif err := ctx.ShouldBindJSON(&request); err != nil {- ctx.JSON(http.StatusBadRequest, gin.H{"error": "Organization name is required"})+ middleware.RejectBindError(ctx, err, "Organization name is required")return}name := strings.TrimSpace(request.Name)- if message := validateOrganizationName(name); message != "" {- ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": message})+ if nameLen := utf8.RuneCountInString(name); nameLen < 1 || nameLen > 100 {+ ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": "Organization name must be between 1 and 100 characters"})return}- // Register binds the timezone as required. Here it is optional: the recovery- // form sends one, but this is a plain authenticated endpoint a CLI or script- // can also reach, and an org whose on-call schedules resolve against UTC is a- // better outcome than refusing to create it at all.timezone := strings.TrimSpace(request.Timezone)if timezone == "" {timezone = "UTC"}- // On-call schedule resolution is tz-aware calendar math, so an unparseable- // zone would surface much later as wrong shift boundaries.if _, err := oncall.LoadTimezone(timezone); err != nil {ctx.JSON(http.StatusUnprocessableEntity, gin.H{"error": err.Error()})return@@ -165,13 +147,7 @@ func (c *organizationController) Create(ctx *gin.Context) {userId := middleware.GetUserId(ctx)tx := db.GetTx(ctx)- // Self-hosted instances allow exactly one organization; Register enforces the- // same rule (as a 409). This answers 422 instead because the message has to- // reach the recovery form, and api.ts only extracts bodies from 401/403/422 --- // a 409 would surface to the user as "API Error: Conflict".- // Without this an authenticated user of any role -- readonly- // included -- could mint an organization here and own it, since the route- // carries no role guard and OrganizationLimitHook is nil outside cloud.+ // One organization per self-hosted instance; 422 rather than Register's 409 so the message reaches the recovery form.if config.Config.CloudMode != "true" {hasOrganizations, err := transactional.OrganizationRepository.HasOrganizations(tx)if err != nil {@@ -217,9 +193,6 @@ func (c *organizationController) Create(ctx *gin.Context) {return}- // Register and FinishSetup both run these for every new org+owner pair; it is- // the cloud build's provisioning seam, so an organization created here must- // not skip it.for _, hook := range PostRegistrationHooks {if err := hook(tx, org, user); err != nil {ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("post-registration hook failed: %w", err))diff --git a/backend/app/db/db_transactional_pg.go b/backend/app/db/db_transactional_pg.goindex 6b7e748a..a1d00d66 100644--- a/backend/app/db/db_transactional_pg.go+++ b/backend/app/db/db_transactional_pg.go@@ -20,11 +20,7 @@ func NewPostgresListener() *pq.Listener {return pq.NewListener(postgresConnectionString(), 10*time.Second, time.Minute, nil)}-func NotifyProjectCacheChanged(tx *sql.Tx, projectId uuid.UUID) error {- return notifyProjectCacheChanged(tx, projectId)-}--func notifyProjectCacheChanged(execer interface {+func NotifyProjectCacheChanged(execer interface {Exec(query string, args ...any) (sql.Result, error)}, projectId uuid.UUID) error {_, err := execer.Exec("SELECT pg_notify($1, $2)", ProjectCacheNotificationChannel, projectId.String())diff --git a/backend/app/db/db_transactional_pg_test.go b/backend/app/db/db_transactional_pg_test.goindex ede4b690..41c2fcd0 100644--- a/backend/app/db/db_transactional_pg_test.go+++ b/backend/app/db/db_transactional_pg_test.go@@ -25,7 +25,7 @@ func TestNotifyProjectCacheChangedIncludesProjectId(t *testing.T) {projectId := uuid.New()execer := &recordingExecer{}- if err := notifyProjectCacheChanged(execer, projectId); err != nil {+ if err := NotifyProjectCacheChanged(execer, projectId); err != nil {t.Fatalf("notify project cache changed: %v", err)}if execer.query != "SELECT pg_notify($1, $2)" {diff --git a/backend/app/migrations/migrate_logger.go b/backend/app/migrations/migrate_logger.goindex c18ef332..7d5de90b 100644--- a/backend/app/migrations/migrate_logger.go+++ b/backend/app/migrations/migrate_logger.go@@ -9,16 +9,9 @@ import ("github.com/tracewayapp/traceway/backend/app/config")-// golang-migrate announces the migration it is about to run only in verbose-// mode, and that is the line an operator needs while a slow index build holds-// up boot. Verbose also emits the read-ahead goroutine's buffering lines, which-// name the *next* migration while the current one is still running and would-// read as progress that is not happening, so they are dropped here.+// Verbose mode is what names the running migration; its read-ahead lines name the next one and are dropped.var migrateReadAheadPrefixes = []string{"Start buffering", "Scheduled"}-// migrateLogger gives the ClickHouse and Postgres paths the per-migration-// progress that runMigrationsOn logs itself on the embedded stores: one line-// naming each migration as it starts, one more with its elapsed time.type migrateLogger struct {store string}diff --git a/backend/app/migrations/sqlite_telemetry/0021_index_session_and_log_trace_time.up.sql b/backend/app/migrations/sqlite_telemetry/0023_index_session_and_log_trace_time.up.sqlsimilarity index 100%rename from backend/app/migrations/sqlite_telemetry/0021_index_session_and_log_trace_time.up.sqlrename to backend/app/migrations/sqlite_telemetry/0023_index_session_and_log_trace_time.up.sqldiff --git a/backend/app/migrations/sqlite_telemetry/0022_index_fired_notifications_project_fired.up.sql b/backend/app/migrations/sqlite_telemetry/0024_index_fired_notifications_project_fired.up.sqlsimilarity index 100%rename from backend/app/migrations/sqlite_telemetry/0022_index_fired_notifications_project_fired.up.sqlrename to backend/app/migrations/sqlite_telemetry/0024_index_fired_notifications_project_fired.up.sqldiff --git a/backend/app/repositories/telemetry/clickhouse/endpoint.repository.go b/backend/app/repositories/telemetry/clickhouse/endpoint.repository.goindex b272081f..004b56fb 100644--- a/backend/app/repositories/telemetry/clickhouse/endpoint.repository.go+++ b/backend/app/repositories/telemetry/clickhouse/endpoint.repository.go@@ -112,9 +112,7 @@ func endpointFilterClause(col, search, rootFilter, methodFilter string) (string,clause += " AND " + col + "is_root = 0"}if methodFilter != "" {- // upper() not upperUTF8(): ASCII is the whole of RFC 9110's method token- // range, and the embedded backends' UPPER() is ASCII-only, so this keeps- // all three answering identically.+ // upper(), not upperUTF8(): method tokens are ASCII, matching the embedded backends' ASCII-only UPPER().clause += " AND startsWith(upper(" + col + "endpoint), ?)"args = append(args, shared.MethodPrefix(methodFilter))}diff --git a/backend/app/repositories/telemetry/duckdb/endpoint.repository.go b/backend/app/repositories/telemetry/duckdb/endpoint.repository.goindex 7357a015..f47e01f8 100644--- a/backend/app/repositories/telemetry/duckdb/endpoint.repository.go+++ b/backend/app/repositories/telemetry/duckdb/endpoint.repository.go@@ -823,10 +823,6 @@ func sortEndpointStats(stats []models.EndpointStats, orderBy string, sortDirectidesc := sortDirection != "asc"sort.Slice(stats, func(i, j int) bool {- // Descending is the ascending comparison with the operands swapped.- // Negating the ascending result instead would return true for both- // (i, j) and (j, i) whenever two rows tie, which is not the strict- // weak ordering sort.Slice documents a requirement for.if desc {i, j = j, i}diff --git a/backend/app/repositories/telemetry/helpers_test.go b/backend/app/repositories/telemetry/helpers_test.goindex e58ae2a3..31458cee 100644--- a/backend/app/repositories/telemetry/helpers_test.go+++ b/backend/app/repositories/telemetry/helpers_test.go@@ -67,9 +67,7 @@ func makeAiTrace(projectId uuid.UUID, traceName string, duration time.Duration,TraceName: traceName,ServerName: "test-server",AppVersion: "1.0.0",- // Unlike makeTask/makeEndpoint, AI traces default to root -- callers that- // need a child span go through makeAiTraceWithRoot.- IsRoot: true,+ IsRoot: true,}}diff --git a/backend/app/repositories/telemetry/shared/sqlfilter.go b/backend/app/repositories/telemetry/shared/sqlfilter.goindex 54ab057d..1e17f571 100644--- a/backend/app/repositories/telemetry/shared/sqlfilter.go+++ b/backend/app/repositories/telemetry/shared/sqlfilter.go@@ -19,20 +19,11 @@ func RootFilterClause(qualifiedCol, rootFilter string) string {}}-// MethodPrefix is the uppercase "GET " form every backend compares against.-// The stored method is whatever the instrumented service reported ---// getHTTPEndpoint concatenates http.request.method verbatim -- so the column-// side is upper-cased at query time rather than the value being trusted.func MethodPrefix(method string) string {return strings.ToUpper(method) + " "}-// MethodFilterClause matches the method case-insensitively. SUBSTR rather than-// LIKE because methodFilter reaches SQL as a value and LIKE would read % and _-// in it as wildcards; UPPER around the column rather than a second bound-// parameter so the comparison cannot depend on how the caller cased its input.-// UPPER is ASCII-only in SQLite, which is exactly the range RFC 9110 allows a-// method token.+// UPPER on the column side so stored lowercase methods still match; SUBSTR rather than LIKE so % and _ in the value stay literal.func MethodFilterClause(qualifiedCol, method string) (clause string, param string) {if method == "" {return "", ""diff --git a/backend/app/repositories/telemetry/sqlite/ai_trace.repository.go b/backend/app/repositories/telemetry/sqlite/ai_trace.repository.goindex 05342baf..07e58a1f 100644--- a/backend/app/repositories/telemetry/sqlite/ai_trace.repository.go+++ b/backend/app/repositories/telemetry/sqlite/ai_trace.repository.go@@ -231,10 +231,6 @@ func (r *aiTraceRepository) InsertAsync(ctx context.Context, lines []models.AiTrreturn tx.Commit()}-// aiTraceFilterClause appends the AI-traces list's filter to a WHERE clause and-// binds what it needs into params. Both the query that selects a trace_name and-// the query that reads its durations go through it, so they cannot filter-// differently.func aiTraceFilterClause(params lit.P, search, rootFilter string) string {clause := ""if search != "" {diff --git a/backend/app/repositories/telemetry/sqlite/endpoint.repository.go b/backend/app/repositories/telemetry/sqlite/endpoint.repository.goindex d547b39b..45c4d47b 100644--- a/backend/app/repositories/telemetry/sqlite/endpoint.repository.go+++ b/backend/app/repositories/telemetry/sqlite/endpoint.repository.go@@ -838,10 +838,6 @@ func (e *endpointRepository) UpsertSlowEndpoint(ctx context.Context, projectId u// --- helpers ----// fetchSortedDurations runs the same filter as the query that selected the-// endpoint, via endpointFilterClause rather than picking out the one leg that-// constrains rows today, so a row-level predicate added there cannot silently-// leave the percentiles describing a different population.func fetchSortedDurations(ctx context.Context, projectId uuid.UUID, endpoint string, from, to time.Time, search, rootFilter, methodFilter string) ([]float64, error) {params := lit.P{"project_id": projectId, "endpoint": endpoint, "from": sqlitetypes.NewSQLiteTime(from), "to": sqlitetypes.NewSQLiteTime(to)}filterClause := endpointFilterClause(params, "", search, rootFilter, methodFilter)@@ -863,10 +859,6 @@ func sortEndpointStats(stats []models.EndpointStats, orderBy string, sortDirectidesc := sortDirection != "asc"sort.Slice(stats, func(i, j int) bool {- // Descending is the ascending comparison with the operands swapped.- // Negating the ascending result instead would return true for both- // (i, j) and (j, i) whenever two rows tie, which is not the strict- // weak ordering sort.Slice documents a requirement for.if desc {i, j = j, i}diff --git a/backend/app/repositories/telemetry/sqlite/exception_stack_trace.repository.go b/backend/app/repositories/telemetry/sqlite/exception_stack_trace.repository.goindex baf066ff..84479747 100644--- a/backend/app/repositories/telemetry/sqlite/exception_stack_trace.repository.go+++ b/backend/app/repositories/telemetry/sqlite/exception_stack_trace.repository.go@@ -492,9 +492,8 @@ func (e *exceptionStackTraceRepository) FindByDistributedTraceId(ctx context.Con}// FindAllBySessionId returns all exceptions/messages stamped with the given session_id, ordered by time.-// The plain `session_id = ?` is what lets SQLite prove the partial idx_exceptions_session_recorded-// (WHERE session_id IS NOT NULL) applies; an IS, IN or COALESCE rewrite would fall back to the-// (project_id, recorded_at) index and re-filter the whole project.+// The plain `session_id = ?` is what lets SQLite use the partial idx_exceptions_session_recorded+// (WHERE session_id IS NOT NULL); an IS, IN or COALESCE rewrite falls back to a whole-project scan.func (e *exceptionStackTraceRepository) FindAllBySessionId(ctx context.Context, projectId, sessionId uuid.UUID) ([]models.ExceptionStackTrace, error) {rows, err := lit.SelectNamed[exceptionRow](db.TelemetryDB,`SELECT id, project_id, trace_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message, distributed_trace_id, session_iddiff --git a/backend/app/repositories/telemetry/sqlite/log_record.repository.go b/backend/app/repositories/telemetry/sqlite/log_record.repository.goindex 995f7cbb..ee752b49 100644--- a/backend/app/repositories/telemetry/sqlite/log_record.repository.go+++ b/backend/app/repositories/telemetry/sqlite/log_record.repository.go@@ -130,7 +130,12 @@ func (r *logRecordRepository) Search(ctx context.Context, params shared.LogSearctraceIdIn := ""if len(params.TraceIds) > 0 {- traceIdIn = " AND " + traceIdInClause(params.TraceIds, args)+ placeholders := make([]string, len(params.TraceIds))+ for i, tid := range params.TraceIds {+ placeholders[i] = fmt.Sprintf(":tid%d", i)+ args[fmt.Sprintf("tid%d", i)] = tid+ }+ traceIdIn = " AND trace_id IN (" + strings.Join(placeholders, ", ") + ")"}countResult, err := lit.SelectSingleNamed[models.CountResult](db.TelemetryDB,@@ -163,6 +168,9 @@ func (r *logRecordRepository) Search(ctx context.Context, params shared.LogSearcargs["limit"] = params.PageSizeargs["offset"] = offset+ // An IN list over trace ids cannot serve ORDER BY from the trace index, so without sqlite_stat1 the+ // planner walks the whole time window (#353): seek each id, keep enough rows to reach the offset,+ // merge. Past the branch cap the + on the sort column forces the same index plus an in-memory sort.var query stringswitch {case len(params.TraceIds) == 0:@@ -170,7 +178,12 @@ func (r *logRecordRepository) Search(ctx context.Context, params shared.LogSearclogRecordColumns, where, orderBy, direction)case len(params.TraceIds) <= maxTraceIdBranches:args["branch_limit"] = offset + params.PageSize- query = perTraceIdPageQuery(where, len(params.TraceIds), orderBy, direction)+ branches := make([]string, len(params.TraceIds))+ for i := range branches {+ branches[i] = fmt.Sprintf(`SELECT * FROM (SELECT %s FROM log_records WHERE %s AND trace_id = :tid%d ORDER BY %s %s, id LIMIT :branch_limit)`,+ logRecordColumns, where, i, orderBy, direction)+ }+ query = strings.Join(branches, " UNION ALL ") + fmt.Sprintf(" ORDER BY %s %s, id LIMIT :limit OFFSET :offset", orderBy, direction)default:query = fmt.Sprintf(`SELECT %s FROM log_records WHERE %s ORDER BY +%s %s, id LIMIT :limit OFFSET :offset`,logRecordColumns, where+traceIdIn, orderBy, direction)@@ -287,35 +300,6 @@ func (r *logRecordRepository) buildWhere(params shared.LogSearchParams) (string,return strings.Join(clauses, " AND "), args}-func traceIdKey(i int) string {- return fmt.Sprintf("tid%d", i)-}--func traceIdInClause(traceIds []string, args lit.P) string {- placeholders := make([]string, len(traceIds))- for i, tid := range traceIds {- key := traceIdKey(i)- placeholders[i] = ":" + key- args[key] = tid- }- return "trace_id IN (" + strings.Join(placeholders, ", ") + ")"-}--// An IN list over trace ids is several disjoint ranges in the trace index, so-// the planner cannot take ORDER BY timestamp from it and, without sqlite_stat1,-// walks the whole time window instead (#353). One branch per id is a single-// seek in index order whatever the stats say; each branch carries enough rows-// to reach the requested offset before the outer sort merges them.-func perTraceIdPageQuery(where string, traceIdCount int, orderBy, direction string) string {- branches := make([]string, traceIdCount)- for i := range branches {- branches[i] = fmt.Sprintf(`SELECT * FROM (SELECT %s FROM log_records WHERE %s AND trace_id = :%s ORDER BY %s %s, id LIMIT :branch_limit)`,- logRecordColumns, where, traceIdKey(i), orderBy, direction)- }- return strings.Join(branches, " UNION ALL ") +- fmt.Sprintf(" ORDER BY %s %s, id LIMIT :limit OFFSET :offset", orderBy, direction)-}-func attrColumn(scope string) string {switch scope {case "resource":diff --git a/backend/app/repositories/telemetry/sqlite/task.repository.go b/backend/app/repositories/telemetry/sqlite/task.repository.goindex 6c2e05a7..de939d31 100644--- a/backend/app/repositories/telemetry/sqlite/task.repository.go+++ b/backend/app/repositories/telemetry/sqlite/task.repository.go@@ -168,9 +168,6 @@ func (e *taskRepository) FindAll(ctx context.Context, projectId uuid.UUID, fromDreturn tasks, count, nil}-// taskFilterClause appends the task list's filter to a WHERE clause and binds-// what it needs into params. Both the query that selects a task and the query-// that reads its durations go through it, so they cannot filter differently.func taskFilterClause(params lit.P, search, rootFilter string) string {clause := ""if search != "" {diff --git a/docs/pages/learn/hardware-requirements.mdx b/docs/pages/learn/hardware-requirements.mdxindex f4efe628..76756428 100644--- a/docs/pages/learn/hardware-requirements.mdx+++ b/docs/pages/learn/hardware-requirements.mdx@@ -8,7 +8,7 @@ There are three shapes:- **SQLite mode** (single binary, zero external dependencies). The whole stack is one process. This is the lightweight option.- **DuckDB mode** (`telemetry_duckdb` build tag, CGO required). Still a single container with zero external dependencies, but the telemetry store is DuckDB's columnar engine, which holds 10 to 100 times more queryable rows than SQLite on the same box. The catch: DuckDB sizes itself from host RAM, so memory-capped containers must set `DUCKDB_MEMORY_LIMIT`.-- **ClickHouse + PostgreSQL mode** (`transactional_pg telemetry_ch` build tags). ClickHouse is built to use a lot of RAM for caches and merges, so this is the heavier option that scales much further.+- **ClickHouse + PostgreSQL mode** (`transactional_pg telemetry_ch` build tags). ClickHouse is built to use a lot of RAM for caches and merges, so this is the heavier option that scales much further. It is also the only mode that supports running more than one backend instance; see [Running more than one instance](/server/minimal#running-more-than-one-instance).## TL;DRdiff --git a/docs/pages/server/all-in-one.mdx b/docs/pages/server/all-in-one.mdxindex 7706e9a9..b1d907a2 100644--- a/docs/pages/server/all-in-one.mdx+++ b/docs/pages/server/all-in-one.mdx@@ -4,6 +4,10 @@ import { Callout } from "nextra/components";Single Docker container with ClickHouse, PostgreSQL, and the Go backend managed by supervisord. Best for simple setups where you want everything in one place.+<Callout type="info">+ Because both databases live inside the container, this image runs as a single instance. To run several backend instances, point the [minimal image](/server/minimal#running-more-than-one-instance) at shared databases.+</Callout>+## Quick StartPull the pre-built image from GitHub Container Registry:diff --git a/docs/pages/server/docker-compose.mdx b/docs/pages/server/docker-compose.mdxindex d19cc37e..23f05a8f 100644--- a/docs/pages/server/docker-compose.mdx+++ b/docs/pages/server/docker-compose.mdx@@ -111,6 +111,8 @@ Edit the environment variables in your compose file as needed:> ClickHouse table TTLs (`metric_points`, `log_records`, etc.) are managed at the schema level; this stack does not run the SQLite retention worker.+To run several backend containers against this stack, see [Running more than one instance](/server/minimal#running-more-than-one-instance): the minimal image is built for that shape, and the databases in this compose file are what the instances share.+## SSO (optional)Adds **Continue with Google** / **Continue with GitHub** buttons to the login and register pages. See the [SSO guide](/learn/sso) for the full provider setup walkthrough.diff --git a/docs/pages/server/duckdb.mdx b/docs/pages/server/duckdb.mdxindex f4bc283b..be4236a2 100644--- a/docs/pages/server/duckdb.mdx+++ b/docs/pages/server/duckdb.mdx@@ -1,9 +1,15 @@# DuckDB+import { Callout } from "nextra/components";+The same single-container shape as the [SQLite deployment](/server/sqlite), but with the telemetry store swapped for DuckDB's columnar engine. The main database (users, organizations, projects, notification rules) stays SQLite; endpoints, spans, exceptions, metrics, and logs go into a DuckDB file next to it. No external Postgres or ClickHouse required.Pick this over the plain SQLite image when your dashboards slow down as data grows: on the same box, DuckDB keeps the dashboard responsive at 10 to 100 times more stored telemetry rows, and columnar compression makes the telemetry file several times smaller for the same data. See [Hardware Requirements](/learn/hardware-requirements#duckdb-mode-the-small-one-that-reads-big) for sizing guidance and benchmark numbers.+<Callout type="info">+ Like the SQLite image, this one runs as a single instance: both database files live under `/data`, and a second container cannot share them or learn about the first one's writes. Running more than one backend instance needs the PostgreSQL + ClickHouse build; see [Running more than one instance](/server/minimal#running-more-than-one-instance).+</Callout>+Two operational differences from the SQLite image:- **The image is Debian, not Alpine.** The DuckDB engine links glibc static libraries, so this build cannot run on musl-based images.diff --git a/docs/pages/server/minimal.mdx b/docs/pages/server/minimal.mdxindex 658cfd11..04b7472b 100644--- a/docs/pages/server/minimal.mdx+++ b/docs/pages/server/minimal.mdx@@ -67,6 +67,24 @@ Then use `traceway:minimal` instead of `ghcr.io/tracewayapp/traceway:minimal` in| `STORAGE_PATH` | Folder for local blob storage. Defaults to `./storage`, which is `/app/storage` in this image. Mount a volume there. Ignored when `STORAGE_TYPE=s3`. | `/app/storage` || `SESSION_RECORDING_RETENTION_DAYS` | Days to keep on-disk session recordings under `STORAGE_PATH/recordings/`. Defaults to `30`. Worker runs hourly and on startup. `0` disables; no effect when `STORAGE_TYPE=s3`. | `30` |+## Running more than one instance++This image is the only one that supports several backend instances behind a load balancer, because everything the instances have to agree on lives in PostgreSQL and ClickHouse:++- **Project tokens.** Each instance keeps the project list in memory to validate SDK tokens. Creating, updating or deleting a project, or rotating a source map token, sends a PostgreSQL `NOTIFY` on the `project_cache_changed` channel inside the write transaction, and every instance listening on that channel reloads the list. An instance that lost its database connection reloads the list as soon as the connection is back, so a change made while it was disconnected is not missed.+- **Background workers.** The notification outbox, the on-call escalator, the synthetics scheduler and the one-time dashboards backfill take a PostgreSQL advisory lock per tick, so only one instance drains the queue or pages someone even though every instance runs the workers.++What the deployment needs:++- Every instance points at the same PostgreSQL and ClickHouse and carries the same `JWT_SECRET`, so a session issued by one instance is valid on all of them.+- `STORAGE_TYPE=s3`. With local storage, a source map uploaded through one instance exists only on that instance's disk. See [Blob Storage](/server/storage).+- `TRUSTED_PROXIES` lists the load balancer. Otherwise every instance sees the balancer's address as the client and all visitors share one rate-limit bucket.+- Migrations run at startup. Roll instances out one at a time so a single instance applies them before the others start.++Two caches stay per instance: the source map artifact cache and the symbolicator's negative cache, which retries a failed artifact within at most 15 minutes on the instances that did not receive the upload. See [Symbolicator performance](/symbolicator/performance).++The [SQLite](/server/sqlite) and [DuckDB](/server/duckdb) images are single-instance by design: their databases are local files, and their project cache has no way to learn about a write made by another process. The [All-in-One](/server/all-in-one) image bundles both databases inside the container, so it is single-instance as well. Run exactly one container of those images per data volume.+## SSO (optional)Adds **Continue with Google** / **Continue with GitHub** buttons to the login and register pages. See the [SSO guide](/learn/sso) for the full provider setup walkthrough.diff --git a/docs/pages/server/sqlite.mdx b/docs/pages/server/sqlite.mdxindex 30706e0b..6bab04ec 100644--- a/docs/pages/server/sqlite.mdx+++ b/docs/pages/server/sqlite.mdx@@ -1,11 +1,17 @@# SQLite+import { Callout } from "nextra/components";+Smallest possible Traceway deployment: a single Alpine container that uses SQLite for both the main database and telemetry, with the frontend embedded into the Go binary. No external Postgres or ClickHouse required. Blob storage (source maps, session recordings, AI traces) is written to a local folder by default and can optionally be backed by S3.If your dashboards slow down as telemetry grows, the [DuckDB deployment](/server/duckdb) keeps this exact single-container setup but swaps the telemetry store for a columnar engine with far more read headroom.All persistent state (both SQLite database files and blob storage) lives under `/data`. Bind-mount that single folder to keep your data across restarts.+<Callout type="info">+ This image runs as a single instance. Both databases are local files under `/data`, so a second container cannot share them or learn about the first one's writes. Running more than one backend instance needs the PostgreSQL + ClickHouse build; see [Running more than one instance](/server/minimal#running-more-than-one-instance).+</Callout>+## Quick StartPull the pre-built image and run it with a persistent data volume:diff --git a/frontend/src/lib/api.ts b/frontend/src/lib/api.tsindex de2a7784..d4d5e4f5 100644--- a/frontend/src/lib/api.ts+++ b/frontend/src/lib/api.ts@@ -32,18 +32,11 @@ async function request(method: string, endpoint: string, data?: unknown, options});if (authState.token !== token) {- throw Object.assign(new Error('Session changed'), { status: 0 });+ throw Object.assign(new Error('Session changed'), { status: 401 });}if (response.status === 401) {authState.logout();- const current = window.location.pathname + window.location.search;- gotoHref(- current === '/' || current.startsWith('/login')- ? '/login'- : `/login?returnTo=${encodeURIComponent(current)}`,- { replaceState: true }- );throw Object.assign(new Error('Unauthorized'), { status: 401 });}@@ -56,16 +49,18 @@ async function request(method: string, endpoint: string, data?: unknown, optionsif (!recovering) {recovering = true;try {- const bundle = await request('GET', '/me/login-bundle').catch(() => null);+ const requestedProjectId = new URL(url, window.location.origin).searchParams.get(+ 'projectId'+ );+ const bundle = requestedProjectId+ ? await request('GET', '/me/login-bundle').catch(() => null)+ : null;if (bundle) {authState.setOrganizations(bundle.organizations ?? []);projectsState.setProjects(bundle.projects ?? []);}if (authState.token === token) {- const requestedProjectId = new URL(url, window.location.origin).searchParams.get(- 'projectId'- );const staleProject =bundle &&requestedProjectId &&diff --git a/frontend/src/lib/utils/links.ts b/frontend/src/lib/utils/links.tsindex d3126e9a..fc45188d 100644--- a/frontend/src/lib/utils/links.ts+++ b/frontend/src/lib/utils/links.ts@@ -10,7 +10,7 @@ function splitHref(href: string): { pathname: string; suffix: string } {// throws on the literal brackets a concrete pathname can carry. Prefixing the// base path is what a concrete href needs instead.export function resolveHref(href: string): string {- if (/^(?:[a-z][a-z\d+.-]*:|\/\/|#)/i.test(href)) return href;+ if (/^(?:https?:|mailto:|tel:|\/\/|#)/i.test(href)) return href;const { pathname, suffix } = splitHref(href);return base + (pathname || '/') + suffix;}diff --git a/frontend/src/routes/finish-setup/+page.svelte b/frontend/src/routes/finish-setup/+page.svelteindex 96202a61..76e03ed5 100644--- a/frontend/src/routes/finish-setup/+page.svelte+++ b/frontend/src/routes/finish-setup/+page.svelte@@ -1,5 +1,5 @@<script lang="ts">- import { base, resolve } from '$app/paths';+ import { resolve } from '$app/paths';import { goto } from '$app/navigation';import { onMount } from 'svelte';import { Button } from '$lib/components/ui/button';diff --git a/frontend/src/routes/setup/+page.svelte b/frontend/src/routes/setup/+page.svelteindex 84bec362..4ffc0263 100644--- a/frontend/src/routes/setup/+page.svelte+++ b/frontend/src/routes/setup/+page.svelte@@ -12,6 +12,7 @@import { authState, type UserOrganizationResponse } from '$lib/state/auth.svelte';import { projectsState, type Project } from '$lib/state/projects.svelte';import { api } from '$lib/api';+ import { getErrorMessage } from '$lib/utils/errors';import SetupProjectsStep from '$lib/components/setup/setup-projects-step.svelte';const writableOrgs = $derived(authState.organizations.filter((o) => o.role !== 'readonly'));@@ -28,37 +29,24 @@const selectedOrgName = $derived(writableOrgs.find((o) => o.id === selectedOrgId)?.name ?? '');- // authState.organizations is hydrated from localStorage and only rewritten on- // login, so a membership removed mid-session is still cached here. The layout- // pins every zero-project account to this page, which makes it the one screen- // that must not act on a stale list.- // Deliberately not a render gate: the cached list is right for every case but- // the mid-session removal this endpoint exists for, so blocking the page would- // tax every onboarding load to avoid one rare branch flip.-+ // The cached organization list can be stale after a mid-session removal.+ let created = false;onMount(async () => {try {const bundle = (await api.get('/me/login-bundle')) as {organizations?: UserOrganizationResponse[];projects?: Project[];};+ if (created) return;authState.setOrganizations(bundle.organizations || []);projectsState.setProjects(bundle.projects || []);} catch {- // Fall back to the cached list rather than stranding the page; a genuinely- // expired token is already handled by the 401 path in api.ts.+ // keep the cached list}});const hasNoOrganizations = $derived(authState.organizations.length === 0);- // A user with no organization is offered a choice before a form, because- // creating one is not always the right answer: on a self-hosted instance that- // already has an organization the endpoint refuses (422, "ask an administrator- // to invite you"), and being removed by an admin is usually followed by an- // invitation rather than by starting a rival organization. So the alternative- // -- log out and come back when the invite lands -- is a first-class action- // here, not something to hunt for in a nav bar this page does not render.let step = $state<'choice' | 'organization'>('choice');let newOrgName = $state('');let timezone = $state(Intl.DateTimeFormat().resolvedOptions().timeZone);@@ -76,17 +64,13 @@name: newOrgName,timezone})) as UserOrganizationResponse;- // Appending flips hasNoOrganizations, which swaps this page over to its- // project-setup half -- the /setup the user is meant to land on. A goto()- // to the route they are already on would remount for no gain and refetch- // the login bundle the response just superseded.+ created = true;authState.setOrganizations([...authState.organizations, organization]);-newOrgName = '';toast.success('Successfully created the Organization', { position: 'top-center' });} catch (e) {- createError =- e instanceof Error && e.message ? e.message : 'Failed to create the organization';+ createError = getErrorMessage(e, 'Failed to create the organization');+ } finally {creating = false;}}diff --git a/frontend/vitest.config.ts b/frontend/vitest.config.tsindex 7008a62c..005410df 100644--- a/frontend/vitest.config.ts+++ b/frontend/vitest.config.ts@@ -4,14 +4,12 @@ import path from 'path';export default defineConfig({plugins: [svelte({ hot: false })],- // These tests run on plain svelte(), not sveltekit(), so nothing supplies the- // $app modules. $app/paths points at SvelteKit's real implementation rather- // than a stub because an identity stub is what kept the suite blind to #325,- // and would keep it blind to the next resolve() misuse. Nothing under test- // calls resolve() today. These are the globals that module reads.+ // Plain svelte(), not sveltekit(), supplies no $app modules. $app/paths is SvelteKit's real+ // implementation rather than an identity stub, which is what kept the suite blind to #325.define: {__SVELTEKIT_PAYLOAD__: 'undefined',__SVELTEKIT_PATHS_BASE__: '""',+ __SVELTEKIT_PATHS_ASSETS__: '""',__SVELTEKIT_APP_DIR__: '"_app"',__SVELTEKIT_HASH_ROUTING__: 'false'},diff --git a/helm/traceway/values.yaml b/helm/traceway/values.yamlindex 1e4538b5..68345509 100644--- a/helm/traceway/values.yaml+++ b/helm/traceway/values.yaml@@ -1,3 +1,4 @@+# More than one replica needs variant=minimal with shared databases; the sqlite variant keeps its databases as local files.replicaCount: 1image:@@ -40,20 +41,10 @@ ingress:resources: {}-# Startup probe. The liveness budget only starts once this succeeds, so a slow-# first boot is not mistaken for a hung process. The server binds its ports only-# after migrations finish, and an upgrade that builds indexes on a large-# telemetry database can take minutes — without this the kubelet would kill the-# pod mid-migration and restart it into the same stall on every retry. The-# default budget (periodSeconds x failureThreshold = 300s) is the same one the-# Docker images give their HEALTHCHECK start period. Raise failureThreshold if-# an upgrade needs longer; the backend logs each migration with its elapsed-# time (boot phases before migrations, such as DuckDB WAL replay, are not yet-# timed). The Deployment's progressDeadlineSeconds follows this budget, but a-# deploy tool's own timeout does not: `helm upgrade --wait` defaults to 5m, and-# `--atomic` (or a Flux HelmRelease remediation) rolls back when it expires,-# scaling the migrating pod away — so raise `--timeout` / `spec.timeout` in-# step with the budget.+# The liveness budget only starts once this succeeds. The server binds its ports+# only after migrations finish, and an index-building upgrade on a large telemetry+# database can take minutes: raise failureThreshold if one needs more than the+# 300s default, and raise `helm upgrade --timeout` in step with it.startupProbe:httpGet:path: /healthdiff --git a/scripts/check-node-pins.sh b/scripts/check-node-pins.shindex c1dc6764..3c295aed 100755--- a/scripts/check-node-pins.sh+++ b/scripts/check-node-pins.sh@@ -11,7 +11,7 @@ set -euo pipefailcd "$(dirname "$0")/.."-pinned=$(tr -d '[:space:]v' < .nvmrc)+pinned=$(tr -d '[:space:]' < .nvmrc)if [[ ! $pinned =~ ^[0-9]+$ ]]; thenecho "check-node-pins: .nvmrc must hold a bare Node major, got '$pinned'" >&2exit 1diff --git a/website/public/blog/reliable-apps-crash-so-why-does-go-recover-panics/boundary-ladder.svg b/website/public/blog/reliable-apps-crash-so-why-does-go-recover-panics/boundary-ladder.svgdeleted file mode 100644index 9c459d91..00000000--- a/website/public/blog/reliable-apps-crash-so-why-does-go-recover-panics/boundary-ladder.svg+++ /dev/null@@ -1,136 +0,0 @@-<?xml version="1.0" encoding="UTF-8" standalone="no"?>-<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"- "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">-<!-- Generated by graphviz version 15.1.1 (20260805.0921)- -->-<!-- Title: boundary_ladder Pages: 1 -->-<svg width="639pt" height="657pt"- viewBox="0.00 0.00 639.00 657.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">-<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(14.4 642.84)">-<title>boundary_ladder</title>-<!-- panic -->-<g id="node1" class="node">-<title>panic</title>-<path fill="#1b1115" stroke="#ff5a5f" stroke-width="1.3" d="M348.45,-628.44C348.45,-628.44 96,-628.44 96,-628.44 90,-628.44 84,-622.44 84,-616.44 84,-616.44 84,-588.52 84,-588.52 84,-582.52 90,-576.52 96,-576.52 96,-576.52 348.45,-576.52 348.45,-576.52 354.45,-576.52 360.45,-582.52 360.45,-588.52 360.45,-588.52 360.45,-616.44 360.45,-616.44 360.45,-622.44 354.45,-628.44 348.45,-628.44"/>-<text xml:space="preserve" text-anchor="start" x="178.73" y="-603.18" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="14.00" fill="#f4f6fb">a panic starts</text>-<text xml:space="preserve" text-anchor="start" x="105.6" y="-591.08" font-family="Helvetica,sans-Serif" font-size="12.00" fill="#ff9a9d">nil pointer · failed type assertion · library bug</text>-</g>-<!-- op -->-<g id="node2" class="node">-<title>op</title>-<path fill="#151a24" stroke="#283041" stroke-width="1.3" d="M377.7,-543.52C377.7,-543.52 66.75,-543.52 66.75,-543.52 60.75,-543.52 54.75,-537.52 54.75,-531.52 54.75,-531.52 54.75,-503.6 54.75,-503.6 54.75,-497.6 60.75,-491.6 66.75,-491.6 66.75,-491.6 377.7,-491.6 377.7,-491.6 383.7,-491.6 389.7,-497.6 389.7,-503.6 389.7,-503.6 389.7,-531.52 389.7,-531.52 389.7,-537.52 383.7,-543.52 377.7,-543.52"/>-<text xml:space="preserve" text-anchor="start" x="191.48" y="-518.26" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="14.00" fill="#f4f6fb">operation</text>-<text xml:space="preserve" text-anchor="start" x="76.35" y="-506.16" font-family="Helvetica,sans-Serif" font-size="12.00" fill="#8a93a6">roll back the partial mutation before the lock is released</text>-</g>-<!-- panic->op -->-<g id="edge1" class="edge">-<title>panic->op</title>-<path fill="none" stroke="#5a6374" stroke-width="1.4" d="M222.23,-576.15C222.23,-569.3 222.23,-561.76 222.23,-554.45"/>-<polygon fill="#5a6374" stroke="#5a6374" stroke-width="1.4" points="225.2,-554.64 222.23,-546.14 219.25,-554.64 225.2,-554.64"/>-</g>-<!-- req -->-<g id="node3" class="node">-<title>req</title>-<path fill="#151a24" stroke="#283041" stroke-width="1.3" d="M388.58,-447.6C388.58,-447.6 55.87,-447.6 55.87,-447.6 49.88,-447.6 43.88,-441.6 43.88,-435.6 43.88,-435.6 43.88,-407.68 43.88,-407.68 43.88,-401.68 49.88,-395.68 55.88,-395.68 55.88,-395.68 388.58,-395.68 388.58,-395.68 394.58,-395.68 400.58,-401.68 400.58,-407.68 400.58,-407.68 400.58,-435.6 400.58,-435.6 400.58,-441.6 394.58,-447.6 388.58,-447.6"/>-<text xml:space="preserve" text-anchor="start" x="197.85" y="-422.34" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="14.00" fill="#f4f6fb">request</text>-<text xml:space="preserve" text-anchor="start" x="65.48" y="-410.24" font-family="Helvetica,sans-Serif" font-size="12.00" fill="#8a93a6">discard everything the request touched (the net/http default)</text>-</g>-<!-- op->req -->-<g id="edge2" class="edge">-<title>op->req</title>-<path fill="none" stroke="#5a6374" stroke-width="1.4" d="M222.23,-491.39C222.23,-481.37 222.23,-469.64 222.23,-458.74"/>-<polygon fill="#5a6374" stroke="#5a6374" stroke-width="1.4" points="225.2,-458.82 222.23,-450.32 219.25,-458.82 225.2,-458.82"/>-<text xml:space="preserve" text-anchor="middle" x="291.98" y="-465.15" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#8a93a6">  side effects survive it: widen</text>-</g>-<!-- ok -->-<g id="node8" class="node">-<title>ok</title>-<path fill="#0c1714" stroke="#22e0a8" stroke-width="1.3" d="M597.83,-351.68C597.83,-351.68 464.62,-351.68 464.62,-351.68 458.62,-351.68 452.62,-345.68 452.62,-339.68 452.62,-339.68 452.62,-299.76 452.62,-299.76 452.62,-293.76 458.62,-287.76 464.62,-287.76 464.62,-287.76 597.83,-287.76 597.83,-287.76 603.83,-287.76 609.83,-293.76 609.83,-299.76 609.83,-299.76 609.83,-339.68 609.83,-339.68 609.83,-345.68 603.83,-351.68 597.83,-351.68"/>-<text xml:space="preserve" text-anchor="start" x="498.98" y="-326.42" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="14.00" fill="#f4f6fb">recovered</text>-<text xml:space="preserve" text-anchor="start" x="474.23" y="-314.32" font-family="Helvetica,sans-Serif" font-size="12.00" fill="#7ff0c8">invariants hold again,</text>-<text xml:space="preserve" text-anchor="start" x="497.1" y="-302.32" font-family="Helvetica,sans-Serif" font-size="12.00" fill="#7ff0c8">keep serving</text>-</g>-<!-- op->ok -->-<g id="edge7" class="edge">-<title>op:e->ok:w</title>-<path fill="none" stroke="#22e0a8" stroke-width="1.4" d="M390.7,-517.56C432.93,-517.56 410.2,-350.85 441.78,-323.5"/>-<polygon fill="#22e0a8" stroke="#22e0a8" stroke-width="1.4" points="442.77,-326.3 449.65,-320.48 440.64,-320.74 442.77,-326.3"/>-<text xml:space="preserve" text-anchor="start" x="422.98" y="-423.19" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#7ff0c8">owns every side effect:</text>-<text xml:space="preserve" text-anchor="start" x="422.98" y="-411.19" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#7ff0c8">recover here</text>-</g>-<!-- job -->-<g id="node4" class="node">-<title>job</title>-<path fill="#151a24" stroke="#283041" stroke-width="1.3" d="M393.45,-345.68C393.45,-345.68 51,-345.68 51,-345.68 45,-345.68 39,-339.68 39,-333.68 39,-333.68 39,-305.76 39,-305.76 39,-299.76 45,-293.76 51,-293.76 51,-293.76 393.45,-293.76 393.45,-293.76 399.45,-293.76 405.45,-299.76 405.45,-305.76 405.45,-305.76 405.45,-333.68 405.45,-333.68 405.45,-339.68 399.45,-345.68 393.45,-345.68"/>-<text xml:space="preserve" text-anchor="start" x="159.23" y="-320.42" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="14.00" fill="#f4f6fb">durable job attempt</text>-<text xml:space="preserve" text-anchor="start" x="60.6" y="-308.32" font-family="Helvetica,sans-Serif" font-size="12.00" fill="#8a93a6">fail the attempt, the queue retries from the last committed step</text>-</g>-<!-- req->job -->-<g id="edge3" class="edge">-<title>req->job</title>-<path fill="none" stroke="#5a6374" stroke-width="1.4" d="M222.23,-395.17C222.23,-383.42 222.23,-369.28 222.23,-356.49"/>-<polygon fill="#5a6374" stroke="#5a6374" stroke-width="1.4" points="225.2,-356.76 222.23,-348.27 219.25,-356.77 225.2,-356.76"/>-<text xml:space="preserve" text-anchor="middle" x="274.73" y="-369.23" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#8a93a6">  side effects survive it</text>-</g>-<!-- req->ok -->-<g id="edge8" class="edge">-<title>req:e->ok:w</title>-<path fill="none" stroke="#22e0a8" stroke-width="1.4" d="M401.58,-421.64C426.67,-421.64 407.22,-390.24 418.23,-367.68 427.83,-347.98 426.51,-327.38 441.47,-321.42"/>-<polygon fill="#22e0a8" stroke="#22e0a8" stroke-width="1.4" points="441.64,-324.41 449.54,-320.07 440.66,-318.54 441.64,-324.41"/>-</g>-<!-- worker -->-<g id="node5" class="node">-<title>worker</title>-<path fill="#151a24" stroke="#283041" stroke-width="1.3" d="M432.45,-243.76C432.45,-243.76 12,-243.76 12,-243.76 6,-243.76 0,-237.76 0,-231.76 0,-231.76 0,-203.84 0,-203.84 0,-197.84 6,-191.84 12,-191.84 12,-191.84 432.45,-191.84 432.45,-191.84 438.45,-191.84 444.45,-197.84 444.45,-203.84 444.45,-203.84 444.45,-231.76 444.45,-231.76 444.45,-237.76 438.45,-243.76 432.45,-243.76"/>-<text xml:space="preserve" text-anchor="start" x="199.73" y="-218.5" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="14.00" fill="#f4f6fb">worker</text>-<text xml:space="preserve" text-anchor="start" x="21.6" y="-206.4" font-family="Helvetica,sans-Serif" font-size="12.00" fill="#8a93a6">the goroutine catches its own panic, a supervisor goroutine starts a new one</text>-</g>-<!-- job->worker -->-<g id="edge4" class="edge">-<title>job->worker</title>-<path fill="none" stroke="#5a6374" stroke-width="1.4" d="M222.23,-293.25C222.23,-281.5 222.23,-267.36 222.23,-254.57"/>-<polygon fill="#5a6374" stroke="#5a6374" stroke-width="1.4" points="225.2,-254.84 222.23,-246.35 219.25,-254.85 225.2,-254.84"/>-<text xml:space="preserve" text-anchor="middle" x="274.73" y="-261.31" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#8a93a6">  side effects survive it</text>-</g>-<!-- job->ok -->-<g id="edge9" class="edge">-<title>job:e->ok:w</title>-<path fill="none" stroke="#22e0a8" stroke-width="1.4" d="M405.45,-319.72C421.99,-319.72 428.75,-319.72 441.16,-319.72"/>-<polygon fill="#22e0a8" stroke="#22e0a8" stroke-width="1.4" points="442.01,-322.7 450.51,-319.72 442.01,-316.75 442.01,-322.7"/>-</g>-<!-- proc -->-<g id="node6" class="node">-<title>proc</title>-<path fill="#1b1115" stroke="#ff5a5f" stroke-width="1.3" d="M399.83,-147.84C399.83,-147.84 44.62,-147.84 44.62,-147.84 38.62,-147.84 32.63,-141.84 32.63,-135.84 32.63,-135.84 32.63,-107.92 32.63,-107.92 32.63,-101.92 38.63,-95.92 44.63,-95.92 44.63,-95.92 399.83,-95.92 399.83,-95.92 405.83,-95.92 411.83,-101.92 411.83,-107.92 411.83,-107.92 411.83,-135.84 411.83,-135.84 411.83,-141.84 405.83,-147.84 399.83,-147.84"/>-<text xml:space="preserve" text-anchor="start" x="196.35" y="-122.58" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="14.00" fill="#f4f6fb">process</text>-<text xml:space="preserve" text-anchor="start" x="54.23" y="-110.48" font-family="Helvetica,sans-Serif" font-size="12.00" fill="#ff9a9d">exit with a nonzero status, nothing smaller could be proven safe</text>-</g>-<!-- worker->proc -->-<g id="edge5" class="edge">-<title>worker->proc</title>-<path fill="none" stroke="#5a6374" stroke-width="1.4" d="M222.23,-191.63C222.23,-181.61 222.23,-169.88 222.23,-158.98"/>-<polygon fill="#5a6374" stroke="#5a6374" stroke-width="1.4" points="225.2,-159.06 222.23,-150.56 219.25,-159.06 225.2,-159.06"/>-<text xml:space="preserve" text-anchor="middle" x="274.73" y="-165.39" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#8a93a6">  side effects survive it</text>-</g>-<!-- worker->ok -->-<g id="edge10" class="edge">-<title>worker:e->ok:w</title>-<path fill="none" stroke="#22e0a8" stroke-width="1.4" d="M445.45,-217.8C486.22,-217.8 422.09,-300.07 441.55,-316.78"/>-<polygon fill="#22e0a8" stroke="#22e0a8" stroke-width="1.4" points="440.6,-319.6 449.59,-319.13 442.26,-313.89 440.6,-319.6"/>-</g>-<!-- sup -->-<g id="node7" class="node">-<title>sup</title>-<path fill="#0c1714" stroke="#22e0a8" stroke-width="1.3" d="M392.33,-51.92C392.33,-51.92 52.12,-51.92 52.12,-51.92 46.13,-51.92 40.13,-45.92 40.13,-39.92 40.13,-39.92 40.13,-12 40.13,-12 40.13,-6 46.13,0 52.13,0 52.13,0 392.33,0 392.33,0 398.33,0 404.33,-6 404.33,-12 404.33,-12 404.33,-39.92 404.33,-39.92 404.33,-45.92 398.33,-51.92 392.33,-51.92"/>-<text xml:space="preserve" text-anchor="start" x="187.73" y="-26.66" font-family="Helvetica,sans-Serif" font-weight="bold" font-size="14.00" fill="#f4f6fb">supervisor</text>-<text xml:space="preserve" text-anchor="start" x="61.73" y="-14.56" font-family="Helvetica,sans-Serif" font-size="12.00" fill="#7ff0c8">systemd · Docker · kubelet start a clean process, with backoff</text>-</g>-<!-- proc->sup -->-<g id="edge6" class="edge">-<title>proc->sup</title>-<path fill="none" stroke="#ff5a5f" stroke-width="1.4" d="M222.23,-95.71C222.23,-85.69 222.23,-73.96 222.23,-63.06"/>-<polygon fill="#ff5a5f" stroke="#ff5a5f" stroke-width="1.4" points="225.2,-63.14 222.23,-54.64 219.25,-63.14 225.2,-63.14"/>-<text xml:space="preserve" text-anchor="middle" x="240.23" y="-69.47" font-family="Helvetica,sans-Serif" font-size="11.00" fill="#ff8a8d">  exit(1)</text>-</g>-</g>-</svg>diff --git a/website/public/blog/reliable-apps-crash-so-why-does-go-recover-panics/og-recovery-boundaries.png b/website/public/blog/reliable-apps-crash-so-why-does-go-recover-panics/og-recovery-boundaries.pngdeleted file mode 100644index 53af2f86..00000000Binary files a/website/public/blog/reliable-apps-crash-so-why-does-go-recover-panics/og-recovery-boundaries.png and /dev/null differ