The only tool you need to know what is happening and how to fix it.
Go alertingexception-trackerissue-trackinglogsmetricsmonitoringobservability-platformopen-telemetryotelsession-replaytelemetrytraces
js-symbolicator-v1 (#188)
narrow-adventure5fad4c5cca44
diff --git a/DOCKER.md b/DOCKER.mddeleted file mode 100644index 535e98b0..00000000--- a/DOCKER.md+++ /dev/null@@ -1,210 +0,0 @@-# Traceway Docker Deployment--Two deployment options:-1. **All-in-One** (`Dockerfile`) — ClickHouse + PostgreSQL + Backend in one container-2. **Minimal** (`Dockerfile.minimal`) — Lightweight Alpine image for external database deployments-------## All-in-One Deployment--Single container with ClickHouse, PostgreSQL, and the Go backend managed by supervisord.--### Build--```bash-docker build -t traceway:latest .-```--### Run--```bash-docker run -d --name traceway \- -p 80:80 \- -v traceway-ch:/var/lib/clickhouse \- -v traceway-pg:/var/lib/postgresql/data \- traceway:latest-```--### Environment Variables--All have working defaults for the all-in-one image. Override as needed:--| Variable | Default | Description |-|----------|---------|-------------|-| `CLICKHOUSE_SERVER` | `localhost:9000` | ClickHouse host:port |-| `CLICKHOUSE_DATABASE` | `traceway` | ClickHouse database name |-| `CLICKHOUSE_USERNAME` | `default` | ClickHouse username |-| `CLICKHOUSE_PASSWORD` | *(empty)* | ClickHouse password |-| `CLICKHOUSE_TLS` | `false` | Enable TLS for ClickHouse |-| `POSTGRES_HOST` | `localhost` | PostgreSQL host |-| `POSTGRES_PORT` | `5432` | PostgreSQL port |-| `POSTGRES_DATABASE` | `traceway` | PostgreSQL database name |-| `POSTGRES_USERNAME` | `traceway` | PostgreSQL username |-| `POSTGRES_PASSWORD` | *(empty)* | PostgreSQL password |-| `POSTGRES_SSLMODE` | `disable` | PostgreSQL SSL mode |-| `JWT_SECRET` | *(built-in default)* | JWT signing secret (min 32 chars) |-| `GIN_MODE` | `release` | Gin framework mode |-| `SESSION_RECORDING_RETENTION_DAYS` | `30` | Days to keep on-disk session recordings under `STORAGE_PATH/recordings/`. Worker runs hourly and on startup. `0` disables; no effect when `STORAGE_TYPE=s3`. |--### Access Points--| URL | Description |-|-----|-------------|-| `http://localhost/` | Frontend dashboard |-| `http://localhost/api/*` | Backend API |-| `http://localhost/health` | Health check |-------## Minimal Deployment--Lightweight Alpine image (~20-30MB) containing only the Go backend with embedded frontend. Connect to external ClickHouse and PostgreSQL instances.--### Build--```bash-docker build -f Dockerfile.minimal -t traceway:minimal .-```--### Required Environment Variables--| Variable | Description | Example |-|----------|-------------|---------|-| `CLICKHOUSE_SERVER` | ClickHouse host:port | `clickhouse:9000` |-| `CLICKHOUSE_DATABASE` | ClickHouse database | `traceway` |-| `CLICKHOUSE_USERNAME` | ClickHouse username | `default` |-| `CLICKHOUSE_PASSWORD` | ClickHouse password | `password` |-| `CLICKHOUSE_TLS` | Enable TLS | `false` |-| `POSTGRES_HOST` | PostgreSQL host | `postgres` |-| `POSTGRES_PORT` | PostgreSQL port | `5432` |-| `POSTGRES_DATABASE` | PostgreSQL database | `traceway` |-| `POSTGRES_USERNAME` | PostgreSQL username | `traceway` |-| `POSTGRES_PASSWORD` | PostgreSQL password | `password` |-| `POSTGRES_SSLMODE` | PostgreSQL SSL mode | `disable` |-| `JWT_SECRET` | JWT signing secret (min 32 chars) | `your-secret-here` |-| `SESSION_RECORDING_RETENTION_DAYS` | Optional. Days to keep on-disk session recordings (default `30`, `0` disables, no effect when `STORAGE_TYPE=s3`). | `30` |--### Run--```bash-docker run -d --name traceway \- -p 80:80 \- -e CLICKHOUSE_SERVER="clickhouse-host:9000" \- -e CLICKHOUSE_DATABASE="traceway" \- -e CLICKHOUSE_USERNAME="default" \- -e CLICKHOUSE_PASSWORD="your-password" \- -e POSTGRES_HOST="postgres-host" \- -e POSTGRES_PORT="5432" \- -e POSTGRES_DATABASE="traceway" \- -e POSTGRES_USERNAME="traceway" \- -e POSTGRES_PASSWORD="your-password" \- -e POSTGRES_SSLMODE="disable" \- -e JWT_SECRET="your-jwt-secret-min-32-characters-long" \- traceway:minimal-```--### Docker Compose Example--```yaml-services:- traceway:- build:- context: .- dockerfile: Dockerfile.minimal- ports:- - "80:80"- environment:- CLICKHOUSE_SERVER: "clickhouse:9000"- CLICKHOUSE_DATABASE: "traceway"- CLICKHOUSE_USERNAME: "default"- CLICKHOUSE_PASSWORD: "clickhouse"- CLICKHOUSE_TLS: "false"- POSTGRES_HOST: "postgres"- POSTGRES_PORT: "5432"- POSTGRES_DATABASE: "traceway"- POSTGRES_USERNAME: "traceway"- POSTGRES_PASSWORD: "traceway"- POSTGRES_SSLMODE: "disable"- JWT_SECRET: "change-this-to-a-secure-secret-at-least-32-chars"- # Optional. Days to keep on-disk session recordings (default 30, 0 disables, no effect when STORAGE_TYPE=s3).- # SESSION_RECORDING_RETENTION_DAYS: "30"- depends_on:- clickhouse:- condition: service_healthy- postgres:- condition: service_healthy- restart: unless-stopped-- clickhouse:- image: clickhouse/clickhouse-server:24.8-alpine- environment:- CLICKHOUSE_DB: traceway- CLICKHOUSE_PASSWORD: clickhouse- volumes:- - clickhouse-data:/var/lib/clickhouse- healthcheck:- test: ["CMD", "clickhouse-client", "--password", "clickhouse", "--query", "SELECT 1"]- interval: 5s- timeout: 3s- start_period: 10s- retries: 10- restart: unless-stopped-- postgres:- image: postgres:17- environment:- POSTGRES_USER: traceway- POSTGRES_PASSWORD: traceway- POSTGRES_DB: traceway- volumes:- - postgres-data:/var/lib/postgresql/data- healthcheck:- test: ["CMD-SHELL", "pg_isready -U traceway"]- interval: 5s- timeout: 3s- start_period: 10s- retries: 10- restart: unless-stopped--volumes:- clickhouse-data:- postgres-data:-```-------## Docker Compose Quick Start--```bash-# Start the full stack (builds images and starts all services)-docker compose up --build--# Clean restart (removes volumes — resets all data)-docker compose down -v && docker compose up --build-```--After starting, open `http://localhost/register` to create your first account.-------## Useful Commands--```bash-# View logs-docker logs traceway-docker logs -f traceway--# Enter container shell-docker exec -it traceway sh # minimal (alpine)-docker exec -it traceway bash # all-in-one (debian)--# Check process status (all-in-one only)-docker exec traceway supervisorctl status--# Health check-curl http://localhost/health--# Stop and remove-docker stop traceway && docker rm traceway-```diff --git a/DOCKER_SIGNATURES.md b/DOCKER_SIGNATURES.mddeleted file mode 100644index c087da3c..00000000--- a/DOCKER_SIGNATURES.md+++ /dev/null@@ -1,204 +0,0 @@-# Docker Image Signatures--All official Traceway Docker images are cryptographically signed using [Cosign](https://docs.sigstore.dev/cosign/overview/), a CNCF tool for container image signing and verification.--## Why Image Signatures Matter--Signed images provide:-- **Authenticity**: Verify the image comes from the official Traceway project-- **Integrity**: Ensure the image hasn't been modified or tampered with-- **Transparency**: Signatures are publicly verifiable using GitHub's OIDC token--## Available Images--Traceway publishes three Docker image variants to GitHub Container Registry (GHCR):--| Image | Purpose | Size | Best For |-|-------|---------|------|----------|-| `ghcr.io/tracewayapp/traceway:v*` | **Full** — ClickHouse + PostgreSQL + supervisord | ~600MB | Large deployments, multi-service setup |-| `ghcr.io/tracewayapp/traceway:v*-minimal` | **Minimal** — single binary with external databases | ~20-30MB | Kubernetes, scalable setups, external database infrastructure |-| `ghcr.io/tracewayapp/traceway:v*-sqlite` | **SQLite** — embedded SQLite, single binary, zero dependencies | ~50-80MB | Small VPS, testing, single-server deployments, embedded mode |--All images are cryptographically signed. Pull the latest release:--```bash-# Full image (all services included)-docker pull ghcr.io/tracewayapp/traceway:latest--# Minimal image (external databases)-docker pull ghcr.io/tracewayapp/traceway:minimal--# SQLite image (embedded, no external dependencies)-docker pull ghcr.io/tracewayapp/traceway:sqlite-```--## Verifying Signatures--### Install Cosign--**macOS:**-```bash-brew install cosign-```--**Linux:**-```bash-wget https://github.com/sigstore/cosign/releases/latest/download/cosign-linux-amd64-sudo mv cosign-linux-amd64 /usr/local/bin/cosign-sudo chmod +x /usr/local/bin/cosign-```--**Windows (PowerShell):**-```powershell-Invoke-WebRequest -Uri "https://github.com/sigstore/cosign/releases/latest/download/cosign-windows-amd64.exe" -OutFile cosign.exe-# Add cosign.exe to your PATH-```--### Verify Signature--```bash-# Full image-cosign verify \- --certificate-identity-regexp '.*' \- --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \- ghcr.io/tracewayapp/traceway:latest--# Minimal image-cosign verify \- --certificate-identity-regexp '.*' \- --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \- ghcr.io/tracewayapp/traceway:minimal--# SQLite image-cosign verify \- --certificate-identity-regexp '.*' \- --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \- ghcr.io/tracewayapp/traceway:sqlite-```--### Example Output--A successful verification looks like:-```-Verification successful!-```--If verification fails, the image may have been tampered with and should not be used.--## How Signatures Are Generated--1. **Build & Push**: GitHub Actions builds the Docker image and pushes it to GHCR-2. **Sign**: Cosign signs the image using GitHub's OIDC token (keyless signing)-3. **Store**: Signature is stored in GHCR alongside the image--## Using in Docker Compose--```yaml-version: '3.8'--services:- traceway:- image: ghcr.io/tracewayapp/traceway:latest- # Optionally verify the signature before running:- # Run: cosign verify ... ghcr.io/tracewayapp/traceway:latest- ports:- - "80:80"- - "8082:8082"- volumes:- - clickhouse_data:/var/lib/clickhouse- - postgres_data:/var/lib/postgresql/data- environment:- GIN_MODE: release--volumes:- clickhouse_data:- postgres_data:-```--## Minimal Image with External Databases--For scalable deployments with external ClickHouse/PostgreSQL:--```yaml-version: '3.8'--services:- traceway:- image: ghcr.io/tracewayapp/traceway:minimal- ports:- - "80:80"- - "8082:8082"- environment:- GIN_MODE: release- CLICKHOUSE_SERVER: clickhouse:9000- POSTGRES_HOST: postgres- POSTGRES_PORT: 5432- # ... other env vars-- clickhouse:- image: clickhouse/clickhouse-server:latest- # ... configuration-- postgres:- image: postgres:15- # ... configuration-```--## SQLite Image (Embedded, Zero Dependencies)--For small deployments, testing, or single-server setups:--```yaml-version: '3.8'--services:- traceway:- image: ghcr.io/tracewayapp/traceway:sqlite- ports:- - "80:80"- - "8082:8082"- volumes:- - traceway_data:/data- environment:- GIN_MODE: release--volumes:- traceway_data:-```--The SQLite image:-- Stores all data in `/data/traceway.db` (SQLite database)-- Stores blobs in `/data/storage` (if using local storage)-- Requires only one volume mount for all persistent state-- Perfect for VPS, small instances, or development--To persist data across restarts, mount a host folder or named volume at `/data`:--```bash-docker run -v /var/lib/traceway:/data ghcr.io/tracewayapp/traceway:sqlite-```--## Troubleshooting--**"Verification failed" error:**-- Ensure you're using the correct image URI (with version tag or `latest`)-- Check your internet connection (Cosign needs to fetch OIDC tokens)-- Try verifying a different version tag--**"cosign not found":**-- Ensure Cosign is installed and in your `$PATH`-- Run `cosign version` to verify installation--**"Certificate verification failed":**-- This indicates the image signature is invalid-- Do not use this image — report it to the Traceway team--## More Information--- [Cosign Documentation](https://docs.sigstore.dev/cosign/overview/)-- [Traceway Self-Hosting Guide](https://docs.tracewayapp.com/server)-- [SBOM (Software Bill of Materials)](https://docs.sigstore.dev/cosign/sbom/)--## Questions?--If you have questions about Docker image security or signatures, [open an issue](https://github.com/tracewayapp/traceway/issues) or ask in the [Traceway Discord](https://discord.gg/RZq9NT62nc).diff --git a/SPEC_NON_ROOT_TRACES.md b/SPEC_NON_ROOT_TRACES.mddeleted file mode 100644index 1730c97f..00000000--- a/SPEC_NON_ROOT_TRACES.md+++ /dev/null@@ -1,296 +0,0 @@-# Non-Root Tasks & AI Traces--## Problem--`backend/app/controllers/otelcontrollers/trace_converter.go:98` classifies a span as an Endpoint, Task, or AiTrace only when `isRoot := len(span.ParentSpanId) == 0`. Three real-world cases violate that assumption:--1. **Queue jobs with propagated trace context.** keepsuit's `QueueInstrumentation::recordJobProcessing` (`vendor/keepsuit/laravel-opentelemetry/src/Instrumentation/QueueInstrumentation.php:106`) — and the equivalents in `opentelemetry-instrumentation-celery`, Symfony Messenger without `messenger_root_spans: true`, etc. — calls `setParent($context)` on the worker's CONSUMER span using the trace context serialized into the job payload by the dispatcher. The consumer span has `parent_span_id = <producer span id>` from the originating request, so `isRoot == false`. It falls through to the generic `spans` table and the Tasks page stays empty.-2. **Child LLM spans.** A request span (root) makes an LLM call as a child span. The child has `gen_ai.*` attributes but a non-empty `parent_span_id`, so it never reaches the `aiTraces = append(...)` branch and the AI Traces page stays empty.-3. **Scheduled console commands.** keepsuit's `ConsoleInstrumentation` (`ConsoleInstrumentation.php:77-79`) emits a root span with kind INTERNAL (the OTel default — no `setSpanKind` call) and the `console.command` attribute. It has no HTTP attrs, so today it falls into the `else { continue }` branch and is dropped. zentigo runs `schedule:run`, `bookings:expire-unconfirmed`, and `bookings:send-reminders` under this instrumentation — all silently dropped.--All three are the same line. The fix promotes spans by their Kind / attributes instead of by their root-ness, and adds enough schema and UI to make the resulting non-root rows distinguishable from real roots.--## Approach--Two new columns on the entity tables, one ingestion change, and one query-time change:--- **`is_root Bool`** on `endpoints` / `tasks` / `ai_traces` — explicit because once `distributed_trace_id` is auto-filled from the OTel trace_id, both the root and its non-root siblings carry the same `distributed_trace_id` and "is_root" is no longer derivable from `distributed_trace_id IS NULL`.-- **`distributed_trace_id Nullable(UUID)`** on `ai_traces` (already exists on the other three) — populated from the OTel trace_id, not a vendor attribute, so it works for any OTLP source out of the box.-- **Converter promotes by Kind/attrs**, not by root-ness. Non-root entities get an id derived from their own `span_id` (not the shared OTel trace_id) so they don't collide with the root entity's id.-- **Spans are re-rooted at ingestion to their nearest enclosing entity within the batch.** `spans.trace_id` (and `exception_stack_traces.trace_id`) is repurposed from "OTel trace_id" to "owning entity id" so each waterfall query stays clean.--### Decisions--- **Drop the `isRoot` gate on _all_ classifications, including endpoints.** Non-root SERVER+HTTP spans (cross-service inbound hops) become endpoint rows with `is_root=false`. The "Non-root" chip applies to endpoints too.-- **Promote console spans to tasks.** Detect via the `console.command` attribute keepsuit attaches. Root INTERNAL spans without HTTP attrs that carry `console.command` become task rows.-- **Keep `rootFilter` on the endpoints API/page** — endpoints can now legitimately be non-root.-- **Tasks search ships in the same PR.** Today the tasks page has no search; this change adds it alongside `rootFilter`.--### Worked example (queue case)--OTel trace_id `X`, spans:-- `A` — SERVER, root, HTTP attrs — request handler-- `B` — PRODUCER, parent=`A` — `send queue`-- `C` — CONSUMER, parent=`B` — `process queue` (arrives in a separate worker OTLP batch)-- `D` — child of `C` — `SELECT …` (arrives in the worker's batch)--Resulting rows:--| | `id` | `distributed_trace_id` | `is_root` |-|---|---|---|---|-| endpoints | `otelTraceIDToUUID(A.trace_id)` = **X** | **X** | true |-| tasks | `otelSpanIDToUUID(C.span_id)` = **Y** | **X** | false |-| spans | `B`'s span_id, `trace_id = X` (owner = endpoint) | — | — |-| spans | `D`'s span_id, `trace_id = Y` (owner = task) | — | — |--`SELECT spans WHERE trace_id = X` returns `B` only. `SELECT spans WHERE trace_id = Y` returns `D` only. `FindByDistributedTraceId(X)` returns both the endpoint and the task.--### Worked example (inline gen_ai case)--`A` is a SERVER root, `L` is a child of `A` with `gen_ai.*` attrs. They arrive in the same batch.--| | `id` | `distributed_trace_id` | `is_root` |-|---|---|---|---|-| endpoints | `X` (from `A.trace_id`) | `X` | true |-| ai_traces | `otelSpanIDToUUID(L.span_id)` = `Z` | `X` | false |--Spans between `A` and `L` (e.g. middleware/DB spans) get `trace_id = X` (owner = endpoint). Anything that turns out to be a descendant of `L` (rare — LLM spans are usually leaves; tool-call children would be the case) gets `trace_id = Z`.--### Worked example (scheduled console command)--`bookings:send-reminders` runs under keepsuit's `ConsoleInstrumentation`. The span has kind INTERNAL, no parent, no HTTP attrs, and attribute `console.command = "bookings:send-reminders"`. Children include the database queries it issues.--| | `id` | `distributed_trace_id` | `is_root` |-|---|---|---|---|-| tasks | `otelTraceIDToUUID(trace_id)` = **X** | **X** | true |-| spans | each DB child, `trace_id = X` (owner = task) | — | — |-------## Backend--### Migrations--ClickHouse — one statement per file (per project rule). Next free number is 0058:--- `0058_add_is_root_to_endpoints.up.sql`- ```sql- ALTER TABLE endpoints ADD COLUMN is_root Bool DEFAULT true- ```-- `0059_add_is_root_to_tasks.up.sql`- ```sql- ALTER TABLE tasks ADD COLUMN is_root Bool DEFAULT true- ```-- `0060_add_distributed_trace_id_to_ai_traces.up.sql`- ```sql- ALTER TABLE ai_traces ADD COLUMN distributed_trace_id Nullable(UUID) DEFAULT NULL- ```-- `0061_add_distributed_trace_id_index_ai_traces.up.sql`- ```sql- ALTER TABLE ai_traces ADD INDEX idx_distributed_trace_id_ai_traces distributed_trace_id TYPE bloom_filter(0.001) GRANULARITY 1- ```-- `0062_add_is_root_to_ai_traces.up.sql`- ```sql- ALTER TABLE ai_traces ADD COLUMN is_root Bool DEFAULT true- ```--SQLite (`backend/app/migrations/sqlite_telemetry/`, next number is 0009). The SQLite migration runner splits on `;` (see `migrations_sqlite.go:80`), so a single multi-statement file is fine:--- `0009_add_is_root_and_distributed_trace_id.up.sql`- ```sql- ALTER TABLE endpoints ADD COLUMN is_root INTEGER NOT NULL DEFAULT 1;- ALTER TABLE tasks ADD COLUMN is_root INTEGER NOT NULL DEFAULT 1;- ALTER TABLE ai_traces ADD COLUMN is_root INTEGER NOT NULL DEFAULT 1;- ALTER TABLE ai_traces ADD COLUMN distributed_trace_id TEXT DEFAULT NULL;- ```--> Default `true`/`1` so backfill is correct — every existing row was promoted under the old `isRoot` gate, so it was a root span by definition.--### Models--`backend/app/models/endpoint.model.go`, `task.model.go`, `ai_trace.model.go`:--- Add `IsRoot bool \`json:"isRoot" ch:"is_root"\``-- For `AiTrace`, also add `DistributedTraceId *uuid.UUID \`json:"distributedTraceId,omitempty" ch:"distributed_trace_id"\``--Aggregate result models — `TaskStats`, `AiTraceStats`, `EndpointStats` (`models/*.go` and inline in repositories) gain `HasRoot bool` and `HasNonRoot bool` so the grouped list page can show the chip on names that contain at least one non-root run.--### Converter (`backend/app/controllers/otelcontrollers/trace_converter.go`)--Rewrite the per-span classification block (currently lines 92–178). Two passes inside the existing `for _, rs := range req.ResourceSpans` loop.--**Pass 1 — classify each span as one of `endpoint`, `task`, `ai_trace`, or `span`.** Drop the `isRoot` gate from all three entity branches. Rule table:--| Condition | Promoted to |-|---|---|-| `Kind ∈ {SERVER, INTERNAL}` AND `hasHTTPAttributes(attrs)` | endpoint |-| `Kind == CONSUMER` | task |-| `Kind == INTERNAL` AND attribute `console.command` is present AND `isRoot` | task |-| `hasGenAiAttributes(attrs)` | ai_trace |-| otherwise | (handled in pass 2) |--For each promoted entity, compute its id:-- `is_root = len(span.ParentSpanId) == 0`-- if root: `id = otelTraceIDToUUID(span.TraceId)` (preserves today's invariant that the root entity's id equals the OTel trace_id)-- if non-root: `id = otelSpanIDToUUID(span.SpanId)` (distinct from any root entity sharing the same trace)-- `distributed_trace_id = otelTraceIDToUUID(span.TraceId)` for **both** root and non-root (this is the change from "vendor attribute only" to "auto-derived"). The vendor attribute `traceway.distributed_trace_id` still wins if present, for backwards compatibility with manually-linked traces from the Go SDK.--Build `spanIdToEntityId map[string]uuid.UUID` from these promotions (keyed by `string(span.SpanId)`).--> Console rule rationale: keepsuit's `ConsoleInstrumentation.commandStarting` sets `console.command` explicitly (`ConsoleInstrumentation.php:78`). We don't use "any root INTERNAL with no HTTP attrs" because it would scoop up arbitrary manual roots from other instrumentations. `console.command` is a concrete signal.--**Pass 2 — emit span rows for the unpromoted spans, re-rooted to their nearest enclosing entity.**--For each unpromoted span, walk parents via `parentMap` until you find one in `spanIdToEntityId`. That entity's id becomes the span's `trace_id`. If no enclosing entity exists in this batch, fall back to `otelTraceIDToUUID(span.TraceId)` (preserves today's behavior for orphan spans whose parent was emitted from another process and never matched any promoted entity — they remain linkable via `distributed_trace_id` on the cross-trace page).--Skip emitting a span row for any span that was itself promoted to an entity (today, only root entities are skipped; the new logic skips all promoted entities including non-root ones).--**Exceptions.** `exc.TraceId` is set to the owning entity's id (the same value the span's `trace_id` would have gotten), not the raw OTel trace_id. `traceType` is `"endpoint"` / `"task"` / `"ai_trace"` based on which kind owned it. Exceptions on unpromoted spans use the entity walked up the parent chain.--### Repositories--Files to update — every read/write that touches `endpoints`, `tasks`, `ai_traces`, `exception_stack_traces`, or `spans`. Each has a ClickHouse variant (`*.repository.go`) and a SQLite variant (`*.repository_sqlite.go`).--- `endpoint.repository.go` + `endpoint.repository_sqlite.go`:- - `BatchInsert` — add `is_root` to the INSERT column list and `Append` call.- - `FindAll`, `FindGroupedByEndpoint`, `FindByEndpoint`, `FindById`, `FindByDistributedTraceId` — add `is_root` to SELECT and Scan.- - Add `rootFilter` parameter to `FindGroupedByEndpoint` (`all` | `root` | `non_root`); when set to `root` add `AND is_root = true`, when `non_root` add `AND is_root = false`. Grouped projection includes `has_root = max(is_root)` and `has_non_root = max(NOT is_root)` so the list row can show the chip if any run in the group is non-root.-- `task.repository.go` + `task.repository_sqlite.go`:- - Same as endpoints: `BatchInsert`, all reads, root filter on `FindGroupedByTaskName`, `has_root` / `has_non_root` in the grouped projection.- - Add `search` parameter (substring match on `task_name` — `positionCaseInsensitive` in ClickHouse, `LIKE` lowercased in SQLite). Currently tasks have no search; this brings them in line with endpoints.- - Add `FindByDistributedTraceId(ctx, distributedTraceId, projectIds)` — mirrors `EndpointRepository.FindByDistributedTraceId`.-- `ai_trace.repository.go` + `ai_trace.repository_sqlite.go`:- - `BatchInsert` — add `is_root` and `distributed_trace_id` columns.- - All reads — add `is_root` and `distributed_trace_id` to SELECT/Scan.- - Same `rootFilter` parameter + grouped projection as tasks.- - Add `FindByDistributedTraceId` (mirrors `EndpointRepository.FindByDistributedTraceId`).-- `exception_stack_trace.repository.go` + `_sqlite`:- - No schema change here — `trace_id` already exists. Only the value being stored changes (owning entity id, not raw OTel trace_id). Existing callers don't need to change.-- `span.repository.go` + `_sqlite`:- - No schema change. `FindByTraceId(projectId, traceId)` still works — it will now be called with `endpoint.id` or `task.id` or `ai_trace.id` (the owning entity id), all of which are valid values to look up.--### Controllers--- `tasks.controller.go` — `FindGrouped` accepts new request fields `search string` and `rootFilter string`. Forward both to the repository. Emit `hasRoot` / `hasNonRoot` per row in the response.-- `ai_trace.controller.go` — same change as tasks (search already exists, just add `rootFilter` + `hasRoot` / `hasNonRoot`).-- `endpoints.controller.go` — `FindGrouped` accepts new `rootFilter` field. (Search already exists.) Emit `hasRoot` / `hasNonRoot` per row.-- `distributed_trace.controller.go` — add a third call to `repositories.AiTraceRepository.FindByDistributedTraceId` and emit `ai_trace` nodes in the response alongside `endpoint` and `task`. Render in the same `Nodes` list.-- Detail controllers (`endpoint.controller.go`, `task.controller.go`, `ai_trace.controller.go`) — include `isRoot` and `distributedTraceId` in their JSON responses so the detail pages can render the chip and the cross-trace link.--### Tests--`trace_converter_test.go` additions:--1. CONSUMER span with non-empty `ParentSpanId` →- - one row in `tasks` with `is_root == false`, `id == otelSpanIDToUUID(span.SpanId)`, `distributed_trace_id == otelTraceIDToUUID(span.TraceId)`- - no row in `endpoints`- - any child spans in the batch land in `spans` with `trace_id == task.id`-2. Child span with `gen_ai.*` attrs under a SERVER root, both in one batch →- - root endpoint row with `is_root == true`, `id == distributed_trace_id == otelTraceIDToUUID(trace_id)`- - ai_trace row with `is_root == false`, `id == otelSpanIDToUUID(child.SpanId)`, same `distributed_trace_id` as the endpoint- - intermediate spans (if any) get `trace_id == endpoint.id`; descendants of the ai_trace span get `trace_id == ai_trace.id`-3. Root INTERNAL span carrying `console.command` attribute, no HTTP attrs →- - one row in `tasks` with `is_root == true`, `id == otelTraceIDToUUID(span.TraceId)`- - child spans get `trace_id == task.id`-4. Non-root SERVER+HTTP span (cross-service inbound hop) →- - one row in `endpoints` with `is_root == false`, `id == otelSpanIDToUUID(span.SpanId)`-5. Existing test — root SERVER + HTTP → endpoint with `is_root == true` — must still pass unchanged.-6. Exception event on the consumer span → `exc.TraceId == task.id`, `exc.TraceType == "task"`.-7. Orphan span whose parent isn't in the batch and which isn't itself promoted → falls back to `trace_id = otelTraceIDToUUID(span.TraceId)` (the OTel trace_id). Document this fallback in the test name.--Repository tests (`endpoint_repository_test.go`, `task_repository_test.go`, the soon-to-exist `ai_trace_repository_test.go`):-- `is_root` round-trips through insert + select.-- `rootFilter` parameter narrows results correctly (`all` / `root` / `non_root`).-- Grouped queries surface `has_root` and `has_non_root`.-- Task `search` parameter filters by task name substring.-------## Frontend--### Shared component--New `frontend/src/lib/components/ui/root-filter/root-filter.svelte` — a `Select.Root` matching the existing inline-select styling, dropped via the SearchBar's `children` snippet so it joins the bar visually (mirroring how `issues/+page.svelte` uses `typeOptions`).--Options:-```ts-const rootFilterOptions = [- { value: 'all', label: 'All' },- { value: 'root', label: 'Root' },- { value: 'non_root', label: 'Non-root' }-];-```--URL param `rootFilter` (`all` is the default and is omitted from the URL, matching how `searchType` is handled on the issues page).--### Chip--New `frontend/src/lib/components/ui/non-root-chip/non-root-chip.svelte` — small `Badge variant="secondary"` reading `Non-root` (or `Mixed` when both `hasRoot` and `hasNonRoot` are true), with tooltip "Triggered by another trace — see the distributed trace view." Rendered:-- On Endpoint, Task, and AI Trace list rows where `hasNonRoot && !hasRoot` ("Non-root") or where both flags are true ("Mixed").-- On detail pages (`/endpoints/[endpoint]/[id]`, `/tasks/[task]/[id]`, `/ai-traces/[traceName]/[id]`) next to the title when the specific instance being displayed has `isRoot === false`.--### `frontend/src/routes/tasks/+page.svelte`--Currently has no search. Add:-- `searchQuery` state + URL persistence (`?search=…`), copying the pattern from `ai-traces/+page.svelte` lines 62–112.-- `rootFilter` state + URL persistence (`?rootFilter=root` etc.).-- `<SearchBar placeholder="Search tasks..." bind:value={searchQuery} onSearch={handleSearch}>` wrapping the new `<RootFilter bind:value={rootFilter} />` in its `children` snippet.-- Send `search` and `rootFilter` in the `/tasks/grouped` POST body.-- Render `<NonRootChip mixed={task.hasRoot && task.hasNonRoot} />` next to `task.taskName` when `task.hasNonRoot` is true.--`/tasks/[task]/+page.svelte` (instance list): pass `rootFilter` through to `/tasks/task` and render the chip per-row using each instance's `isRoot`.--### `frontend/src/routes/ai-traces/+page.svelte`--Already has search. Add:-- `rootFilter` state + URL persistence.-- Pass the filter via the SearchBar `children` snippet (same as tasks).-- Send `rootFilter` in the API call.-- Render `<NonRootChip>` next to `trace.traceName` when `trace.hasNonRoot`.--`/ai-traces/[traceName]/+page.svelte` — chip per instance row using `aiTrace.isRoot`.--### `frontend/src/routes/endpoints/+page.svelte`--- `rootFilter` state + URL persistence (endpoints can now be non-root after this change).-- Render `<NonRootChip>` next to the endpoint name when `endpoint.hasNonRoot`.--`/endpoints/[endpoint]/+page.svelte` — chip per instance row using `endpoint.isRoot`.--### Distributed-trace page--Wherever `DistributedTraceResponse.Nodes` are rendered, add a case for `traceType === "ai_trace"` so the new third entity type shows up alongside endpoints and tasks. Reuse the existing node card styling.--### Detail pages — cross-trace link--On `/tasks/[task]/[id]`, `/ai-traces/[traceName]/[id]`, and `/endpoints/[endpoint]/[id]`, when `distributedTraceId` is set and there are sibling entities, show a "View distributed trace" link to `/distributed-traces/<distributedTraceId>`. (Today this link is only meaningful when the vendor attribute was set; after this change it's meaningful for every OTel-sourced trace.)-------## Verification--1. **Unit/integration tests** — `cd backend && go test ./...` covers converter + repo round-trips on both ClickHouse and SQLite paths.-2. **End-to-end against a real Laravel/keepsuit app (zentigo is the canonical reproduction):**- - Build backend on this branch, point the app at it via `OTEL_EXPORTER_OTLP_ENDPOINT`.- - **Queue task (non-root):** `php artisan tinker --execute='\App\Jobs\TestTracewayJob::dispatch();'`, then run a queue worker. Confirm a new row appears on `/tasks` with the **Non-root** chip. Open it and confirm the "View distributed trace" link reaches a page showing both endpoint (dispatcher request) and task (worker) nodes.- - **Console task (root):** `php artisan bookings:send-reminders`. Confirm a Tasks row with the command name, no chip (root).- - **HTTP+queue together:** trigger a request that dispatches a notification (e.g. `NewReviewNotification`). On the endpoint detail page confirm the cross-trace link; on the distributed-trace view confirm both nodes.- - **AI trace (non-root):** easiest is a synthetic OTLP payload (`curl` a hand-rolled JSON with a SERVER root and a child span carrying `gen_ai.system="openai"` etc.). Confirm the AI Traces page shows the child with the chip and that the cross-trace view includes endpoint + ai_trace nodes.-3. **`rootFilter` UI** — toggle the dropdown on `/tasks`, `/ai-traces`, `/endpoints` and confirm URL persistence (`?rootFilter=non_root` etc., `all` omitted from URL).-4. **Backfill sanity** — after migration run, query `SELECT count(), sum(is_root) FROM endpoints / tasks / ai_traces`; both numbers should match (every row defaults to `is_root=1`).-------## Migration / Backfill Notes--- All existing rows default to `is_root = true` — correct because the old converter only ever inserted root spans.-- `ai_traces.distributed_trace_id` is `NULL` on existing rows. That's fine — the cross-trace lookup degrades gracefully and there's no UI that requires it to be present on historical rows.-- The change to `spans.trace_id` semantics (owning entity id, not OTel trace_id) is forward-only — existing span rows still have OTel trace_id stored there, which equals the endpoint's id for the single-process case (the only case the old converter produced), so existing waterfalls keep working.-------## Out of Scope--- Cross-service SERVER hops where you'd want one node per hop in the cross-trace view — endpoints now accept non-root SERVER spans, but the UX for showing per-hop endpoint nodes (e.g., a tree view of cross-service requests) is a separate decision.-- Reconciling the legacy `/api/report` distributed-trace flow with the new auto-derivation. The Go SDK's opt-in `distributedTraceId` field on `clientmodels` continues to work unchanged.-- A "show me only the root trace" view at the distributed-trace page — current behavior shows everything, which is what users want for debugging.diff --git a/backend/app/controllers/clientcontrollers/client.controller.go b/backend/app/controllers/clientcontrollers/client.controller.goindex 3a9eccfd..bc6cba72 100644--- a/backend/app/controllers/clientcontrollers/client.controller.go+++ b/backend/app/controllers/clientcontrollers/client.controller.go@@ -3,7 +3,6 @@ package clientcontrollersimport ("bytes""crypto/sha256"- "database/sql""encoding/hex""encoding/json""fmt"@@ -12,7 +11,6 @@ import ("strings""time"- "github.com/tracewayapp/traceway/backend/app/db""github.com/tracewayapp/traceway/backend/app/hooks""github.com/tracewayapp/traceway/backend/app/middleware""github.com/tracewayapp/traceway/backend/app/models"@@ -144,26 +142,13 @@ func (e clientController) Report(c *gin.Context) {spansToInsert = append(spansToInsert, span)}}- var sourceMaps *[]*models.SourceMap- if project != nil && isJsFramework(project.Framework) {- loadSpan := traceway.StartSpan(c, "report.load_source_maps")- sourceMapsLoaded, err := db.ExecuteTransaction(func(tx *sql.Tx) ([]*models.SourceMap, error) {- if request.AppVersion != "" {- return repositories.SourceMapRepository.FindByProjectAndVersion(tx, projectId, request.AppVersion)- }- return repositories.SourceMapRepository.FindLatestByProject(tx, projectId)- })- loadSpan.End()- if err == nil && len(sourceMapsLoaded) > 0 {- sourceMaps = &sourceMapsLoaded- }- }+ resolveJs := project != nil && isJsFramework(project.Framework)resolveSpan := traceway.StartSpan(c, "report.resolve_stack_traces")for _, cst := range cf.StackTraces {resolvedStackTrace := cst.StackTrace- if sourceMaps != nil {- resolvedStackTrace = services.ResolveStackTrace(c, projectId, cst.StackTrace, *sourceMaps)+ if resolveJs {+ resolvedStackTrace = services.ResolveStackTrace(c, projectId, cst.StackTrace)}est := cst.ToExceptionStackTrace(ComputeExceptionHash(resolvedStackTrace, cst.IsMessage), request.AppVersion, request.ServerName)est.StackTrace = resolvedStackTrace@@ -385,6 +370,7 @@ var jsFrameworks = map[string]bool{"react": true,"svelte": true,"vuejs": true,+ "jquery": true,"nextjs": true,"nestjs": true,"express": true,diff --git a/backend/app/controllers/otelcontrollers/helpers.go b/backend/app/controllers/otelcontrollers/helpers.goindex 04cc40d6..a3fc7566 100644--- a/backend/app/controllers/otelcontrollers/helpers.go+++ b/backend/app/controllers/otelcontrollers/helpers.go@@ -162,3 +162,54 @@ func getStringValues(attrs []*commonpb.KeyValue, key string) []string {}return out}++func getStringArray(attrs []*commonpb.KeyValue, key string) []string {+ for _, kv := range attrs {+ if kv.Key != key || kv.Value == nil {+ continue+ }+ av, ok := kv.Value.Value.(*commonpb.AnyValue_ArrayValue)+ if !ok || av.ArrayValue == nil {+ continue+ }+ out := make([]string, 0, len(av.ArrayValue.Values))+ for _, item := range av.ArrayValue.Values {+ s := ""+ if item != nil {+ if sv, ok := item.Value.(*commonpb.AnyValue_StringValue); ok {+ s = sv.StringValue+ }+ }+ out = append(out, s)+ }+ return out+ }+ return nil+}++func getIntArray(attrs []*commonpb.KeyValue, key string) []int64 {+ for _, kv := range attrs {+ if kv.Key != key || kv.Value == nil {+ continue+ }+ av, ok := kv.Value.Value.(*commonpb.AnyValue_ArrayValue)+ if !ok || av.ArrayValue == nil {+ continue+ }+ out := make([]int64, 0, len(av.ArrayValue.Values))+ for _, item := range av.ArrayValue.Values {+ var n int64+ if item != nil {+ switch iv := item.Value.(type) {+ case *commonpb.AnyValue_IntValue:+ n = iv.IntValue+ case *commonpb.AnyValue_DoubleValue:+ n = int64(iv.DoubleValue)+ }+ }+ out = append(out, n)+ }+ return out+ }+ return nil+}diff --git a/backend/app/controllers/otelcontrollers/otel.controller.go b/backend/app/controllers/otelcontrollers/otel.controller.goindex a4aa498d..1b57b442 100644--- a/backend/app/controllers/otelcontrollers/otel.controller.go+++ b/backend/app/controllers/otelcontrollers/otel.controller.go@@ -57,8 +57,15 @@ func (o otelController) ExportTraces(c *gin.Context) {return}+ symbolicate := func(ctx context.Context, stackTrace, language string) string {+ if !isJsLanguage(language) {+ return stackTrace+ }+ return services.ResolveStackTrace(ctx, projectId, stackTrace)+ }+convertStart := time.Now()- endpoints, tasks, spans, exceptions, aiTraces, aiConversations := convertTraces(projectId, req)+ endpoints, tasks, spans, exceptions, aiTraces, aiConversations := convertTraces(c, projectId, req, symbolicate)var droppedHealthchecks intendpoints, spans, droppedHealthchecks = services.FilterHealthchecks(project, endpoints, spans, exceptions)diff --git a/backend/app/controllers/otelcontrollers/trace_converter.go b/backend/app/controllers/otelcontrollers/trace_converter.goindex 5be76c5d..59457ba0 100644--- a/backend/app/controllers/otelcontrollers/trace_converter.go+++ b/backend/app/controllers/otelcontrollers/trace_converter.go@@ -1,6 +1,7 @@package otelcontrollersimport (+ "context""encoding/json""fmt""net/http"@@ -41,7 +42,7 @@ func (k entityKind) traceType() string {return ""}-func convertTraces(projectId uuid.UUID, req *coltracepb.ExportTraceServiceRequest) (+func convertTraces(ctx context.Context, projectId uuid.UUID, req *coltracepb.ExportTraceServiceRequest, symbolicate symbolicateFunc) (endpoints []models.Endpoint,tasks []models.Task,spans []models.Span,@@ -54,6 +55,7 @@ func convertTraces(projectId uuid.UUID, req *coltracepb.ExportTraceServiceRequesresourceAttrs := rs.GetResource().GetAttributes()serverName := getStringAttribute(resourceAttrs, "service.name")appVersion := getStringAttribute(resourceAttrs, "service.version")+ language := getStringAttribute(resourceAttrs, "telemetry.sdk.language")if appVersion == "" {if scriptVersionId := getStringAttribute(resourceAttrs, "cloudflare.script_version.id"); scriptVersionId != "" {if idx := strings.LastIndex(scriptVersionId, "-"); idx != -1 {@@ -264,8 +266,8 @@ func convertTraces(projectId uuid.UUID, req *coltracepb.ExportTraceServiceRequesfor _, event := range span.Events {if event.Name == "exception" {exc := buildException(- projectId, ownerId, traceType, event,- allAttrs, serverName, appVersion,+ ctx, projectId, ownerId, traceType, event,+ allAttrs, serverName, appVersion, language, symbolicate,)if owner != nil {exc.DistributedTraceId = owner.distributedTraceId@@ -439,21 +441,39 @@ func buildTask(}}+type symbolicateFunc func(ctx context.Context, stackTrace, language string) string+func buildException(+ ctx context.Context,projectId, traceId uuid.UUID,traceType string,event *tracepb.Span_Event,spanAttrs map[string]string,- serverName, appVersion string,+ serverName, appVersion, language string,+ symbolicate symbolicateFunc,) models.ExceptionStackTrace {eventAttrs := event.AttributesexcType := getStringAttribute(eventAttrs, "exception.type")excMessage := getStringAttribute(eventAttrs, "exception.message")- excStacktrace := getStringAttribute(eventAttrs, "exception.stacktrace")- stackTrace := formatExceptionStackTrace(excType, excMessage, excStacktrace)+ stackTrace, ok := buildHoneycombStackTrace(excType, excMessage, eventAttrs)+ if !ok {+ excStacktrace := getStringAttribute(eventAttrs, "exception.stacktrace")+ stackTrace = formatExceptionStackTrace(excType, excMessage, excStacktrace)+ }++ if symbolicate != nil {+ stackTrace = symbolicate(ctx, stackTrace, language)+ }+hash := clientcontrollers.ComputeExceptionHash(stackTrace, false)+ attrs := spanAttrs+ if isJsLanguage(language) {+ attrs = cloneStringMap(spanAttrs)+ attrs["telemetry.sdk.language"] = language+ }+return models.ExceptionStackTrace{Id: uuid.New(),ProjectId: projectId,@@ -462,12 +482,65 @@ func buildException(ExceptionHash: hash,StackTrace: stackTrace,RecordedAt: nanoToTime(event.TimeUnixNano),- Attributes: spanAttrs,+ Attributes: attrs,AppVersion: appVersion,ServerName: serverName,}}+func cloneStringMap(m map[string]string) map[string]string {+ out := make(map[string]string, len(m)+1)+ for k, v := range m {+ out[k] = v+ }+ return out+}++func isJsLanguage(lang string) bool {+ switch strings.ToLower(lang) {+ case "webjs", "nodejs", "javascript", "typescript":+ return true+ }+ return false+}++func buildHoneycombStackTrace(excType, excMessage string, eventAttrs []*commonpb.KeyValue) (string, bool) {+ urls := getStringArray(eventAttrs, "exception.structured_stacktrace.urls")+ if len(urls) == 0 {+ return "", false+ }+ functions := getStringArray(eventAttrs, "exception.structured_stacktrace.functions")+ lines := getIntArray(eventAttrs, "exception.structured_stacktrace.lines")+ columns := getIntArray(eventAttrs, "exception.structured_stacktrace.columns")++ at := func(s []string, i int) string {+ if i < len(s) {+ return s[i]+ }+ return ""+ }+ atInt := func(s []int64, i int) int64 {+ if i < len(s) {+ return s[i]+ }+ return 0+ }++ var b strings.Builder+ if header := formatExceptionStackTrace(excType, excMessage, ""); header != "unknown exception" {+ b.WriteString(header)+ b.WriteByte('\n')+ }+ for i, url := range urls {+ if fn := at(functions, i); fn != "" {+ b.WriteString(fn)+ b.WriteString("()\n")+ }+ fmt.Fprintf(&b, " %s:%d:%d\n", url, atInt(lines, i), atInt(columns, i))+ }+ return strings.TrimRight(b.String(), "\n"), true+}+func hasGenAiAttributes(attrs []*commonpb.KeyValue) bool {for _, kv := range attrs {if strings.HasPrefix(kv.Key, "gen_ai.") {diff --git a/backend/app/controllers/otelcontrollers/trace_converter_test.go b/backend/app/controllers/otelcontrollers/trace_converter_test.goindex 4c49437c..b2e2f6ee 100644--- a/backend/app/controllers/otelcontrollers/trace_converter_test.go+++ b/backend/app/controllers/otelcontrollers/trace_converter_test.go@@ -1,6 +1,7 @@package otelcontrollersimport (+ "context""encoding/json""flag""os"@@ -94,7 +95,7 @@ func TestConvertTraces_Snapshot(t *testing.T) {t.Fatalf("failed to unmarshal fixture %s: %v", tt.fixture, err)}- endpoints, _, spans, exceptions, aiTraces, aiConversations := convertTraces(testProjectId, req)+ endpoints, _, spans, exceptions, aiTraces, aiConversations := convertTraces(context.Background(), testProjectId, req, nil)// Check if all child spans share a trace ID with an endpointendpointIds := map[uuid.UUID]bool{}@@ -375,7 +376,7 @@ func TestTraceIdResolution_CrossScope(t *testing.T) {},}- endpoints, _, spans, _, _, _ := convertTraces(testProjectId, req)+ endpoints, _, spans, _, _, _ := convertTraces(context.Background(), testProjectId, req, nil)if len(endpoints) != 1 {t.Fatalf("expected 1 endpoint, got %d", len(endpoints))@@ -468,7 +469,7 @@ func TestConvertTraces_ConsumerNonRoot_BecomesTask(t *testing.T) {}},}- endpoints, tasks, spans, _, _, _ := convertTraces(testProjectId, req)+ endpoints, tasks, spans, _, _, _ := convertTraces(context.Background(), testProjectId, req, nil)if len(endpoints) != 0 {t.Fatalf("expected 0 endpoints, got %d", len(endpoints))@@ -518,7 +519,7 @@ func TestConvertTraces_ConsoleCommand_BecomesTask(t *testing.T) {}},}- _, tasks, _, _, _, _ := convertTraces(testProjectId, req)+ _, tasks, _, _, _, _ := convertTraces(context.Background(), testProjectId, req, nil)if len(tasks) != 1 {t.Fatalf("expected 1 task, got %d", len(tasks))}@@ -550,7 +551,7 @@ func TestConvertTraces_InlineGenAi_BecomesAiTrace(t *testing.T) {}},}- endpoints, _, _, _, aiTraces, _ := convertTraces(testProjectId, req)+ endpoints, _, _, _, aiTraces, _ := convertTraces(context.Background(), testProjectId, req, nil)if len(endpoints) != 1 {t.Fatalf("expected 1 endpoint, got %d", len(endpoints))}@@ -604,7 +605,7 @@ func TestConvertTraces_ExceptionOnConsumer_TraceTypeIsTask(t *testing.T) {}},}- _, tasks, _, exceptions, _, _ := convertTraces(testProjectId, req)+ _, tasks, _, exceptions, _, _ := convertTraces(context.Background(), testProjectId, req, nil)if len(tasks) != 1 || len(exceptions) != 1 {t.Fatalf("expected 1 task + 1 exception, got %d / %d", len(tasks), len(exceptions))}@@ -638,7 +639,7 @@ func TestConvertTraces_OrphanSpan_FallsBackToTraceId(t *testing.T) {}},}- _, _, spans, _, _, _ := convertTraces(testProjectId, req)+ _, _, spans, _, _, _ := convertTraces(context.Background(), testProjectId, req, nil)if len(spans) != 1 {t.Fatalf("expected 1 span row, got %d", len(spans))}@@ -660,3 +661,100 @@ func strKV(key, val string) *commonpb.KeyValue {Value: &commonpb.AnyValue{Value: &commonpb.AnyValue_StringValue{StringValue: val}},}}++func strArrayKV(key string, vals ...string) *commonpb.KeyValue {+ items := make([]*commonpb.AnyValue, len(vals))+ for i, v := range vals {+ items[i] = &commonpb.AnyValue{Value: &commonpb.AnyValue_StringValue{StringValue: v}}+ }+ return &commonpb.KeyValue{Key: key, Value: &commonpb.AnyValue{Value: &commonpb.AnyValue_ArrayValue{ArrayValue: &commonpb.ArrayValue{Values: items}}}}+}++func intArrayKV(key string, vals ...int64) *commonpb.KeyValue {+ items := make([]*commonpb.AnyValue, len(vals))+ for i, v := range vals {+ items[i] = &commonpb.AnyValue{Value: &commonpb.AnyValue_IntValue{IntValue: v}}+ }+ return &commonpb.KeyValue{Key: key, Value: &commonpb.AnyValue{Value: &commonpb.AnyValue_ArrayValue{ArrayValue: &commonpb.ArrayValue{Values: items}}}}+}++func TestBuildHoneycombStackTrace(t *testing.T) {+ attrs := []*commonpb.KeyValue{+ strArrayKV("exception.structured_stacktrace.urls", "https://x/app.js", "https://x/app.js"),+ strArrayKV("exception.structured_stacktrace.functions", "foo", ""),+ intArrayKV("exception.structured_stacktrace.lines", 10, 20),+ intArrayKV("exception.structured_stacktrace.columns", 5, 7),+ }+ got, ok := buildHoneycombStackTrace("Error", "boom", attrs)+ if !ok {+ t.Fatal("expected ok=true when structured stacktrace present")+ }+ want := "Error: boom\nfoo()\n https://x/app.js:10:5\n https://x/app.js:20:7"+ if got != want {+ t.Errorf("got %q\nwant %q", got, want)+ }++ if _, ok := buildHoneycombStackTrace("Error", "boom", makeAttrs("exception.stacktrace", "x")); ok {+ t.Error("expected ok=false when the structured field is absent")+ }+}++func TestConvertTraces_HoneycombJsExceptionSymbolicates(t *testing.T) {+ traceId := []byte{0xa0, 0xa1, 0xa2, 0xa3, 0xa4, 0xa5, 0xa6, 0xa7, 0xa8, 0xa9, 0xaa, 0xab, 0xac, 0xad, 0xae, 0xaf}+ spanId := []byte{0xb0, 0xb1, 0xb2, 0xb3, 0xb4, 0xb5, 0xb6, 0xb7}+ now := uint64(1_700_000_000_000_000_000)++ req := &coltracepb.ExportTraceServiceRequest{+ ResourceSpans: []*tracepb.ResourceSpans{{+ Resource: &resourcepb.Resource{Attributes: []*commonpb.KeyValue{+ strKV("service.name", "web"),+ strKV("telemetry.sdk.language", "webjs"),+ }},+ ScopeSpans: []*tracepb.ScopeSpans{{+ Spans: []*tracepb.Span{{+ TraceId: traceId, SpanId: spanId,+ Name: "GET /", Kind: tracepb.Span_SPAN_KIND_SERVER,+ StartTimeUnixNano: now, EndTimeUnixNano: now + 1_000_000,+ Attributes: makeAttrs("http.route", "/"),+ Events: []*tracepb.Span_Event{{+ Name: "exception",+ TimeUnixNano: now + 500_000,+ Attributes: []*commonpb.KeyValue{+ strKV("exception.type", "Error"),+ strKV("exception.message", "user has no name"),+ strArrayKV("exception.structured_stacktrace.urls", "app.min.js", "app.min.js"),+ strArrayKV("exception.structured_stacktrace.functions", "n", ""),+ intArrayKV("exception.structured_stacktrace.lines", 1, 1),+ intArrayKV("exception.structured_stacktrace.columns", 63, 146),+ },+ }},+ }},+ }},+ }},+ }++ var symInput, symLang string+ symbolicate := func(_ context.Context, stackTrace, language string) string {+ symInput, symLang = stackTrace, language+ return "RESOLVED"+ }++ _, _, _, exceptions, _, _ := convertTraces(context.Background(), testProjectId, req, symbolicate)+ if len(exceptions) != 1 {+ t.Fatalf("expected 1 exception, got %d", len(exceptions))+ }+ exc := exceptions[0]+ if symLang != "webjs" {+ t.Errorf("expected language webjs passed to symbolicate, got %q", symLang)+ }+ wantParsed := "Error: user has no name\nn()\n app.min.js:1:63\n app.min.js:1:146"+ if symInput != wantParsed {+ t.Errorf("honeycomb parse:\n got %q\nwant %q", symInput, wantParsed)+ }+ if exc.StackTrace != "RESOLVED" {+ t.Errorf("expected symbolicated stack trace stored, got %q", exc.StackTrace)+ }+ if exc.Attributes["telemetry.sdk.language"] != "webjs" {+ t.Errorf("expected stamped telemetry.sdk.language=webjs, got %q", exc.Attributes["telemetry.sdk.language"])+ }+}diff --git a/backend/app/controllers/sourcemap.controller.go b/backend/app/controllers/sourcemap.controller.goindex 2aa506c4..205dd826 100644--- a/backend/app/controllers/sourcemap.controller.go+++ b/backend/app/controllers/sourcemap.controller.go@@ -1,17 +1,13 @@package controllersimport (+ "fmt""github.com/tracewayapp/traceway/backend/app/middleware"- "github.com/tracewayapp/traceway/backend/app/models"- "github.com/tracewayapp/traceway/backend/app/db"- "github.com/tracewayapp/traceway/backend/app/repositories"+ "github.com/tracewayapp/traceway/backend/app/services""github.com/tracewayapp/traceway/backend/app/storage"- "database/sql"- "fmt""io""net/http"- "strings"- "time"+ "path/filepath""github.com/gin-gonic/gin"traceway "go.tracewayapp.com"@@ -19,6 +15,15 @@ import (type sourceMapController struct{}+func isSourceArtifact(name string) bool {+ switch filepath.Ext(name) {+ case ".map", ".js", ".cjs", ".mjs":+ return true+ default:+ return false+ }+}+func (s sourceMapController) Upload(c *gin.Context) {projectId, err := middleware.GetProjectId(c)if err != nil {@@ -31,11 +36,6 @@ func (s sourceMapController) Upload(c *gin.Context) {return}- version := c.Request.FormValue("version")- if version == "" {- version = "unversioned"- }-files := c.Request.MultipartForm.File["files"]if len(files) == 0 {c.JSON(http.StatusBadRequest, gin.H{"error": "No files uploaded"})@@ -44,7 +44,7 @@ func (s sourceMapController) Upload(c *gin.Context) {uploaded := 0for _, fileHeader := range files {- if !strings.HasSuffix(fileHeader.Filename, ".map") {+ if !isSourceArtifact(fileHeader.Filename) {continue}@@ -66,36 +66,12 @@ func (s sourceMapController) Upload(c *gin.Context) {return}- storageKey := fmt.Sprintf("sourcemaps/%s/%s/%s", projectId, version, fileHeader.Filename)-+ storageKey := fmt.Sprintf("sourcemaps/%s/%s", projectId, fileHeader.Filename)if err := storage.Store.Write(c, storageKey, data); err != nil {c.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to write source map to storage: %w", err))return}-- _, err = db.ExecuteTransaction(func(tx *sql.Tx) (*models.SourceMap, error) {- existing, err := repositories.SourceMapRepository.FindByProjectVersionAndFileName(tx, projectId, version, fileHeader.Filename)- if err != nil {- return nil, err- }- if existing != nil {- existing.StorageKey = storageKey- existing.FileSize = fileHeader.Size- existing.UploadedAt = time.Now().UTC()- return existing, repositories.SourceMapRepository.Update(tx, existing)- }- return repositories.SourceMapRepository.Create(tx, &models.SourceMap{- ProjectId: projectId,- Version: version,- FileName: fileHeader.Filename,- StorageKey: storageKey,- FileSize: fileHeader.Size,- })- })- if err != nil {- c.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to upsert source map metadata: %w", err))- return- }+ services.InvalidateSourceMap(projectId, fileHeader.Filename)uploaded++}diff --git a/backend/app/migrations/pg/0039_create_source_map_flatten_migrations.up.sql b/backend/app/migrations/pg/0039_create_source_map_flatten_migrations.up.sqlnew file mode 100644index 00000000..ada47ea3--- /dev/null+++ b/backend/app/migrations/pg/0039_create_source_map_flatten_migrations.up.sql@@ -0,0 +1,5 @@+CREATE TABLE IF NOT EXISTS source_map_flatten_migrations (+ id SERIAL PRIMARY KEY,+ project_id UUID NOT NULL UNIQUE,+ migrated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW()+)diff --git a/backend/app/migrations/sqlite/0013_create_source_map_flatten_migrations.up.sql b/backend/app/migrations/sqlite/0013_create_source_map_flatten_migrations.up.sqlnew file mode 100644index 00000000..cdd5afe3--- /dev/null+++ b/backend/app/migrations/sqlite/0013_create_source_map_flatten_migrations.up.sql@@ -0,0 +1,5 @@+CREATE TABLE IF NOT EXISTS source_map_flatten_migrations (+ id INTEGER PRIMARY KEY,+ project_id TEXT NOT NULL UNIQUE,+ migrated_at DATETIME NOT NULL DEFAULT (datetime('now'))+)diff --git a/backend/app/models/models.go b/backend/app/models/models.goindex 420ad9dc..6501cf08 100644--- a/backend/app/models/models.go+++ b/backend/app/models/models.go@@ -29,6 +29,8 @@ func Init(driver lit.Driver) {lit.RegisterModel[UserOrganizationResponse](driver)lit.RegisterModel[CountResult](driver)lit.RegisterModel[SourceMap](driver)+ lit.RegisterModel[SourceMapFlattenMigration](driver)+ lit.RegisterModel[SourceMapProjectId](driver)lit.RegisterModelWithNaming[MetricRegistry](driver, metricRegistryNaming{})lit.RegisterModel[WidgetGroup](driver)lit.RegisterModel[WidgetGroupWidget](driver)diff --git a/backend/app/models/source_map.model.go b/backend/app/models/source_map.model.goindex 6f13345f..1e02ecbf 100644--- a/backend/app/models/source_map.model.go+++ b/backend/app/models/source_map.model.go@@ -7,11 +7,21 @@ import ()type SourceMap struct {- Id int `json:"id"`- ProjectId uuid.UUID `json:"projectId"`- Version string `json:"version"`- FileName string `json:"fileName"`- StorageKey string `json:"storageKey"`- FileSize int64 `json:"fileSize"`+ Id int `json:"id"`+ ProjectId uuid.UUID `json:"projectId"`+ Version string `json:"version"`+ FileName string `json:"fileName"`+ StorageKey string `json:"storageKey"`+ FileSize int64 `json:"fileSize"`UploadedAt time.Time `json:"uploadedAt"`}++type SourceMapFlattenMigration struct {+ Id int `json:"id"`+ ProjectId uuid.UUID `json:"projectId"`+ MigratedAt time.Time `json:"migratedAt"`+}++type SourceMapProjectId struct {+ ProjectId uuid.UUID `json:"projectId"`+}diff --git a/backend/app/monitoring/backend_reporter.go b/backend/app/monitoring/backend_reporter.goindex 3469d5b7..0796bcba 100644--- a/backend/app/monitoring/backend_reporter.go+++ b/backend/app/monitoring/backend_reporter.go@@ -14,11 +14,13 @@ import (const backendReportInterval = 30 * time.Secondtype backendBaselines struct {- smHits uint64- smMisses uint64- smEvictions uint64- smFailures uint64- first bool+ smHits uint64+ smMisses uint64+ smEvictions uint64+ smFailures uint64+ smNotFound uint64+ smNegativeHits uint64+ first bool}func StartBackendReporter(ctx context.Context) {@@ -53,6 +55,7 @@ func reportBackendOnce(b *backendBaselines) {smStats := services.SourceMapStats()traceway.CaptureMetric("traceway.sourcemap.entries", float64(smStats.Entries))traceway.CaptureMetric("traceway.sourcemap.bytes", float64(smStats.Bytes))+ traceway.CaptureMetric("traceway.sourcemap.negative_entries", float64(smStats.NegativeEntries))traceway.CaptureMetric("traceway.sourcemap.parse_ms", smStats.LastParseMs)if !b.first {@@ -60,11 +63,15 @@ func reportBackendOnce(b *backendBaselines) {traceway.CaptureMetric("traceway.sourcemap.misses.delta", float64(safeDelta(b.smMisses, smStats.Misses)))traceway.CaptureMetric("traceway.sourcemap.evictions.delta", float64(safeDelta(b.smEvictions, smStats.Evictions)))traceway.CaptureMetric("traceway.sourcemap.load_failures.delta", float64(safeDelta(b.smFailures, smStats.Failures)))+ traceway.CaptureMetric("traceway.sourcemap.not_found.delta", float64(safeDelta(b.smNotFound, smStats.NotFound)))+ traceway.CaptureMetric("traceway.sourcemap.negative_hits.delta", float64(safeDelta(b.smNegativeHits, smStats.NegativeHits)))}b.smHits = smStats.Hitsb.smMisses = smStats.Missesb.smEvictions = smStats.Evictionsb.smFailures = smStats.Failures+ b.smNotFound = smStats.NotFound+ b.smNegativeHits = smStats.NegativeHitsb.first = falsetraceway.CaptureMetric("traceway.cache.projects.entries", float64(cache.ProjectCache.Entries()))diff --git a/backend/app/repositories/source_map.repository.go b/backend/app/repositories/source_map.repository.godeleted file mode 100644index cdca9a50..00000000--- a/backend/app/repositories/source_map.repository.go+++ /dev/null@@ -1,53 +0,0 @@-package repositories--import (- "database/sql"- "time"-- "github.com/tracewayapp/traceway/backend/app/models"-- "github.com/google/uuid"- "github.com/tracewayapp/lit/v2"-)--type sourceMapRepository struct{}--func (s *sourceMapRepository) Create(tx *sql.Tx, sm *models.SourceMap) (*models.SourceMap, error) {- sm.UploadedAt = time.Now().UTC()- id, err := lit.Insert(tx, sm)- if err != nil {- return nil, err- }- sm.Id = id- return sm, nil-}--func (s *sourceMapRepository) Update(tx *sql.Tx, sm *models.SourceMap) error {- return lit.UpdateNamed[models.SourceMap](tx, sm, "id = :id", lit.P{"id": sm.Id})-}--func (s *sourceMapRepository) FindByProjectAndVersion(tx *sql.Tx, projectId uuid.UUID, version string) ([]*models.SourceMap, error) {- return lit.SelectNamed[models.SourceMap](- tx,- "SELECT * FROM source_maps WHERE project_id = :project_id AND version = :version",- lit.P{"project_id": projectId, "version": version},- )-}--func (s *sourceMapRepository) FindLatestByProject(tx *sql.Tx, projectId uuid.UUID) ([]*models.SourceMap, error) {- return lit.SelectNamed[models.SourceMap](- tx,- "SELECT * FROM source_maps WHERE project_id = :project_id AND version = (SELECT version FROM source_maps WHERE project_id = :project_id ORDER BY uploaded_at DESC LIMIT 1)",- lit.P{"project_id": projectId},- )-}--func (s *sourceMapRepository) FindByProjectVersionAndFileName(tx *sql.Tx, projectId uuid.UUID, version, fileName string) (*models.SourceMap, error) {- return lit.SelectSingleNamed[models.SourceMap](- tx,- "SELECT * FROM source_maps WHERE project_id = :project_id AND version = :version AND file_name = :file_name",- lit.P{"project_id": projectId, "version": version, "file_name": fileName},- )-}--var SourceMapRepository = sourceMapRepository{}diff --git a/backend/app/services/sourcemap_cache_test.go b/backend/app/services/sourcemap_cache_test.goindex 81569cc2..026363ea 100644--- a/backend/app/services/sourcemap_cache_test.go+++ b/backend/app/services/sourcemap_cache_test.go@@ -5,12 +5,13 @@ import ("context""errors""fmt"+ "strings""sync""testing""time"- "github.com/tracewayapp/traceway/backend/app/models""github.com/tracewayapp/traceway/backend/app/storage"+ "github.com/tracewayapp/traceway/backend/app/symbolicator""github.com/google/uuid")@@ -19,12 +20,27 @@ func newTestSourceMapCache(maxEntries int, maxBytes int64) *sourceMapCache {return &sourceMapCache{items: make(map[string]*list.Element),order: list.New(),- loading: make(map[string]*sourceMapLoad),+ loading: make(map[string]*resolverLoad),+ negative: make(map[string]*negativeEntry),maxEntries: maxEntries,maxBytes: maxBytes,}}+func storageResolverBuild(key string) resolverBuild {+ return func(ctx context.Context) (*symbolicator.Resolver, int64, error) {+ data, err := storage.Store.Read(ctx, key)+ if err != nil {+ return nil, 0, err+ }+ r, err := symbolicator.NewResolver(data, nil)+ if err != nil {+ return nil, 0, err+ }+ return r, int64(len(data)), nil+ }+}+type countingStorage struct {mu sync.Mutexreads map[string]int@@ -45,7 +61,7 @@ func (c *countingStorage) Read(_ context.Context, key string) ([]byte, error) {time.Sleep(10 * time.Millisecond)d, ok := c.data[key]if !ok {- return nil, errors.New("not found")+ return nil, storage.ErrNotFound}return d, nil}@@ -60,18 +76,18 @@ func TestSourceMapCacheSingleflight(t *testing.T) {c := newTestSourceMapCache(10, 1<<20)const n = 16- results := make([]*parsedSourceMap, n)+ results := make([]*symbolicator.Resolver, n)var wg sync.WaitGroupstart := make(chan struct{})for i := range n {wg.Go(func() {<-start- sm, err := c.getOrLoad(context.Background(), "singleflight-test.js.map")+ r, err := c.getOrBuild(context.Background(), "singleflight-test.js.map", storageResolverBuild("singleflight-test.js.map"))if err != nil {t.Error(err)return}- results[i] = sm+ results[i] = r})}close(start)@@ -82,11 +98,11 @@ func TestSourceMapCacheSingleflight(t *testing.T) {}for i := 1; i < n; i++ {if results[i] != results[0] {- t.Fatal("expected all callers to share a single parsed instance")+ t.Fatal("expected all callers to share a single resolver instance")}}- if _, err := c.getOrLoad(context.Background(), "singleflight-test.js.map"); err != nil {+ if _, err := c.getOrBuild(context.Background(), "singleflight-test.js.map", storageResolverBuild("singleflight-test.js.map")); err != nil {t.Fatal(err)}if got := cs.reads["singleflight-test.js.map"]; got != 1 {@@ -107,16 +123,17 @@ func TestSourceMapCacheDistinctKeysConcurrent(t *testing.T) {var wg sync.WaitGroupstart := make(chan struct{})- results := make([]*parsedSourceMap, keys*4)+ results := make([]*symbolicator.Resolver, keys*4)for i := range keys * 4 {wg.Go(func() {<-start- sm, err := c.getOrLoad(context.Background(), fmt.Sprintf("distinct-%d.js.map", i%keys))+ key := fmt.Sprintf("distinct-%d.js.map", i%keys)+ r, err := c.getOrBuild(context.Background(), key, storageResolverBuild(key))if err != nil {t.Error(err)return}- results[i] = sm+ results[i] = r})}close(start)@@ -161,7 +178,7 @@ func TestSourceMapCacheFailedLoadRetries(t *testing.T) {storage.Store = fsc := newTestSourceMapCache(10, 1<<20)- if _, err := c.getOrLoad(context.Background(), "flaky.js.map"); err == nil {+ if _, err := c.getOrBuild(context.Background(), "flaky.js.map", storageResolverBuild("flaky.js.map")); err == nil {t.Fatal("expected first load to fail")}c.mu.Lock()@@ -169,14 +186,14 @@ func TestSourceMapCacheFailedLoadRetries(t *testing.T) {t.Errorf("expected 1 recorded failure, got %d", c.failures)}c.mu.Unlock()- sm, err := c.getOrLoad(context.Background(), "flaky.js.map")- if err != nil || sm == nil {+ r, err := c.getOrBuild(context.Background(), "flaky.js.map", storageResolverBuild("flaky.js.map"))+ if err != nil || r == nil {t.Fatalf("expected retry to succeed, got %v", err)}if fs.reads != 2 {t.Errorf("expected 2 storage reads (fail then retry), got %d", fs.reads)}- if _, err := c.getOrLoad(context.Background(), "flaky.js.map"); err != nil {+ if _, err := c.getOrBuild(context.Background(), "flaky.js.map", storageResolverBuild("flaky.js.map")); err != nil {t.Fatal(err)}if fs.reads != 2 {@@ -203,13 +220,13 @@ func TestSourceMapCachePanicRecovery(t *testing.T) {c := newTestSourceMapCache(10, 1<<20)type result struct {- sm *parsedSourceMap- err error+ resolver *symbolicator.Resolver+ err error}leader := make(chan result, 1)go func() {- sm, err := c.getOrLoad(context.Background(), "boom.js.map")- leader <- result{sm, err}+ r, err := c.getOrBuild(context.Background(), "boom.js.map", storageResolverBuild("boom.js.map"))+ leader <- result{r, err}}()deadline := time.Now().Add(2 * time.Second)@@ -229,21 +246,21 @@ func TestSourceMapCachePanicRecovery(t *testing.T) {waiters := make(chan result, 2)for range 2 {go func() {- sm, err := c.getOrLoad(context.Background(), "boom.js.map")- waiters <- result{sm, err}+ r, err := c.getOrBuild(context.Background(), "boom.js.map", storageResolverBuild("boom.js.map"))+ waiters <- result{r, err}}()}close(release)r := <-leader- if r.err == nil || r.sm != nil {- t.Fatalf("expected leader to receive panic error, got sm=%v err=%v", r.sm, r.err)+ if r.err == nil || r.resolver != nil {+ t.Fatalf("expected leader to receive panic error, got resolver=%v err=%v", r.resolver, r.err)}for range 2 {r := <-waiters- if r.err == nil || r.sm != nil {- t.Fatalf("expected waiter to receive an error, got sm=%v err=%v", r.sm, r.err)+ if r.err == nil || r.resolver != nil {+ t.Fatalf("expected waiter to receive an error, got resolver=%v err=%v", r.resolver, r.err)}}@@ -266,18 +283,176 @@ func TestResolveStackTraceFailedMapAttemptedOncePerTrace(t *testing.T) {cs := &countingStorage{reads: map[string]int{}, data: map[string][]byte{}}storage.Store = cs- sourceMaps := []*models.SourceMap{{FileName: "dead.js.map", StorageKey: "dead.js.map"}}+ projectId := uuid.New()trace := "Error: boom\n fn()\n dead.js:1:10\n fn2()\n dead.js:1:20\n fn3()\n dead.js:1:30"- resolved := ResolveStackTrace(context.Background(), uuid.New(), trace, sourceMaps)+ resolved := ResolveStackTrace(context.Background(), projectId, trace)if resolved != trace {t.Error("trace should be stored as-is when its source map cannot be loaded")}- if got := cs.reads["dead.js.map"]; got != 1 {+ mapKey := fmt.Sprintf("sourcemaps/%s/dead.js.map", projectId)+ if got := cs.reads[mapKey]; got != 1 {t.Errorf("expected 1 load attempt per trace for a failing map, got %d", got)}}+func TestSourceMapNegativeBackoffEscalates(t *testing.T) {+ c := newTestSourceMapCache(10, 1<<20)++ c.mu.Lock()+ c.markNegativeLocked("k", sourceMapNegativeBaseTTL)+ first := time.Until(c.negative["k"].expiresAt)+ c.mu.Unlock()+ if first > sourceMapNegativeBaseTTL || first < sourceMapNegativeBaseTTL-time.Second {+ t.Errorf("first failure should use the base TTL, got %v", first)+ }++ c.mu.Lock()+ c.markNegativeLocked("k", sourceMapNegativeBaseTTL)+ second := time.Until(c.negative["k"].expiresAt)+ for range 20 {+ c.markNegativeLocked("k", sourceMapNegativeBaseTTL)+ }+ capped := time.Until(c.negative["k"].expiresAt)+ c.mu.Unlock()++ if second < first {+ t.Errorf("TTL should escalate on repeat failures: first %v, second %v", first, second)+ }+ if capped > sourceMapNegativeMaxTTL || capped < sourceMapNegativeMaxTTL-time.Second {+ t.Errorf("TTL should cap at %v, got %v", sourceMapNegativeMaxTTL, capped)+ }+}++func TestSourceMapNegativeMapPrunesAtCap(t *testing.T) {+ c := newTestSourceMapCache(10, 1<<20)+ c.mu.Lock()+ for i := range sourceMapNegativeMaxKeys {+ c.markNegativeLocked(fmt.Sprintf("k-%d", i), sourceMapNegativeBaseTTL)+ }+ c.markNegativeLocked("one-more", sourceMapNegativeBaseTTL)+ size := len(c.negative)+ c.mu.Unlock()+ if size > sourceMapNegativeMaxKeys {+ t.Errorf("negative map should stay at or below %d keys, got %d", sourceMapNegativeMaxKeys, size)+ }+}++func TestResolveStackTraceTransientFailureNegativeCached(t *testing.T) {+ prev := storage.Store+ defer func() { storage.Store = prev }()+ fs := &flakyStorage{failures: 1 << 30}+ storage.Store = fs++ projectId := uuid.New()+ trace := "Error: boom\n fn()\n down.js:1:10"++ _ = ResolveStackTrace(context.Background(), projectId, trace)+ _ = ResolveStackTrace(context.Background(), projectId, trace)++ if fs.reads != 1 {+ t.Errorf("transient storage failures should be negative cached, got %d reads", fs.reads)+ }+}++func TestInvalidateSourceMapClearsNegative(t *testing.T) {+ InitSourceMapCache(100, 64<<20)+ prev := storage.Store+ defer func() { storage.Store = prev }()+ cs := &countingStorage{reads: map[string]int{}, data: map[string][]byte{}}+ storage.Store = cs++ projectId := uuid.New()+ mapKey := fmt.Sprintf("sourcemaps/%s/late.js.map", projectId)+ trace := "Error: boom\n fn()\n late.js:1:1"++ if got := ResolveStackTrace(context.Background(), projectId, trace); got != trace {+ t.Error("trace should pass through while the map is missing")+ }++ cs.data[mapKey] = []byte(`{"version":3,"sources":["a.js"],"names":[],"mappings":"AAAA"}`)+ InvalidateSourceMap(projectId, "late.js.map")++ resolved := ResolveStackTrace(context.Background(), projectId, trace)+ if !strings.Contains(resolved, "a.js:1:1") {+ t.Errorf("upload should clear the negative entry immediately, got %q", resolved)+ }+}++func TestInvalidateSourceMapEvictsStaleResolver(t *testing.T) {+ InitSourceMapCache(100, 64<<20)+ prev := storage.Store+ defer func() { storage.Store = prev }()+ cs := &countingStorage{reads: map[string]int{}, data: map[string][]byte{}}+ storage.Store = cs++ projectId := uuid.New()+ mapKey := fmt.Sprintf("sourcemaps/%s/stable.js.map", projectId)+ cs.data[mapKey] = []byte(`{"version":3,"sources":["a.js"],"names":[],"mappings":"AAAA"}`)+ trace := "Error: boom\n fn()\n stable.js:1:1"++ if got := ResolveStackTrace(context.Background(), projectId, trace); !strings.Contains(got, "a.js:1:1") {+ t.Fatalf("expected initial resolution against a.js, got %q", got)+ }++ cs.data[mapKey] = []byte(`{"version":3,"sources":["b.js"],"names":[],"mappings":"AAAA"}`)+ if got := ResolveStackTrace(context.Background(), projectId, trace); !strings.Contains(got, "a.js:1:1") {+ t.Fatalf("re-upload without invalidation should still serve the cached resolver, got %q", got)+ }++ InvalidateSourceMap(projectId, "stable.js")+ if got := ResolveStackTrace(context.Background(), projectId, trace); !strings.Contains(got, "b.js:1:1") {+ t.Errorf("invalidation should evict the cached resolver, got %q", got)+ }+}++type bundleFailStorage struct {+ data map[string][]byte+ failBundle bool+}++func (b *bundleFailStorage) Write(_ context.Context, key string, data []byte) error {+ b.data[key] = data+ return nil+}++func (b *bundleFailStorage) Read(_ context.Context, key string) ([]byte, error) {+ if b.failBundle && !strings.HasSuffix(key, ".map") {+ return nil, errors.New("storage down")+ }+ d, ok := b.data[key]+ if !ok {+ return nil, storage.ErrNotFound+ }+ return d, nil+}++func TestTransientBundleReadFailureFailsBuild(t *testing.T) {+ InitSourceMapCache(100, 64<<20)+ prev := storage.Store+ defer func() { storage.Store = prev }()+ projectId := uuid.New()+ prefix := fmt.Sprintf("sourcemaps/%s/", projectId)+ bs := &bundleFailStorage{failBundle: true, data: map[string][]byte{+ prefix + "app.js.map": []byte(`{"version":3,"sources":["a.js"],"names":[],"mappings":"AAAA"}`),+ prefix + "app.js": []byte(`var x=1;`),+ }}+ storage.Store = bs++ trace := "Error: boom\n fn()\n app.js:1:1"++ if got := ResolveStackTrace(context.Background(), projectId, trace); got != trace {+ t.Errorf("a transient bundle read failure must not cache a names-less resolver, got %q", got)+ }++ bs.failBundle = false+ InvalidateSourceMap(projectId, "app.js.map")++ if got := ResolveStackTrace(context.Background(), projectId, trace); !strings.Contains(got, "a.js:1:1") {+ t.Errorf("expected full resolution once the bundle read recovers, got %q", got)+ }+}+func TestSourceMapCacheByteCapEviction(t *testing.T) {prev := storage.Storedefer func() { storage.Store = prev }()@@ -293,7 +468,7 @@ func TestSourceMapCacheByteCapEviction(t *testing.T) {for i := range 3 {key := fmt.Sprintf("evict-%d.js.map", i)cs.data[key] = fmt.Appendf(nil, `{"version":3,"sources":["a.js"],"sourcesContent":[%q],"names":[],"mappings":"AAAA"}`, content)- if _, err := c.getOrLoad(context.Background(), key); err != nil {+ if _, err := c.getOrBuild(context.Background(), key, storageResolverBuild(key)); err != nil {t.Fatal(err)}}diff --git a/backend/app/services/sourcemap_parse.go b/backend/app/services/sourcemap_parse.godeleted file mode 100644index 3363bc82..00000000--- a/backend/app/services/sourcemap_parse.go+++ /dev/null@@ -1,352 +0,0 @@-package services--import (- "encoding/json"- "errors"- "fmt"- "net/url"- "path"- "sort"- "strings"-)--type smMapping struct {- genLine int32- genCol int32- srcInd int32- srcLine int32- srcCol int32- nameInd int32-}--const smMappingBytes = 24--type smSection struct {- offsetLine int- offsetCol int- sources []string- sourcesContent []string- names []string- mappings []smMapping-}--type parsedSourceMap struct {- sections []smSection- size int64-}--type smSourceMapJSON struct {- Version int `json:"version"`- SourceRoot string `json:"sourceRoot"`- Sources []string `json:"sources"`- SourcesContent []string `json:"sourcesContent"`- Names []json.RawMessage `json:"names"`- Mappings string `json:"mappings"`-}--type smSectionJSON struct {- Offset struct {- Line int `json:"line"`- Column int `json:"column"`- } `json:"offset"`- Map *smSourceMapJSON `json:"map"`-}--type smFileJSON struct {- smSourceMapJSON- Sections []smSectionJSON `json:"sections"`-}--func parseSourceMap(data []byte) (*parsedSourceMap, error) {- var f smFileJSON- if err := json.Unmarshal(data, &f); err != nil {- return nil, err- }- if err := checkSourceMapVersion(f.Version); err != nil {- return nil, err- }-- sectionsJSON := f.Sections- if len(sectionsJSON) == 0 {- sectionsJSON = []smSectionJSON{{Map: &f.smSourceMapJSON}}- }-- p := &parsedSourceMap{sections: make([]smSection, 0, len(sectionsJSON))}- for _, sj := range sectionsJSON {- if sj.Map == nil {- return nil, errors.New("sourcemap: section without map")- }- s, err := parseSourceMapSection(sj)- if err != nil {- return nil, err- }- p.sections = append(p.sections, s)- }- for i, j := 0, len(p.sections)-1; i < j; i, j = i+1, j-1 {- p.sections[i], p.sections[j] = p.sections[j], p.sections[i]- }- p.computeSize()- return p, nil-}--func parseSourceMapSection(sj smSectionJSON) (smSection, error) {- m := sj.Map- if err := checkSourceMapVersion(m.Version); err != nil {- return smSection{}, err- }-- var rootURL *url.URL- if m.SourceRoot != "" {- u, err := url.Parse(m.SourceRoot)- if err != nil {- return smSection{}, err- }- if u.IsAbs() {- rootURL = u- }- }-- sources := make([]string, len(m.Sources))- for i, src := range m.Sources {- sources[i] = absSourceName(rootURL, m.SourceRoot, src)- }-- names := make([]string, len(m.Names))- for i, raw := range m.Names {- names[i] = decodeSourceMapName(raw)- }-- mappings, err := parseSourceMapMappings(m.Mappings)- if err != nil {- return smSection{}, err- }-- return smSection{- offsetLine: sj.Offset.Line,- offsetCol: sj.Offset.Column,- sources: sources,- sourcesContent: m.SourcesContent,- names: names,- mappings: mappings,- }, nil-}--func checkSourceMapVersion(version int) error {- if version == 3 || version == 0 {- return nil- }- return fmt.Errorf("sourcemap: got version=%d, but only 3rd version is supported", version)-}--func absSourceName(rootURL *url.URL, sourceRoot, source string) string {- if path.IsAbs(source) {- return source- }- if u, err := url.Parse(source); err == nil && u.IsAbs() {- return source- }- if rootURL != nil {- u := *rootURL- u.Path = path.Join(u.Path, source)- return u.String()- }- if sourceRoot != "" {- return path.Join(sourceRoot, source)- }- return source-}--func decodeSourceMapName(raw json.RawMessage) string {- if len(raw) == 0 {- return ""- }- if raw[0] == '"' && raw[len(raw)-1] == '"' {- var s string- if err := json.Unmarshal(raw, &s); err == nil {- return s- }- }- return string(raw)-}--var vlqDecodeMap [256]byte--func init() {- const alphabet = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"- for i := range len(alphabet) {- vlqDecodeMap[alphabet[i]] = byte(i)- }-}--func decodeVLQ(s string, i int) (int32, int, error) {- var n int32- shift := uint(0)- for {- if i >= len(s) {- return 0, i, errors.New("sourcemap: unexpected end of mappings")- }- c := vlqDecodeMap[s[i]]- i++- n += int32(c&31) << shift- shift += 5- if c&32 == 0 {- break- }- }- if n&1 != 0 {- return -(n >> 1), i, nil- }- return n >> 1, i, nil-}--func parseSourceMapMappings(s string) ([]smMapping, error) {- if s == "" {- return nil, errors.New("sourcemap: mappings are empty")- }-- values := make([]smMapping, 0, strings.Count(s, ",")+strings.Count(s, ";"))- cur := smMapping{genLine: 1, srcLine: 1}- hasValue := false- hasName := false- field := 0-- push := func() {- if !hasValue {- return- }- hasValue = false- m := cur- if hasName {- hasName = false- } else {- m.nameInd = -1- }- values = append(values, m)- }-- i := 0- for i < len(s) {- switch s[i] {- case ',':- push()- field = 0- i++- case ';':- push()- cur.genLine++- cur.genCol = 0- field = 0- i++- default:- n, next, err := decodeVLQ(s, i)- if err != nil {- return nil, err- }- i = next- switch field {- case 0:- cur.genCol += n- case 1:- cur.srcInd += n- case 2:- cur.srcLine += n- case 3:- cur.srcCol += n- case 4:- cur.nameInd += n- hasName = true- }- if field == 4 {- field = 0- } else {- field++- }- hasValue = true- }- }- push()- return values, nil-}--func (p *parsedSourceMap) source(genLine, genCol int) (string, string, int, int, bool) {- for i := range p.sections {- s := &p.sections[i]- if s.offsetLine < genLine || (s.offsetLine+1 == genLine && s.offsetCol <= genCol) {- return s.lookup(genLine-s.offsetLine, genCol-s.offsetCol)- }- }- return "", "", 0, 0, false-}--func (s *smSection) lookup(genLine, genCol int) (string, string, int, int, bool) {- if len(s.mappings) == 0 {- return "", "", 0, 0, false- }-- i := sort.Search(len(s.mappings), func(i int) bool {- m := &s.mappings[i]- if int(m.genLine) == genLine {- return int(m.genCol) >= genCol- }- return int(m.genLine) >= genLine- })-- var match *smMapping- if i == len(s.mappings) {- match = &s.mappings[i-1]- if int(match.genLine) != genLine {- return "", "", 0, 0, false- }- } else {- match = &s.mappings[i]- if int(match.genLine) > genLine || int(match.genCol) > genCol {- if i == 0 {- return "", "", 0, 0, false- }- match = &s.mappings[i-1]- }- }-- source := ""- if match.srcInd >= 0 && int(match.srcInd) < len(s.sources) {- source = s.sources[match.srcInd]- }- name := ""- if match.nameInd >= 0 && int(match.nameInd) < len(s.names) {- name = s.names[match.nameInd]- }- return source, name, int(match.srcLine), int(match.srcCol), true-}--func (p *parsedSourceMap) sourceContent(file string) string {- for i := range p.sections {- s := &p.sections[i]- for j, src := range s.sources {- if src == file {- if j < len(s.sourcesContent) {- return s.sourcesContent[j]- }- break- }- }- }- return ""-}--func (p *parsedSourceMap) computeSize() {- size := int64(64)- for i := range p.sections {- s := &p.sections[i]- size += 64 + int64(len(s.mappings))*smMappingBytes- for _, v := range s.sources {- size += int64(len(v)) + 16- }- for _, v := range s.sourcesContent {- size += int64(len(v)) + 16- }- for _, v := range s.names {- size += int64(len(v)) + 16- }- }- p.size = size-}diff --git a/backend/app/services/sourcemap_parse_test.go b/backend/app/services/sourcemap_parse_test.godeleted file mode 100644index da7a62e3..00000000--- a/backend/app/services/sourcemap_parse_test.go+++ /dev/null@@ -1,103 +0,0 @@-package services--import (- "fmt"- "strings"- "testing"-)--func TestParseSourceMapSections(t *testing.T) {- data := []byte(`{"version":3,"sections":[- {"offset":{"line":0,"column":0},"map":{"version":3,"sources":["first.js"],"names":[],"mappings":"AAAA"}},- {"offset":{"line":5,"column":0},"map":{"version":3,"sources":["second.js"],"names":[],"mappings":"AAAA"}}]}`)- p, err := parseSourceMap(data)- if err != nil {- t.Fatal(err)- }-- file, _, line, col, ok := p.source(1, 0)- if !ok || file != "first.js" || line != 1 || col != 0 {- t.Errorf("line 1 should resolve via the first section, got file=%q line=%d col=%d ok=%v", file, line, col, ok)- }-- file, _, line, col, ok = p.source(6, 0)- if !ok || file != "second.js" || line != 1 || col != 0 {- t.Errorf("line 6 should resolve via the second section (offset 5), got file=%q line=%d col=%d ok=%v", file, line, col, ok)- }-- if _, _, _, _, ok = p.source(7, 0); ok {- t.Error("line past the second section's mappings should not resolve")- }-}--func TestParseSourceMapErrors(t *testing.T) {- cases := []struct {- name string- data string- wantErr string- }{- {"version rejected", `{"version":2,"sources":[],"names":[],"mappings":"AAAA"}`, "version"},- {"empty mappings", `{"version":3,"sources":[],"names":[],"mappings":""}`, "mappings are empty"},- {"truncated vlq", `{"version":3,"sources":[],"names":[],"mappings":"g"}`, "unexpected end of mappings"},- {"section without map", `{"version":3,"sections":[{"offset":{"line":0,"column":0}}]}`, "section without map"},- {"section version rejected", `{"version":3,"sections":[{"offset":{"line":0,"column":0},"map":{"version":2,"sources":[],"names":[],"mappings":"AAAA"}}]}`, "version"},- }- for _, tc := range cases {- t.Run(tc.name, func(t *testing.T) {- _, err := parseSourceMap([]byte(tc.data))- if err == nil || !strings.Contains(err.Error(), tc.wantErr) {- t.Errorf("expected error containing %q, got %v", tc.wantErr, err)- }- })- }-}--func TestParseSourceMapNonStringNames(t *testing.T) {- p, err := parseSourceMap([]byte(`{"version":3,"sources":["a.js"],"names":[42],"mappings":"AAAAA"}`))- if err != nil {- t.Fatal(err)- }- _, name, _, _, ok := p.source(1, 0)- if !ok || name != "42" {- t.Errorf("expected numeric name decoded to %q, got %q ok=%v", "42", name, ok)- }-}--func TestParseSourceMapSourceIndexOutOfBounds(t *testing.T) {- p, err := parseSourceMap([]byte(`{"version":3,"sources":["a.js"],"names":[],"mappings":"ACAA"}`))- if err != nil {- t.Fatal(err)- }- file, _, line, col, ok := p.source(1, 0)- if !ok || file != "" || line != 1 || col != 0 {- t.Errorf("out-of-bounds source index should resolve with empty file, got file=%q line=%d col=%d ok=%v", file, line, col, ok)- }-}--func TestParseSourceMapSourceRoot(t *testing.T) {- cases := []struct {- name string- sourceRoot string- source string- want string- }{- {"path join", "src", "a.js", "src/a.js"},- {"url join", "http://example.com/assets/", "a.js", "http://example.com/assets/a.js"},- {"absolute path kept", "src", "/abs/a.js", "/abs/a.js"},- {"absolute url kept", "src", "webpack:///./foo.js", "webpack:///./foo.js"},- {"no root", "", "../src/a.js", "../src/a.js"},- }- for _, tc := range cases {- t.Run(tc.name, func(t *testing.T) {- data := fmt.Sprintf(`{"version":3,"sourceRoot":%q,"sources":[%q],"names":[],"mappings":"AAAA"}`, tc.sourceRoot, tc.source)- p, err := parseSourceMap([]byte(data))- if err != nil {- t.Fatal(err)- }- file, _, _, _, ok := p.source(1, 0)- if !ok || file != tc.want {- t.Errorf("expected source %q, got %q ok=%v", tc.want, file, ok)- }- })- }-}diff --git a/backend/app/services/sourcemap_resolver.go b/backend/app/services/sourcemap_resolver.goindex 1e2f296d..351c8da1 100644--- a/backend/app/services/sourcemap_resolver.go+++ b/backend/app/services/sourcemap_resolver.go@@ -3,6 +3,7 @@ package servicesimport ("container/list""context"+ "errors""fmt""path/filepath""regexp"@@ -11,8 +12,8 @@ import ("sync""time"- "github.com/tracewayapp/traceway/backend/app/models""github.com/tracewayapp/traceway/backend/app/storage"+ "github.com/tracewayapp/traceway/backend/app/symbolicator""github.com/google/uuid"traceway "go.tracewayapp.com"@@ -20,12 +21,19 @@ import (const sourceMapLoadTimeout = 5 * time.Secondconst sourceMapFailReportInterval = time.Minute+const sourceMapNegativeBaseTTL = time.Minute+const sourceMapTransientNegativeBaseTTL = 15 * time.Second+const sourceMapNegativeMaxTTL = 15 * time.Minute+const sourceMapNegativeMaxKeys = 10000++type resolverBuild func(context.Context) (*symbolicator.Resolver, int64, error)type sourceMapCache struct {mu sync.Mutexitems map[string]*list.Elementorder *list.List- loading map[string]*sourceMapLoad+ loading map[string]*resolverLoad+ negative map[string]*negativeEntrymaxEntries intmaxBytes int64curBytes int64@@ -33,26 +41,35 @@ type sourceMapCache struct {misses uint64evictions uint64failures uint64+ notFound uint64+ negativeHits uint64lastParseMs float64failuresSinceReport uint64lastFailAt time.Time}+type negativeEntry struct {+ expiresAt time.Time+ failures uint32+}+type sourceMapCacheEntry struct {- key string- sm *parsedSourceMap+ key string+ resolver *symbolicator.Resolver+ size int64}-type sourceMapLoad struct {- done chan struct{}- sm *parsedSourceMap- err error+type resolverLoad struct {+ done chan struct{}+ resolver *symbolicator.Resolver+ err error}var smCache = &sourceMapCache{items: make(map[string]*list.Element),order: list.New(),- loading: make(map[string]*sourceMapLoad),+ loading: make(map[string]*resolverLoad),+ negative: make(map[string]*negativeEntry),maxEntries: 200,maxBytes: 500 << 20,}@@ -65,12 +82,12 @@ func InitSourceMapCache(maxEntries int, maxBytes int64) {smCache.evictLocked()}-func (c *sourceMapCache) getOrLoad(ctx context.Context, key string) (sm *parsedSourceMap, err error) {+func (c *sourceMapCache) getOrBuild(ctx context.Context, key string, build resolverBuild) (resolver *symbolicator.Resolver, err error) {c.mu.Lock()if el, ok := c.items[key]; ok {c.hits++c.order.MoveToFront(el)- cached := el.Value.(*sourceMapCacheEntry).sm+ cached := el.Value.(*sourceMapCacheEntry).resolverc.mu.Unlock()return cached, nil}@@ -82,53 +99,107 @@ func (c *sourceMapCache) getOrLoad(ctx context.Context, key string) (sm *parsedSc.hits++c.mu.Unlock()}- return l.sm, l.err+ return l.resolver, l.err}c.misses++- l := &sourceMapLoad{done: make(chan struct{})}+ l := &resolverLoad{done: make(chan struct{})}c.loading[key] = lc.mu.Unlock()- var parseMs float64+ var size int64+ var buildMs float64defer func() {if r := recover(); r != nil {- l.sm = nil- l.err = fmt.Errorf("source map load panicked (key=%s): %v", key, r)+ l.resolver = nil+ l.err = fmt.Errorf("source map resolver build panicked (key=%s): %v", key, r)c.reportLoadFailure(l.err)- sm, err = nil, l.err+ resolver, err = nil, l.err}c.mu.Lock()delete(c.loading, key)- if l.err == nil && l.sm != nil {- c.lastParseMs = parseMs- c.insertLocked(key, l.sm)- } else {- c.failures+++ if l.err == nil && l.resolver != nil {+ c.lastParseMs = buildMs+ c.insertLocked(key, l.resolver, size)+ delete(c.negative, key)+ } else if l.err != nil {+ if errors.Is(l.err, storage.ErrNotFound) {+ c.notFound+++ c.markNegativeLocked(key, sourceMapNegativeBaseTTL)+ } else {+ c.failures+++ c.markNegativeLocked(key, sourceMapTransientNegativeBaseTTL)+ }}c.mu.Unlock()close(l.done)}()- l.sm, parseMs, l.err = c.load(ctx, key)- return l.sm, l.err+ start := time.Now()+ l.resolver, size, l.err = build(ctx)+ buildMs = float64(time.Since(start).Microseconds()) / 1000.0+ if l.err != nil && !errors.Is(l.err, storage.ErrNotFound) {+ c.reportLoadFailure(fmt.Errorf("failed to build source map resolver (key=%s): %w", key, l.err))+ }+ return l.resolver, l.err}-func (c *sourceMapCache) load(ctx context.Context, key string) (*parsedSourceMap, float64, error) {- readCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), sourceMapLoadTimeout)- defer cancel()- data, err := storage.Store.Read(readCtx, key)- if err != nil {- c.reportLoadFailure(fmt.Errorf("failed to read source map from storage (key=%s): %w", key, err))- return nil, 0, err+func (c *sourceMapCache) isNegative(key string) bool {+ c.mu.Lock()+ defer c.mu.Unlock()+ e, ok := c.negative[key]+ if !ok || time.Now().After(e.expiresAt) {+ return false}+ c.negativeHits+++ return true+}- parseStart := time.Now()- sm, err := parseSourceMap(data)- if err != nil {- c.reportLoadFailure(fmt.Errorf("failed to parse source map (key=%s): %w", key, err))- return nil, 0, err+func (c *sourceMapCache) markNegativeLocked(key string, base time.Duration) {+ e := c.negative[key]+ if e == nil {+ if len(c.negative) >= sourceMapNegativeMaxKeys {+ c.pruneNegativeLocked()+ }+ e = &negativeEntry{}+ c.negative[key] = e+ }+ ttl := min(base<<min(e.failures, 16), sourceMapNegativeMaxTTL)+ e.failures+++ e.expiresAt = time.Now().Add(ttl)+}++func (c *sourceMapCache) pruneNegativeLocked() {+ now := time.Now()+ for k, e := range c.negative {+ if now.After(e.expiresAt) {+ delete(c.negative, k)+ }+ }+ for k := range c.negative {+ if len(c.negative) < sourceMapNegativeMaxKeys {+ break+ }+ delete(c.negative, k)+ }+}++func (c *sourceMapCache) invalidate(key string) {+ c.mu.Lock()+ defer c.mu.Unlock()+ delete(c.negative, key)+ if el, ok := c.items[key]; ok {+ evicted := c.order.Remove(el).(*sourceMapCacheEntry)+ delete(c.items, key)+ c.curBytes -= evicted.size}- return sm, float64(time.Since(parseStart).Microseconds()) / 1000.0, nil+}++func InvalidateSourceMap(projectId uuid.UUID, fileName string) {+ base := filepath.Base(fileName)+ if !strings.HasSuffix(base, ".map") {+ base += ".map"+ }+ smCache.invalidate(fmt.Sprintf("sourcemaps/%s/%s", projectId, base))}func (c *sourceMapCache) reportLoadFailure(err error) {@@ -142,14 +213,14 @@ func (c *sourceMapCache) reportLoadFailure(err error) {}c.mu.Unlock()if report > 0 {- traceway.CaptureException(fmt.Errorf("source map loads failed %d time(s) since last report: %w", report, err))+ traceway.CaptureException(fmt.Errorf("source map resolver builds failed %d time(s) since last report: %w", report, err))}}-func (c *sourceMapCache) insertLocked(key string, sm *parsedSourceMap) {- el := c.order.PushFront(&sourceMapCacheEntry{key: key, sm: sm})+func (c *sourceMapCache) insertLocked(key string, resolver *symbolicator.Resolver, size int64) {+ el := c.order.PushFront(&sourceMapCacheEntry{key: key, resolver: resolver, size: size})c.items[key] = el- c.curBytes += sm.size+ c.curBytes += sizec.evictLocked()}@@ -161,69 +232,56 @@ func (c *sourceMapCache) evictLocked() {}evicted := c.order.Remove(back).(*sourceMapCacheEntry)delete(c.items, evicted.key)- c.curBytes -= evicted.sm.size+ c.curBytes -= evicted.sizec.evictions++}}type SourceMapCacheStats struct {- Entries int- Bytes int64- MaxEntries int- MaxBytes int64- Hits uint64- Misses uint64- Evictions uint64- Failures uint64- LastParseMs float64+ Entries int+ Bytes int64+ MaxEntries int+ MaxBytes int64+ Hits uint64+ Misses uint64+ Evictions uint64+ Failures uint64+ NotFound uint64+ NegativeHits uint64+ NegativeEntries int+ LastParseMs float64}func SourceMapStats() SourceMapCacheStats {smCache.mu.Lock()defer smCache.mu.Unlock()return SourceMapCacheStats{- Entries: smCache.order.Len(),- Bytes: smCache.curBytes,- MaxEntries: smCache.maxEntries,- MaxBytes: smCache.maxBytes,- Hits: smCache.hits,- Misses: smCache.misses,- Evictions: smCache.evictions,- Failures: smCache.failures,- LastParseMs: smCache.lastParseMs,+ Entries: smCache.order.Len(),+ Bytes: smCache.curBytes,+ MaxEntries: smCache.maxEntries,+ MaxBytes: smCache.maxBytes,+ Hits: smCache.hits,+ Misses: smCache.misses,+ Evictions: smCache.evictions,+ Failures: smCache.failures,+ NotFound: smCache.notFound,+ NegativeHits: smCache.negativeHits,+ NegativeEntries: len(smCache.negative),+ LastParseMs: smCache.lastParseMs,}}var stackFrameRe = regexp.MustCompile(`^(\s{4})(.+):(\d+):(\d+)$`)-var jsFuncDeclRe = regexp.MustCompile(- `(?:(?:export\s+(?:default\s+)?)?function\s+(\w+)` +- `|(?:const|let|var)\s+(\w+)\s*=` +- `|^\s*(?:async\s+)?(\w+)\s*\([^)]*\)\s*\{)`,-)--var jsControlFlowKeywords = map[string]bool{- "if": true, "for": true, "while": true, "switch": true,- "catch": true, "return": true, "throw": true, "else": true,-}--func ResolveStackTrace(ctx context.Context, projectId uuid.UUID, stackTrace string, sourceMaps []*models.SourceMap) string {- if len(sourceMaps) == 0 {- return stackTrace- }- smByBasename := make(map[string]*models.SourceMap)- for _, sm := range sourceMaps {- smByBasename[sm.FileName] = sm- base := filepath.Base(sm.FileName)- smByBasename[base] = sm- }+func ResolveStackTrace(ctx context.Context, projectId uuid.UUID, stackTrace string) string {+ prefix := fmt.Sprintf("sourcemaps/%s/", projectId)lines := strings.Split(stackTrace, "\n")resolved := make([]string, 0, len(lines))framesResolved := 0maxFrames := 50- localMaps := make(map[string]*parsedSourceMap)+ localResolvers := make(map[string]*symbolicator.Resolver)for _, line := range lines {if framesResolved >= maxFrames {@@ -242,43 +300,45 @@ func ResolveStackTrace(ctx context.Context, projectId uuid.UUID, stackTrace strilineNum, _ := strconv.Atoi(matches[3])colNum, _ := strconv.Atoi(matches[4])- sm := findSourceMap(fileName, smByBasename)- if sm == nil {+ base := mapBasename(fileName)+ if base == "" {resolved = append(resolved, line)continue}+ mapKey := prefix + base + ".map"+ bundleKey := prefix + base- pm, err := getSourceMap(ctx, sm.StorageKey, localMaps)- if err != nil || pm == nil {+ if smCache.isNegative(mapKey) {resolved = append(resolved, line)continue}- origFile, origName, origLine, origCol, ok := pm.source(lineNum, colNum-1)- if !ok {+ resolver, err := getResolver(ctx, mapKey, buildResolver(mapKey, bundleKey), localResolvers)+ if err != nil || resolver == nil {resolved = append(resolved, line)continue}- if content := pm.sourceContent(origFile); content != "" {- if extracted := extractFunctionName(content, origLine); extracted != "" {- origName = extracted- }+ frame, ok := resolver.Lookup(uint32(lineNum-1), uint32(colNum-1))+ if !ok {+ resolved = append(resolved, line)+ continue}- if origFile == "" {- origFile = "<unknown>"+ file := frame.File+ if file == "" {+ file = "<unknown>"}- resolved = append(resolved, fmt.Sprintf("%s%s:%d:%d", indent, origFile, origLine, origCol+1))+ resolved = append(resolved, fmt.Sprintf("%s%s:%d:%d", indent, file, frame.Line, frame.Col))framesResolved++- if origName != "" && len(resolved) >= 2 {+ if frame.Fn != "" && len(resolved) >= 2 {prev := resolved[len(resolved)-2]if strings.HasSuffix(strings.TrimSpace(prev), "()") {trimmed := strings.TrimSpace(prev)- indent := prev[:len(prev)-len(trimmed)]- resolved[len(resolved)-2] = indent + origName + "()"+ prevIndent := prev[:len(prev)-len(trimmed)]+ resolved[len(resolved)-2] = prevIndent + frame.Fn + "()"}}}@@ -286,53 +346,53 @@ func ResolveStackTrace(ctx context.Context, projectId uuid.UUID, stackTrace strireturn strings.Join(resolved, "\n")}-func findSourceMap(stackFile string, smByBasename map[string]*models.SourceMap) *models.SourceMap {- mapName := stackFile + ".map"- if sm, ok := smByBasename[mapName]; ok {- return sm- }-- base := filepath.Base(stackFile) + ".map"- if sm, ok := smByBasename[base]; ok {- return sm+func getResolver(ctx context.Context, cacheKey string, build resolverBuild, local map[string]*symbolicator.Resolver) (*symbolicator.Resolver, error) {+ if r, ok := local[cacheKey]; ok {+ return r, nil}-- cleanName := stackFile- if idx := strings.IndexAny(cleanName, "?#"); idx != -1 {- cleanName = cleanName[:idx]- }- mapName = filepath.Base(cleanName) + ".map"- if sm, ok := smByBasename[mapName]; ok {- return sm+ r, err := smCache.getOrBuild(ctx, cacheKey, build)+ if err != nil {+ local[cacheKey] = nil+ return nil, err}-- return nil+ local[cacheKey] = r+ return r, nil}-func getSourceMap(ctx context.Context, storageKey string, localMaps map[string]*parsedSourceMap) (*parsedSourceMap, error) {- if m, ok := localMaps[storageKey]; ok {- return m, nil+func mapBasename(stackFile string) string {+ clean := stackFile+ if idx := strings.IndexAny(clean, "?#"); idx != -1 {+ clean = clean[:idx]}- m, err := smCache.getOrLoad(ctx, storageKey)- if err != nil {- localMaps[storageKey] = nil- return nil, err- }- localMaps[storageKey] = m- return m, nil+ return filepath.Base(clean)}-func extractFunctionName(sourceContent string, line int) string {- lines := strings.Split(sourceContent, "\n")- for i := line - 1; i >= 0 && i >= line-50; i-- {- matches := jsFuncDeclRe.FindStringSubmatch(lines[i])- if matches != nil {- for _, m := range matches[1:] {- if m != "" && !jsControlFlowKeywords[m] {- return m- }- }+func buildResolver(mapKey, bundleKey string) resolverBuild {+ return func(ctx context.Context) (*symbolicator.Resolver, int64, error) {+ base := context.WithoutCancel(ctx)++ mapBytes, err := readWithTimeout(base, mapKey)+ if err != nil {+ return nil, 0, err}++ var bundleBytes []byte+ if b, readErr := readWithTimeout(base, bundleKey); readErr == nil {+ bundleBytes = b+ } else if !errors.Is(readErr, storage.ErrNotFound) {+ return nil, 0, fmt.Errorf("failed to read bundle (key=%s): %w", bundleKey, readErr)+ }++ resolver, err := symbolicator.NewResolver(mapBytes, bundleBytes)+ if err != nil {+ return nil, 0, err+ }+ return resolver, resolver.ApproxSize(), nil}- return ""+}++func readWithTimeout(ctx context.Context, key string) ([]byte, error) {+ readCtx, cancel := context.WithTimeout(ctx, sourceMapLoadTimeout)+ defer cancel()+ return storage.Store.Read(readCtx, key)}diff --git a/backend/app/services/sourcemap_resolver_symbolic_test.go b/backend/app/services/sourcemap_resolver_symbolic_test.godeleted file mode 100644index 07a41133..00000000--- a/backend/app/services/sourcemap_resolver_symbolic_test.go+++ /dev/null@@ -1,267 +0,0 @@-package services--import (- "context"- "flag"- "fmt"- "os"- "path/filepath"- "strings"- "testing"-- "github.com/tracewayapp/traceway/backend/app/models"- "github.com/tracewayapp/traceway/backend/app/storage"-- "github.com/google/uuid"-)--var recordSymbolic = flag.Bool("record-symbolic", false, "write testdata/symbolic_parity_results.txt")--type symbolicFrame struct {- inFile string- inLine int- inCol int- file string- line int- col int- name string- pinnedName string- unresolved bool-}--type symbolicCase struct {- name string- maps []string- frames []symbolicFrame-}--type symbolicResult struct {- caseName string- input string- expectedLocation string- actualLocation string- locationPass bool- expectedName string- parityName string- actualName string- namePass bool- nameChecked bool- pinned bool-}--var symbolicCases = []symbolicCase{- {- name: "resolves_inlined_function",- maps: []string{"sourcemapcache/inlining/module.js.map"},- frames: []symbolicFrame{- {inFile: "module.js", inLine: 1, inCol: 63, file: "../src/app.js", line: 3, col: 30, name: "buttonCallback"},- {inFile: "module.js", inLine: 1, inCol: 47, file: "../src/bar.js", line: 4, col: 3, name: "bar"},- {inFile: "module.js", inLine: 1, inCol: 34, file: "../src/foo.js", line: 2, col: 9},- {inFile: "module.js", inLine: 2, inCol: 18, unresolved: true},- },- },- {- name: "writes_simple_cache",- maps: []string{"sourcemapcache/simple/minified.js.map"},- frames: []symbolicFrame{- {inFile: "minified.js", inLine: 1, inCol: 11, file: "tests/fixtures/simple/original.js", line: 2, col: 10, name: "abcd"},- },- },- {- name: "resolves_location_from_cache",- maps: []string{"sourcemapcache/preact.module.js.map"},- frames: []symbolicFrame{- {inFile: "preact.module.js", inLine: 1, inCol: 50, file: "../src/constants.js", line: 3, col: 35},- {inFile: "preact.module.js", inLine: 1, inCol: 133, file: "../src/util.js", line: 12, col: 23, name: "assign"},- {inFile: "preact.module.js", inLine: 1, inCol: 482, file: "../src/create-element.js", line: 40, col: 9, name: "createElement", pinnedName: "normalizedProps"},- {inFile: "preact.module.js", inLine: 1, inCol: 9780, file: "../src/component.js", line: 182, col: 5},- {inFile: "preact.module.js", inLine: 1, inCol: 9795, file: "../src/create-context.js", line: 3, col: 12},- },- },- {- name: "missing_source_names",- maps: []string{"sourcemapcache/nofiles.js.map"},- frames: []symbolicFrame{- {inFile: "nofiles.js", inLine: 1, inCol: 39, file: "<unknown>", line: 3, col: 9, name: "add"},- },- },- {- name: "missing_source_contents",- maps: []string{"sourcemapcache/preact-missing-source-contents.module.js.map"},- frames: []symbolicFrame{- {inFile: "preact-missing-source-contents.module.js", inLine: 1, inCol: 133, file: "../src/util.js", line: 12, col: 23},- },- },- {- name: "hermes_scope_lookup",- maps: []string{"sourcemapcache/hermes-metro/react-native-hermes.map"},- frames: []symbolicFrame{- {inFile: "react-native-hermes", inLine: 1, inCol: 11940, file: "module.js", line: 2, col: 11, name: "foo"},- {inFile: "react-native-hermes", inLine: 1, inCol: 11858, file: "input.js", line: 3, col: 1, name: "<global>", pinnedName: "anonymous"},- },- },- {- name: "metro_scope_lookup",- maps: []string{"sourcemapcache/hermes-metro/react-native-metro.js.map"},- frames: []symbolicFrame{- {inFile: "react-native-metro.js", inLine: 7, inCol: 101, file: "module.js", line: 2, col: 11, name: "foo"},- {inFile: "react-native-metro.js", inLine: 6, inCol: 44, file: "input.js", line: 3, col: 1, name: "<global>", pinnedName: "foo"},- },- },- {- name: "webpack_scope_lookup",- maps: []string{"sourcemapcache/webpack/bundle.js.map"},- frames: []symbolicFrame{- {inFile: "bundle.js", inLine: 1, inCol: 85, file: "webpack:///./foo.js", line: 2, col: 9, name: "module.exports", pinnedName: "foo"},- {inFile: "bundle.js", inLine: 1, inCol: 44, file: "webpack:///./bar.js", line: 2, col: 3, name: "module.exports", pinnedName: "bar"},- },- },-}--func TestResolveStackTraceSymbolicParity(t *testing.T) {- InitSourceMapCache(100, 64<<20)- store, err := storage.NewLocalStorage("testdata")- if err != nil {- t.Fatal(err)- }- storage.Store = store- projectId := uuid.New()-- var results []symbolicResult-- for _, tc := range symbolicCases {- t.Run(tc.name, func(t *testing.T) {- sourceMaps := make([]*models.SourceMap, 0, len(tc.maps))- for _, mapPath := range tc.maps {- sourceMaps = append(sourceMaps, &models.SourceMap{- ProjectId: projectId,- Version: "1.0.0",- FileName: filepath.Base(mapPath),- StorageKey: mapPath,- })- }-- lines := []string{"Error: symbolic fixture test"}- for _, f := range tc.frames {- lines = append(lines, "anonymous()")- lines = append(lines, fmt.Sprintf(" %s:%d:%d", f.inFile, f.inLine, f.inCol))- }- input := strings.Join(lines, "\n")-- resolved := ResolveStackTrace(context.Background(), projectId, input, sourceMaps)- outLines := strings.Split(resolved, "\n")- if len(outLines) != len(lines) {- t.Fatalf("expected %d output lines, got %d:\n%s", len(lines), len(outLines), resolved)- }-- for i, f := range tc.frames {- nameLine := outLines[1+2*i]- locationLine := outLines[2+2*i]- inputFrame := fmt.Sprintf("%s:%d:%d", f.inFile, f.inLine, f.inCol)-- var expectedLocation string- var expectedName string- parityName := ""- pinned := false- nameChecked := true- if f.unresolved {- expectedLocation = inputFrame- expectedName = "anonymous"- } else {- expectedLocation = fmt.Sprintf("%s:%d:%d", f.file, f.line, f.col)- expectedName = f.name- parityName = f.name- if f.pinnedName != "" {- expectedName = f.pinnedName- pinned = true- }- nameChecked = expectedName != ""- }-- actualLocation := strings.TrimPrefix(locationLine, " ")- actualName := strings.TrimSuffix(strings.TrimPrefix(nameLine, " "), "()")-- locationPass := actualLocation == expectedLocation- namePass := actualName == expectedName-- if !locationPass {- t.Errorf("frame %s: location expected %q, got %q", inputFrame, expectedLocation, actualLocation)- }- if nameChecked && !namePass {- if pinned {- t.Errorf("frame %s: name expected %q (pinned deviation, parity target %q), got %q", inputFrame, expectedName, parityName, actualName)- } else {- t.Errorf("frame %s: name expected %q, got %q", inputFrame, expectedName, actualName)- }- }-- results = append(results, symbolicResult{- caseName: tc.name,- input: inputFrame,- expectedLocation: expectedLocation,- actualLocation: actualLocation,- locationPass: locationPass,- expectedName: expectedName,- parityName: parityName,- actualName: actualName,- namePass: namePass,- nameChecked: nameChecked,- pinned: pinned,- })- }- })- }-- if *recordSymbolic {- if err := writeSymbolicResults(results); err != nil {- t.Fatalf("failed to write results file: %v", err)- }- }-}--func writeSymbolicResults(results []symbolicResult) error {- var b strings.Builder- parity := 0- pinnedCount := 0- failed := 0- for _, r := range results {- switch {- case !r.locationPass || (r.nameChecked && !r.namePass):- failed++- case r.pinned:- pinnedCount++- default:- parity++- }- }-- b.WriteString("symbolic parity results\n")- b.WriteString("fixtures: getsentry/symbolic 925230e878ea25f6a0af88171fce93b6272cd30d\n")- b.WriteString("regenerate: go test ./app/services/ -run TestResolveStackTraceSymbolicParity -record-symbolic\n")- b.WriteString("PINNED frames intentionally deviate from symbolic; see sourcemapcache/README.md\n")- fmt.Fprintf(&b, "frames: %d parity, %d pinned deviations, %d failing, %d total\n\n", parity, pinnedCount, failed, len(results))-- for _, r := range results {- status := "PASS"- if !r.locationPass || (r.nameChecked && !r.namePass) {- status = "FAIL"- } else if r.pinned {- status = "PINNED"- }- fmt.Fprintf(&b, "[%s] %s :: %s\n", status, r.caseName, r.input)- fmt.Fprintf(&b, " location expected: %s\n", r.expectedLocation)- fmt.Fprintf(&b, " location actual: %s\n", r.actualLocation)- if r.pinned {- fmt.Fprintf(&b, " name parity target: %s\n", r.parityName)- fmt.Fprintf(&b, " name pinned: %s\n", r.expectedName)- } else if r.nameChecked {- fmt.Fprintf(&b, " name expected: %s\n", r.expectedName)- } else {- b.WriteString(" name expected: (not asserted)\n")- }- fmt.Fprintf(&b, " name actual: %s\n", r.actualName)- b.WriteString("\n")- }-- return os.WriteFile("testdata/symbolic_parity_results.txt", []byte(b.String()), 0644)-}diff --git a/backend/app/services/sourcemap_resolver_test.go b/backend/app/services/sourcemap_resolver_test.goindex f91d41f2..37095bf4 100644--- a/backend/app/services/sourcemap_resolver_test.go+++ b/backend/app/services/sourcemap_resolver_test.go@@ -1,129 +1,86 @@package services-import "testing"--func TestExtractFunctionName(t *testing.T) {- tests := []struct {- name string- content string- line int- expected string- }{- {- name: "arrow function above throw",- content: `import React from 'react';-const handleEventError = () => {- try {- log("Triggering error...")- throw new Error("Test error")- } catch (e) {- throw e- }-}`,- line: 5,- expected: "handleEventError",- },- {- name: "regular function declaration",- content: `function processData(input) {- transform(input)- return input-}`,- line: 3,- expected: "processData",- },- {- name: "export function",- content: `export function handleError(err) {- console.error(err)- reportError(err)-}`,- line: 3,- expected: "handleError",- },- {- name: "export default function",- content: `export default function App() {- return <div>Hello</div>-}`,- line: 2,- expected: "App",- },- {- name: "class method",- content: `class MyComponent {- handleClick(event) {- event.preventDefault()- this.setState({ clicked: true })- }-}`,- line: 4,- expected: "handleClick",- },- {- name: "async method",- content: `class ApiClient {- async fetchData(url) {- await fetch(url)- return null- }-}`,- line: 4,- expected: "fetchData",- },- {- name: "skips function calls",- content: `const fn = () => {- log("hello")- doSomething()- throw new Error("fail")-}`,- line: 4,- expected: "fn",- },- {- name: "skips control flow keywords",- content: `function foo() {- if (x) {- doSomething()- }-}`,- line: 3,- expected: "foo",- },- {- name: "returns empty when no function found",- content: `x = 1-y = 2-z = x + y`,- line: 3,- expected: "",- },- {- name: "var declaration with function expression",- content: `var onSubmit = function(e) {- e.preventDefault()- submit(data)-}`,- line: 3,- expected: "onSubmit",- },- {- name: "let declaration with arrow",- content: `let compute = (a, b) => {- return a + b-}`,- line: 2,- expected: "compute",- },+import (+ "context"+ "fmt"+ "os"+ "strings"+ "testing"++ "github.com/tracewayapp/traceway/backend/app/storage"++ "github.com/google/uuid"+)++func seedFixture(t *testing.T, cs *countingStorage, key, fixturePath string) {+ t.Helper()+ data, err := os.ReadFile(fixturePath)+ if err != nil {+ t.Fatal(err)}+ cs.data[key] = data+}++func TestResolveStackTraceWithBundle(t *testing.T) {+ InitSourceMapCache(100, 64<<20)+ prev := storage.Store+ defer func() { storage.Store = prev }()+ cs := &countingStorage{reads: map[string]int{}, data: map[string][]byte{}}+ storage.Store = cs++ projectId := uuid.New()+ prefix := fmt.Sprintf("sourcemaps/%s/", projectId)+ seedFixture(t, cs, prefix+"minified.js.map", "testdata/sourcemapcache/simple/minified.js.map")+ seedFixture(t, cs, prefix+"minified.js", "testdata/sourcemapcache/simple/minified.js")++ input := "Error: boom\nanonymous()\n minified.js:1:11"+ lines := strings.Split(ResolveStackTrace(context.Background(), projectId, input), "\n")++ if got, want := lines[2], " tests/fixtures/simple/original.js:2:10"; got != want {+ t.Errorf("location: got %q, want %q", got, want)+ }+ if got, want := lines[1], "abcd()"; got != want {+ t.Errorf("function name (from bundle): got %q, want %q", got, want)+ }+}++func TestResolveStackTraceLocationOnlyWithoutBundle(t *testing.T) {+ InitSourceMapCache(100, 64<<20)+ prev := storage.Store+ defer func() { storage.Store = prev }()+ cs := &countingStorage{reads: map[string]int{}, data: map[string][]byte{}}+ storage.Store = cs++ projectId := uuid.New()+ prefix := fmt.Sprintf("sourcemaps/%s/", projectId)+ seedFixture(t, cs, prefix+"preact-missing-source-contents.module.js.map", "testdata/sourcemapcache/preact-missing-source-contents.module.js.map")++ input := "Error: boom\nanonymous()\n preact-missing-source-contents.module.js:1:133"+ lines := strings.Split(ResolveStackTrace(context.Background(), projectId, input), "\n")++ if got, want := lines[2], " ../src/util.js:12:23"; got != want {+ t.Errorf("location: got %q, want %q", got, want)+ }+ if got, want := lines[1], "anonymous()"; got != want {+ t.Errorf("function name should stay unresolved without a bundle: got %q, want %q", got, want)+ }+}++func TestResolveStackTraceNegativeCacheAvoidsRepeatReads(t *testing.T) {+ InitSourceMapCache(100, 64<<20)+ prev := storage.Store+ defer func() { storage.Store = prev }()+ cs := &countingStorage{reads: map[string]int{}, data: map[string][]byte{}}+ storage.Store = cs++ projectId := uuid.New()+ input := "Error: boom\n foo()\n missing.js:1:5"++ _ = ResolveStackTrace(context.Background(), projectId, input)+ _ = ResolveStackTrace(context.Background(), projectId, input)- for _, tt := range tests {- t.Run(tt.name, func(t *testing.T) {- got := extractFunctionName(tt.content, tt.line)- if got != tt.expected {- t.Errorf("extractFunctionName() = %q, want %q", got, tt.expected)- }- })+ mapKey := fmt.Sprintf("sourcemaps/%s/missing.js.map", projectId)+ if got := cs.reads[mapKey]; got != 1 {+ t.Errorf("negative cache should prevent repeat reads, got %d reads", got)}}diff --git a/backend/app/services/testdata/symbolic_parity_results.txt b/backend/app/services/testdata/symbolic_parity_results.txtdeleted file mode 100644index b15aeb3f..00000000--- a/backend/app/services/testdata/symbolic_parity_results.txt+++ /dev/null@@ -1,119 +0,0 @@-symbolic parity results-fixtures: getsentry/symbolic 925230e878ea25f6a0af88171fce93b6272cd30d-regenerate: go test ./app/services/ -run TestResolveStackTraceSymbolicParity -record-symbolic-PINNED frames intentionally deviate from symbolic; see sourcemapcache/README.md-frames: 13 parity, 5 pinned deviations, 0 failing, 18 total--[PASS] resolves_inlined_function :: module.js:1:63- location expected: ../src/app.js:3:30- location actual: ../src/app.js:3:30- name expected: buttonCallback- name actual: buttonCallback--[PASS] resolves_inlined_function :: module.js:1:47- location expected: ../src/bar.js:4:3- location actual: ../src/bar.js:4:3- name expected: bar- name actual: bar--[PASS] resolves_inlined_function :: module.js:1:34- location expected: ../src/foo.js:2:9- location actual: ../src/foo.js:2:9- name expected: (not asserted)- name actual: foo--[PASS] resolves_inlined_function :: module.js:2:18- location expected: module.js:2:18- location actual: module.js:2:18- name expected: anonymous- name actual: anonymous--[PASS] writes_simple_cache :: minified.js:1:11- location expected: tests/fixtures/simple/original.js:2:10- location actual: tests/fixtures/simple/original.js:2:10- name expected: abcd- name actual: abcd--[PASS] resolves_location_from_cache :: preact.module.js:1:50- location expected: ../src/constants.js:3:35- location actual: ../src/constants.js:3:35- name expected: (not asserted)- name actual: IS_NON_DIMENSIONAL--[PASS] resolves_location_from_cache :: preact.module.js:1:133- location expected: ../src/util.js:12:23- location actual: ../src/util.js:12:23- name expected: assign- name actual: assign--[PINNED] resolves_location_from_cache :: preact.module.js:1:482- location expected: ../src/create-element.js:40:9- location actual: ../src/create-element.js:40:9- name parity target: createElement- name pinned: normalizedProps- name actual: normalizedProps--[PASS] resolves_location_from_cache :: preact.module.js:1:9780- location expected: ../src/component.js:182:5- location actual: ../src/component.js:182:5- name expected: (not asserted)- name actual: defer--[PASS] resolves_location_from_cache :: preact.module.js:1:9795- location expected: ../src/create-context.js:3:12- location actual: ../src/create-context.js:3:12- name expected: (not asserted)- name actual: i--[PASS] missing_source_names :: nofiles.js:1:39- location expected: <unknown>:3:9- location actual: <unknown>:3:9- name expected: add- name actual: add--[PASS] missing_source_contents :: preact-missing-source-contents.module.js:1:133- location expected: ../src/util.js:12:23- location actual: ../src/util.js:12:23- name expected: (not asserted)- name actual: obj--[PASS] hermes_scope_lookup :: react-native-hermes:1:11940- location expected: module.js:2:11- location actual: module.js:2:11- name expected: foo- name actual: foo--[PINNED] hermes_scope_lookup :: react-native-hermes:1:11858- location expected: input.js:3:1- location actual: input.js:3:1- name parity target: <global>- name pinned: anonymous- name actual: anonymous--[PASS] metro_scope_lookup :: react-native-metro.js:7:101- location expected: module.js:2:11- location actual: module.js:2:11- name expected: foo- name actual: foo--[PINNED] metro_scope_lookup :: react-native-metro.js:6:44- location expected: input.js:3:1- location actual: input.js:3:1- name parity target: <global>- name pinned: foo- name actual: foo--[PINNED] webpack_scope_lookup :: bundle.js:1:85- location expected: webpack:///./foo.js:2:9- location actual: webpack:///./foo.js:2:9- name parity target: module.exports- name pinned: foo- name actual: foo--[PINNED] webpack_scope_lookup :: bundle.js:1:44- location expected: webpack:///./bar.js:2:3- location actual: webpack:///./bar.js:2:3- name parity target: module.exports- name pinned: bar- name actual: bar-diff --git a/backend/app/sourcemapbackfill/backfill.go b/backend/app/sourcemapbackfill/backfill.gonew file mode 100644index 00000000..17403097--- /dev/null+++ b/backend/app/sourcemapbackfill/backfill.go@@ -0,0 +1,130 @@+package sourcemapbackfill++import (+ "context"+ "errors"+ "fmt"+ "path/filepath"+ "time"++ "github.com/tracewayapp/lit/v2"+ "github.com/tracewayapp/traceway/backend/app/config"+ "github.com/tracewayapp/traceway/backend/app/db"+ "github.com/tracewayapp/traceway/backend/app/models"+ "github.com/tracewayapp/traceway/backend/app/services"+ "github.com/tracewayapp/traceway/backend/app/storage"++ "github.com/google/uuid"+ traceway "go.tracewayapp.com"+)++const storageOpTimeout = 15 * time.Second++func Start(ctx context.Context) {+ if db.DB == nil || storage.Store == nil {+ return+ }+ go func() {+ defer traceway.Recover()+ run(ctx)+ }()+}++func run(ctx context.Context) {+ projects, err := lit.SelectNamed[models.SourceMapProjectId](+ db.DB,+ "SELECT DISTINCT project_id FROM source_maps WHERE project_id NOT IN (SELECT project_id FROM source_map_flatten_migrations)",+ lit.P{},+ )+ if err != nil {+ traceway.CaptureException(fmt.Errorf("source map flatten: failed to list un-migrated projects: %w", err))+ return+ }+ if len(projects) == 0 {+ return+ }++ migrated := 0+ for _, p := range projects {+ if ctx.Err() != nil {+ return+ }+ if migrateProject(ctx, p.ProjectId) {+ migrated+++ }+ }+ if migrated > 0 {+ config.Logf("source map flatten: migrated %d project(s) to versionless layout", migrated)+ }+}++func migrateProject(ctx context.Context, projectId uuid.UUID) bool {+ rows, err := lit.SelectNamed[models.SourceMap](+ db.DB,+ `SELECT s.id, s.project_id, s.version, s.file_name, s.storage_key, s.file_size, s.uploaded_at FROM source_maps s+ JOIN (SELECT file_name, MAX(uploaded_at) AS mx FROM source_maps WHERE project_id = :pid GROUP BY file_name) g+ ON s.file_name = g.file_name AND s.uploaded_at = g.mx+ WHERE s.project_id = :pid`,+ lit.P{"pid": projectId},+ )+ if err != nil {+ traceway.CaptureException(fmt.Errorf("source map flatten: failed to read source maps for project %s: %w", projectId, err))+ return false+ }++ seen := make(map[string]bool, len(rows))+ for _, row := range rows {+ base := filepath.Base(row.FileName)+ if seen[base] {+ continue+ }+ seen[base] = true++ oldKey := row.StorageKey+ flatKey := fmt.Sprintf("sourcemaps/%s/%s", projectId, base)+ if oldKey == flatKey {+ continue+ }++ if !copyObject(ctx, projectId, oldKey, flatKey) {+ return false+ }+ services.InvalidateSourceMap(projectId, base)+ }++ flattenRow := models.SourceMapFlattenMigration{+ ProjectId: projectId,+ MigratedAt: time.Now().UTC(),+ }+ if _, err := lit.Insert(db.DB, &flattenRow); err != nil {+ traceway.CaptureException(fmt.Errorf("source map flatten: failed to record migration for project %s: %w", projectId, err))+ return false+ }+ return true+}++func copyObject(ctx context.Context, projectId uuid.UUID, oldKey, flatKey string) bool {+ opCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), storageOpTimeout)+ defer cancel()++ if _, err := storage.Store.Read(opCtx, flatKey); err == nil {+ return true+ } else if !errors.Is(err, storage.ErrNotFound) {+ traceway.CaptureException(fmt.Errorf("source map flatten: failed to check %s for project %s: %w", flatKey, projectId, err))+ return false+ }++ data, err := storage.Store.Read(opCtx, oldKey)+ if errors.Is(err, storage.ErrNotFound) {+ return true+ }+ if err != nil {+ traceway.CaptureException(fmt.Errorf("source map flatten: failed to read %s for project %s: %w", oldKey, projectId, err))+ return false+ }+ if err := storage.Store.Write(opCtx, flatKey, data); err != nil {+ traceway.CaptureException(fmt.Errorf("source map flatten: failed to write %s for project %s: %w", flatKey, projectId, err))+ return false+ }+ return true+}diff --git a/backend/app/sourcemapbackfill/backfill_test.go b/backend/app/sourcemapbackfill/backfill_test.gonew file mode 100644index 00000000..dbb2ca3f--- /dev/null+++ b/backend/app/sourcemapbackfill/backfill_test.go@@ -0,0 +1,187 @@+//go:build !pgch++package sourcemapbackfill++import (+ "context"+ "database/sql"+ "fmt"+ "sync"+ "testing"+ "time"++ "github.com/google/uuid"+ "github.com/tracewayapp/lit/v2"+ "github.com/tracewayapp/traceway/backend/app/db"+ "github.com/tracewayapp/traceway/backend/app/models"+ "github.com/tracewayapp/traceway/backend/app/storage"+ _ "modernc.org/sqlite"+)++type memStore struct {+ mu sync.Mutex+ data map[string][]byte+}++func (m *memStore) Write(_ context.Context, key string, data []byte) error {+ m.mu.Lock()+ m.data[key] = append([]byte(nil), data...)+ m.mu.Unlock()+ return nil+}++func (m *memStore) Read(_ context.Context, key string) ([]byte, error) {+ m.mu.Lock()+ defer m.mu.Unlock()+ d, ok := m.data[key]+ if !ok {+ return nil, storage.ErrNotFound+ }+ return append([]byte(nil), d...), nil+}++func (m *memStore) get(key string) (string, bool) {+ m.mu.Lock()+ defer m.mu.Unlock()+ d, ok := m.data[key]+ return string(d), ok+}++func setup(t *testing.T) *memStore {+ t.Helper()++ mainDB, err := sql.Open("sqlite", ":memory:")+ if err != nil {+ t.Fatalf("open main db: %v", err)+ }+ mainDB.SetMaxOpenConns(1)++ ddl := []string{+ `CREATE TABLE source_maps (id INTEGER PRIMARY KEY, project_id TEXT NOT NULL, version TEXT NOT NULL, file_name TEXT NOT NULL, storage_key TEXT NOT NULL, file_size INTEGER NOT NULL, uploaded_at DATETIME NOT NULL)`,+ `CREATE TABLE source_map_flatten_migrations (id INTEGER PRIMARY KEY, project_id TEXT NOT NULL UNIQUE, migrated_at DATETIME NOT NULL)`,+ }+ for _, stmt := range ddl {+ if _, err := mainDB.Exec(stmt); err != nil {+ t.Fatalf("ddl: %v", err)+ }+ }++ prevDB, prevDriver, prevStore := db.DB, db.Driver, storage.Store+ db.DB = mainDB+ db.Driver = lit.SQLite+ models.Init(db.Driver)++ ms := &memStore{data: map[string][]byte{}}+ storage.Store = ms++ t.Cleanup(func() {+ mainDB.Close()+ db.DB = prevDB+ db.Driver = prevDriver+ storage.Store = prevStore+ })+ return ms+}++func seedMap(t *testing.T, ms *memStore, projectId uuid.UUID, version, fileName string, body string, uploadedAt time.Time) {+ t.Helper()+ storageKey := fmt.Sprintf("sourcemaps/%s/%s/%s", projectId, version, fileName)+ sm := models.SourceMap{+ ProjectId: projectId,+ Version: version,+ FileName: fileName,+ StorageKey: storageKey,+ FileSize: int64(len(body)),+ UploadedAt: uploadedAt,+ }+ if _, err := lit.Insert(db.DB, &sm); err != nil {+ t.Fatalf("seed source_map: %v", err)+ }+ ms.data[storageKey] = []byte(body)+}++func countMigrations(t *testing.T) int {+ t.Helper()+ var n int+ if err := db.DB.QueryRow("SELECT count(*) FROM source_map_flatten_migrations").Scan(&n); err != nil {+ t.Fatalf("count migrations: %v", err)+ }+ return n+}++func TestFlattenCopiesNewestVersionToFlatKey(t *testing.T) {+ ms := setup(t)+ p := uuid.New()+ now := time.Now().UTC()+ seedMap(t, ms, p, "v1", "app.js.map", "OLD", now.Add(-2*time.Hour))+ seedMap(t, ms, p, "v2", "app.js.map", "NEW", now.Add(-1*time.Hour))++ run(context.Background())++ flatKey := fmt.Sprintf("sourcemaps/%s/app.js.map", p)+ got, ok := ms.get(flatKey)+ if !ok || got != "NEW" {+ t.Errorf("flat key = %q (present=%v), want newest content NEW", got, ok)+ }+ if n := countMigrations(t); n != 1 {+ t.Errorf("migration rows = %d, want 1", n)+ }+ if _, ok := ms.get(fmt.Sprintf("sourcemaps/%s/v1/app.js.map", p)); !ok {+ t.Error("old v1 object should be left in place (copy-only)")+ }+ if _, ok := ms.get(fmt.Sprintf("sourcemaps/%s/v2/app.js.map", p)); !ok {+ t.Error("old v2 object should be left in place (copy-only)")+ }+}++func TestFlattenIsIdempotentAcrossRuns(t *testing.T) {+ ms := setup(t)+ p := uuid.New()+ seedMap(t, ms, p, "v1", "app.js.map", "BODY", time.Now().UTC())++ run(context.Background())+ run(context.Background())++ if n := countMigrations(t); n != 1 {+ t.Errorf("migration rows after two runs = %d, want 1", n)+ }+}++func TestFlattenRetryDoesNotOverwriteFreshUploads(t *testing.T) {+ ms := setup(t)+ p := uuid.New()+ seedMap(t, ms, p, "v1", "app.js.map", "STALE", time.Now().UTC())++ flatKey := fmt.Sprintf("sourcemaps/%s/app.js.map", p)+ ms.data[flatKey] = []byte("FRESH_UPLOAD")++ run(context.Background())++ if got, _ := ms.get(flatKey); got != "FRESH_UPLOAD" {+ t.Errorf("flat key = %q, a backfill retry must not overwrite a newer upload", got)+ }+ if n := countMigrations(t); n != 1 {+ t.Errorf("migration rows = %d, want 1 (project should still be marked migrated)", n)+ }+}++func TestFlattenSkipsAlreadyMigratedProjects(t *testing.T) {+ ms := setup(t)+ p := uuid.New()+ seedMap(t, ms, p, "v1", "app.js.map", "FIRST", time.Now().UTC())++ run(context.Background())++ flatKey := fmt.Sprintf("sourcemaps/%s/app.js.map", p)+ ms.data[flatKey] = []byte("SECOND")+ seedMap(t, ms, p, "v2", "app.js.map", "FROM_OLD_ROW", time.Now().UTC().Add(time.Hour))++ run(context.Background())++ if got, _ := ms.get(flatKey); got != "SECOND" {+ t.Errorf("flat key = %q, want SECOND untouched", got)+ }+ if n := countMigrations(t); n != 1 {+ t.Errorf("migration rows = %d, want 1", n)+ }+}diff --git a/backend/app/storage/local.go b/backend/app/storage/local.goindex 604e3410..ee2e4cc0 100644--- a/backend/app/storage/local.go+++ b/backend/app/storage/local.go@@ -39,7 +39,7 @@ func (l *localStorage) Read(_ context.Context, key string) ([]byte, error) {data, err := os.ReadFile(fullPath)if err != nil {if os.IsNotExist(err) {- return nil, fmt.Errorf("file not found: %s", key)+ return nil, ErrNotFound}return nil, fmt.Errorf("failed to read file %s: %w", fullPath, err)}diff --git a/backend/app/storage/s3.go b/backend/app/storage/s3.goindex a424b82e..e3aba66c 100644--- a/backend/app/storage/s3.go+++ b/backend/app/storage/s3.go@@ -3,6 +3,7 @@ package storageimport ("bytes""context"+ "errors""fmt""io"@@ -10,6 +11,8 @@ import ("github.com/aws/aws-sdk-go-v2/config""github.com/aws/aws-sdk-go-v2/credentials""github.com/aws/aws-sdk-go-v2/service/s3"+ s3types "github.com/aws/aws-sdk-go-v2/service/s3/types"+ "github.com/aws/smithy-go")type s3Storage struct {@@ -61,6 +64,9 @@ func (s *s3Storage) Read(ctx context.Context, key string) ([]byte, error) {Key: aws.String(key),})if err != nil {+ if isS3NotFound(err) {+ return nil, ErrNotFound+ }return nil, fmt.Errorf("failed to get object %s: %w", key, err)}defer output.Body.Close()@@ -71,3 +77,18 @@ func (s *s3Storage) Read(ctx context.Context, key string) ([]byte, error) {}return data, nil}++func isS3NotFound(err error) bool {+ var nsk *s3types.NoSuchKey+ if errors.As(err, &nsk) {+ return true+ }+ var apiErr smithy.APIError+ if errors.As(err, &apiErr) {+ switch apiErr.ErrorCode() {+ case "NoSuchKey", "NotFound", "404":+ return true+ }+ }+ return false+}diff --git a/backend/app/storage/storage.go b/backend/app/storage/storage.goindex 09fbe013..918943c1 100644--- a/backend/app/storage/storage.go+++ b/backend/app/storage/storage.go@@ -1,11 +1,14 @@package storageimport (- "github.com/tracewayapp/traceway/backend/app/config""context"+ "errors""fmt"+ "github.com/tracewayapp/traceway/backend/app/config")+var ErrNotFound = errors.New("storage: object not found")+type Storage interface {Write(ctx context.Context, key string, data []byte) errorRead(ctx context.Context, key string) ([]byte, error)diff --git a/backend/app/symbolicator/bundle_parser_goja.go b/backend/app/symbolicator/bundle_parser_goja.gonew file mode 100644index 00000000..b1138972--- /dev/null+++ b/backend/app/symbolicator/bundle_parser_goja.go@@ -0,0 +1,529 @@+package symbolicator++import (+ "regexp"+ "slices"+ "unicode/utf16"++ "github.com/dop251/goja/ast"+ "github.com/dop251/goja/file"+ "github.com/dop251/goja/parser"+)++func parseFunctionScopes(bundle []byte) (*functionScopes, error) {+ src := string(bundle)+ prog, err := parser.ParseFile(nil, "", src, 0, parser.WithDisableSourceMaps)+ if err != nil {+ neutralized := neutralizeModuleSyntax(src)+ if neutralized == src {+ return nil, err+ }+ prog, err = parser.ParseFile(nil, "", neutralized, 0, parser.WithDisableSourceMaps)+ if err != nil {+ return nil, err+ }+ src = neutralized+ }++ c := &scopeCollector{}+ c.walk(prog)++ offsets := make([]uint32, 0, len(c.raw)*3)+ for _, s := range c.raw {+ offsets = append(offsets, s.start, s.end, s.namePos)+ }+ pos := convertOffsets(src, offsets)++ scopes := make([]genScope, len(c.raw))+ for i, s := range c.raw {+ start, end, name := pos[s.start], pos[s.end], pos[s.namePos]+ scopes[i] = genScope{+ startLine: start.line, startCol: start.col,+ endLine: end.line, endCol: end.col,+ nameLine: name.line, nameCol: name.col,+ }+ }+ return &functionScopes{transitions: buildTransitions(scopes)}, nil+}++type genScope struct {+ startLine, startCol uint32+ endLine, endCol uint32+ nameLine, nameCol uint32+}++type functionScopes struct {+ transitions []scopeTransition // sorted by generated position+}++// scopeTransition marks that, from this generated position onward, the+// innermost enclosing function's name token is at (nameLine, nameCol). has is+// false when no function encloses the range (global scope).+type scopeTransition struct {+ line, col uint32+ nameLine, nameCol uint32+ has bool+}++type scopeEvent struct {+ line, col uint32+ start bool+ scope int+}++// buildTransitions flattens the well-nested function scopes into a sorted list+// of transitions in a single sweep, so the resolver can find the enclosing+// function of any token with a linear merge instead of scanning every scope per+// token.+func buildTransitions(scopes []genScope) []scopeTransition {+ events := make([]scopeEvent, 0, len(scopes)*2)+ for i := range scopes {+ s := scopes[i]+ if !less(s.startLine, s.startCol, s.endLine, s.endCol) {+ continue+ }+ events = append(events,+ scopeEvent{line: s.startLine, col: s.startCol, start: true, scope: i},+ scopeEvent{line: s.endLine, col: s.endCol, start: false, scope: i},+ )+ }+ slices.SortFunc(events, func(a, b scopeEvent) int {+ if a.line != b.line {+ return int(a.line) - int(b.line)+ }+ if a.col != b.col {+ return int(a.col) - int(b.col)+ }+ if a.start == b.start {+ return 0+ }+ if !a.start { // a scope ending here closes before another opens+ return -1+ }+ return 1+ })++ var transitions []scopeTransition+ stack := make([]int, 0, 16)+ var lastNameLine, lastNameCol uint32+ lastHas, haveLast := false, false++ for i := 0; i < len(events); {+ line, col := events[i].line, events[i].col+ for i < len(events) && events[i].line == line && events[i].col == col {+ if events[i].start {+ stack = append(stack, events[i].scope)+ } else {+ stack = removeFromStack(stack, events[i].scope)+ }+ i+++ }++ var nameLine, nameCol uint32+ has := false+ if len(stack) > 0 {+ top := scopes[stack[len(stack)-1]]+ nameLine, nameCol, has = top.nameLine, top.nameCol, true+ }+ if !haveLast || has != lastHas || nameLine != lastNameLine || nameCol != lastNameCol {+ transitions = append(transitions, scopeTransition{line: line, col: col, nameLine: nameLine, nameCol: nameCol, has: has})+ lastNameLine, lastNameCol, lastHas, haveLast = nameLine, nameCol, has, true+ }+ }+ return transitions+}++func removeFromStack(stack []int, scope int) []int {+ for i := len(stack) - 1; i >= 0; i-- {+ if stack[i] == scope {+ return append(stack[:i], stack[i+1:]...)+ }+ }+ return stack+}++func less(aLine, aCol, bLine, bCol uint32) bool {+ return aLine < bLine || (aLine == bLine && aCol < bCol)+}++type genPos struct {+ line, col uint32+}++func convertOffsets(src string, offsets []uint32) map[uint32]genPos {+ sorted := append([]uint32(nil), offsets...)+ slices.Sort(sorted)++ out := make(map[uint32]genPos, len(sorted))+ var line, col uint32+ oi := 0+ for i, r := range src {+ for oi < len(sorted) && sorted[oi] == uint32(i) {+ out[sorted[oi]] = genPos{line, col}+ oi+++ }+ if oi >= len(sorted) {+ return out+ }+ if r == '\n' {+ line+++ col = 0+ } else {+ col += uint32(utf16.RuneLen(r))+ }+ }+ for oi < len(sorted) {+ out[sorted[oi]] = genPos{line, col}+ oi+++ }+ return out+}++type rawScope struct {+ start, end, namePos uint32+}++type scopeCollector struct {+ raw []rawScope+}++func off(idx file.Idx) uint32 {+ if idx <= 1 {+ return 0+ }+ return uint32(idx - 1)+}++func (c *scopeCollector) recordFunction(lit *ast.FunctionLiteral) {+ namePos := lit.Idx0()+ if lit.Name != nil {+ namePos = lit.Name.Idx+ }+ c.raw = append(c.raw, rawScope{start: off(lit.Idx0()), end: off(lit.Idx1()), namePos: off(namePos)})+}++func (c *scopeCollector) recordArrow(lit *ast.ArrowFunctionLiteral) {+ c.raw = append(c.raw, rawScope{start: off(lit.Idx0()), end: off(lit.Idx1()), namePos: off(lit.Idx0())})+}++func (c *scopeCollector) recordClass(lit *ast.ClassLiteral) {+ namePos := lit.Idx0()+ if lit.Name != nil {+ namePos = lit.Name.Idx+ }+ c.raw = append(c.raw, rawScope{start: off(lit.Idx0()), end: off(lit.Idx1()), namePos: off(namePos)})+}++func (c *scopeCollector) walk(n ast.Node) {+ if n == nil {+ return+ }+ switch t := n.(type) {+ case *ast.FunctionLiteral:+ c.recordFunction(t)+ case *ast.ArrowFunctionLiteral:+ c.recordArrow(t)+ case *ast.ClassLiteral:+ c.recordClass(t)+ }+ c.walkChildren(n)+}++func (c *scopeCollector) walkExprs(exprs []ast.Expression) {+ for _, e := range exprs {+ c.walkExpr(e)+ }+}++func (c *scopeCollector) walkStmts(stmts []ast.Statement) {+ for _, s := range stmts {+ c.walkStmt(s)+ }+}++func (c *scopeCollector) walkExpr(e ast.Expression) {+ if e != nil {+ c.walk(e)+ }+}++func (c *scopeCollector) walkStmt(s ast.Statement) {+ if s != nil {+ c.walk(s)+ }+}++func (c *scopeCollector) walkBindings(bindings []*ast.Binding) {+ for _, b := range bindings {+ if b != nil {+ c.walk(b)+ }+ }+}++func (c *scopeCollector) walkChildren(n ast.Node) {+ switch t := n.(type) {+ case *ast.Program:+ c.walkStmts(t.Body)++ case *ast.BlockStatement:+ c.walkStmts(t.List)+ case *ast.ExpressionStatement:+ c.walkExpr(t.Expression)+ case *ast.IfStatement:+ c.walkExpr(t.Test)+ c.walkStmt(t.Consequent)+ c.walkStmt(t.Alternate)+ case *ast.ForStatement:+ if t.Initializer != nil {+ c.walk(t.Initializer.(ast.Node))+ }+ c.walkExpr(t.Test)+ c.walkExpr(t.Update)+ c.walkStmt(t.Body)+ case *ast.ForInStatement:+ c.walkForInto(t.Into)+ c.walkExpr(t.Source)+ c.walkStmt(t.Body)+ case *ast.ForOfStatement:+ c.walkForInto(t.Into)+ c.walkExpr(t.Source)+ c.walkStmt(t.Body)+ case *ast.WhileStatement:+ c.walkExpr(t.Test)+ c.walkStmt(t.Body)+ case *ast.DoWhileStatement:+ c.walkStmt(t.Body)+ c.walkExpr(t.Test)+ case *ast.SwitchStatement:+ c.walkExpr(t.Discriminant)+ for _, cs := range t.Body {+ if cs != nil {+ c.walk(cs)+ }+ }+ case *ast.CaseStatement:+ c.walkExpr(t.Test)+ c.walkStmts(t.Consequent)+ case *ast.TryStatement:+ if t.Body != nil {+ c.walk(t.Body)+ }+ if t.Catch != nil {+ c.walk(t.Catch)+ }+ if t.Finally != nil {+ c.walk(t.Finally)+ }+ case *ast.CatchStatement:+ if t.Parameter != nil {+ c.walk(t.Parameter)+ }+ if t.Body != nil {+ c.walk(t.Body)+ }+ case *ast.ThrowStatement:+ c.walkExpr(t.Argument)+ case *ast.ReturnStatement:+ c.walkExpr(t.Argument)+ case *ast.LabelledStatement:+ c.walkStmt(t.Statement)+ case *ast.WithStatement:+ c.walkExpr(t.Object)+ c.walkStmt(t.Body)+ case *ast.VariableStatement:+ c.walkBindings(t.List)+ case *ast.LexicalDeclaration:+ c.walkBindings(t.List)+ case *ast.VariableDeclaration:+ c.walkBindings(t.List)+ case *ast.FunctionDeclaration:+ if t.Function != nil {+ c.walk(t.Function)+ }+ case *ast.ClassDeclaration:+ if t.Class != nil {+ c.walk(t.Class)+ }++ case *ast.ForLoopInitializerExpression:+ c.walkExpr(t.Expression)+ case *ast.ForLoopInitializerVarDeclList:+ c.walkBindings(t.List)+ case *ast.ForLoopInitializerLexicalDecl:+ c.walk(&t.LexicalDeclaration)+ case *ast.ForIntoVar:+ if t.Binding != nil {+ c.walk(t.Binding)+ }+ case *ast.ForIntoExpression:+ c.walkExpr(t.Expression)+ case *ast.ForDeclaration:+ if t.Target != nil {+ c.walk(t.Target)+ }++ case *ast.FunctionLiteral:+ if t.ParameterList != nil {+ c.walkBindings(t.ParameterList.List)+ c.walkExpr(t.ParameterList.Rest)+ }+ if t.Body != nil {+ c.walk(t.Body)+ }+ case *ast.ArrowFunctionLiteral:+ if t.ParameterList != nil {+ c.walkBindings(t.ParameterList.List)+ c.walkExpr(t.ParameterList.Rest)+ }+ if t.Body != nil {+ c.walk(t.Body)+ }+ case *ast.ExpressionBody:+ c.walkExpr(t.Expression)+ case *ast.ClassLiteral:+ c.walkExpr(t.SuperClass)+ for _, el := range t.Body {+ if el != nil {+ c.walk(el)+ }+ }+ case *ast.MethodDefinition:+ c.walkExpr(t.Key)+ if t.Body != nil {+ c.walk(t.Body)+ }+ case *ast.FieldDefinition:+ c.walkExpr(t.Key)+ c.walkExpr(t.Initializer)+ case *ast.ClassStaticBlock:+ if t.Block != nil {+ c.walk(t.Block)+ }+ case *ast.Binding:+ if t.Target != nil {+ c.walk(t.Target)+ }+ c.walkExpr(t.Initializer)+ case *ast.AssignExpression:+ c.walkExpr(t.Left)+ c.walkExpr(t.Right)+ case *ast.BinaryExpression:+ c.walkExpr(t.Left)+ c.walkExpr(t.Right)+ case *ast.ConditionalExpression:+ c.walkExpr(t.Test)+ c.walkExpr(t.Consequent)+ c.walkExpr(t.Alternate)+ case *ast.UnaryExpression:+ c.walkExpr(t.Operand)+ case *ast.CallExpression:+ c.walkExpr(t.Callee)+ c.walkExprs(t.ArgumentList)+ case *ast.NewExpression:+ c.walkExpr(t.Callee)+ c.walkExprs(t.ArgumentList)+ case *ast.DotExpression:+ c.walkExpr(t.Left)+ case *ast.PrivateDotExpression:+ c.walkExpr(t.Left)+ case *ast.BracketExpression:+ c.walkExpr(t.Left)+ c.walkExpr(t.Member)+ case *ast.SequenceExpression:+ c.walkExprs(t.Sequence)+ case *ast.ArrayLiteral:+ c.walkExprs(t.Value)+ case *ast.ArrayPattern:+ c.walkExprs(t.Elements)+ c.walkExpr(t.Rest)+ case *ast.ObjectLiteral:+ for _, p := range t.Value {+ if p != nil {+ c.walk(p)+ }+ }+ case *ast.ObjectPattern:+ for _, p := range t.Properties {+ if p != nil {+ c.walk(p)+ }+ }+ c.walkExpr(t.Rest)+ case *ast.PropertyKeyed:+ c.walkExpr(t.Key)+ c.walkExpr(t.Value)+ case *ast.PropertyShort:+ c.walkExpr(t.Initializer)+ case *ast.SpreadElement:+ c.walkExpr(t.Expression)+ case *ast.TemplateLiteral:+ c.walkExpr(t.Tag)+ c.walkExprs(t.Expressions)+ case *ast.YieldExpression:+ c.walkExpr(t.Argument)+ case *ast.AwaitExpression:+ c.walkExpr(t.Argument)+ case *ast.OptionalChain:+ c.walkExpr(t.Expression)+ case *ast.Optional:+ c.walkExpr(t.Expression)+ }+}++func (c *scopeCollector) walkForInto(into ast.ForInto) {+ if into != nil {+ c.walk(into)+ }+}++var moduleStmtPatterns = []*regexp.Regexp{+ regexp.MustCompile(`export\s*\{[^}]*\}\s*(?:from\s*("[^"]*"|'[^']*'))?\s*;?`),+ regexp.MustCompile(`export\s+default\s`),+ regexp.MustCompile(`export\s+(const|let|var|function|class|async)\b`),+ regexp.MustCompile(`import\s*(?:[\w$*{},\s]+?from\s*)?("[^"]*"|'[^']*')\s*;?`),+}++func neutralizeModuleSyntax(src string) string {+ buf := []byte(src)+ changed := false+ for _, re := range moduleStmtPatterns {+ for _, m := range re.FindAllIndex(buf, -1) {+ if !atStatementPosition(buf, m[0]) {+ continue+ }+ keep := 0+ if re == moduleStmtPatterns[2] {+ sub := re.FindSubmatchIndex(buf[m[0]:m[1]])+ if len(sub) >= 4 && sub[2] >= 0 {+ keep = m[1] - m[0] - sub[2]+ }+ }+ for i := m[0]; i < m[1]-keep; i++ {+ if buf[i] != '\n' && buf[i] != '\r' {+ buf[i] = ' '+ }+ }+ changed = true+ }+ }+ if !changed {+ return src+ }+ return string(buf)+}++func atStatementPosition(buf []byte, offset int) bool {+ i := offset - 1+ for i >= 0 && (buf[i] == ' ' || buf[i] == '\t') {+ i--+ }+ if i < 0 {+ return true+ }+ switch buf[i] {+ case ';', '}', '\n', '\r':+ return true+ }+ return false+}diff --git a/backend/app/symbolicator/sourcemap_parser/sourcemap.go b/backend/app/symbolicator/sourcemap_parser/sourcemap.gonew file mode 100644index 00000000..e843255f--- /dev/null+++ b/backend/app/symbolicator/sourcemap_parser/sourcemap.go@@ -0,0 +1,203 @@+package sourcemap_parser++import (+ "encoding/json"+ "fmt"+ "sort"+ "strings"+)++const NoID int32 = -1+const unmapped uint32 = ^uint32(0)++type Token struct {+ GenLine, GenCol uint32+ SrcID int32+ SrcLine, SrcCol uint32+ NameID int32+}++type ParsedMap struct {+ Sources []string+ Names []string+ Tokens []Token // sorted by (GenLine, GenCol)+}++type rawMap struct {+ Version int `json:"version"`+ SourceRoot string `json:"sourceRoot"`+ Sources []*string `json:"sources"`+ Names []json.RawMessage `json:"names"`+ Mappings string `json:"mappings"`+ Sections []rawSection `json:"sections"`+}++type rawSection struct {+ Offset struct {+ Line uint32 `json:"line"`+ Column uint32 `json:"column"`+ } `json:"offset"`+ Map *json.RawMessage `json:"map"`+}++func Parse(data []byte) (*ParsedMap, error) {+ var raw rawMap+ if err := json.Unmarshal(data, &raw); err != nil {+ return nil, fmt.Errorf("sourcemap: invalid JSON: %w", err)+ }+ if raw.Sections != nil {+ return parseIndexed(&raw)+ }+ return parseRegular(&raw)+}++func parseRegular(raw *rawMap) (*ParsedMap, error) {+ m := &ParsedMap{+ Sources: make([]string, len(raw.Sources)),+ Names: make([]string, 0, len(raw.Names)),+ }+ for i, s := range raw.Sources {+ if s == nil {+ continue+ }+ m.Sources[i] = joinSourceRoot(raw.SourceRoot, *s)+ }+ for _, n := range raw.Names {+ m.Names = append(m.Names, decodeName(n))+ }+ if err := m.decodeMappings(raw.Mappings); err != nil {+ return nil, err+ }+ sortTokens(m.Tokens)+ return m, nil+}++func parseIndexed(raw *rawMap) (*ParsedMap, error) {+ out := &ParsedMap{}+ for i, section := range raw.Sections {+ if section.Map == nil {+ return nil, fmt.Errorf("sourcemap: section %d has no embedded map", i)+ }+ inner, err := Parse(*section.Map)+ if err != nil {+ return nil, fmt.Errorf("sourcemap: section %d: %w", i, err)+ }+ srcOffset := int32(len(out.Sources))+ nameOffset := int32(len(out.Names))+ out.Sources = append(out.Sources, inner.Sources...)+ out.Names = append(out.Names, inner.Names...)+ for _, t := range inner.Tokens {+ if t.GenLine == 0 {+ t.GenCol += section.Offset.Column+ }+ t.GenLine += section.Offset.Line+ if t.SrcID != NoID {+ t.SrcID += srcOffset+ }+ if t.NameID != NoID {+ t.NameID += nameOffset+ }+ out.Tokens = append(out.Tokens, t)+ }+ }+ sortTokens(out.Tokens)+ return out, nil+}++func (m *ParsedMap) decodeMappings(mappings string) error {+ var genLine uint32+ var genCol, srcID, srcLine, srcCol, nameID int64+ nums := make([]int64, 0, 5)++ for _, line := range strings.Split(mappings, ";") {+ genCol = 0+ for _, seg := range strings.Split(line, ",") {+ if seg == "" {+ continue+ }+ nums = nums[:0]+ var err error+ nums, err = decodeVLQ(seg, nums)+ if err != nil {+ return err+ }+ genCol += nums[0]+ t := Token{+ GenLine: genLine,+ GenCol: uint32(genCol),+ SrcID: NoID,+ SrcLine: unmapped,+ SrcCol: unmapped,+ NameID: NoID,+ }+ if len(nums) >= 4 {+ srcID += nums[1]+ srcLine += nums[2]+ srcCol += nums[3]+ t.SrcID = int32(srcID)+ t.SrcLine = uint32(srcLine)+ t.SrcCol = uint32(srcCol)+ if int(srcID) >= len(m.Sources) || srcID < 0 {+ t.SrcID = NoID+ }+ }+ if len(nums) >= 5 {+ nameID += nums[4]+ if nameID >= 0 && int(nameID) < len(m.Names) {+ t.NameID = int32(nameID)+ }+ }+ m.Tokens = append(m.Tokens, t)+ }+ genLine+++ }+ return nil+}++// FloorToken returns the token at the greatest (GenLine, GenCol) <= (line, col),+// walking back to the first of several tokens that share that position+// (rust-sourcemap greatest_lower_bound). Returns nil before the first token.+func (m *ParsedMap) FloorToken(line, col uint32) *Token {+ idx := sort.Search(len(m.Tokens), func(i int) bool {+ t := m.Tokens[i]+ return t.GenLine > line || (t.GenLine == line && t.GenCol > col)+ })+ if idx == 0 {+ return nil+ }+ idx--+ for idx > 0 {+ a, b := m.Tokens[idx-1], m.Tokens[idx]+ if a.GenLine == b.GenLine && a.GenCol == b.GenCol {+ idx--+ } else {+ break+ }+ }+ return &m.Tokens[idx]+}++func sortTokens(tokens []Token) {+ sort.SliceStable(tokens, func(i, j int) bool {+ a, b := tokens[i], tokens[j]+ if a.GenLine != b.GenLine {+ return a.GenLine < b.GenLine+ }+ return a.GenCol < b.GenCol+ })+}++func decodeName(raw json.RawMessage) string {+ var s string+ if err := json.Unmarshal(raw, &s); err == nil {+ return s+ }+ return strings.Trim(string(raw), `"`)+}++func joinSourceRoot(root, source string) string {+ if root == "" || strings.Contains(source, "://") || strings.HasPrefix(source, "/") {+ return source+ }+ return strings.TrimSuffix(root, "/") + "/" + source+}diff --git a/backend/app/symbolicator/sourcemap_parser/vlq.go b/backend/app/symbolicator/sourcemap_parser/vlq.gonew file mode 100644index 00000000..e885302a--- /dev/null+++ b/backend/app/symbolicator/sourcemap_parser/vlq.go@@ -0,0 +1,47 @@+package sourcemap_parser++import "fmt"++const vlqAlphabet = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"++var vlqLookup [128]int8++func init() {+ for i := range vlqLookup {+ vlqLookup[i] = -1+ }+ for i := 0; i < len(vlqAlphabet); i++ {+ vlqLookup[vlqAlphabet[i]] = int8(i)+ }+}++func decodeVLQ(seg string, dst []int64) ([]int64, error) {+ var cur uint64+ var shift uint+ for i := 0; i < len(seg); i++ {+ c := seg[i]+ if c >= 128 || vlqLookup[c] < 0 {+ return dst, fmt.Errorf("vlq: invalid character %q in segment %q", c, seg)+ }+ d := uint64(vlqLookup[c])+ if shift > 58 {+ return dst, fmt.Errorf("vlq: segment %q overflows", seg)+ }+ cur |= (d & 31) << shift+ if d&32 != 0 {+ shift += 5+ } else {+ val := int64(cur >> 1)+ if cur&1 != 0 {+ val = -val+ }+ dst = append(dst, val)+ cur = 0+ shift = 0+ }+ }+ if shift != 0 {+ return dst, fmt.Errorf("vlq: truncated segment %q", seg)+ }+ return dst, nil+}diff --git a/backend/app/symbolicator/symbolic_parity_test.go b/backend/app/symbolicator/symbolic_parity_test.gonew file mode 100644index 00000000..522c13fe--- /dev/null+++ b/backend/app/symbolicator/symbolic_parity_test.go@@ -0,0 +1,167 @@+package symbolicator++import (+ "os"+ "path/filepath"+ "testing"+)++func fixtureRoot(t *testing.T) string {+ t.Helper()+ root, err := filepath.Abs(filepath.Join("..", "..", "..", "..", "symbolic"))+ if err != nil {+ t.Fatal(err)+ }+ if _, err := os.Stat(root); err != nil {+ t.Skipf("symbolic fixture checkout not found at %s", root)+ }+ return root+}++func fixture(t *testing.T, parts ...string) string {+ t.Helper()+ return filepath.Join(append([]string{fixtureRoot(t), "symbolic-testutils", "fixtures"}, parts...)...)+}++func mustRead(t *testing.T, path string) []byte {+ t.Helper()+ data, err := os.ReadFile(path)+ if err != nil {+ t.Fatalf("reading fixture: %v", err)+ }+ return data+}++func readIfSet(t *testing.T, parts []string) []byte {+ if parts == nil {+ return nil+ }+ return mustRead(t, fixture(t, parts...))+}++type expectation struct {+ line, col uint32++ wantNone bool+ file string+ srcLine uint32+ srcCol uint32+ fn string+}++type parityCase struct {+ name string+ minifiedPath []string+ mapPath []string+ expectations []expectation+}++var parityCases = []parityCase{+ {+ name: "resolves_inlined_function",+ minifiedPath: []string{"sourcemapcache", "inlining", "module.js"},+ mapPath: []string{"sourcemapcache", "inlining", "module.js.map"},+ expectations: []expectation{+ {line: 0, col: 62, file: "../src/app.js", srcLine: 2, srcCol: 29, fn: ""},+ {line: 0, col: 46, file: "../src/bar.js", srcLine: 3, srcCol: 2, fn: "bar"},+ {line: 0, col: 33, file: "../src/foo.js", srcLine: 1, srcCol: 8, fn: ""},+ {line: 1, col: 17, wantNone: true},+ },+ },+ {+ name: "writes_simple_cache",+ minifiedPath: []string{"sourcemapcache", "simple", "minified.js"},+ mapPath: []string{"sourcemapcache", "simple", "minified.js.map"},+ expectations: []expectation{+ {line: 0, col: 10, file: "tests/fixtures/simple/original.js", srcLine: 1, srcCol: 9, fn: "abcd"},+ },+ },+ {+ name: "resolves_location_from_cache",+ minifiedPath: []string{"sourcemapcache", "preact.module.js"},+ mapPath: []string{"sourcemapcache", "preact.module.js.map"},+ expectations: []expectation{+ {line: 0, col: 49, file: "../src/constants.js", srcLine: 2, srcCol: 34, fn: ""},+ {line: 0, col: 132, file: "../src/util.js", srcLine: 11, srcCol: 22, fn: "assign"},+ {line: 0, col: 481, file: "../src/create-element.js", srcLine: 39, srcCol: 8, fn: "createElement"},+ {line: 0, col: 9779, file: "../src/component.js", srcLine: 181, srcCol: 4, fn: ""},+ {line: 0, col: 9794, file: "../src/create-context.js", srcLine: 2, srcCol: 11, fn: ""},+ },+ },+ {+ name: "missing_source_names",+ minifiedPath: []string{"sourcemapcache", "nofiles.js"},+ mapPath: []string{"sourcemapcache", "nofiles.js.map"},+ expectations: []expectation{+ {line: 0, col: 38, srcLine: 2, srcCol: 8, fn: "add"},+ },+ },+ {+ name: "hermes_scope_lookup",+ mapPath: []string{"sourcemapcache", "hermes-metro", "react-native-hermes.map"},+ expectations: []expectation{+ {line: 0, col: 11939, file: "module.js", srcLine: 1, srcCol: 10, fn: ""},+ {line: 0, col: 11857, file: "input.js", srcLine: 2, srcCol: 0, fn: ""},+ },+ },+ {+ name: "metro_scope_lookup",+ minifiedPath: []string{"sourcemapcache", "hermes-metro", "react-native-metro.js"},+ mapPath: []string{"sourcemapcache", "hermes-metro", "react-native-metro.js.map"},+ expectations: []expectation{+ {line: 6, col: 100, file: "module.js", srcLine: 1, srcCol: 10, fn: ""},+ {line: 5, col: 43, file: "input.js", srcLine: 2, srcCol: 0, fn: ""},+ },+ },+ {+ name: "webpack_scope_lookup",+ minifiedPath: []string{"sourcemapcache", "webpack", "bundle.js"},+ mapPath: []string{"sourcemapcache", "webpack", "bundle.js.map"},+ expectations: []expectation{+ {line: 0, col: 84, file: "webpack:///./foo.js", srcLine: 1, srcCol: 8, fn: ""},+ {line: 0, col: 43, file: "webpack:///./bar.js", srcLine: 1, srcCol: 2, fn: ""},+ },+ },+}++func TestSymbolicParity(t *testing.T) {+ for _, tc := range parityCases {+ t.Run(tc.name, func(t *testing.T) {+ mapBytes := mustRead(t, fixture(t, tc.mapPath...))+ bundle := readIfSet(t, tc.minifiedPath)++ r, err := NewResolver(mapBytes, bundle)+ if err != nil {+ t.Fatalf("NewResolver: %v", err)+ }++ for _, exp := range tc.expectations {+ frame, ok := r.Lookup(exp.line, exp.col)++ if exp.wantNone {+ if ok {+ t.Errorf("lookup(%d,%d): got %+v, want no mapping", exp.line, exp.col, frame)+ }+ continue+ }+ if !ok {+ t.Errorf("lookup(%d,%d): got no mapping, want a result", exp.line, exp.col)+ continue+ }++ if exp.file != "" && frame.File != exp.file {+ t.Errorf("lookup(%d,%d) file: got %q, want %q", exp.line, exp.col, frame.File, exp.file)+ }+ if frame.Line != exp.srcLine+1 {+ t.Errorf("lookup(%d,%d) line: got %d, want %d", exp.line, exp.col, frame.Line, exp.srcLine+1)+ }+ if frame.Col != exp.srcCol+1 {+ t.Errorf("lookup(%d,%d) col: got %d, want %d", exp.line, exp.col, frame.Col, exp.srcCol+1)+ }+ if frame.Fn != exp.fn {+ t.Errorf("lookup(%d,%d) fn: got %q, want %q", exp.line, exp.col, frame.Fn, exp.fn)+ }+ }+ })+ }+}diff --git a/backend/app/symbolicator/symbolicator.go b/backend/app/symbolicator/symbolicator.gonew file mode 100644index 00000000..a4a67d91--- /dev/null+++ b/backend/app/symbolicator/symbolicator.go@@ -0,0 +1,144 @@+package symbolicator++import (+ "sort"++ "github.com/tracewayapp/traceway/backend/app/symbolicator/sourcemap_parser"+)++type StackTraceFrame struct {+ File string+ Line uint32+ Col uint32+ Fn string+}++type resolvedToken struct {+ genLine, genCol uint32+ srcLine, srcCol uint32+ fileIdx int32+ fnIdx int32+}++type Resolver struct {+ tokens []resolvedToken+ files []string+ fns []string+}++func NewResolver(sourceMap, bundle []byte) (*Resolver, error) {+ m, err := sourcemap_parser.Parse(sourceMap)+ if err != nil {+ return nil, err+ }++ var scopes *functionScopes+ if len(bundle) > 0 {+ if fs, err := parseFunctionScopes(bundle); err == nil {+ scopes = fs+ }+ }++ files := newInterner()+ fns := newInterner()++ var transitions []scopeTransition+ if scopes != nil {+ transitions = scopes.transitions+ }+ ti := -1+ var curFnIdx int32 = -1+ curResolved := false++ tokens := make([]resolvedToken, len(m.Tokens))+ for i := range m.Tokens {+ t := m.Tokens[i]++ // Tokens are sorted by generated position, so the floor transition+ // pointer only moves forward across the whole map (linear merge).+ for ti+1 < len(transitions) && !less(t.GenLine, t.GenCol, transitions[ti+1].line, transitions[ti+1].col) {+ ti+++ curResolved = false+ }++ rt := resolvedToken{genLine: t.GenLine, genCol: t.GenCol, fileIdx: -1, fnIdx: -1}+ if t.SrcID != sourcemap_parser.NoID && int(t.SrcID) < len(m.Sources) {+ rt.fileIdx = files.intern(m.Sources[t.SrcID])+ rt.srcLine = t.SrcLine + 1+ rt.srcCol = t.SrcCol + 1+ if ti >= 0 && transitions[ti].has {+ // Resolve the enclosing function's name once per transition,+ // not once per token.+ if !curResolved {+ curFnIdx = -1+ tr := transitions[ti]+ if nt := m.FloorToken(tr.nameLine, tr.nameCol); nt != nil && nt.NameID != sourcemap_parser.NoID {+ curFnIdx = fns.intern(m.Names[nt.NameID])+ }+ curResolved = true+ }+ rt.fnIdx = curFnIdx+ }+ }+ tokens[i] = rt+ }++ return &Resolver{tokens: tokens, files: files.list, fns: fns.list}, nil+}++func (r *Resolver) Lookup(genLine, genCol uint32) (StackTraceFrame, bool) {+ toks := r.tokens+ idx := sort.Search(len(toks), func(i int) bool {+ return toks[i].genLine > genLine || (toks[i].genLine == genLine && toks[i].genCol > genCol)+ })+ if idx == 0 {+ return StackTraceFrame{}, false+ }+ idx--+ for idx > 0 && toks[idx-1].genLine == toks[idx].genLine && toks[idx-1].genCol == toks[idx].genCol {+ idx--+ }++ t := toks[idx]+ if t.genLine < genLine || t.fileIdx < 0 {+ return StackTraceFrame{}, false+ }+ frame := StackTraceFrame{File: r.files[t.fileIdx], Line: t.srcLine, Col: t.srcCol}+ if t.fnIdx >= 0 {+ frame.Fn = r.fns[t.fnIdx]+ }+ return frame, true+}++// ApproxSize estimates the resolver's retained heap footprint, for cache+// accounting. The flat token table dominates; the bundle and source map are+// not retained.+func (r *Resolver) ApproxSize() int64 {+ n := int64(len(r.tokens)) * 24+ for _, s := range r.files {+ n += int64(len(s)) + 16+ }+ for _, s := range r.fns {+ n += int64(len(s)) + 16+ }+ return n+}++type interner struct {+ list []string+ idx map[string]int32+}++func newInterner() *interner {+ return &interner{idx: make(map[string]int32)}+}++func (in *interner) intern(s string) int32 {+ if i, ok := in.idx[s]; ok {+ return i+ }+ i := int32(len(in.list))+ in.list = append(in.list, s)+ in.idx[s] = i+ return i+}diff --git a/backend/cmd/run.go b/backend/cmd/run.goindex a4fd6589..faabd458 100644--- a/backend/cmd/run.go+++ b/backend/cmd/run.go@@ -23,6 +23,7 @@ import ("github.com/tracewayapp/traceway/backend/app/recordings""github.com/tracewayapp/traceway/backend/app/retention""github.com/tracewayapp/traceway/backend/app/services"+ "github.com/tracewayapp/traceway/backend/app/sourcemapbackfill""github.com/tracewayapp/traceway/backend/app/storage""github.com/tracewayapp/traceway/backend/static"@@ -132,6 +133,7 @@ func Run(opts ...Option) {notifications.StartEvaluator(ctx)retention.Start(ctx)recordings.Start(ctx)+ sourcemapbackfill.Start(ctx)var router *gin.Engineif o != nil && o.disableLogging {diff --git a/backend/go.mod b/backend/go.modindex 256c5630..861e21ca 100644--- a/backend/go.mod+++ b/backend/go.mod@@ -8,6 +8,7 @@ require (github.com/aws/aws-sdk-go-v2/credentials v1.19.7github.com/aws/aws-sdk-go-v2/service/s3 v1.96.0github.com/coreos/go-systemd/v22 v22.6.0+ github.com/dop251/goja v0.0.0-20260607120635-348e6bea910dgithub.com/golang-jwt/jwt/v5 v5.3.0github.com/gorilla/securecookie v1.1.2github.com/gorilla/sessions v1.4.0@@ -22,6 +23,7 @@ require (require (cloud.google.com/go/compute/metadata v0.8.0 // indirectgithub.com/go-chi/chi/v5 v5.2.2 // indirect+ github.com/go-sourcemap/sourcemap v2.1.3+incompatible // indirectgithub.com/gorilla/mux v1.7.4 // indirectgithub.com/grpc-ecosystem/grpc-gateway/v2 v2.27.2 // indirectgithub.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 // indirect@@ -58,7 +60,7 @@ require (github.com/aws/aws-sdk-go-v2/service/sso v1.30.9 // indirectgithub.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.13 // indirectgithub.com/aws/aws-sdk-go-v2/service/sts v1.41.6 // indirect- github.com/aws/smithy-go v1.24.0 // indirect+ github.com/aws/smithy-go v1.24.0github.com/bytedance/sonic v1.14.0 // indirectgithub.com/bytedance/sonic/loader v0.3.0 // indirectgithub.com/cespare/xxhash/v2 v2.3.0 // indirect@@ -71,7 +73,7 @@ require (github.com/go-playground/universal-translator v0.18.1 // indirectgithub.com/go-playground/validator/v10 v10.27.0 // indirectgithub.com/goccy/go-json v0.10.2 // indirect- github.com/goccy/go-yaml v1.18.0 // indirect+ github.com/goccy/go-yaml v1.19.2 // indirectgithub.com/json-iterator/go v1.1.12 // indirectgithub.com/klauspost/compress v1.18.0 // indirectgithub.com/klauspost/cpuid/v2 v2.3.0 // indirectdiff --git a/backend/go.sum b/backend/go.sumindex 261c6acb..c6288887 100644--- a/backend/go.sum+++ b/backend/go.sum@@ -82,6 +82,8 @@ github.com/docker/go-connections v0.6.0 h1:LlMG9azAe1TqfR7sO+NJttz1gy6KO7VJBh+pMgithub.com/docker/go-connections v0.6.0/go.mod h1:AahvXYshr6JgfUJGdDCs2b5EZG/vmaMAntpSFH5BFKE=github.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4=github.com/docker/go-units v0.5.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=+github.com/dop251/goja v0.0.0-20260607120635-348e6bea910d h1:xbM5U2EvWKkHxzEQJ2DEn20FwolWZahuTnVHr6WL3Q4=+github.com/dop251/goja v0.0.0-20260607120635-348e6bea910d/go.mod h1:Sc+QOu1WruvaaeT/cxFez/pXHpI9ZDjg/E8QNfSVveI=github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=@@ -111,10 +113,12 @@ github.com/go-playground/universal-translator v0.18.1 h1:Bcnm0ZwsGyWbCzImXv+pAJngithub.com/go-playground/universal-translator v0.18.1/go.mod h1:xekY+UJKNuX9WP91TpwSH2VMlDf28Uj24BCp08ZFTUY=github.com/go-playground/validator/v10 v10.27.0 h1:w8+XrWVMhGkxOaaowyKH35gFydVHOvC0/uWoy2Fzwn4=github.com/go-playground/validator/v10 v10.27.0/go.mod h1:I5QpIEbmr8On7W0TktmJAumgzX4CA1XNl4ZmDuVHKKo=+github.com/go-sourcemap/sourcemap v2.1.3+incompatible h1:W1iEw64niKVGogNgBN3ePyLFfuisuzeidWPMPWmECqU=+github.com/go-sourcemap/sourcemap v2.1.3+incompatible/go.mod h1:F8jJfvm2KbVjc5NqelyYJmf/v5J0dwNLS2mL4sNA1Jg=github.com/goccy/go-json v0.10.2 h1:CrxCmQqYDkv1z7lO7Wbh2HN93uovUHgrECaO5ZrCXAU=github.com/goccy/go-json v0.10.2/go.mod h1:6MelG93GURQebXPDq3khkgXZkazVtN9CRI+MGFi0w8I=-github.com/goccy/go-yaml v1.18.0 h1:8W7wMFS12Pcas7KU+VVkaiCng+kG8QiFeFwzFb+rwuw=-github.com/goccy/go-yaml v1.18.0/go.mod h1:XBurs7gK8ATbW4ZPGKgcbrY1Br56PdM69F7LkFRi1kA=+github.com/goccy/go-yaml v1.19.2 h1:PmFC1S6h8ljIz6gMRBopkjP1TVT7xuwrButHID66PoM=+github.com/goccy/go-yaml v1.19.2/go.mod h1:XBurs7gK8ATbW4ZPGKgcbrY1Br56PdM69F7LkFRi1kA=github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=github.com/golang-jwt/jwt/v5 v5.3.0 h1:pv4AsKCKKZuqlgs5sUmn4x8UlGa0kEVt/puTpKx9vvo=github.com/golang-jwt/jwt/v5 v5.3.0/go.mod h1:fxCRLWMO43lRc8nhHWY6LGqRcf+1gQWArsqaEUEa5bE=diff --git a/docs/pages/client/js-sdk/sourcemap-upload.mdx b/docs/pages/client/js-sdk/sourcemap-upload.mdxindex 24e74661..faf1c30c 100644--- a/docs/pages/client/js-sdk/sourcemap-upload.mdx+++ b/docs/pages/client/js-sdk/sourcemap-upload.mdx@@ -16,11 +16,10 @@ This provides the `traceway-sourcemaps` CLI command.traceway-sourcemaps \--url https://your-traceway-instance.com \--token your-sourcemap-token \- --version 1.2.3 \--directory ./dist```-The CLI recursively finds all `*.map` files in the specified directory and uploads them to Traceway via `POST /api/sourcemaps/upload`.+The CLI recursively finds all `*.map` files in the specified directory and uploads them to Traceway via `POST /api/sourcemaps/upload`, together with the sibling `.js`/`.cjs`/`.mjs` bundle next to each map when one exists. The bundle is what lets the backend resolve the enclosing function name for each frame; without it, symbolication still resolves file, line, and column but leaves function names as the browser reported them.## CLI Options@@ -28,7 +27,6 @@ The CLI recursively finds all `*.map` files in the specified directory and uploa|------|----------|-------------|| `--url <url>` | Yes | Traceway backend URL || `--token <token>` | Yes | Source map upload token |-| `--version <version>` | Yes | App version to associate with the source maps || `--directory <dir>` | No | Directory to search (default: current directory) |## Environment Variables@@ -71,7 +69,7 @@ jobs:- run: npm run build- name: Upload source maps- run: npx traceway-sourcemaps --version ${{ github.sha }} --directory ./dist+ run: npx traceway-sourcemaps --directory ./distenv:TRACEWAY_URL: ${{ secrets.TRACEWAY_URL }}TRACEWAY_SOURCEMAP_TOKEN: ${{ secrets.TRACEWAY_SOURCEMAP_TOKEN }}@@ -85,26 +83,20 @@ Add the upload step to your `package.json` scripts:{"scripts": {"build": "vite build",- "postbuild": "traceway-sourcemaps --version $npm_package_version --directory ./dist"+ "postbuild": "traceway-sourcemaps --directory ./dist"}}```Make sure `TRACEWAY_URL` and `TRACEWAY_SOURCEMAP_TOKEN` are set in your environment.-## Version Matching+## How Maps Are Matched-The `--version` value must match the `version` option passed to the Traceway SDK during initialization:+There is nothing to configure at runtime. When an exception is captured, Traceway takes each minified frame's filename and looks up the map you uploaded under that same name, deobfuscating the frame. The most recent upload of each filename wins, so content-hashed bundle names (the default in Vite, Next, Angular, and most bundlers) keep every build's maps distinct automatically.-```javascript-import { init } from "@tracewayapp/frontend";+Uploads take effect immediately: the backend evicts any cached state for the uploaded filenames, so exceptions arriving right after a deploy resolve against the new maps. See [Symbolication](/learn/symbolication-js) for the full resolution pipeline.-init("your-token@https://traceway.example.com/api/report", {- version: "1.2.3", // Must match the --version used during upload-});-```--When an exception is captured, Traceway uses the version to look up the correct source maps and deobfuscate the stack trace.+The SDK's `version` option is unrelated to source maps: it is plain metadata shown on exceptions in the dashboard for filtering by build, and can be set or omitted independently of uploads.## Next Stepsdiff --git a/docs/pages/learn/symbolication-js.mdx b/docs/pages/learn/symbolication-js.mdxindex b8811c13..7a032e67 100644--- a/docs/pages/learn/symbolication-js.mdx+++ b/docs/pages/learn/symbolication-js.mdx@@ -26,16 +26,18 @@ handleClick()src/pages/Profile.tsx:108:5```-This page explains exactly how that translation works today, what is cached and when, how the pipeline is observable through metrics, where the current heuristics fall short, and how we verify the output. For how to upload source maps in the first place, see [Source Maps](/client/js-sdk/sourcemap-upload).+This page explains exactly how that translation works today, what is cached and when, how the pipeline is observable through metrics, where the current limits are, and how we verify the output. For how to upload source maps in the first place, see [Source Maps](/client/js-sdk/sourcemap-upload).## When Symbolication RunsSymbolication happens **once, at ingest**, inside `POST /api/report`:-1. The report arrives from the JS SDK. If the project is a JS framework, the backend loads the project's source maps from the database: the set matching the report's `version` if one was sent, otherwise the latest uploaded set. This is why the `--version` flag on upload must match the SDK's `version` option.+1. The report arrives from the JS SDK. If the project is a JS framework, each frame is resolved directly against object storage at `sourcemaps/{projectId}/{filename}`: the map is addressed by the frame's filename, with no version and no database lookup. The most recent upload of each filename wins.2. Each stack trace in the report is resolved frame by frame.3. The **resolved** trace is what gets hashed for grouping and stored in ClickHouse.+The same resolution runs for exceptions arriving over the OTLP endpoint when the resource reports a JavaScript SDK (`telemetry.sdk.language` of `webjs`, `nodejs`, `javascript`, or `typescript`).+Nothing is re-resolved when you view an issue. The trade-off: traces are immutable snapshots of the symbolication quality at the moment they arrived. Uploading better source maps improves future events, not past ones.## The Frame Format@@ -61,19 +63,21 @@ For each location line, in order:### 1. Find the right source map-The frame's file name is matched against the uploaded maps by appending `.map`:+The frame's file name is reduced to its basename (any query string or fragment stripped first), and the map is addressed in object storage at a deterministic key:-| Frame file | Lookup attempts |+| Frame file | Storage key ||---|---|-| `app.bundle.js` | `app.bundle.js.map`, then basename `app.bundle.js.map` |-| `https://cdn.example.com/assets/app.bundle.js` | full name, then basename `app.bundle.js.map` |-| `app.bundle.js?v=abc123` | query/fragment stripped, then `app.bundle.js.map` |+| `app.bundle.js` | `sourcemaps/{projectId}/app.bundle.js.map` |+| `https://cdn.example.com/assets/app.bundle.js` | `sourcemaps/{projectId}/app.bundle.js.map` |+| `app.bundle.js?v=abc123` | `sourcemaps/{projectId}/app.bundle.js.map` |++The minified bundle, when the upload CLI found one next to the map, lives at the same key without the `.map` suffix and feeds function name resolution (next section).-No match: the line passes through unchanged.+No map in storage: the line passes through unchanged, and that filename enters a miss cooldown so subsequent exceptions skip the storage read entirely (see "When loading fails" below).-### 2. Get the parsed map+### 2. Get the resolver-The map comes from the in-memory cache described below. First use pays a storage read plus a parse; everything after is a memory lookup.+The parsed lookup structure comes from the in-memory cache described below. First use pays a storage read plus a parse; everything after is a memory lookup.### 3. Look up the position@@ -115,34 +119,17 @@ If the source map carries `"sources": [null]` entries (older uglify pipelines),A maximum of **50 frames** per trace are resolved; deeper frames pass through minified.-## Function Names: The Regex Heuristic--Locations come straight from the source map and are exact. Function names are harder, because a source map's `names` token at a position is the identifier *at that token*, not the name of the *enclosing function* that a stack trace conventionally shows. Getting the enclosing function right requires scope analysis: parsing the minified bundle into an AST and mapping positions to function scopes, which is what Sentry's symbolic does. Traceway does not store the minified bundle, only the `.map`, so scope analysis is not possible today.+## Function Names: Scope Analysis-Instead we use a heuristic. Since most maps embed the original source (`sourcesContent`), the resolver takes the resolved original line and scans **backwards up to 50 lines** for something that looks like a function declaration:+Locations come straight from the source map and are exact. Function names are harder, because a source map's `names` token at a position is the identifier *at that token*, not the name of the *enclosing function* that a stack trace conventionally shows. Getting the enclosing function right requires scope analysis: parsing the minified bundle into an AST and mapping positions to function scopes. This is the same approach Sentry's symbolic takes, and it is why the upload CLI sends the minified bundle alongside each `.map`.-```-(?:export\s+(?:default\s+)?)?function\s+(\w+) match: function declarations-(?:const|let|var)\s+(\w+)\s*= match: variable assignments-^\s*(?:async\s+)?(\w+)\s*\([^)]*\)\s*\{ match: method shorthand-```+When a bundle is available, the resolver:-The first match wins, with control flow keywords (`if`, `for`, `while`, `switch`, `catch`, `return`, `throw`, `else`) filtered out. When the heuristic finds something, it overrides the source map's name token. When it finds nothing, the name token is used as-is.+1. Parses it with a pure-Go JavaScript parser into every function in the file: declarations, function expressions, arrow functions, and class methods, each with the generated position of its name. If the bundle fails to parse as a script (ESM syntax), it is retried with module syntax neutralized.+2. Flattens the nested scopes into a sorted list of transitions: "from this generated position onward, the innermost enclosing function's name sits at line L, column C".+3. For each mapping in the source map, resolves the enclosing function's *name position* back through the map itself, yielding the original, pre-minification name.-### Where it works--```js-export function assign(obj, props) {- for (let i in props) obj[i] = props[i]; // error thrown here- return obj;-}-```--Scanning backwards from the error line immediately hits `function assign`, which is correct. Simple module-level functions, exported functions, and arrow functions assigned to `const` all resolve well.--### Where it fails, concretely--The middle alternation matches **any** variable assignment, not just function expressions, and the scan has no notion of scope boundaries. From preact's `create-element.js`:+Take preact's `create-element.js`, which the old name heuristic famously got wrong:```jsexport function createElement(type, props, children) {@@ -154,19 +141,12 @@ export function createElement(type, props, children) {}```-Scanning backwards from line 40, the first match is `let normalizedProps =`, so the frame is named `normalizedProps` instead of `createElement`. Similar failure shapes:--| Case | Correct name | Heuristic produces |-|---|---|---|-| Nearest declaration is a local variable | `createElement` | `normalizedProps` |-| Top-level/module code (Hermes, Metro bundles) | `<global>` | `anonymous` or a nearby declaration |-| CommonJS export assignment | `module.exports` | the inner function's name |-| Sibling function that closed above the error line | enclosing function | the sibling |+A frame inside that loop now resolves to `createElement`, because the scope walk knows which function *contains* the error position rather than guessing from nearby text.-These are known, pinned deviations in our test suite (more on that below). Two design decisions contain the blast radius:+When no bundle was uploaded (a map-only upload from an older CLI, or a `.map` with no sibling bundle in the build output), there is no scope information and **no name is resolved**: the name line stays whatever the SDK reported (typically `anonymous()`), while file, line, and column still resolve exactly. Two design decisions contain the remaining gaps:1. **Names never affect grouping.** See the next section.-2. **Locations are never heuristic.** The file, line, and column shown are exact source map output, so the click-through is right even when the label is wrong.+2. **Locations never depend on the bundle.** The file, line, and column shown are exact source map output, so the click-through is right even when the label is missing.## How Symbolication Interacts with Grouping@@ -187,7 +167,7 @@ Error: boom Error: boom../src/create-element.js:40:9 ../src/create-element.js:40:9```-This means name quality can keep improving (including the future scope-analysis work) without re-bucketing your existing issues.+This means name quality can keep improving (for example, by uploading bundles where you previously uploaded only maps) without re-bucketing your existing issues.**Columns are kept.** Minified bundles put everything on line 1, so when no source map matched, the column is the only thing distinguishing two different errors. `bundle.min.js:1:48211` and `bundle.min.js:1:91567` therefore hash differently, on purpose.@@ -195,48 +175,51 @@ One consequence to be aware of: a trace that fails symbolication (storage outage## What Is Cached, How, and When-Source maps for real applications are large (a 20 MB `.map` is unremarkable), and the parsed lookup structure is what actually serves frames. Traceway keeps **one in-memory cache** of parsed maps, keyed by the map's storage key.+Source maps for real applications are large (a 20 MB `.map` is unremarkable), and the compiled lookup structure is what actually serves frames. Traceway keeps **one in-memory cache** of built resolvers, keyed by the map's storage key.### What a cache entry contains-When a map is parsed, only the four things the resolver ever reads are kept:+Building a resolver consumes the `.map` and the bundle and keeps only what lookups need:| Field | Used for ||---|---|-| `mappings` (decoded to 24-byte segments) | position lookup via binary search |-| `sources` | original file names |-| `sourcesContent` | the function name heuristic |-| `names` | function name tokens |+| token table (one 24-byte token per mapping) | position lookup via binary search |+| interned file names | original file names |+| interned function names | enclosing function names, pre-resolved per scope at build time |-Everything else in the JSON (raw VLQ text, `file`, `sourceRoot` after resolution) is discarded, and **the raw file bytes exist only for the duration of the parse**. A parsed entry is typically 2 to 5 times the raw file size, and its actual in-memory size is measured at parse time, which is what the byte cap counts.+Everything else is discarded: the raw map and bundle bytes, `sourcesContent`, the bundle's AST, and the scope transitions all exist **only for the duration of the build**. Scope analysis runs once per build, not per lookup, so serving a frame is a binary search over the token table. The entry's in-memory size is estimated from the token count plus the interned strings, which is what the byte cap counts.### Cache policy| Property | Value ||---|---|-| Keying | storage key, one entry per uploaded map file |-| Capacity | `SOURCEMAP_CACHE_MAX_ENTRIES` (default 200) and `SOURCEMAP_CACHE_MAX_BYTES_MB` (default 500, measured parsed size) |+| Keying | map storage key, one entry per bundle filename per project |+| Capacity | `SOURCEMAP_CACHE_MAX_ENTRIES` (default 200) and `SOURCEMAP_CACHE_MAX_BYTES_MB` (default 500, estimated resolver size) || Eviction | least recently used, whenever either cap is exceeded |-| Loading | deduplicated (singleflight): concurrent requests for the same map share one storage read and one parse |+| Loading | deduplicated (singleflight): concurrent requests for the same map share one storage read and one build || Load timeout | 5 seconds per storage read |-| Failures | never cached globally; retried on the next exception |+| Failures | negative-cached with an escalating cooldown (below) |+| Invalidation | uploading a file evicts its resolver and clears its cooldown immediately |### The lifecycle, end to endSay two exceptions referencing `app.bundle.js.map` arrive at the same time, on a cold cache:1. Both miss the cache and race to a mutex. The winner registers itself as the **loader** for that key; the loser sees the in-flight registration and parks.-2. The loader reads the `.map` from storage (S3 or local disk, 5 second budget), parses it, throws the raw bytes away, and inserts the slim parsed entry.-3. The parked request wakes and receives the exact same parsed instance. One read, one parse, one copy in memory, no matter how many requests piled up.+2. The loader reads the `.map` and the bundle from storage (S3 or local disk, 5 second budget per read), builds the resolver (parse map, parse bundle, pre-resolve scope names), throws the raw bytes away, and inserts the slim entry.+3. The parked request wakes and receives the exact same resolver instance. One read, one build, one copy in memory, no matter how many requests piled up.4. Every subsequent exception is a pure memory hit, which also bumps the entry to the front of the LRU.-5. If the cache later exceeds a cap, the least recently used entries are dropped. A dropped map is not an error; the next exception that needs it repeats steps 1 to 3.+5. If the cache later exceeds a cap, the least recently used entries are dropped. A dropped resolver is not an error; the next exception that needs it repeats steps 1 to 3.+6. Re-uploading `app.bundle.js.map` (or `app.bundle.js`) evicts the cached resolver, so the next exception rebuilds from the new upload rather than serving the stale one.### When loading fails-If the read times out, storage errors, or the map fails to parse:+If the map does not exist, the read times out, storage errors, or the map fails to parse:-- The affected frames pass through **unchanged**, and the trace is stored as-is. Ingest is never blocked beyond the 5 second budget and never errors because of symbolication.-- The failure is **not cached**, so the next exception retries immediately; recovery after an outage is instant.+- The affected frames pass through **unchanged**, and the trace is stored as-is. Ingest is never blocked beyond the per-read budgets (5 seconds each for the map and the bundle) and never errors because of symbolication. A transient failure reading the *bundle* fails the build too, rather than silently caching a resolver without function names; the cooldown retries it within seconds.+- The failure is **negative-cached per filename**: while the cooldown is active, frames referencing that filename skip the storage read entirely. A missing map starts at a 1 minute cooldown; a transient failure (timeout, storage error, unparseable map) starts at 15 seconds, so brief storage hiccups recover fast. Each consecutive failure doubles the cooldown, capped at 15 minutes, so a project that never uploads maps converges to a few storage reads per hour per filename instead of one per minute.+- The cooldown clears the moment that filename is uploaded (the upload endpoint invalidates it in-process) or a load succeeds. In a multi-instance deployment, instances other than the one that served the upload retry within their remaining cooldown, at most 15 minutes.+- The negative cache is bounded at 10,000 filenames; expired entries are pruned when the bound is hit.- Within a single trace, a failed map is attempted **once**, not once per frame, so a 50-frame trace cannot stack fifty timeouts.- Traceway self-reports the failure into its own monitoring project, throttled to one exception per minute with a count of suppressed occurrences.@@ -246,41 +229,43 @@ The backend self-reports cache metrics every 30 seconds (see [Capacity Monitorin| Metric | Meaning ||---|---|-| `traceway.sourcemap.entries` | parsed maps currently in memory |-| `traceway.sourcemap.bytes` | measured total size of those entries |+| `traceway.sourcemap.entries` | built resolvers currently in memory |+| `traceway.sourcemap.bytes` | estimated total size of those entries |+| `traceway.sourcemap.negative_entries` | filenames currently under a miss/failure cooldown || `traceway.sourcemap.hits.delta` | cache hits since the last tick |-| `traceway.sourcemap.misses.delta` | loads triggered since the last tick |+| `traceway.sourcemap.misses.delta` | builds triggered since the last tick || `traceway.sourcemap.evictions.delta` | LRU evictions since the last tick |-| `traceway.sourcemap.load_failures.delta` | failed loads (storage error, 5s timeout, unparseable map); each one is a trace stored unsymbolicated |-| `traceway.sourcemap.parse_ms` | duration of the most recent parse |+| `traceway.sourcemap.load_failures.delta` | failed builds (storage error, 5s timeout, unparseable map); each one is a trace stored unsymbolicated |+| `traceway.sourcemap.not_found.delta` | lookups that found no uploaded map for the frame's filename since the last tick |+| `traceway.sourcemap.negative_hits.delta` | frame lookups skipped by an active cooldown since the last tick |+| `traceway.sourcemap.parse_ms` | duration of the most recent resolver build |-What healthy looks like: `misses.delta` near zero in steady state (every miss is a storage read plus a re-parse), `load_failures.delta` at zero (anything above zero means minified traces are reaching your issues), and `bytes` comfortably under the cap. A useful alert is `load_failures.delta` max greater than 0 over 5 minutes.+What healthy looks like: `misses.delta` near zero in steady state (every miss is a storage read plus a rebuild), `load_failures.delta` at zero (anything above zero means minified traces are reaching your issues), and `bytes` comfortably under the cap. A useful alert is `load_failures.delta` max greater than 0 over 5 minutes.++`not_found.delta` is a configuration signal rather than a failure signal: sustained nonzero values for a project that should be symbolicated mean exceptions reference filenames that were never uploaded, most commonly a missing upload step in CI or a `--directory` pointing at the wrong build output. `negative_hits.delta` shows the cooldown doing its job; high values alongside high `not_found.delta` are normal for JS projects that simply do not upload maps.## Limitations TodayHonest list, in rough order of impact:-1. **Function names are heuristic.** No scope analysis; the regex picks the nearest declaration-looking line above the error, which is wrong in the patterns shown above. Locations are unaffected.-2. **The minified bundle is not stored**, only the `.map`. This is what blocks real scope analysis (and is the prerequisite for fixing limitation 1).-3. **Symbolication is ingest-only.** Maps uploaded after an event arrived do not improve that event.+1. **Function names require the bundle.** The current upload CLI sends it automatically, but map-only uploads (older CLI versions, or maps whose sibling bundle was not in the build output) resolve locations without names.+2. **Symbolication is ingest-only.** Maps uploaded after an event arrived do not improve that event.+3. **Maps are matched by basename.** Two bundles with the same filename in different directories collide, and the most recent upload wins. Content-hashed bundle names (the default in Vite, Next, Angular, and most bundlers) avoid this entirely; stable names like `app.js` shared across concurrently deployed versions resolve against the latest upload only.4. **The 5 second load budget trades completeness for latency.** A map that takes longer to fetch produces an unsymbolicated trace (recorded in `load_failures.delta`) rather than a slow ingest.-5. **Outages split grouping transiently.** Unsymbolicated and symbolicated versions of the same error hash differently.-6. **Maps with multiple `null` sources are ambiguous.** Frames resolve as `<unknown>`, and the name heuristic may read the wrong embedded source, since empty file names cannot be told apart.-7. **50 resolved frames per trace.** Deeper frames stay minified.-8. **Hermes/Metro global code** is labeled `anonymous` rather than `<global>`.+5. **Failure cooldowns can delay recovery on other instances.** Uploading a map clears its cooldown instantly on the instance that handled the upload; in multi-instance deployments, the rest retry within their remaining cooldown, at most 15 minutes.+6. **Outages split grouping transiently.** Unsymbolicated and symbolicated versions of the same error hash differently.+7. **Maps with `null` sources are ambiguous.** Frames resolve as `<unknown>`, though line, column, and name still resolve.+8. **50 resolved frames per trace.** Deeper frames stay minified.+9. **Module-level code has no name.** Errors thrown outside any function (Hermes/Metro global prologue, top-level module code) resolve location-only rather than being labeled `<global>`.## How We Verify It-The resolver is tested for parity against [getsentry/symbolic](https://github.com/getsentry/symbolic), the symbolication engine behind Sentry, using its own fixture suite (webpack, Metro, Hermes, preact, inlined functions, null-source maps). Every frame's expected file, line, column, and name is asserted, and the committed baseline currently reads:--```-frames: 13 parity, 5 pinned deviations, 0 failing, 18 total-```+The resolver is tested for parity against [getsentry/symbolic](https://github.com/getsentry/symbolic), the symbolication engine behind Sentry, using its own fixture suite (webpack, Metro, Hermes, preact, inlined functions, null-source maps). Every fixture frame's expected file, line, column, and name is asserted exactly, so any drift fails the build.-All 18 frames match symbolic on **location**. The 5 deviations are **name-only**, are exactly the heuristic failures documented above, and are pinned: the tests assert our current output, so any drift fails the build, and each pinned frame records the symbolic name as the parity target to restore once scope analysis lands.+All frames match symbolic on **location**. With scope analysis, names now match too for frames inside a function, including the cases the old text-scanning heuristic got wrong (preact's `createElement` and `assign` among them). The remaining name gap is module-level code, where symbolic reports `<global>` and Traceway reports no name (limitation 9 above).## Related Pages-- [Source Maps](/client/js-sdk/sourcemap-upload): uploading maps from CI and matching versions+- [Source Maps](/client/js-sdk/sourcemap-upload): uploading maps from CI- [Issues](/learn/issues): grouping, lifecycle, and archiving- [Capacity Monitoring](/server/capacity-monitoring): cache sizing and alertingdiff --git a/docs/pages/server/_meta.json b/docs/pages/server/_meta.jsonindex c4193d3c..968770d6 100644--- a/docs/pages/server/_meta.json+++ b/docs/pages/server/_meta.json@@ -6,5 +6,6 @@"sqlite": "SQLite","local-setup": "Local Setup","haloy": "Haloy",- "capacity-monitoring": "Capacity & Self-Monitoring"+ "capacity-monitoring": "Capacity & Self-Monitoring",+ "upgrade-1-8": "Upgrading to 1.8"}diff --git a/docs/pages/server/capacity-monitoring.mdx b/docs/pages/server/capacity-monitoring.mdxindex a0d55831..ed75c804 100644--- a/docs/pages/server/capacity-monitoring.mdx+++ b/docs/pages/server/capacity-monitoring.mdx@@ -76,25 +76,32 @@ Not emitted in SQLite mode.| Metric | Meaning ||--------|---------|-| `traceway.sourcemap.entries` / `traceway.sourcemap.bytes` | Parsed source maps held in memory and their estimated total size |+| `traceway.sourcemap.entries` / `traceway.sourcemap.bytes` | Built source map resolvers held in memory and their estimated total size |+| `traceway.sourcemap.negative_entries` | Filenames currently under a miss/failure cooldown (no map uploaded, or recent load failures) || `traceway.sourcemap.hits.delta` / `misses.delta` / `evictions.delta` | Cache traffic since the last tick |-| `traceway.sourcemap.load_failures.delta` | Failed source map loads (storage error, 5s timeout, unparseable map) since the last tick; affected traces are stored unsymbolicated |-| `traceway.sourcemap.parse_ms` | Duration of the most recent source map parse |+| `traceway.sourcemap.load_failures.delta` | Failed resolver builds (storage error, 5s timeout, unparseable map) since the last tick; affected traces are stored unsymbolicated |+| `traceway.sourcemap.not_found.delta` | Lookups since the last tick where no map was uploaded for the frame's filename; sustained nonzero for a project that should symbolicate usually means a missing CI upload step |+| `traceway.sourcemap.negative_hits.delta` | Frame lookups skipped by an active cooldown since the last tick; high values are normal for JS projects that do not upload maps |+| `traceway.sourcemap.parse_ms` | Duration of the most recent resolver build |## Source map cache sizing-Source maps are cached parsed, in a single LRU keyed by file. Only the fields the resolver uses are kept (mappings, source names, embedded source contents); the raw `.map` bytes are held only while parsing. Concurrent requests for the same map share one storage read and one parse, so a given map exists in memory at most once. A parsed map is typically 2-5x its raw size; the cache cap is on the estimated in-memory size, not the raw file size.+Symbolication resolves stack frames against object storage at `sourcemaps/{projectId}/{filename}`, with one in-memory LRU of built resolvers in front of it. A resolver keeps only a flat token table plus interned file and function names; the raw `.map` and bundle bytes are held only while building. Concurrent requests for the same map share one storage read and one build, so a given map exists in memory at most once. The cache cap is on the estimated in-memory size, not the raw file size.++Lookups that find no map (or fail to load one) are negative-cached per filename with an escalating cooldown: a missing map starts at 1 minute, a transient failure at 15 seconds, and each consecutive failure doubles the cooldown up to a 15 minute cap. Uploading a file clears its cooldown and evicts any cached resolver on the instance that handled the upload; in multi-instance deployments the other instances retry within their remaining cooldown. This keeps JS projects that never upload maps from generating a steady stream of storage reads, and keeps a storage outage from adding read-timeout latency to every exception report.++Upgrading from 1.7? The flat storage layout is new in 1.8 and existing maps are migrated automatically at startup; see [Upgrading to 1.8](/server/upgrade-1-8).| Variable | Default | Description ||----------|---------|-------------|-| `SOURCEMAP_CACHE_MAX_ENTRIES` | `200` | Max parsed source maps kept in memory |-| `SOURCEMAP_CACHE_MAX_BYTES_MB` | `500` | Max estimated in-memory size of all parsed maps |+| `SOURCEMAP_CACHE_MAX_ENTRIES` | `200` | Max built resolvers kept in memory |+| `SOURCEMAP_CACHE_MAX_BYTES_MB` | `500` | Max estimated in-memory size of all resolvers |Sizing guidance:- **Small box (1-2 GB RAM):** `SOURCEMAP_CACHE_MAX_BYTES_MB=100`.-- **Plenty of RAM, many active source maps:** raise `SOURCEMAP_CACHE_MAX_BYTES_MB` until `traceway.sourcemap.misses.delta` stays near zero in steady state; every miss is a storage read plus a re-parse, which burns CPU and briefly doubles that map's memory footprint.-- Watch `traceway.proc.rss_mb` against your host RAM; if it climbs after exception views with stack traces, the source map cache is the usual suspect.+- **Plenty of RAM, many active source maps:** raise `SOURCEMAP_CACHE_MAX_BYTES_MB` until `traceway.sourcemap.misses.delta` stays near zero in steady state; every miss is a storage read plus a rebuild, which burns CPU and briefly doubles that map's memory footprint.+- Watch `traceway.proc.rss_mb` against your host RAM; if it climbs during exception ingest with stack traces, the source map cache is the usual suspect.## Alerting on these metrics@@ -122,6 +129,7 @@ Two more rule types complete the picture:| `traceway.ch.disk.used_pcnt` | max | gt | 80 | 15 | warning | ClickHouse disk filling up || `traceway.ch.pool.open` | avg | gte | 15 | 10 | warning | Connection pool saturated — check `insert_ms` next || `traceway.recordings.dropped.delta` | max | gt | 0 | 5 | warning | Recording segments being dropped — ingestion overload |+| `traceway.sourcemap.load_failures.delta` | max | gt | 0 | 10 | warning | Source map loads failing (storage errors/timeouts); traces stored unsymbolicated || `cpu.used_pcnt` | avg | gt | 80 | 10 | warning | Backend CPU-bound || `traceway.proc.rss_mb` | avg | gt | 80% of host RAM | 10 | warning | Memory pressure / OOM risk (e.g. 800 on a 1 GB box) || `traceway.sourcemap.bytes` | avg | gt | ~half of `SOURCEMAP_CACHE_MAX_BYTES_MB`×1MB | 30 | info | Source map cache growing, revisit sizing |diff --git a/docs/pages/server/upgrade-1-8.mdx b/docs/pages/server/upgrade-1-8.mdxnew file mode 100644index 00000000..9198e83f--- /dev/null+++ b/docs/pages/server/upgrade-1-8.mdx@@ -0,0 +1,52 @@+# Upgrading from 1.7 to 1.8++This guide covers the one breaking change in 1.8 that touches persisted data: the source map storage migration. If you do not use JS source maps, 1.8 is a drop-in upgrade and you can stop reading.++## What changed++1.7 stored uploaded source maps in object storage under per-version paths and tracked them in the database:++```+sourcemaps/{projectId}/{version}/{filename}+```++At ingest, the backend looked up the right set of maps by the `version` reported by the SDK, which meant the `--version` flag passed to the upload CLI had to match the `version` option passed to the SDK at runtime.++1.8 removes the version concept from source maps entirely. Maps live at a flat, deterministic key and are matched by the stack frame's filename alone:++```+sourcemaps/{projectId}/{filename}+```++The most recent upload of each filename wins. Content-hashed bundle names (the default in Vite, Next, Angular, and most bundlers) keep every build's maps distinct automatically. The SDK's `version` option still exists, but it is now plain metadata for filtering exceptions in the dashboard and has no effect on symbolication.++## The automatic storage migration++On first startup, 1.8 migrates existing maps to the flat layout. No operator action is required.++For each project that has source maps recorded in the database:++1. The most recent upload of each filename (across all versions) is copied to its flat key.+2. A flat key that already exists is never overwritten, so anything uploaded with the 1.8 CLI always wins over backfilled content, even if the migration retries after a partial failure.+3. Completion is recorded per project in the `source_map_flatten_migrations` table. Migrated projects are skipped on subsequent startups.+4. If any copy fails (for example a transient S3 error), the project is not marked migrated and the whole project is retried on the next startup. Failures are reported to your monitoring project as exceptions.++The migration **copies**, it does not move: the old `sourcemaps/{projectId}/{version}/...` objects are left in place. Once the backfill has completed for all projects (every project has a row in `source_map_flatten_migrations`), the versioned paths are dead data and can be deleted from your bucket or `STORAGE_PATH` to reclaim space.++## What this means for symbolication during the upgrade++If a project ships multiple app versions concurrently **and** uses stable bundle filenames (such as `app.js` rather than `app.[hash].js`), 1.7 could resolve each report against its own version's maps. 1.8 resolves every report against the most recent upload of that filename. Switch your bundler to content-hashed filenames to keep concurrent versions distinct.++## Update the upload CLI++Upgrade `@tracewayapp/sourcemap-upload` together with the backend:++- The `--version` flag is gone. Old CI scripts that still pass it keep working; the flag is ignored.+- The new CLI also uploads the minified `.js`/`.cjs`/`.mjs` bundle next to each map, which is what enables function names in resolved stack traces.+- Do not point the new CLI at a 1.7 backend: 1.7 stores its uploads as `unversioned`, and any SDK that reports a `version` will then match no maps and silently lose symbolication.++## Related pages++- [Symbolication](/learn/symbolication-js): the full resolution pipeline in 1.8+- [Source Maps](/client/js-sdk/sourcemap-upload): CLI usage and CI examples+- [Capacity & Self-Monitoring](/server/capacity-monitoring): cache sizing and the symbolication metricsdiff --git a/examples/devtesting-embedded/go.mod b/examples/devtesting-embedded/go.modindex 43865d40..96455a10 100644--- a/examples/devtesting-embedded/go.mod+++ b/examples/devtesting-embedded/go.mod@@ -47,6 +47,7 @@ require (github.com/cespare/xxhash/v2 v2.3.0 // indirectgithub.com/cloudwego/base64x v0.1.6 // indirectgithub.com/coreos/go-systemd/v22 v22.6.0 // indirect+ github.com/dop251/goja v0.0.0-20260607120635-348e6bea910d // indirectgithub.com/gabriel-vasile/mimetype v1.4.13 // indirectgithub.com/gin-contrib/sse v1.1.1 // indirectgithub.com/go-chi/chi/v5 v5.2.2 // indirect@@ -57,6 +58,7 @@ require (github.com/go-playground/locales v0.14.1 // indirectgithub.com/go-playground/universal-translator v0.18.1 // indirectgithub.com/go-playground/validator/v10 v10.30.2 // indirect+ github.com/go-sourcemap/sourcemap v2.1.3+incompatible // indirectgithub.com/goccy/go-json v0.10.6 // indirectgithub.com/goccy/go-yaml v1.19.2 // indirectgithub.com/golang-jwt/jwt/v5 v5.3.0 // indirectdiff --git a/examples/devtesting-embedded/go.sum b/examples/devtesting-embedded/go.sumindex c8415819..112647ef 100644--- a/examples/devtesting-embedded/go.sum+++ b/examples/devtesting-embedded/go.sum@@ -86,6 +86,8 @@ github.com/docker/go-connections v0.6.0 h1:LlMG9azAe1TqfR7sO+NJttz1gy6KO7VJBh+pMgithub.com/docker/go-connections v0.6.0/go.mod h1:AahvXYshr6JgfUJGdDCs2b5EZG/vmaMAntpSFH5BFKE=github.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4=github.com/docker/go-units v0.5.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=+github.com/dop251/goja v0.0.0-20260607120635-348e6bea910d h1:xbM5U2EvWKkHxzEQJ2DEn20FwolWZahuTnVHr6WL3Q4=+github.com/dop251/goja v0.0.0-20260607120635-348e6bea910d/go.mod h1:Sc+QOu1WruvaaeT/cxFez/pXHpI9ZDjg/E8QNfSVveI=github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=@@ -116,6 +118,8 @@ github.com/go-playground/universal-translator v0.18.1 h1:Bcnm0ZwsGyWbCzImXv+pAJngithub.com/go-playground/universal-translator v0.18.1/go.mod h1:xekY+UJKNuX9WP91TpwSH2VMlDf28Uj24BCp08ZFTUY=github.com/go-playground/validator/v10 v10.30.2 h1:JiFIMtSSHb2/XBUbWM4i/MpeQm9ZK2xqPNk8vgvu5JQ=github.com/go-playground/validator/v10 v10.30.2/go.mod h1:mAf2pIOVXjTEBrwUMGKkCWKKPs9NheYGabeB04txQSc=+github.com/go-sourcemap/sourcemap v2.1.3+incompatible h1:W1iEw64niKVGogNgBN3ePyLFfuisuzeidWPMPWmECqU=+github.com/go-sourcemap/sourcemap v2.1.3+incompatible/go.mod h1:F8jJfvm2KbVjc5NqelyYJmf/v5J0dwNLS2mL4sNA1Jg=github.com/goccy/go-json v0.10.6 h1:p8HrPJzOakx/mn/bQtjgNjdTcN+/S6FcG2CTtQOrHVU=github.com/goccy/go-json v0.10.6/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=github.com/goccy/go-yaml v1.19.2 h1:PmFC1S6h8ljIz6gMRBopkjP1TVT7xuwrButHID66PoM=diff --git a/frontend/src/lib/components/issues/stack-trace-card.svelte b/frontend/src/lib/components/issues/stack-trace-card.svelteindex f2a52f21..259264aa 100644--- a/frontend/src/lib/components/issues/stack-trace-card.svelte+++ b/frontend/src/lib/components/issues/stack-trace-card.svelte@@ -4,11 +4,12 @@import { getTimezone } from '$lib/state/timezone.svelte';import Button from '../ui/button/button.svelte';import { Archive, ChevronRight, ChevronDown } from 'lucide-svelte';- import { parseStackTrace } from '$lib/utils/stack-trace-parser';+ import { parseStackTrace, type StackFrame } from '$lib/utils/stack-trace-parser';interface Props {stackTrace: string;isMessage?: boolean;+ isJavaScript?: boolean;firstSeen?: string;lastSeen?: string;totalCount?: number;@@ -20,6 +21,7 @@let {stackTrace,isMessage = false,+ isJavaScript = false,firstSeen,lastSeen,totalCount,@@ -31,6 +33,7 @@const tz = $derived(timezone ?? getTimezone());const showStats = $derived(firstSeen && lastSeen && totalCount !== undefined);const parsed = $derived(parseStackTrace(stackTrace));+ const usePretty = $derived(isJavaScript && parsed.groups.length > 0);let expandedGroups = $state<Set<number>>(new Set());@@ -43,6 +46,23 @@}expandedGroups = next;}++ function formatFrame(frame: StackFrame) {+ const fn = (frame.functionName ?? '').replace(/\(\)\s*$/, '').trim() || '<anonymous>';+ const m = frame.location.match(/^(.*):(\d+):(\d+)$/);+ if (!m) {+ return { fn, dir: '', file: frame.location, lineCol: '', raw: frame.location };+ }+ const [, path, line, col] = m;+ const slash = path.lastIndexOf('/');+ return {+ fn,+ dir: slash >= 0 ? path.slice(0, slash + 1) : '',+ file: slash >= 0 ? path.slice(slash + 1) : path,+ lineCol: `:${line}:${col}`,+ raw: frame.location+ };+ }</script><Card.Root>@@ -71,58 +91,89 @@</div>{#if showStats}<Card.Description>- First seen: {formatDateTime(firstSeen!, { timezone: tz })} · Last seen: {formatDateTime(- lastSeen!,- { timezone: tz }- )} · Total occurrences: {totalCount}+ <span class="tabular-nums"+ >First seen: {formatDateTime(firstSeen!, { timezone: tz })} · Last seen: {formatDateTime(+ lastSeen!,+ { timezone: tz }+ )} · Total occurrences: {totalCount}</span+ ></Card.Description>{/if}</Card.Header><Card.Content>- <div class="overflow-x-auto rounded-md bg-muted p-4 font-mono text-sm whitespace-pre-wrap">- {#if parsed.errorMessage}<div>{parsed.errorMessage}</div>{/if}- {#each parsed.groups as group, i}- {#if group.type === 'app'}- <div>- {#if group.frame.functionName}- <div>{group.frame.functionName}</div>- {/if}- <div>{group.frame.location}</div>- </div>- {:else}- <div- class="-mb-4 cursor-pointer pt-2 text-muted-foreground select-none"- role="button"- tabindex="0"- onclick={() => toggleGroup(i)}- onkeydown={(e) => {- if (e.key === 'Enter' || e.key === ' ') {- e.preventDefault();- toggleGroup(i);- }- }}- >- <span class="inline-flex items-center">+ {#if usePretty}+ <div class="overflow-hidden rounded-lg border">+ {#if parsed.errorMessage}+ <div class="border-b bg-muted/60 px-4 py-3">+ <p class="font-mono text-sm font-medium break-words whitespace-pre-wrap text-foreground">+ {parsed.errorMessage}+ </p>+ </div>+ {/if}++ <ol role="list" class="divide-y divide-border">+ {#each parsed.groups as group, i}+ {#if group.type === 'app'}+ {@const f = formatFrame(group.frame)}+ <li+ class="flex flex-wrap items-baseline gap-x-2.5 gap-y-0.5 border-l-2 px-4 py-2.5 {i ===+ 0+ ? 'border-l-primary'+ : 'border-l-primary/35'}"+ >+ <div class="min-w-0 font-mono text-sm font-medium break-all text-foreground">+ {f.fn}+ </div>+ <div class="min-w-0 font-mono text-xs break-all text-muted-foreground tabular-nums" title={f.raw}>+ {f.dir}<span class="text-foreground/85">{f.file}</span>{f.lineCol}+ </div>+ </li>+ {:else}+ <li class="border-l-2 border-l-transparent">+ <button+ type="button"+ class="flex w-full items-center gap-1.5 px-4 py-2 text-left text-xs text-muted-foreground hover:bg-muted/60"+ onclick={() => toggleGroup(i)}+ >+ {#if expandedGroups.has(i)}+ <ChevronDown class="size-3.5 shrink-0" />+ {:else}+ <ChevronRight class="size-3.5 shrink-0" />+ {/if}+ <span class="tabular-nums"+ >{group.frames.length} library {group.frames.length === 1 ? 'frame' : 'frames'}</span+ >+ <span class="rounded bg-muted px-1.5 py-0.5 font-mono text-foreground/70"+ >{group.packageName}</span+ >+ </button>{#if expandedGroups.has(i)}- <ChevronDown class="mr-0.5 h-3.5 w-3.5" />- {:else}- <ChevronRight class="mr-0.5 h-3.5 w-3.5" />+ <ol role="list" class="divide-y divide-border/50 border-t border-border/50 bg-muted/20">+ {#each group.frames as frame}+ {@const f = formatFrame(frame)}+ <li class="flex flex-wrap items-baseline gap-x-2.5 gap-y-0.5 py-2 pr-4 pl-9">+ <div class="min-w-0 font-mono text-sm break-all text-muted-foreground">+ {f.fn}+ </div>+ <div+ class="min-w-0 font-mono text-xs break-all text-muted-foreground/70 tabular-nums"+ title={f.raw}+ >+ {f.dir}<span class="text-foreground/60">{f.file}</span>{f.lineCol}+ </div>+ </li>+ {/each}+ </ol>{/if}- {group.frames.length} library {group.frames.length === 1 ? 'frame' : 'frames'} ({group.packageName})- </span>- </div>- {#if expandedGroups.has(i)}- {#each group.frames as frame}- <div class="text-muted-foreground">- {#if frame.functionName}- <div>{frame.functionName}</div>- {/if}- <div>{frame.location}</div>- </div>- {/each}+ </li>{/if}- {/if}- {/each}+ {/each}+ </ol>+ </div>+ {:else}+ <div class="overflow-x-auto rounded-lg border bg-muted/40 p-4">+ <pre class="font-mono text-sm break-words whitespace-pre-wrap text-foreground">{stackTrace}</pre></div>+ {/if}</Card.Content></Card.Root>diff --git a/frontend/src/lib/state/projects.svelte.ts b/frontend/src/lib/state/projects.svelte.tsindex 7701d626..3a2161a9 100644--- a/frontend/src/lib/state/projects.svelte.ts+++ b/frontend/src/lib/state/projects.svelte.ts@@ -50,6 +50,12 @@ export function isJsFramework(fw: Framework): boolean {return JS_FRAMEWORKS.includes(fw);}+export const JS_LANGUAGES = ['webjs', 'nodejs', 'javascript', 'typescript'];++export function isJsLanguage(lang?: string | null): boolean {+ return !!lang && JS_LANGUAGES.includes(lang.toLowerCase());+}+export function isOtelFramework(fw: Framework): boolean {return fw === 'opentelemetry' || fw === 'hono';}diff --git a/frontend/src/routes/issues/[exceptionHash]/+page.svelte b/frontend/src/routes/issues/[exceptionHash]/+page.svelteindex 7490d6bf..64044b19 100644--- a/frontend/src/routes/issues/[exceptionHash]/+page.svelte+++ b/frontend/src/routes/issues/[exceptionHash]/+page.svelte@@ -6,7 +6,7 @@import { Button } from '$lib/components/ui/button';import { LoadingCircle } from '$lib/components/ui/loading-circle';import { ErrorDisplay } from '$lib/components/ui/error-display';- import { projectsState } from '$lib/state/projects.svelte';+ import { projectsState, isJsFramework, isJsLanguage } from '$lib/state/projects.svelte';import { StackTraceCard, EventCard, EventsTable, PageHeader } from '$lib/components/issues';import { toast } from 'svelte-sonner';import ArchiveConfirmationDialog from '$lib/components/archive-confirmation-dialog.svelte';@@ -35,6 +35,11 @@const exceptionHash = $derived(page.params.exceptionHash ?? '');const latestOccurrence = $derived(occurrences[0]);const isMessage = $derived(latestOccurrence?.isMessage ?? false);+ const isJavaScript = $derived(+ (projectsState.currentProject?.framework+ ? isJsFramework(projectsState.currentProject.framework)+ : false) || isJsLanguage(latestOccurrence?.attributes?.['telemetry.sdk.language'])+ );const hasMoreOccurrences = $derived(total > 10);const firstLineOfStackTrace = $derived(latestOccurrence?.stackTrace.split('\n')[0] || 'Exception');@@ -163,6 +168,7 @@<StackTraceCardstackTrace={latestOccurrence?.stackTrace ?? group.stackTrace}{isMessage}+ {isJavaScript}firstSeen={group.firstSeen}lastSeen={group.lastSeen}totalCount={group.count}diff --git a/frontend/src/routes/issues/[exceptionHash]/[exceptionId]/+page.svelte b/frontend/src/routes/issues/[exceptionHash]/[exceptionId]/+page.svelteindex b31f8cf3..6dae9f18 100644--- a/frontend/src/routes/issues/[exceptionHash]/[exceptionId]/+page.svelte+++ b/frontend/src/routes/issues/[exceptionHash]/[exceptionId]/+page.svelte@@ -2,7 +2,7 @@import { api } from '$lib/api';import { LoadingCircle } from "$lib/components/ui/loading-circle";import { ErrorDisplay } from "$lib/components/ui/error-display";- import { projectsState } from '$lib/state/projects.svelte';+ import { projectsState, isJsFramework, isJsLanguage } from '$lib/state/projects.svelte';import { getTimezone } from '$lib/state/timezone.svelte';import { formatDateTime } from '$lib/utils/formatters';import { StackTraceCard, EventCard, EventsTable, PageHeader } from '$lib/components/issues';@@ -31,6 +31,11 @@const isMessage = $derived(occurrence?.isMessage ?? false);+ const isJavaScript = $derived(+ (projectsState.currentProject?.framework+ ? isJsFramework(projectsState.currentProject.framework)+ : false) || isJsLanguage(occurrence?.attributes?.['telemetry.sdk.language'])+ );const firstLineOfStackTrace = $derived(occurrence?.stackTrace.split('\n')[0] || 'Exception');const hasMoreOccurrences = $derived(total > 10);const subtitleText = $derived(occurrence ? `Event from ${formatDateTime(occurrence.recordedAt, { timezone })}` : 'Loading...');@@ -169,6 +174,7 @@<StackTraceCardstackTrace={occurrence.stackTrace}{isMessage}+ {isJavaScript}bind:showArchiveDialog={showArchiveDialog}bind:archiving={archiving}/>