Skip to content

The only tool you need to know what is happening and how to fix it.

docs + better looking chips

ddux9419800babc8
diff --git a/docs/pages/learn/index.mdx b/docs/pages/learn/index.mdx
index 4e7a4fa9..f0572ef3 100644
--- a/docs/pages/learn/index.mdx
+++ b/docs/pages/learn/index.mdx
@@ -1,42 +1,39 @@
# Architecture Overview
-Traceway organizes your application telemetry around two building blocks: **Traces** and **Issues**. Everything else — spans, attributes, metrics — enriches these two concepts.
+Traceway's mental model is small. There's one underlying primitive — **spans** — and a handful of **materialized views** over those spans that make the dashboard fast.
-## Building Blocks
+## The primitive: spans
-<img src="/architecture.png" alt="Traceway Architecture" style={{ maxWidth: '100%', height: 'auto' }} />
-
-### Traces
-
-A **Trace** is a timed unit of work. There are two specializations:
+Every piece of trace data Traceway ingests is an OpenTelemetry span. Spans carry a name, a kind, a duration, attributes, events (exceptions live here), and parent/trace ids. The shape is OTel-native — Traceway speaks OTLP directly and doesn't impose its own trace model on top.
-- **Endpoints** — HTTP requests, automatically created by middleware (e.g., `GET /api/users/:id`)
-- **Tasks** — Background jobs, manually created in your code (e.g., cron jobs, queue workers)
+[Learn more about spans →](/learn/spans)
-Both carry duration, attributes, and spans. [Learn more →](/learn/traces)
+## The materialized views
-### Issues
+On ingestion, Traceway classifies each span and writes a row into a dedicated table when the kind or attributes match a rule. These tables are the top-level dashboard concepts:
-An **Issue** captures a problem — an exception, a panic, or an explicitly sent message. Issues can be **connected to a trace** via the trace's context, linking the error to the request or task that caused it.
+- **Endpoints** — inbound HTTP entry points (`GET /api/users/:id`).
+- **Tasks** — queue consumers, scheduled commands, anything that runs as a "thing that completes".
+- **AI Traces** — any span carrying `gen_ai.*` attributes, root or child.
+- **Issues** — exception events lifted off whichever entity owned the span.
-Similar exceptions are automatically grouped by normalized stack trace hash. [Learn more →](/learn/issues)
+Each table is indexed and shaped for one job: answering "what's slow?", "what's expensive?", "what's failing?" without scanning the full span store. They're projections, not parallel universes — the underlying span data is the source of truth.
-### Attributes
+[Trace classification rules and the views →](/learn/traces) · [Issues →](/learn/issues) · [AI Tracing →](/learn/ai-tracing)
-**Attributes** are key-value metadata attached to traces and issues. Values are strings — if the string is valid JSON, the dashboard displays it with proper formatting.
+## Root and non-root entities
-Attributes flow from global scope to per-request scope. [Learn more →](/learn/attributes)
+Endpoints, Tasks, and AI Traces all carry an `is_root` flag. A row is **non-root** when the span that produced it was triggered by another trace — a queue worker started by an HTTP request, an LLM call made inside that same handler, a downstream service hop. Non-root rows link back to the originating trace via the auto-derived `distributed_trace_id`, so the distributed-trace view shows the full picture across all the entities that share one OTel trace id.
-### Spans
+[More on root vs non-root →](/learn/traces#root-and-non-root-entities)
-**Spans** are timed sub-operations within a trace. Use them to measure where time is spent — database queries, API calls, business logic steps.
+## Enrichments
-Spans display as a timeline within a trace in the dashboard. [Learn more →](/learn/spans)
+- **Attributes** — key-value metadata on spans and the entities derived from them. Flow from global → per-request scope. [Details →](/learn/attributes)
+- **Sessions** — user-session grouping for browser/mobile clients, separate from traces. [Details →](/learn/sessions)
+- **Metrics** — time-series measurements, ingested via OTLP metrics. Independent of the trace pipeline. [Details →](/learn/metrics)
+- **Logs** — OTLP logs ingestion, joinable to traces by trace id. [Details →](/learn/logs)
-### Metrics
-
-**Metrics** are time-series measurements of your application's health. System metrics (CPU, memory, goroutines) are collected automatically, and you can capture custom metrics with optional tags for filtering and grouping.
-
-Metrics are displayed on a configurable dashboard with widget groups and multiple chart types. [Learn more →](/learn/metrics)
+<img src="/architecture.png" alt="Traceway Architecture" style={{ maxWidth: '100%', height: 'auto' }} />
-See [Data Flow](/learn/data-flow) for how these concepts connect during request processing, task execution, and metrics collection.
+See [Data Flow](/learn/data-flow) for how these connect during request processing, task execution, and metrics collection.
diff --git a/docs/pages/learn/spans.mdx b/docs/pages/learn/spans.mdx
index 25070b04..8fad2a42 100644
--- a/docs/pages/learn/spans.mdx
+++ b/docs/pages/learn/spans.mdx
@@ -1,44 +1,44 @@
# Spans
-Spans are timed sub-operations within a trace. They show where time is spent during request processing or background task execution.
+Spans are the underlying primitive Traceway stores. Everything else — Endpoints, Tasks, AI Traces, Issues — is a [materialized view](/learn/traces#spans-are-the-primitive) derived from the spans you ingest. The shape is OpenTelemetry-native, so your existing OTel instrumentation works without translation.
-## What Spans Capture
+## What a span captures
-Each span records:
+| Field | Description |
+| --- | --- |
+| `span_id` | 8-byte OTel span id, stored as the low 8 bytes of a UUID |
+| `trace_id` | The id of the owning entity (endpoint, task, or ai_trace) — see "Owning entity" below |
+| `parent_span_id` | The span's direct parent within the trace, if any |
+| `name` | Descriptive label (`db.users.find`, `POST https://api.stripe.com/v1/charges`, etc.) |
+| `kind` | One of OTel's `SERVER`, `CLIENT`, `INTERNAL`, `PRODUCER`, `CONSUMER` |
+| `start_time` / `duration` | When it started and how long it ran |
+| `attributes` | Arbitrary key-value metadata (HTTP, DB, AI, custom) |
+| `events` | Timestamped events on the span — `"exception"` events become [Issues](/learn/issues) |
-| Field | Description |
-| ---------- | ------------------------------------ |
-| ID | Unique span identifier |
-| Name | Descriptive name you provide |
-| Start Time | When the span started |
-| Duration | Time elapsed (calculated on `End()`) |
+## Owning entity, not OTel trace id
-## Use Cases
+Each row in the `spans` table carries a `trace_id` column — but in Traceway it points to the **owning entity** (an endpoint, task, or ai_trace id), not the raw OTel trace id. This is what makes the waterfall view cheap:
-Spans are useful for timing:
+- "Show me the children of this endpoint" is a single indexed read on `spans.trace_id = <endpoint.id>`.
+- The full cross-process trace is still reachable via the entity's `distributed_trace_id`, which equals the OTel trace id.
-- Database queries
-- External API calls
-- Cache operations
-- File I/O
-- Business logic steps
+Re-rooting happens at ingestion. When a non-root entity (e.g. a CONSUMER task) is promoted, its descendant spans get the task's id as their `trace_id`. Sibling spans under the originating endpoint keep the endpoint's id. So a single OTel trace that produced an endpoint + a task ends up with two clean waterfalls — one rooted at the endpoint, one at the task — joinable through `distributed_trace_id`.
-## Viewing Spans
+## What spans are good for
-In the dashboard:
+Spans tell you **where the time went**. If an endpoint takes 500ms and 450ms of that is in a `db.users.find` span, you know what to optimize. The dashboard's drill-down — Endpoints / Tasks / AI Traces → instance → waterfall — bottoms out in spans.
-1. Navigate to the **Endpoints** or **Tasks** page
-2. Click on an entry to see its details
-3. Individual traces show their spans as a timeline
-4. Each span displays its name and duration
+Common things to wrap in a span:
-<img src="/spans.png" alt="Spans Timeline" style={{ maxWidth: '100%', height: 'auto', marginTop: '16px', borderRadius: '8px', border: '1px solid #e5e7eb' }} />
-
-This helps identify bottlenecks — if a request takes 500ms and 450ms is in a database span, you know where to optimize.
+- Database queries (auto-instrumented in most stacks)
+- External HTTP calls (`http.client` auto-instrumentation)
+- Cache operations
+- File I/O
+- Business-logic steps you want to time
-## Naming Conventions
+## Naming
-Use hierarchical, descriptive names for easy filtering:
+Hierarchical names group well in the dashboard's filters:
```
db.users.find
@@ -49,15 +49,21 @@ cache.redis.get
logic.pricing.calculate
```
-The pattern `category.resource.action` keeps span names consistent and searchable across your codebase.
+`category.resource.action` is a useful default — pick something and stay consistent.
## Nesting
-Spans can not be nested. The nesting is achieved purely by the start time of the child spans:
+In Traceway's storage, span nesting is reconstructed from `parent_span_id`. The waterfall view renders children indented under their parent in chronological order, exactly the way the OTel spec defines a trace tree.
```
-Total: 200ms
-├── process.checkout.validate: 20ms to 150ms
-├── process.checkout.charge: 150ms to 160ms
-└── process.checkout.fulfill: 160ms to 200ms
+endpoint POST /api/checkout: 200ms
+├── validate.cart 20ms
+├── pricing.calculate 80ms
+│ └── db.products.batch 40ms
+├── stripe.charge 60ms
+└── db.orders.insert 30ms
```
+
+## Manual spans
+
+The Traceway SDKs and any standard OTel SDK can create spans manually. See the per-framework client docs for the exact API; the underlying shape is always the same OTel span model described above.
diff --git a/docs/pages/learn/traces.mdx b/docs/pages/learn/traces.mdx
index bde89c43..5e317bad 100644
--- a/docs/pages/learn/traces.mdx
+++ b/docs/pages/learn/traces.mdx
@@ -1,70 +1,100 @@
# Traces
-A **Trace** is a timed unit of work in your application. Every HTTP request and background job that Traceway records is a trace.
+A **trace** is the lifeline of one logical operation as it moves through your system — a request, a queued job, a scheduled command, an LLM call. Traceway models traces the same way OpenTelemetry does: a tree of **spans** identified by a shared trace id.
-## Trace Types
+## Spans are the primitive
-### Endpoints
-
-Endpoints are traces created automatically by middleware for HTTP requests. Each endpoint is identified by its HTTP method and route pattern:
+Spans are the ground truth. Every other concept in Traceway — Endpoints, Tasks, AI Traces, Issues — is **derived from spans on ingestion**, then stored in a dedicated table so the dashboard can answer "what's slow?" or "what's expensive?" in milliseconds without scanning every span.
```
-GET /api/users/:id
-POST /api/orders
-DELETE /api/sessions/:token
+OTLP span batch
+ │
+ ▼
+┌───────────────────────────┐
+│ classify by kind + attrs │ ← promotion rules
+└───────────────────────────┘
+ │
+ ├──► endpoints ─┐
+ ├──► tasks ─┤ ←─ materialized views,
+ ├──► ai_traces ─┤ indexed for fast aggregation
+ ├──► exception_… ─┘
+ │
+ └──► spans (everything else, the raw waterfall)
```
-Endpoints are ordered in the dashboard by **Impact**, which factors in both performance and error frequency.
+You can think of `endpoints`, `tasks`, and `ai_traces` as **cached projections** of the spans table. They exist for one reason: aggregations like "P95 of every endpoint over 7 days" or "total LLM cost grouped by trace_name" need to be cheap. Scanning the full spans table to compute those every dashboard load would not be cheap.
-<img src="/endpoints.png" alt="Endpoints Dashboard" style={{ maxWidth: '100%', height: 'auto', marginTop: '16px', borderRadius: '8px', border: '1px solid #e5e7eb' }} />
+The trade-off this design makes:
+- Top-level dashboards (Endpoints, Tasks, AI Traces) are **fast** — they read a small, pre-aggregated table.
+- The detail/waterfall view falls back to `spans` for the full timeline of children under any entity.
+- Promotion happens **at write time**, not at query time, so the rules are baked in once per span.
-The middleware captures:
+## What gets promoted
-| Field | Description |
-| ----------- | ---------------------------- |
-| Endpoint | HTTP method + route template |
-| Duration | Request processing time |
-| Status Code | HTTP response status |
-| Body Size | Response body size in bytes |
-| Client IP | Request origin IP address |
-| Attributes | Key-value metadata |
-| Spans | Timed sub-operations |
+A span lands in a top-level table when its kind or attributes match a rule. The classifier doesn't care whether the span is a root or a child — it cares about what the span represents.
-### Tasks
+| Condition on the OTel span | Promoted to |
+|---|---|
+| `SpanKind = SERVER` / `INTERNAL` + HTTP attributes, and the span is an inbound entry point (root, or parent lives in a different process) | **Endpoint** |
+| `SpanKind = CONSUMER` (queue / messenger workers, root or child) | **Task** |
+| Root `INTERNAL` span with a `console.command` attribute (artisan / artisan-style runners) | **Task** |
+| Any span carrying `gen_ai.*` attributes (root or child) | **AI Trace** |
+| `"exception"` event on any span | **Issue** (attributed to the owning entity above) |
+| Anything else with a parent | **Span** in the waterfall, re-rooted to its nearest enclosing entity |
-Tasks are traces created manually for background operations that run outside HTTP requests:
+In-process sub-handler spans (e.g. a framework's "handler /path" INTERNAL+HTTP child) stay in the spans table — endpoints are only promoted for actual entry points, so frameworks don't silently double-emit.
-- Cron jobs
-- Queue workers
-- Data migrations
-- Scheduled reports
+## Root and non-root entities
-Tasks carry the same data as endpoints (duration, attributes, spans) but without HTTP-specific fields like status code or client IP.
+Each Endpoint / Task / AI Trace row carries an `is_root` flag.
-<img src="/tasks.png" alt="Tasks Dashboard" style={{ maxWidth: '100%', height: 'auto', marginTop: '16px', borderRadius: '8px', border: '1px solid #e5e7eb' }} />
+- **Root** — the span had no parent in any process. This is the typical case: an inbound HTTP request, a standalone cron run, a fresh LLM call.
+- **Non-root** — the span was triggered by another trace. Examples:
+ - A queue worker's CONSUMER span parented to the dispatcher's PRODUCER span via the trace context serialized into the job payload.
+ - A child gen_ai span made inside a request handler.
+ - A cross-service inbound HTTP hop where the upstream service propagated `traceparent`.
+
+Non-root rows surface a **Non-root** chip in the dashboard list. On the row's detail page, a **View distributed trace** link jumps to the full picture across all the entities sharing that trace id.
-## Trace Lifecycle
+The id rules avoid collisions when a root and a non-root entity share the same trace id:
+- Root entity id = OTel trace id (so the request and its lineage are easy to look up by trace id).
+- Non-root entity id = the entity's own span id (distinct from the root's id even within the same trace).
-1. A trace is created — either automatically by the client sdk (endpoints) or manually by starting a trace or measuring a task
-2. During execution, spans and attributes are added to the trace via context
-3. If an error occurs, an issue is captured and linked to the trace via its ID
-4. When the work completes, the trace is recorded with its final duration and all collected data
+## Distributed traces
-## Trace Identity
+Every Endpoint / Task / AI Trace stores a **`distributed_trace_id`** equal to the OTel trace id, auto-derived on ingestion. So:
-Each trace gets a unique UUID. This ID links:
+- A request that dispatches a queue job and the worker that processes it share the same `distributed_trace_id`. They show up as two rows — the endpoint and the task — and a single page (`/distributed-traces/<id>`) renders them side by side with their spans.
+- A request handler that calls an LLM produces an endpoint row and an ai_trace row with the same `distributed_trace_id`. Same cross-trace page.
+- Cross-service HTTP hops (each service exports its own SERVER span) all share the trace id. Each hop appears as its own endpoint row.
-- The trace record itself (endpoint name, duration, status)
-- Any spans created during execution
-- Any issues (exceptions/messages) captured during the trace
+You don't need to set any vendor attribute for this to work — OTLP's native trace_id is enough. (The Go SDK's manual `distributedTraceId` field continues to work for legacy `/api/report` flows and overrides the auto-derived value when set.)
-When you view an issue in the dashboard, you can navigate to the trace that caused it, and vice versa.
+## What each entity stores
-## Sampling
+The shape mirrors what's useful for the dashboard's aggregations on that entity.
-Not every trace needs to be recorded. Traceway supports two sample rates:
+### Endpoints
+Identified by HTTP method + route template (`GET /api/users/:id`). Ranked by **Impact** — a composite of Apdex, error rate, P99, client-error rate, and volume-weighted error rate. Streaming responses (SSE / WebSocket) are flagged and excluded from latency-based components.
-- **Sample rate** — Controls what percentage of normal traces are recorded (default: 100%)
-- **Error sample rate** — Controls what percentage of error traces are recorded (default: 100%)
+### Tasks
+Identified by a task name (job class, scheduled command, agent operation). Ranked by count × (P95 − P50) as a rough impact proxy. CONSUMER spans from any queue driver (database, Redis, SQS, Beanstalk, …) land here, as do root `console.command` spans.
-Error traces (exceptions, panics, status >= 500, errors) use the error sample rate, which you typically keep at 100% to capture all problems.
+### AI Traces
+Identified by `trace.name` (your agent or workflow). Carries model, provider, input/output/cached/reasoning tokens, input/output/total cost, finish reason, and a pointer to the full conversation in object storage. Ranked by total cost by default.
+
+## Lifecycle and sampling
+
+1. A span is created in your app (by a framework's auto-instrumentation, the Traceway SDK, or manual OTel calls).
+2. Child spans, attributes, and events accumulate against it.
+3. The trace is exported via OTLP (or `/api/report` for the legacy Go SDK).
+4. Traceway classifies each span, inserts into the matching entity table, and re-roots the remaining children into `spans` under their nearest enclosing entity.
+5. Exceptions recorded as `"exception"` events become Issues attributed to the owning entity.
+
+Sampling:
+- **Sample rate** — percentage of normal traces recorded (default: 100%).
+- **Error sample rate** — percentage of error traces recorded (default: 100%). Keep this at 100% to catch every failure.
+
+<img src="/endpoints.png" alt="Endpoints Dashboard" style={{ maxWidth: '100%', height: 'auto', marginTop: '16px', borderRadius: '8px', border: '1px solid #e5e7eb' }} />
+
+<img src="/tasks.png" alt="Tasks Dashboard" style={{ maxWidth: '100%', height: 'auto', marginTop: '16px', borderRadius: '8px', border: '1px solid #e5e7eb' }} />
diff --git a/frontend/src/lib/components/ui/non-root-chip/non-root-chip.svelte b/frontend/src/lib/components/ui/non-root-chip/non-root-chip.svelte
index 65d04df8..adc6856c 100644
--- a/frontend/src/lib/components/ui/non-root-chip/non-root-chip.svelte
+++ b/frontend/src/lib/components/ui/non-root-chip/non-root-chip.svelte
@@ -8,7 +8,7 @@
let { mixed = false }: Props = $props();
- const label = $derived(mixed ? 'Mixed' : 'Non-root');
+ const label = $derived(mixed ? 'Mixed' : 'Non Root');
const tooltip = $derived(
mixed
? 'Some runs were triggered by another trace — see the distributed trace view.'
@@ -19,7 +19,10 @@
<Tooltip.Provider>
<Tooltip.Root>
<Tooltip.Trigger>
- <Badge variant="secondary" class="ml-2 text-[10px]">
+ <Badge
+ variant="outline"
+ class="ml-2 text-xs border-amber-500 bg-amber-50 text-amber-700 dark:border-yellow-500 dark:bg-transparent dark:text-yellow-400"
+ >
{label}
</Badge>
</Tooltip.Trigger>