The only tool you need to know what is happening and how to fix it.
Go alertingexception-trackerissue-trackinglogsmetricsmonitoringobservability-platformopen-telemetryotelsession-replaytelemetrytraces
docs + better looking chips
ddux9419800babc8
diff --git a/docs/pages/learn/index.mdx b/docs/pages/learn/index.mdxindex 4e7a4fa9..f0572ef3 100644--- a/docs/pages/learn/index.mdx+++ b/docs/pages/learn/index.mdx@@ -1,42 +1,39 @@# Architecture Overview-Traceway organizes your application telemetry around two building blocks: **Traces** and **Issues**. Everything else — spans, attributes, metrics — enriches these two concepts.+Traceway's mental model is small. There's one underlying primitive — **spans** — and a handful of **materialized views** over those spans that make the dashboard fast.-## Building Blocks+## The primitive: spans-<img src="/architecture.png" alt="Traceway Architecture" style={{ maxWidth: '100%', height: 'auto' }} />--### Traces--A **Trace** is a timed unit of work. There are two specializations:+Every piece of trace data Traceway ingests is an OpenTelemetry span. Spans carry a name, a kind, a duration, attributes, events (exceptions live here), and parent/trace ids. The shape is OTel-native — Traceway speaks OTLP directly and doesn't impose its own trace model on top.-- **Endpoints** — HTTP requests, automatically created by middleware (e.g., `GET /api/users/:id`)-- **Tasks** — Background jobs, manually created in your code (e.g., cron jobs, queue workers)+[Learn more about spans →](/learn/spans)-Both carry duration, attributes, and spans. [Learn more →](/learn/traces)+## The materialized views-### Issues+On ingestion, Traceway classifies each span and writes a row into a dedicated table when the kind or attributes match a rule. These tables are the top-level dashboard concepts:-An **Issue** captures a problem — an exception, a panic, or an explicitly sent message. Issues can be **connected to a trace** via the trace's context, linking the error to the request or task that caused it.+- **Endpoints** — inbound HTTP entry points (`GET /api/users/:id`).+- **Tasks** — queue consumers, scheduled commands, anything that runs as a "thing that completes".+- **AI Traces** — any span carrying `gen_ai.*` attributes, root or child.+- **Issues** — exception events lifted off whichever entity owned the span.-Similar exceptions are automatically grouped by normalized stack trace hash. [Learn more →](/learn/issues)+Each table is indexed and shaped for one job: answering "what's slow?", "what's expensive?", "what's failing?" without scanning the full span store. They're projections, not parallel universes — the underlying span data is the source of truth.-### Attributes+[Trace classification rules and the views →](/learn/traces) · [Issues →](/learn/issues) · [AI Tracing →](/learn/ai-tracing)-**Attributes** are key-value metadata attached to traces and issues. Values are strings — if the string is valid JSON, the dashboard displays it with proper formatting.+## Root and non-root entities-Attributes flow from global scope to per-request scope. [Learn more →](/learn/attributes)+Endpoints, Tasks, and AI Traces all carry an `is_root` flag. A row is **non-root** when the span that produced it was triggered by another trace — a queue worker started by an HTTP request, an LLM call made inside that same handler, a downstream service hop. Non-root rows link back to the originating trace via the auto-derived `distributed_trace_id`, so the distributed-trace view shows the full picture across all the entities that share one OTel trace id.-### Spans+[More on root vs non-root →](/learn/traces#root-and-non-root-entities)-**Spans** are timed sub-operations within a trace. Use them to measure where time is spent — database queries, API calls, business logic steps.+## Enrichments-Spans display as a timeline within a trace in the dashboard. [Learn more →](/learn/spans)+- **Attributes** — key-value metadata on spans and the entities derived from them. Flow from global → per-request scope. [Details →](/learn/attributes)+- **Sessions** — user-session grouping for browser/mobile clients, separate from traces. [Details →](/learn/sessions)+- **Metrics** — time-series measurements, ingested via OTLP metrics. Independent of the trace pipeline. [Details →](/learn/metrics)+- **Logs** — OTLP logs ingestion, joinable to traces by trace id. [Details →](/learn/logs)-### Metrics--**Metrics** are time-series measurements of your application's health. System metrics (CPU, memory, goroutines) are collected automatically, and you can capture custom metrics with optional tags for filtering and grouping.--Metrics are displayed on a configurable dashboard with widget groups and multiple chart types. [Learn more →](/learn/metrics)+<img src="/architecture.png" alt="Traceway Architecture" style={{ maxWidth: '100%', height: 'auto' }} />-See [Data Flow](/learn/data-flow) for how these concepts connect during request processing, task execution, and metrics collection.+See [Data Flow](/learn/data-flow) for how these connect during request processing, task execution, and metrics collection.diff --git a/docs/pages/learn/spans.mdx b/docs/pages/learn/spans.mdxindex 25070b04..8fad2a42 100644--- a/docs/pages/learn/spans.mdx+++ b/docs/pages/learn/spans.mdx@@ -1,44 +1,44 @@# Spans-Spans are timed sub-operations within a trace. They show where time is spent during request processing or background task execution.+Spans are the underlying primitive Traceway stores. Everything else — Endpoints, Tasks, AI Traces, Issues — is a [materialized view](/learn/traces#spans-are-the-primitive) derived from the spans you ingest. The shape is OpenTelemetry-native, so your existing OTel instrumentation works without translation.-## What Spans Capture+## What a span captures-Each span records:+| Field | Description |+| --- | --- |+| `span_id` | 8-byte OTel span id, stored as the low 8 bytes of a UUID |+| `trace_id` | The id of the owning entity (endpoint, task, or ai_trace) — see "Owning entity" below |+| `parent_span_id` | The span's direct parent within the trace, if any |+| `name` | Descriptive label (`db.users.find`, `POST https://api.stripe.com/v1/charges`, etc.) |+| `kind` | One of OTel's `SERVER`, `CLIENT`, `INTERNAL`, `PRODUCER`, `CONSUMER` |+| `start_time` / `duration` | When it started and how long it ran |+| `attributes` | Arbitrary key-value metadata (HTTP, DB, AI, custom) |+| `events` | Timestamped events on the span — `"exception"` events become [Issues](/learn/issues) |-| Field | Description |-| ---------- | ------------------------------------ |-| ID | Unique span identifier |-| Name | Descriptive name you provide |-| Start Time | When the span started |-| Duration | Time elapsed (calculated on `End()`) |+## Owning entity, not OTel trace id-## Use Cases+Each row in the `spans` table carries a `trace_id` column — but in Traceway it points to the **owning entity** (an endpoint, task, or ai_trace id), not the raw OTel trace id. This is what makes the waterfall view cheap:-Spans are useful for timing:+- "Show me the children of this endpoint" is a single indexed read on `spans.trace_id = <endpoint.id>`.+- The full cross-process trace is still reachable via the entity's `distributed_trace_id`, which equals the OTel trace id.-- Database queries-- External API calls-- Cache operations-- File I/O-- Business logic steps+Re-rooting happens at ingestion. When a non-root entity (e.g. a CONSUMER task) is promoted, its descendant spans get the task's id as their `trace_id`. Sibling spans under the originating endpoint keep the endpoint's id. So a single OTel trace that produced an endpoint + a task ends up with two clean waterfalls — one rooted at the endpoint, one at the task — joinable through `distributed_trace_id`.-## Viewing Spans+## What spans are good for-In the dashboard:+Spans tell you **where the time went**. If an endpoint takes 500ms and 450ms of that is in a `db.users.find` span, you know what to optimize. The dashboard's drill-down — Endpoints / Tasks / AI Traces → instance → waterfall — bottoms out in spans.-1. Navigate to the **Endpoints** or **Tasks** page-2. Click on an entry to see its details-3. Individual traces show their spans as a timeline-4. Each span displays its name and duration+Common things to wrap in a span:-<img src="/spans.png" alt="Spans Timeline" style={{ maxWidth: '100%', height: 'auto', marginTop: '16px', borderRadius: '8px', border: '1px solid #e5e7eb' }} />--This helps identify bottlenecks — if a request takes 500ms and 450ms is in a database span, you know where to optimize.+- Database queries (auto-instrumented in most stacks)+- External HTTP calls (`http.client` auto-instrumentation)+- Cache operations+- File I/O+- Business-logic steps you want to time-## Naming Conventions+## Naming-Use hierarchical, descriptive names for easy filtering:+Hierarchical names group well in the dashboard's filters:```db.users.find@@ -49,15 +49,21 @@ cache.redis.getlogic.pricing.calculate```-The pattern `category.resource.action` keeps span names consistent and searchable across your codebase.+`category.resource.action` is a useful default — pick something and stay consistent.## Nesting-Spans can not be nested. The nesting is achieved purely by the start time of the child spans:+In Traceway's storage, span nesting is reconstructed from `parent_span_id`. The waterfall view renders children indented under their parent in chronological order, exactly the way the OTel spec defines a trace tree.```-Total: 200ms-├── process.checkout.validate: 20ms to 150ms-├── process.checkout.charge: 150ms to 160ms-└── process.checkout.fulfill: 160ms to 200ms+endpoint POST /api/checkout: 200ms+├── validate.cart 20ms+├── pricing.calculate 80ms+│ └── db.products.batch 40ms+├── stripe.charge 60ms+└── db.orders.insert 30ms```++## Manual spans++The Traceway SDKs and any standard OTel SDK can create spans manually. See the per-framework client docs for the exact API; the underlying shape is always the same OTel span model described above.diff --git a/docs/pages/learn/traces.mdx b/docs/pages/learn/traces.mdxindex bde89c43..5e317bad 100644--- a/docs/pages/learn/traces.mdx+++ b/docs/pages/learn/traces.mdx@@ -1,70 +1,100 @@# Traces-A **Trace** is a timed unit of work in your application. Every HTTP request and background job that Traceway records is a trace.+A **trace** is the lifeline of one logical operation as it moves through your system — a request, a queued job, a scheduled command, an LLM call. Traceway models traces the same way OpenTelemetry does: a tree of **spans** identified by a shared trace id.-## Trace Types+## Spans are the primitive-### Endpoints--Endpoints are traces created automatically by middleware for HTTP requests. Each endpoint is identified by its HTTP method and route pattern:+Spans are the ground truth. Every other concept in Traceway — Endpoints, Tasks, AI Traces, Issues — is **derived from spans on ingestion**, then stored in a dedicated table so the dashboard can answer "what's slow?" or "what's expensive?" in milliseconds without scanning every span.```-GET /api/users/:id-POST /api/orders-DELETE /api/sessions/:token+OTLP span batch+ │+ ▼+┌───────────────────────────┐+│ classify by kind + attrs │ ← promotion rules+└───────────────────────────┘+ │+ ├──► endpoints ─┐+ ├──► tasks ─┤ ←─ materialized views,+ ├──► ai_traces ─┤ indexed for fast aggregation+ ├──► exception_… ─┘+ │+ └──► spans (everything else, the raw waterfall)```-Endpoints are ordered in the dashboard by **Impact**, which factors in both performance and error frequency.+You can think of `endpoints`, `tasks`, and `ai_traces` as **cached projections** of the spans table. They exist for one reason: aggregations like "P95 of every endpoint over 7 days" or "total LLM cost grouped by trace_name" need to be cheap. Scanning the full spans table to compute those every dashboard load would not be cheap.-<img src="/endpoints.png" alt="Endpoints Dashboard" style={{ maxWidth: '100%', height: 'auto', marginTop: '16px', borderRadius: '8px', border: '1px solid #e5e7eb' }} />+The trade-off this design makes:+- Top-level dashboards (Endpoints, Tasks, AI Traces) are **fast** — they read a small, pre-aggregated table.+- The detail/waterfall view falls back to `spans` for the full timeline of children under any entity.+- Promotion happens **at write time**, not at query time, so the rules are baked in once per span.-The middleware captures:+## What gets promoted-| Field | Description |-| ----------- | ---------------------------- |-| Endpoint | HTTP method + route template |-| Duration | Request processing time |-| Status Code | HTTP response status |-| Body Size | Response body size in bytes |-| Client IP | Request origin IP address |-| Attributes | Key-value metadata |-| Spans | Timed sub-operations |+A span lands in a top-level table when its kind or attributes match a rule. The classifier doesn't care whether the span is a root or a child — it cares about what the span represents.-### Tasks+| Condition on the OTel span | Promoted to |+|---|---|+| `SpanKind = SERVER` / `INTERNAL` + HTTP attributes, and the span is an inbound entry point (root, or parent lives in a different process) | **Endpoint** |+| `SpanKind = CONSUMER` (queue / messenger workers, root or child) | **Task** |+| Root `INTERNAL` span with a `console.command` attribute (artisan / artisan-style runners) | **Task** |+| Any span carrying `gen_ai.*` attributes (root or child) | **AI Trace** |+| `"exception"` event on any span | **Issue** (attributed to the owning entity above) |+| Anything else with a parent | **Span** in the waterfall, re-rooted to its nearest enclosing entity |-Tasks are traces created manually for background operations that run outside HTTP requests:+In-process sub-handler spans (e.g. a framework's "handler /path" INTERNAL+HTTP child) stay in the spans table — endpoints are only promoted for actual entry points, so frameworks don't silently double-emit.-- Cron jobs-- Queue workers-- Data migrations-- Scheduled reports+## Root and non-root entities-Tasks carry the same data as endpoints (duration, attributes, spans) but without HTTP-specific fields like status code or client IP.+Each Endpoint / Task / AI Trace row carries an `is_root` flag.-<img src="/tasks.png" alt="Tasks Dashboard" style={{ maxWidth: '100%', height: 'auto', marginTop: '16px', borderRadius: '8px', border: '1px solid #e5e7eb' }} />+- **Root** — the span had no parent in any process. This is the typical case: an inbound HTTP request, a standalone cron run, a fresh LLM call.+- **Non-root** — the span was triggered by another trace. Examples:+ - A queue worker's CONSUMER span parented to the dispatcher's PRODUCER span via the trace context serialized into the job payload.+ - A child gen_ai span made inside a request handler.+ - A cross-service inbound HTTP hop where the upstream service propagated `traceparent`.++Non-root rows surface a **Non-root** chip in the dashboard list. On the row's detail page, a **View distributed trace** link jumps to the full picture across all the entities sharing that trace id.-## Trace Lifecycle+The id rules avoid collisions when a root and a non-root entity share the same trace id:+- Root entity id = OTel trace id (so the request and its lineage are easy to look up by trace id).+- Non-root entity id = the entity's own span id (distinct from the root's id even within the same trace).-1. A trace is created — either automatically by the client sdk (endpoints) or manually by starting a trace or measuring a task-2. During execution, spans and attributes are added to the trace via context-3. If an error occurs, an issue is captured and linked to the trace via its ID-4. When the work completes, the trace is recorded with its final duration and all collected data+## Distributed traces-## Trace Identity+Every Endpoint / Task / AI Trace stores a **`distributed_trace_id`** equal to the OTel trace id, auto-derived on ingestion. So:-Each trace gets a unique UUID. This ID links:+- A request that dispatches a queue job and the worker that processes it share the same `distributed_trace_id`. They show up as two rows — the endpoint and the task — and a single page (`/distributed-traces/<id>`) renders them side by side with their spans.+- A request handler that calls an LLM produces an endpoint row and an ai_trace row with the same `distributed_trace_id`. Same cross-trace page.+- Cross-service HTTP hops (each service exports its own SERVER span) all share the trace id. Each hop appears as its own endpoint row.-- The trace record itself (endpoint name, duration, status)-- Any spans created during execution-- Any issues (exceptions/messages) captured during the trace+You don't need to set any vendor attribute for this to work — OTLP's native trace_id is enough. (The Go SDK's manual `distributedTraceId` field continues to work for legacy `/api/report` flows and overrides the auto-derived value when set.)-When you view an issue in the dashboard, you can navigate to the trace that caused it, and vice versa.+## What each entity stores-## Sampling+The shape mirrors what's useful for the dashboard's aggregations on that entity.-Not every trace needs to be recorded. Traceway supports two sample rates:+### Endpoints+Identified by HTTP method + route template (`GET /api/users/:id`). Ranked by **Impact** — a composite of Apdex, error rate, P99, client-error rate, and volume-weighted error rate. Streaming responses (SSE / WebSocket) are flagged and excluded from latency-based components.-- **Sample rate** — Controls what percentage of normal traces are recorded (default: 100%)-- **Error sample rate** — Controls what percentage of error traces are recorded (default: 100%)+### Tasks+Identified by a task name (job class, scheduled command, agent operation). Ranked by count × (P95 − P50) as a rough impact proxy. CONSUMER spans from any queue driver (database, Redis, SQS, Beanstalk, …) land here, as do root `console.command` spans.-Error traces (exceptions, panics, status >= 500, errors) use the error sample rate, which you typically keep at 100% to capture all problems.+### AI Traces+Identified by `trace.name` (your agent or workflow). Carries model, provider, input/output/cached/reasoning tokens, input/output/total cost, finish reason, and a pointer to the full conversation in object storage. Ranked by total cost by default.++## Lifecycle and sampling++1. A span is created in your app (by a framework's auto-instrumentation, the Traceway SDK, or manual OTel calls).+2. Child spans, attributes, and events accumulate against it.+3. The trace is exported via OTLP (or `/api/report` for the legacy Go SDK).+4. Traceway classifies each span, inserts into the matching entity table, and re-roots the remaining children into `spans` under their nearest enclosing entity.+5. Exceptions recorded as `"exception"` events become Issues attributed to the owning entity.++Sampling:+- **Sample rate** — percentage of normal traces recorded (default: 100%).+- **Error sample rate** — percentage of error traces recorded (default: 100%). Keep this at 100% to catch every failure.++<img src="/endpoints.png" alt="Endpoints Dashboard" style={{ maxWidth: '100%', height: 'auto', marginTop: '16px', borderRadius: '8px', border: '1px solid #e5e7eb' }} />++<img src="/tasks.png" alt="Tasks Dashboard" style={{ maxWidth: '100%', height: 'auto', marginTop: '16px', borderRadius: '8px', border: '1px solid #e5e7eb' }} />diff --git a/frontend/src/lib/components/ui/non-root-chip/non-root-chip.svelte b/frontend/src/lib/components/ui/non-root-chip/non-root-chip.svelteindex 65d04df8..adc6856c 100644--- a/frontend/src/lib/components/ui/non-root-chip/non-root-chip.svelte+++ b/frontend/src/lib/components/ui/non-root-chip/non-root-chip.svelte@@ -8,7 +8,7 @@let { mixed = false }: Props = $props();- const label = $derived(mixed ? 'Mixed' : 'Non-root');+ const label = $derived(mixed ? 'Mixed' : 'Non Root');const tooltip = $derived(mixed? 'Some runs were triggered by another trace — see the distributed trace view.'@@ -19,7 +19,10 @@<Tooltip.Provider><Tooltip.Root><Tooltip.Trigger>- <Badge variant="secondary" class="ml-2 text-[10px]">+ <Badge+ variant="outline"+ class="ml-2 text-xs border-amber-500 bg-amber-50 text-amber-700 dark:border-yellow-500 dark:bg-transparent dark:text-yellow-400"+ >{label}</Badge></Tooltip.Trigger>