Skip to content

The only tool you need to know what is happening and how to fix it.

progress

dduxf97ed864184f
diff --git a/SPEC_NON_ROOT_TRACES.md b/SPEC_NON_ROOT_TRACES.md
new file mode 100644
index 00000000..1730c97f
--- /dev/null
+++ b/SPEC_NON_ROOT_TRACES.md
@@ -0,0 +1,296 @@
+# Non-Root Tasks & AI Traces
+
+## Problem
+
+`backend/app/controllers/otelcontrollers/trace_converter.go:98` classifies a span as an Endpoint, Task, or AiTrace only when `isRoot := len(span.ParentSpanId) == 0`. Three real-world cases violate that assumption:
+
+1. **Queue jobs with propagated trace context.** keepsuit's `QueueInstrumentation::recordJobProcessing` (`vendor/keepsuit/laravel-opentelemetry/src/Instrumentation/QueueInstrumentation.php:106`) — and the equivalents in `opentelemetry-instrumentation-celery`, Symfony Messenger without `messenger_root_spans: true`, etc. — calls `setParent($context)` on the worker's CONSUMER span using the trace context serialized into the job payload by the dispatcher. The consumer span has `parent_span_id = <producer span id>` from the originating request, so `isRoot == false`. It falls through to the generic `spans` table and the Tasks page stays empty.
+2. **Child LLM spans.** A request span (root) makes an LLM call as a child span. The child has `gen_ai.*` attributes but a non-empty `parent_span_id`, so it never reaches the `aiTraces = append(...)` branch and the AI Traces page stays empty.
+3. **Scheduled console commands.** keepsuit's `ConsoleInstrumentation` (`ConsoleInstrumentation.php:77-79`) emits a root span with kind INTERNAL (the OTel default — no `setSpanKind` call) and the `console.command` attribute. It has no HTTP attrs, so today it falls into the `else { continue }` branch and is dropped. zentigo runs `schedule:run`, `bookings:expire-unconfirmed`, and `bookings:send-reminders` under this instrumentation — all silently dropped.
+
+All three are the same line. The fix promotes spans by their Kind / attributes instead of by their root-ness, and adds enough schema and UI to make the resulting non-root rows distinguishable from real roots.
+
+## Approach
+
+Two new columns on the entity tables, one ingestion change, and one query-time change:
+
+- **`is_root Bool`** on `endpoints` / `tasks` / `ai_traces` — explicit because once `distributed_trace_id` is auto-filled from the OTel trace_id, both the root and its non-root siblings carry the same `distributed_trace_id` and "is_root" is no longer derivable from `distributed_trace_id IS NULL`.
+- **`distributed_trace_id Nullable(UUID)`** on `ai_traces` (already exists on the other three) — populated from the OTel trace_id, not a vendor attribute, so it works for any OTLP source out of the box.
+- **Converter promotes by Kind/attrs**, not by root-ness. Non-root entities get an id derived from their own `span_id` (not the shared OTel trace_id) so they don't collide with the root entity's id.
+- **Spans are re-rooted at ingestion to their nearest enclosing entity within the batch.** `spans.trace_id` (and `exception_stack_traces.trace_id`) is repurposed from "OTel trace_id" to "owning entity id" so each waterfall query stays clean.
+
+### Decisions
+
+- **Drop the `isRoot` gate on _all_ classifications, including endpoints.** Non-root SERVER+HTTP spans (cross-service inbound hops) become endpoint rows with `is_root=false`. The "Non-root" chip applies to endpoints too.
+- **Promote console spans to tasks.** Detect via the `console.command` attribute keepsuit attaches. Root INTERNAL spans without HTTP attrs that carry `console.command` become task rows.
+- **Keep `rootFilter` on the endpoints API/page** — endpoints can now legitimately be non-root.
+- **Tasks search ships in the same PR.** Today the tasks page has no search; this change adds it alongside `rootFilter`.
+
+### Worked example (queue case)
+
+OTel trace_id `X`, spans:
+- `A` — SERVER, root, HTTP attrs — request handler
+- `B` — PRODUCER, parent=`A` — `send queue`
+- `C` — CONSUMER, parent=`B` — `process queue` (arrives in a separate worker OTLP batch)
+- `D` — child of `C` — `SELECT …` (arrives in the worker's batch)
+
+Resulting rows:
+
+| | `id` | `distributed_trace_id` | `is_root` |
+|---|---|---|---|
+| endpoints | `otelTraceIDToUUID(A.trace_id)` = **X** | **X** | true |
+| tasks | `otelSpanIDToUUID(C.span_id)` = **Y** | **X** | false |
+| spans | `B`'s span_id, `trace_id = X` (owner = endpoint) | — | — |
+| spans | `D`'s span_id, `trace_id = Y` (owner = task) | — | — |
+
+`SELECT spans WHERE trace_id = X` returns `B` only. `SELECT spans WHERE trace_id = Y` returns `D` only. `FindByDistributedTraceId(X)` returns both the endpoint and the task.
+
+### Worked example (inline gen_ai case)
+
+`A` is a SERVER root, `L` is a child of `A` with `gen_ai.*` attrs. They arrive in the same batch.
+
+| | `id` | `distributed_trace_id` | `is_root` |
+|---|---|---|---|
+| endpoints | `X` (from `A.trace_id`) | `X` | true |
+| ai_traces | `otelSpanIDToUUID(L.span_id)` = `Z` | `X` | false |
+
+Spans between `A` and `L` (e.g. middleware/DB spans) get `trace_id = X` (owner = endpoint). Anything that turns out to be a descendant of `L` (rare — LLM spans are usually leaves; tool-call children would be the case) gets `trace_id = Z`.
+
+### Worked example (scheduled console command)
+
+`bookings:send-reminders` runs under keepsuit's `ConsoleInstrumentation`. The span has kind INTERNAL, no parent, no HTTP attrs, and attribute `console.command = "bookings:send-reminders"`. Children include the database queries it issues.
+
+| | `id` | `distributed_trace_id` | `is_root` |
+|---|---|---|---|
+| tasks | `otelTraceIDToUUID(trace_id)` = **X** | **X** | true |
+| spans | each DB child, `trace_id = X` (owner = task) | — | — |
+
+---
+
+## Backend
+
+### Migrations
+
+ClickHouse — one statement per file (per project rule). Next free number is 0058:
+
+- `0058_add_is_root_to_endpoints.up.sql`
+ ```sql
+ ALTER TABLE endpoints ADD COLUMN is_root Bool DEFAULT true
+ ```
+- `0059_add_is_root_to_tasks.up.sql`
+ ```sql
+ ALTER TABLE tasks ADD COLUMN is_root Bool DEFAULT true
+ ```
+- `0060_add_distributed_trace_id_to_ai_traces.up.sql`
+ ```sql
+ ALTER TABLE ai_traces ADD COLUMN distributed_trace_id Nullable(UUID) DEFAULT NULL
+ ```
+- `0061_add_distributed_trace_id_index_ai_traces.up.sql`
+ ```sql
+ ALTER TABLE ai_traces ADD INDEX idx_distributed_trace_id_ai_traces distributed_trace_id TYPE bloom_filter(0.001) GRANULARITY 1
+ ```
+- `0062_add_is_root_to_ai_traces.up.sql`
+ ```sql
+ ALTER TABLE ai_traces ADD COLUMN is_root Bool DEFAULT true
+ ```
+
+SQLite (`backend/app/migrations/sqlite_telemetry/`, next number is 0009). The SQLite migration runner splits on `;` (see `migrations_sqlite.go:80`), so a single multi-statement file is fine:
+
+- `0009_add_is_root_and_distributed_trace_id.up.sql`
+ ```sql
+ ALTER TABLE endpoints ADD COLUMN is_root INTEGER NOT NULL DEFAULT 1;
+ ALTER TABLE tasks ADD COLUMN is_root INTEGER NOT NULL DEFAULT 1;
+ ALTER TABLE ai_traces ADD COLUMN is_root INTEGER NOT NULL DEFAULT 1;
+ ALTER TABLE ai_traces ADD COLUMN distributed_trace_id TEXT DEFAULT NULL;
+ ```
+
+> Default `true`/`1` so backfill is correct — every existing row was promoted under the old `isRoot` gate, so it was a root span by definition.
+
+### Models
+
+`backend/app/models/endpoint.model.go`, `task.model.go`, `ai_trace.model.go`:
+
+- Add `IsRoot bool \`json:"isRoot" ch:"is_root"\``
+- For `AiTrace`, also add `DistributedTraceId *uuid.UUID \`json:"distributedTraceId,omitempty" ch:"distributed_trace_id"\``
+
+Aggregate result models — `TaskStats`, `AiTraceStats`, `EndpointStats` (`models/*.go` and inline in repositories) gain `HasRoot bool` and `HasNonRoot bool` so the grouped list page can show the chip on names that contain at least one non-root run.
+
+### Converter (`backend/app/controllers/otelcontrollers/trace_converter.go`)
+
+Rewrite the per-span classification block (currently lines 92–178). Two passes inside the existing `for _, rs := range req.ResourceSpans` loop.
+
+**Pass 1 — classify each span as one of `endpoint`, `task`, `ai_trace`, or `span`.** Drop the `isRoot` gate from all three entity branches. Rule table:
+
+| Condition | Promoted to |
+|---|---|
+| `Kind ∈ {SERVER, INTERNAL}` AND `hasHTTPAttributes(attrs)` | endpoint |
+| `Kind == CONSUMER` | task |
+| `Kind == INTERNAL` AND attribute `console.command` is present AND `isRoot` | task |
+| `hasGenAiAttributes(attrs)` | ai_trace |
+| otherwise | (handled in pass 2) |
+
+For each promoted entity, compute its id:
+- `is_root = len(span.ParentSpanId) == 0`
+- if root: `id = otelTraceIDToUUID(span.TraceId)` (preserves today's invariant that the root entity's id equals the OTel trace_id)
+- if non-root: `id = otelSpanIDToUUID(span.SpanId)` (distinct from any root entity sharing the same trace)
+- `distributed_trace_id = otelTraceIDToUUID(span.TraceId)` for **both** root and non-root (this is the change from "vendor attribute only" to "auto-derived"). The vendor attribute `traceway.distributed_trace_id` still wins if present, for backwards compatibility with manually-linked traces from the Go SDK.
+
+Build `spanIdToEntityId map[string]uuid.UUID` from these promotions (keyed by `string(span.SpanId)`).
+
+> Console rule rationale: keepsuit's `ConsoleInstrumentation.commandStarting` sets `console.command` explicitly (`ConsoleInstrumentation.php:78`). We don't use "any root INTERNAL with no HTTP attrs" because it would scoop up arbitrary manual roots from other instrumentations. `console.command` is a concrete signal.
+
+**Pass 2 — emit span rows for the unpromoted spans, re-rooted to their nearest enclosing entity.**
+
+For each unpromoted span, walk parents via `parentMap` until you find one in `spanIdToEntityId`. That entity's id becomes the span's `trace_id`. If no enclosing entity exists in this batch, fall back to `otelTraceIDToUUID(span.TraceId)` (preserves today's behavior for orphan spans whose parent was emitted from another process and never matched any promoted entity — they remain linkable via `distributed_trace_id` on the cross-trace page).
+
+Skip emitting a span row for any span that was itself promoted to an entity (today, only root entities are skipped; the new logic skips all promoted entities including non-root ones).
+
+**Exceptions.** `exc.TraceId` is set to the owning entity's id (the same value the span's `trace_id` would have gotten), not the raw OTel trace_id. `traceType` is `"endpoint"` / `"task"` / `"ai_trace"` based on which kind owned it. Exceptions on unpromoted spans use the entity walked up the parent chain.
+
+### Repositories
+
+Files to update — every read/write that touches `endpoints`, `tasks`, `ai_traces`, `exception_stack_traces`, or `spans`. Each has a ClickHouse variant (`*.repository.go`) and a SQLite variant (`*.repository_sqlite.go`).
+
+- `endpoint.repository.go` + `endpoint.repository_sqlite.go`:
+ - `BatchInsert` — add `is_root` to the INSERT column list and `Append` call.
+ - `FindAll`, `FindGroupedByEndpoint`, `FindByEndpoint`, `FindById`, `FindByDistributedTraceId` — add `is_root` to SELECT and Scan.
+ - Add `rootFilter` parameter to `FindGroupedByEndpoint` (`all` | `root` | `non_root`); when set to `root` add `AND is_root = true`, when `non_root` add `AND is_root = false`. Grouped projection includes `has_root = max(is_root)` and `has_non_root = max(NOT is_root)` so the list row can show the chip if any run in the group is non-root.
+- `task.repository.go` + `task.repository_sqlite.go`:
+ - Same as endpoints: `BatchInsert`, all reads, root filter on `FindGroupedByTaskName`, `has_root` / `has_non_root` in the grouped projection.
+ - Add `search` parameter (substring match on `task_name` — `positionCaseInsensitive` in ClickHouse, `LIKE` lowercased in SQLite). Currently tasks have no search; this brings them in line with endpoints.
+ - Add `FindByDistributedTraceId(ctx, distributedTraceId, projectIds)` — mirrors `EndpointRepository.FindByDistributedTraceId`.
+- `ai_trace.repository.go` + `ai_trace.repository_sqlite.go`:
+ - `BatchInsert` — add `is_root` and `distributed_trace_id` columns.
+ - All reads — add `is_root` and `distributed_trace_id` to SELECT/Scan.
+ - Same `rootFilter` parameter + grouped projection as tasks.
+ - Add `FindByDistributedTraceId` (mirrors `EndpointRepository.FindByDistributedTraceId`).
+- `exception_stack_trace.repository.go` + `_sqlite`:
+ - No schema change here — `trace_id` already exists. Only the value being stored changes (owning entity id, not raw OTel trace_id). Existing callers don't need to change.
+- `span.repository.go` + `_sqlite`:
+ - No schema change. `FindByTraceId(projectId, traceId)` still works — it will now be called with `endpoint.id` or `task.id` or `ai_trace.id` (the owning entity id), all of which are valid values to look up.
+
+### Controllers
+
+- `tasks.controller.go` — `FindGrouped` accepts new request fields `search string` and `rootFilter string`. Forward both to the repository. Emit `hasRoot` / `hasNonRoot` per row in the response.
+- `ai_trace.controller.go` — same change as tasks (search already exists, just add `rootFilter` + `hasRoot` / `hasNonRoot`).
+- `endpoints.controller.go` — `FindGrouped` accepts new `rootFilter` field. (Search already exists.) Emit `hasRoot` / `hasNonRoot` per row.
+- `distributed_trace.controller.go` — add a third call to `repositories.AiTraceRepository.FindByDistributedTraceId` and emit `ai_trace` nodes in the response alongside `endpoint` and `task`. Render in the same `Nodes` list.
+- Detail controllers (`endpoint.controller.go`, `task.controller.go`, `ai_trace.controller.go`) — include `isRoot` and `distributedTraceId` in their JSON responses so the detail pages can render the chip and the cross-trace link.
+
+### Tests
+
+`trace_converter_test.go` additions:
+
+1. CONSUMER span with non-empty `ParentSpanId` →
+ - one row in `tasks` with `is_root == false`, `id == otelSpanIDToUUID(span.SpanId)`, `distributed_trace_id == otelTraceIDToUUID(span.TraceId)`
+ - no row in `endpoints`
+ - any child spans in the batch land in `spans` with `trace_id == task.id`
+2. Child span with `gen_ai.*` attrs under a SERVER root, both in one batch →
+ - root endpoint row with `is_root == true`, `id == distributed_trace_id == otelTraceIDToUUID(trace_id)`
+ - ai_trace row with `is_root == false`, `id == otelSpanIDToUUID(child.SpanId)`, same `distributed_trace_id` as the endpoint
+ - intermediate spans (if any) get `trace_id == endpoint.id`; descendants of the ai_trace span get `trace_id == ai_trace.id`
+3. Root INTERNAL span carrying `console.command` attribute, no HTTP attrs →
+ - one row in `tasks` with `is_root == true`, `id == otelTraceIDToUUID(span.TraceId)`
+ - child spans get `trace_id == task.id`
+4. Non-root SERVER+HTTP span (cross-service inbound hop) →
+ - one row in `endpoints` with `is_root == false`, `id == otelSpanIDToUUID(span.SpanId)`
+5. Existing test — root SERVER + HTTP → endpoint with `is_root == true` — must still pass unchanged.
+6. Exception event on the consumer span → `exc.TraceId == task.id`, `exc.TraceType == "task"`.
+7. Orphan span whose parent isn't in the batch and which isn't itself promoted → falls back to `trace_id = otelTraceIDToUUID(span.TraceId)` (the OTel trace_id). Document this fallback in the test name.
+
+Repository tests (`endpoint_repository_test.go`, `task_repository_test.go`, the soon-to-exist `ai_trace_repository_test.go`):
+- `is_root` round-trips through insert + select.
+- `rootFilter` parameter narrows results correctly (`all` / `root` / `non_root`).
+- Grouped queries surface `has_root` and `has_non_root`.
+- Task `search` parameter filters by task name substring.
+
+---
+
+## Frontend
+
+### Shared component
+
+New `frontend/src/lib/components/ui/root-filter/root-filter.svelte` — a `Select.Root` matching the existing inline-select styling, dropped via the SearchBar's `children` snippet so it joins the bar visually (mirroring how `issues/+page.svelte` uses `typeOptions`).
+
+Options:
+```ts
+const rootFilterOptions = [
+ { value: 'all', label: 'All' },
+ { value: 'root', label: 'Root' },
+ { value: 'non_root', label: 'Non-root' }
+];
+```
+
+URL param `rootFilter` (`all` is the default and is omitted from the URL, matching how `searchType` is handled on the issues page).
+
+### Chip
+
+New `frontend/src/lib/components/ui/non-root-chip/non-root-chip.svelte` — small `Badge variant="secondary"` reading `Non-root` (or `Mixed` when both `hasRoot` and `hasNonRoot` are true), with tooltip "Triggered by another trace — see the distributed trace view." Rendered:
+- On Endpoint, Task, and AI Trace list rows where `hasNonRoot && !hasRoot` ("Non-root") or where both flags are true ("Mixed").
+- On detail pages (`/endpoints/[endpoint]/[id]`, `/tasks/[task]/[id]`, `/ai-traces/[traceName]/[id]`) next to the title when the specific instance being displayed has `isRoot === false`.
+
+### `frontend/src/routes/tasks/+page.svelte`
+
+Currently has no search. Add:
+- `searchQuery` state + URL persistence (`?search=…`), copying the pattern from `ai-traces/+page.svelte` lines 62–112.
+- `rootFilter` state + URL persistence (`?rootFilter=root` etc.).
+- `<SearchBar placeholder="Search tasks..." bind:value={searchQuery} onSearch={handleSearch}>` wrapping the new `<RootFilter bind:value={rootFilter} />` in its `children` snippet.
+- Send `search` and `rootFilter` in the `/tasks/grouped` POST body.
+- Render `<NonRootChip mixed={task.hasRoot && task.hasNonRoot} />` next to `task.taskName` when `task.hasNonRoot` is true.
+
+`/tasks/[task]/+page.svelte` (instance list): pass `rootFilter` through to `/tasks/task` and render the chip per-row using each instance's `isRoot`.
+
+### `frontend/src/routes/ai-traces/+page.svelte`
+
+Already has search. Add:
+- `rootFilter` state + URL persistence.
+- Pass the filter via the SearchBar `children` snippet (same as tasks).
+- Send `rootFilter` in the API call.
+- Render `<NonRootChip>` next to `trace.traceName` when `trace.hasNonRoot`.
+
+`/ai-traces/[traceName]/+page.svelte` — chip per instance row using `aiTrace.isRoot`.
+
+### `frontend/src/routes/endpoints/+page.svelte`
+
+- `rootFilter` state + URL persistence (endpoints can now be non-root after this change).
+- Render `<NonRootChip>` next to the endpoint name when `endpoint.hasNonRoot`.
+
+`/endpoints/[endpoint]/+page.svelte` — chip per instance row using `endpoint.isRoot`.
+
+### Distributed-trace page
+
+Wherever `DistributedTraceResponse.Nodes` are rendered, add a case for `traceType === "ai_trace"` so the new third entity type shows up alongside endpoints and tasks. Reuse the existing node card styling.
+
+### Detail pages — cross-trace link
+
+On `/tasks/[task]/[id]`, `/ai-traces/[traceName]/[id]`, and `/endpoints/[endpoint]/[id]`, when `distributedTraceId` is set and there are sibling entities, show a "View distributed trace" link to `/distributed-traces/<distributedTraceId>`. (Today this link is only meaningful when the vendor attribute was set; after this change it's meaningful for every OTel-sourced trace.)
+
+---
+
+## Verification
+
+1. **Unit/integration tests** — `cd backend && go test ./...` covers converter + repo round-trips on both ClickHouse and SQLite paths.
+2. **End-to-end against a real Laravel/keepsuit app (zentigo is the canonical reproduction):**
+ - Build backend on this branch, point the app at it via `OTEL_EXPORTER_OTLP_ENDPOINT`.
+ - **Queue task (non-root):** `php artisan tinker --execute='\App\Jobs\TestTracewayJob::dispatch();'`, then run a queue worker. Confirm a new row appears on `/tasks` with the **Non-root** chip. Open it and confirm the "View distributed trace" link reaches a page showing both endpoint (dispatcher request) and task (worker) nodes.
+ - **Console task (root):** `php artisan bookings:send-reminders`. Confirm a Tasks row with the command name, no chip (root).
+ - **HTTP+queue together:** trigger a request that dispatches a notification (e.g. `NewReviewNotification`). On the endpoint detail page confirm the cross-trace link; on the distributed-trace view confirm both nodes.
+ - **AI trace (non-root):** easiest is a synthetic OTLP payload (`curl` a hand-rolled JSON with a SERVER root and a child span carrying `gen_ai.system="openai"` etc.). Confirm the AI Traces page shows the child with the chip and that the cross-trace view includes endpoint + ai_trace nodes.
+3. **`rootFilter` UI** — toggle the dropdown on `/tasks`, `/ai-traces`, `/endpoints` and confirm URL persistence (`?rootFilter=non_root` etc., `all` omitted from URL).
+4. **Backfill sanity** — after migration run, query `SELECT count(), sum(is_root) FROM endpoints / tasks / ai_traces`; both numbers should match (every row defaults to `is_root=1`).
+
+---
+
+## Migration / Backfill Notes
+
+- All existing rows default to `is_root = true` — correct because the old converter only ever inserted root spans.
+- `ai_traces.distributed_trace_id` is `NULL` on existing rows. That's fine — the cross-trace lookup degrades gracefully and there's no UI that requires it to be present on historical rows.
+- The change to `spans.trace_id` semantics (owning entity id, not OTel trace_id) is forward-only — existing span rows still have OTel trace_id stored there, which equals the endpoint's id for the single-process case (the only case the old converter produced), so existing waterfalls keep working.
+
+---
+
+## Out of Scope
+
+- Cross-service SERVER hops where you'd want one node per hop in the cross-trace view — endpoints now accept non-root SERVER spans, but the UX for showing per-hop endpoint nodes (e.g., a tree view of cross-service requests) is a separate decision.
+- Reconciling the legacy `/api/report` distributed-trace flow with the new auto-derivation. The Go SDK's opt-in `distributedTraceId` field on `clientmodels` continues to work unchanged.
+- A "show me only the root trace" view at the distributed-trace page — current behavior shows everything, which is what users want for debugging.
diff --git a/backend/app/controllers/ai_trace.controller.go b/backend/app/controllers/ai_trace.controller.go
index f141dc67..7fbf5398 100644
--- a/backend/app/controllers/ai_trace.controller.go
+++ b/backend/app/controllers/ai_trace.controller.go
@@ -24,6 +24,7 @@ type AiTraceSearchRequest struct {
SortDirection string `json:"sortDirection"`
Pagination PaginationParams `json:"pagination"`
Search string `json:"search"`
+ RootFilter string `json:"rootFilter"`
}
type AiTraceInstancesRequest struct {
@@ -58,7 +59,7 @@ func (a aiTraceController) FindGroupedByTraceName(c *gin.Context) {
return
}
- stats, total, err := repositories.AiTraceRepository.FindGroupedByTraceName(c, projectId, request.FromDate, request.ToDate, request.Pagination.Page, request.Pagination.PageSize, request.OrderBy, request.SortDirection, request.Search)
+ stats, total, err := repositories.AiTraceRepository.FindGroupedByTraceName(c, projectId, request.FromDate, request.ToDate, request.Pagination.Page, request.Pagination.PageSize, request.OrderBy, request.SortDirection, request.Search, request.RootFilter)
if err != nil {
c.AbortWithError(500, traceway.NewStackTraceErrorf("error loading ai trace stats: %w", err))
return
diff --git a/backend/app/controllers/distributed_trace.controller.go b/backend/app/controllers/distributed_trace.controller.go
index 472b09bc..43cbfcb4 100644
--- a/backend/app/controllers/distributed_trace.controller.go
+++ b/backend/app/controllers/distributed_trace.controller.go
@@ -23,6 +23,7 @@ type DistributedTraceNode struct {
TraceType string `json:"traceType"`
Endpoint *models.Endpoint `json:"endpoint,omitempty"`
Task *models.Task `json:"task,omitempty"`
+ AiTrace *models.AiTrace `json:"aiTrace,omitempty"`
Spans []models.Span `json:"spans"`
Exception *EndpointExceptionInfo `json:"exception,omitempty"`
}
@@ -79,6 +80,12 @@ func (d distributedTraceController) GetDistributedTrace(c *gin.Context) {
return
}
+ aiTraces, err := repositories.AiTraceRepository.FindByDistributedTraceId(ctx, distributedTraceId, projectIds)
+ if err != nil {
+ c.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to query ai traces: %w", err))
+ return
+ }
+
exceptions, err := repositories.ExceptionStackTraceRepository.FindByDistributedTraceId(ctx, distributedTraceId, projectIds)
if err != nil {
c.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to query exceptions: %w", err))
@@ -105,6 +112,9 @@ func (d distributedTraceController) GetDistributedTrace(c *gin.Context) {
for _, t := range tasks {
matchedIds[t.Id] = true
}
+ for _, a := range aiTraces {
+ matchedIds[a.Id] = true
+ }
var nodes []DistributedTraceNode
@@ -132,6 +142,18 @@ func (d distributedTraceController) GetDistributedTrace(c *gin.Context) {
nodes = append(nodes, node)
}
+ for _, a := range aiTraces {
+ node := DistributedTraceNode{
+ ProjectId: a.ProjectId,
+ ProjectName: projectNameMap[a.ProjectId],
+ TraceType: "ai_trace",
+ AiTrace: &a,
+ Spans: []models.Span{},
+ Exception: exceptionByTraceId[a.Id],
+ }
+ nodes = append(nodes, node)
+ }
+
for _, exc := range exceptions {
if exc.TraceId != nil && matchedIds[*exc.TraceId] {
continue
diff --git a/backend/app/controllers/endpoint.controller.go b/backend/app/controllers/endpoint.controller.go
index 5c2bf4d1..7c90d62a 100644
--- a/backend/app/controllers/endpoint.controller.go
+++ b/backend/app/controllers/endpoint.controller.go
@@ -23,6 +23,7 @@ type EndpointSearchRequest struct {
SortDirection string `json:"sortDirection"`
Pagination PaginationParams `json:"pagination"`
Search string `json:"search"`
+ RootFilter string `json:"rootFilter"`
}
type EndpointInstancesRequest struct {
@@ -92,7 +93,7 @@ func (e endpointController) FindGroupedByEndpoint(c *gin.Context) {
}
span := traceway.StartSpan(c, "loading grouped endpoints")
- stats, total, err := repositories.EndpointRepository.FindGroupedByEndpoint(c, projectId, request.FromDate, request.ToDate, request.Pagination.Page, request.Pagination.PageSize, request.OrderBy, request.SortDirection, request.Search)
+ stats, total, err := repositories.EndpointRepository.FindGroupedByEndpoint(c, projectId, request.FromDate, request.ToDate, request.Pagination.Page, request.Pagination.PageSize, request.OrderBy, request.SortDirection, request.Search, request.RootFilter)
span.End()
if err != nil {
c.AbortWithError(500, traceway.NewStackTraceErrorf("error loading stats: %w", err))
diff --git a/backend/app/controllers/otelcontrollers/testdata/node_sign_in.json.golden.json b/backend/app/controllers/otelcontrollers/testdata/node_sign_in.json.golden.json
index 40da201c..22ca730f 100644
--- a/backend/app/controllers/otelcontrollers/testdata/node_sign_in.json.golden.json
+++ b/backend/app/controllers/otelcontrollers/testdata/node_sign_in.json.golden.json
@@ -36,7 +36,7 @@
"exceptions": [
{
"stackTrace": "APIError: Invalid email or password",
- "traceType": "task"
+ "traceType": "endpoint"
}
],
"aiTraceCount": 0,
diff --git a/backend/app/controllers/otelcontrollers/trace_converter.go b/backend/app/controllers/otelcontrollers/trace_converter.go
index dc540825..5202a4ba 100644
--- a/backend/app/controllers/otelcontrollers/trace_converter.go
+++ b/backend/app/controllers/otelcontrollers/trace_converter.go
@@ -20,6 +20,27 @@ type aiTraceConversation struct {
Content []byte
}
+type entityKind int
+
+const (
+ entityNone entityKind = iota
+ entityEndpoint
+ entityTask
+ entityAiTrace
+)
+
+func (k entityKind) traceType() string {
+ switch k {
+ case entityEndpoint:
+ return "endpoint"
+ case entityTask:
+ return "task"
+ case entityAiTrace:
+ return "ai_trace"
+ }
+ return ""
+}
+
func convertTraces(projectId uuid.UUID, req *coltracepb.ExportTraceServiceRequest) (
endpoints []models.Endpoint,
tasks []models.Task,
@@ -41,14 +62,13 @@ func convertTraces(projectId uuid.UUID, req *coltracepb.ExportTraceServiceReques
}
}
- // First pass: collect all spans and build parent→child relationships
type spanEntry struct {
span *tracepb.Span
scopeName string
}
var allSpans []spanEntry
- parentMap := map[string]string{} // childSpanId → parentSpanId
- spanById := map[string]*tracepb.Span{} // spanId → span (for trace_id lookup)
+ parentMap := map[string]string{}
+ spanById := map[string]*tracepb.Span{}
for _, ss := range rs.ScopeSpans {
for _, span := range ss.Spans {
allSpans = append(allSpans, spanEntry{span: span, scopeName: ss.GetScope().GetName()})
@@ -59,88 +79,157 @@ func convertTraces(projectId uuid.UUID, req *coltracepb.ExportTraceServiceReques
}
}
- // Build spanToTraceId. Prefer each span's native OTel trace_id so logs on
- // the wire (which carry trace_id as hex) can be joined against the
- // converted traces via traceIdUuidToHex(endpoint.id). Fall back to walking
- // parents + minting a UUID only when the OTel trace_id is missing or
- // malformed — an unusual case but worth handling defensively.
- spanToTraceId := map[string]uuid.UUID{}
- var resolveTraceId func(spanIdStr string) uuid.UUID
- resolveTraceId = func(spanIdStr string) uuid.UUID {
- if tid, ok := spanToTraceId[spanIdStr]; ok {
- return tid
+ // Pass 1: classify each span by Kind/attrs and assign an entity id.
+ // Roots get id = otelTraceIDToUUID(trace_id); non-roots get id = otelSpanIDToUUID(span_id).
+ // distributed_trace_id = otelTraceIDToUUID(trace_id) for both, unless overridden by
+ // the vendor `traceway.distributed_trace_id` attribute.
+ type promotion struct {
+ kind entityKind
+ id uuid.UUID
+ isRoot bool
+ distributedTraceId *uuid.UUID
+ }
+ spanIdToPromotion := map[string]promotion{}
+
+ for _, entry := range allSpans {
+ span := entry.span
+ kind := classifySpan(span, spanById)
+ if kind == entityNone {
+ continue
}
- if sp, ok := spanById[spanIdStr]; ok && len(sp.TraceId) > 0 {
- if tid := otelTraceIDToUUID(sp.TraceId); tid != uuid.Nil {
- spanToTraceId[spanIdStr] = tid
- return tid
+
+ isRoot := len(span.ParentSpanId) == 0
+
+ var id uuid.UUID
+ if isRoot {
+ id = otelTraceIDToUUID(span.TraceId)
+ if id == uuid.Nil {
+ id = otelSpanIDToUUID(span.SpanId)
+ }
+ } else {
+ id = otelSpanIDToUUID(span.SpanId)
+ if id == uuid.Nil {
+ id = uuid.New()
+ }
+ }
+
+ dtId := otelTraceIDToUUID(span.TraceId)
+ var distributedTraceId *uuid.UUID
+ if dtId != uuid.Nil {
+ distributedTraceId = &dtId
+ }
+ if override := getStringAttribute(span.Attributes, "traceway.distributed_trace_id"); override != "" {
+ if parsed, err := uuid.Parse(override); err == nil {
+ distributedTraceId = &parsed
}
}
- if parentId, hasParent := parentMap[spanIdStr]; hasParent {
- tid := resolveTraceId(parentId)
- spanToTraceId[spanIdStr] = tid
- return tid
+
+ spanIdToPromotion[string(span.SpanId)] = promotion{
+ kind: kind,
+ id: id,
+ isRoot: isRoot,
+ distributedTraceId: distributedTraceId,
}
- tid := uuid.New()
- spanToTraceId[spanIdStr] = tid
- return tid
}
- for _, entry := range allSpans {
- resolveTraceId(string(entry.span.SpanId))
+
+ // resolveOwner walks parents until it finds a promoted entity. Returns the
+ // promotion (so callers know the kind for trace_type) and whether one was
+ // found. Cached per span id within this resource batch.
+ ownerCache := map[string]*promotion{}
+ var resolveOwner func(spanIdStr string) *promotion
+ resolveOwner = func(spanIdStr string) *promotion {
+ if cached, ok := ownerCache[spanIdStr]; ok {
+ return cached
+ }
+ if p, ok := spanIdToPromotion[spanIdStr]; ok {
+ pp := p
+ ownerCache[spanIdStr] = &pp
+ return &pp
+ }
+ parentId, hasParent := parentMap[spanIdStr]
+ if !hasParent {
+ ownerCache[spanIdStr] = nil
+ return nil
+ }
+ owner := resolveOwner(parentId)
+ ownerCache[spanIdStr] = owner
+ return owner
}
- // Second pass: process spans and create records
+ // Pass 2: emit entity rows + span rows + exceptions.
for _, entry := range allSpans {
span := entry.span
spanAttrs := span.Attributes
allAttrs := extractAttributes(spanAttrs)
-
- isRoot := len(span.ParentSpanId) == 0
- traceId := spanToTraceId[string(span.SpanId)]
-
- spanId := otelSpanIDToUUID(span.SpanId)
startTime := nanoToTime(span.StartTimeUnixNano)
endTime := nanoToTime(span.EndTimeUnixNano)
duration := endTime.Sub(startTime)
- var distributedTraceId *uuid.UUID
- if dtid := getStringAttribute(spanAttrs, "traceway.distributed_trace_id"); dtid != "" {
- if parsed, err := uuid.Parse(dtid); err == nil {
- distributedTraceId = &parsed
+ prom, promoted := spanIdToPromotion[string(span.SpanId)]
+
+ // Determine the owning entity for span/exception trace_id.
+ var owner *promotion
+ if promoted {
+ p := prom
+ owner = &p
+ } else {
+ owner = resolveOwner(string(span.SpanId))
+ }
+
+ // trace_id for span rows / exceptions: owning entity id when known;
+ // otherwise fall back to the OTel trace_id (orphan path — preserves
+ // today's behavior for cross-process children whose parent never
+ // matched a promoted entity within this batch).
+ var ownerId uuid.UUID
+ var ownerTraceType string
+ if owner != nil {
+ ownerId = owner.id
+ ownerTraceType = owner.kind.traceType()
+ } else {
+ ownerId = otelTraceIDToUUID(span.TraceId)
+ if ownerId == uuid.Nil {
+ ownerId = uuid.New()
}
}
- if isRoot {
- rootSpanId := spanId
- if (span.Kind == tracepb.Span_SPAN_KIND_SERVER || span.Kind == tracepb.Span_SPAN_KIND_INTERNAL) && hasHTTPAttributes(spanAttrs) {
+ if promoted {
+ rootSpanId := otelSpanIDToUUID(span.SpanId)
+ switch prom.kind {
+ case entityEndpoint:
ep := buildEndpoint(
- traceId, projectId, span, spanAttrs, allAttrs,
+ prom.id, projectId, span, spanAttrs, allAttrs,
startTime, duration, serverName, appVersion,
)
- ep.DistributedTraceId = distributedTraceId
+ ep.DistributedTraceId = prom.distributedTraceId
ep.SpanId = &rootSpanId
+ ep.IsRoot = prom.isRoot
endpoints = append(endpoints, ep)
- } else if span.Kind == tracepb.Span_SPAN_KIND_CONSUMER {
+ case entityTask:
t := buildTask(
- traceId, projectId, span, allAttrs,
+ prom.id, projectId, span, allAttrs,
startTime, endTime, duration, serverName, appVersion,
)
- t.DistributedTraceId = distributedTraceId
+ t.DistributedTraceId = prom.distributedTraceId
t.SpanId = &rootSpanId
+ t.IsRoot = prom.isRoot
tasks = append(tasks, t)
- } else if hasGenAiAttributes(spanAttrs) {
+ case entityAiTrace:
aiTrace := buildAiTrace(
- traceId, projectId, span, spanAttrs, allAttrs,
+ prom.id, projectId, span, spanAttrs, allAttrs,
startTime, duration, serverName, appVersion,
)
+ aiTrace.DistributedTraceId = prom.distributedTraceId
+ aiTrace.IsRoot = prom.isRoot
aiTraces = append(aiTraces, aiTrace)
- if conv := extractConversation(spanAttrs, projectId, traceId); conv != nil {
+ if conv := extractConversation(spanAttrs, projectId, prom.id); conv != nil {
aiConversations = append(aiConversations, *conv)
}
- } else {
- continue
}
- } else {
+ } else if len(span.ParentSpanId) > 0 {
+ // Non-root, unpromoted span → goes to the generic spans table,
+ // re-rooted to its nearest enclosing entity (or the OTel
+ // trace_id as fallback when the parent chain doesn't reach a
+ // promoted span in this batch).
spanName := span.Name
if dbQuery := getStringAttribute(spanAttrs, "db.query.text"); dbQuery != "" {
spanName = dbQuery
@@ -149,8 +238,8 @@ func convertTraces(projectId uuid.UUID, req *coltracepb.ExportTraceServiceReques
}
spans = append(spans, models.Span{
- Id: spanId,
- TraceId: traceId,
+ Id: otelSpanIDToUUID(span.SpanId),
+ TraceId: ownerId,
ProjectId: projectId,
Name: spanName,
StartTime: startTime,
@@ -158,20 +247,28 @@ func convertTraces(projectId uuid.UUID, req *coltracepb.ExportTraceServiceReques
RecordedAt: startTime,
ParentSpanId: ptrSpanUUID(span.ParentSpanId),
})
+ } else {
+ // Unpromoted root span — match historical behavior and drop
+ // it (no entity row, no span row, no exception). Common case:
+ // CLIENT-kind roots or non-HTTP SERVER roots from custom
+ // instrumentation that we don't have a dedicated page for.
+ continue
}
- traceType := "task"
- if isRoot && (span.Kind == tracepb.Span_SPAN_KIND_SERVER || span.Kind == tracepb.Span_SPAN_KIND_INTERNAL) && hasHTTPAttributes(spanAttrs) {
- traceType = "endpoint"
+ traceType := ownerTraceType
+ if traceType == "" {
+ traceType = "task"
}
for _, event := range span.Events {
if event.Name == "exception" {
exc := buildException(
- projectId, traceId, traceType, event,
+ projectId, ownerId, traceType, event,
allAttrs, serverName, appVersion,
)
- exc.DistributedTraceId = distributedTraceId
+ if owner != nil {
+ exc.DistributedTraceId = owner.distributedTraceId
+ }
exceptions = append(exceptions, exc)
}
}
@@ -180,6 +277,31 @@ func convertTraces(projectId uuid.UUID, req *coltracepb.ExportTraceServiceReques
return
}
+func classifySpan(span *tracepb.Span, spanById map[string]*tracepb.Span) entityKind {
+ attrs := span.Attributes
+ if (span.Kind == tracepb.Span_SPAN_KIND_SERVER || span.Kind == tracepb.Span_SPAN_KIND_INTERNAL) && hasHTTPAttributes(attrs) {
+ if len(span.ParentSpanId) == 0 {
+ return entityEndpoint
+ }
+ if _, parentInBatch := spanById[string(span.ParentSpanId)]; !parentInBatch {
+ return entityEndpoint
+ }
+ }
+ if span.Kind == tracepb.Span_SPAN_KIND_CONSUMER {
+ return entityTask
+ }
+ // keepsuit's ConsoleInstrumentation (and equivalents) emits a root INTERNAL
+ // span with `console.command` set. Only promote when it is a root span —
+ // otherwise we would scoop up arbitrary manual roots from other code paths.
+ if span.Kind == tracepb.Span_SPAN_KIND_INTERNAL && len(span.ParentSpanId) == 0 && getStringAttribute(attrs, "console.command") != "" {
+ return entityTask
+ }
+ if hasGenAiAttributes(attrs) {
+ return entityAiTrace
+ }
+ return entityNone
+}
+
func hasHTTPAttributes(attrs []*commonpb.KeyValue) bool {
for _, kv := range attrs {
switch kv.Key {
diff --git a/backend/app/controllers/otelcontrollers/trace_converter_test.go b/backend/app/controllers/otelcontrollers/trace_converter_test.go
index d55f2b32..6d4355bb 100644
--- a/backend/app/controllers/otelcontrollers/trace_converter_test.go
+++ b/backend/app/controllers/otelcontrollers/trace_converter_test.go
@@ -445,6 +445,206 @@ func TestFormatExceptionStackTrace(t *testing.T) {
}
}
+// --- Non-root classification tests ---
+
+func TestConvertTraces_ConsumerNonRoot_BecomesTask(t *testing.T) {
+ // Worker batch: CONSUMER span with a parent that lives in the producer's
+ // batch (not present in our scope), plus a child DB span.
+ traceId := []byte{0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, 0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f}
+ producerSpanId := []byte{0xA0, 0xA1, 0xA2, 0xA3, 0xA4, 0xA5, 0xA6, 0xA7}
+ consumerSpanId := []byte{0xB0, 0xB1, 0xB2, 0xB3, 0xB4, 0xB5, 0xB6, 0xB7}
+ childSpanId := []byte{0xC0, 0xC1, 0xC2, 0xC3, 0xC4, 0xC5, 0xC6, 0xC7}
+ now := uint64(1_700_000_000_000_000_000)
+
+ req := &coltracepb.ExportTraceServiceRequest{
+ ResourceSpans: []*tracepb.ResourceSpans{{
+ Resource: &resourcepb.Resource{Attributes: []*commonpb.KeyValue{strKV("service.name", "worker")}},
+ ScopeSpans: []*tracepb.ScopeSpans{{
+ Spans: []*tracepb.Span{
+ {TraceId: traceId, SpanId: consumerSpanId, ParentSpanId: producerSpanId, Name: "process job", Kind: tracepb.Span_SPAN_KIND_CONSUMER, StartTimeUnixNano: now, EndTimeUnixNano: now + 1_000_000},
+ {TraceId: traceId, SpanId: childSpanId, ParentSpanId: consumerSpanId, Name: "SELECT users", Kind: tracepb.Span_SPAN_KIND_INTERNAL, StartTimeUnixNano: now, EndTimeUnixNano: now + 500_000},
+ },
+ }},
+ }},
+ }
+
+ endpoints, tasks, spans, _, _, _ := convertTraces(testProjectId, req)
+
+ if len(endpoints) != 0 {
+ t.Fatalf("expected 0 endpoints, got %d", len(endpoints))
+ }
+ if len(tasks) != 1 {
+ t.Fatalf("expected 1 task, got %d", len(tasks))
+ }
+ if tasks[0].IsRoot {
+ t.Errorf("expected task.IsRoot == false, got true")
+ }
+ wantTaskId := otelSpanIDToUUID(consumerSpanId)
+ if tasks[0].Id != wantTaskId {
+ t.Errorf("expected task.Id %s (from span_id), got %s", wantTaskId, tasks[0].Id)
+ }
+ wantDtId := otelTraceIDToUUID(traceId)
+ if tasks[0].DistributedTraceId == nil || *tasks[0].DistributedTraceId != wantDtId {
+ t.Errorf("expected task.DistributedTraceId %s (from trace_id), got %v", wantDtId, tasks[0].DistributedTraceId)
+ }
+
+ if len(spans) != 1 {
+ t.Fatalf("expected 1 span row, got %d", len(spans))
+ }
+ if spans[0].TraceId != wantTaskId {
+ t.Errorf("expected child span.TraceId == task.Id %s, got %s", wantTaskId, spans[0].TraceId)
+ }
+}
+
+func TestConvertTraces_ConsoleCommand_BecomesTask(t *testing.T) {
+ traceId := []byte{0x20, 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, 0x28, 0x29, 0x2a, 0x2b, 0x2c, 0x2d, 0x2e, 0x2f}
+ rootSpanId := []byte{0xD0, 0xD1, 0xD2, 0xD3, 0xD4, 0xD5, 0xD6, 0xD7}
+ now := uint64(1_700_000_000_000_000_000)
+
+ req := &coltracepb.ExportTraceServiceRequest{
+ ResourceSpans: []*tracepb.ResourceSpans{{
+ Resource: &resourcepb.Resource{Attributes: []*commonpb.KeyValue{strKV("service.name", "scheduler")}},
+ ScopeSpans: []*tracepb.ScopeSpans{{
+ Spans: []*tracepb.Span{{
+ TraceId: traceId, SpanId: rootSpanId,
+ Name: "bookings:send-reminders", Kind: tracepb.Span_SPAN_KIND_INTERNAL,
+ StartTimeUnixNano: now, EndTimeUnixNano: now + 2_000_000,
+ Attributes: []*commonpb.KeyValue{strKV("console.command", "bookings:send-reminders")},
+ }},
+ }},
+ }},
+ }
+
+ _, tasks, _, _, _, _ := convertTraces(testProjectId, req)
+ if len(tasks) != 1 {
+ t.Fatalf("expected 1 task, got %d", len(tasks))
+ }
+ if !tasks[0].IsRoot {
+ t.Errorf("expected task.IsRoot == true, got false")
+ }
+ if tasks[0].Id != otelTraceIDToUUID(traceId) {
+ t.Errorf("expected task.Id == otelTraceIDToUUID(trace_id), got %s", tasks[0].Id)
+ }
+}
+
+func TestConvertTraces_InlineGenAi_BecomesAiTrace(t *testing.T) {
+ traceId := []byte{0x30, 0x31, 0x32, 0x33, 0x34, 0x35, 0x36, 0x37, 0x38, 0x39, 0x3a, 0x3b, 0x3c, 0x3d, 0x3e, 0x3f}
+ rootSpanId := []byte{0xE0, 0xE1, 0xE2, 0xE3, 0xE4, 0xE5, 0xE6, 0xE7}
+ llmSpanId := []byte{0xE8, 0xE9, 0xEA, 0xEB, 0xEC, 0xED, 0xEE, 0xEF}
+ now := uint64(1_700_000_000_000_000_000)
+
+ req := &coltracepb.ExportTraceServiceRequest{
+ ResourceSpans: []*tracepb.ResourceSpans{{
+ Resource: &resourcepb.Resource{Attributes: []*commonpb.KeyValue{strKV("service.name", "api")}},
+ ScopeSpans: []*tracepb.ScopeSpans{{
+ Spans: []*tracepb.Span{
+ {TraceId: traceId, SpanId: rootSpanId, Name: "POST /chat", Kind: tracepb.Span_SPAN_KIND_SERVER, StartTimeUnixNano: now, EndTimeUnixNano: now + 10_000_000,
+ Attributes: []*commonpb.KeyValue{strKV("http.request.method", "POST"), strKV("http.route", "/chat")}},
+ {TraceId: traceId, SpanId: llmSpanId, ParentSpanId: rootSpanId, Name: "chat openai", Kind: tracepb.Span_SPAN_KIND_INTERNAL, StartTimeUnixNano: now, EndTimeUnixNano: now + 5_000_000,
+ Attributes: []*commonpb.KeyValue{strKV("gen_ai.system", "openai"), strKV("gen_ai.request.model", "gpt-4")}},
+ },
+ }},
+ }},
+ }
+
+ endpoints, _, _, _, aiTraces, _ := convertTraces(testProjectId, req)
+ if len(endpoints) != 1 {
+ t.Fatalf("expected 1 endpoint, got %d", len(endpoints))
+ }
+ if !endpoints[0].IsRoot {
+ t.Errorf("expected endpoint.IsRoot == true")
+ }
+ wantEndpointId := otelTraceIDToUUID(traceId)
+ if endpoints[0].Id != wantEndpointId {
+ t.Errorf("expected endpoint.Id %s, got %s", wantEndpointId, endpoints[0].Id)
+ }
+
+ if len(aiTraces) != 1 {
+ t.Fatalf("expected 1 ai_trace, got %d", len(aiTraces))
+ }
+ if aiTraces[0].IsRoot {
+ t.Errorf("expected aiTrace.IsRoot == false")
+ }
+ wantAiId := otelSpanIDToUUID(llmSpanId)
+ if aiTraces[0].Id != wantAiId {
+ t.Errorf("expected aiTrace.Id %s, got %s", wantAiId, aiTraces[0].Id)
+ }
+ if aiTraces[0].DistributedTraceId == nil || *aiTraces[0].DistributedTraceId != wantEndpointId {
+ t.Errorf("expected aiTrace.DistributedTraceId %s, got %v", wantEndpointId, aiTraces[0].DistributedTraceId)
+ }
+}
+
+func TestConvertTraces_ExceptionOnConsumer_TraceTypeIsTask(t *testing.T) {
+ traceId := []byte{0x40, 0x41, 0x42, 0x43, 0x44, 0x45, 0x46, 0x47, 0x48, 0x49, 0x4a, 0x4b, 0x4c, 0x4d, 0x4e, 0x4f}
+ producerSpanId := []byte{0xF0, 0xF1, 0xF2, 0xF3, 0xF4, 0xF5, 0xF6, 0xF7}
+ consumerSpanId := []byte{0xF8, 0xF9, 0xFA, 0xFB, 0xFC, 0xFD, 0xFE, 0xFF}
+ now := uint64(1_700_000_000_000_000_000)
+
+ req := &coltracepb.ExportTraceServiceRequest{
+ ResourceSpans: []*tracepb.ResourceSpans{{
+ Resource: &resourcepb.Resource{Attributes: []*commonpb.KeyValue{strKV("service.name", "worker")}},
+ ScopeSpans: []*tracepb.ScopeSpans{{
+ Spans: []*tracepb.Span{{
+ TraceId: traceId, SpanId: consumerSpanId, ParentSpanId: producerSpanId,
+ Name: "process job", Kind: tracepb.Span_SPAN_KIND_CONSUMER,
+ StartTimeUnixNano: now, EndTimeUnixNano: now + 1_000_000,
+ Events: []*tracepb.Span_Event{{
+ Name: "exception",
+ TimeUnixNano: now + 500_000,
+ Attributes: []*commonpb.KeyValue{
+ strKV("exception.type", "RuntimeError"),
+ strKV("exception.message", "boom"),
+ },
+ }},
+ }},
+ }},
+ }},
+ }
+
+ _, tasks, _, exceptions, _, _ := convertTraces(testProjectId, req)
+ if len(tasks) != 1 || len(exceptions) != 1 {
+ t.Fatalf("expected 1 task + 1 exception, got %d / %d", len(tasks), len(exceptions))
+ }
+ if exceptions[0].TraceType != "task" {
+ t.Errorf("expected exception.TraceType == 'task', got %q", exceptions[0].TraceType)
+ }
+ if exceptions[0].TraceId == nil || *exceptions[0].TraceId != tasks[0].Id {
+ t.Errorf("expected exception.TraceId == task.Id %s, got %v", tasks[0].Id, exceptions[0].TraceId)
+ }
+}
+
+func TestConvertTraces_OrphanSpan_FallsBackToTraceId(t *testing.T) {
+ // A non-root span whose parent is not in this batch and the span itself
+ // isn't classified as an entity. Should land in the spans table with
+ // trace_id == otelTraceIDToUUID(trace_id) (orphan fallback path).
+ traceId := []byte{0x50, 0x51, 0x52, 0x53, 0x54, 0x55, 0x56, 0x57, 0x58, 0x59, 0x5a, 0x5b, 0x5c, 0x5d, 0x5e, 0x5f}
+ orphanParentId := []byte{0x60, 0x61, 0x62, 0x63, 0x64, 0x65, 0x66, 0x67}
+ orphanSpanId := []byte{0x70, 0x71, 0x72, 0x73, 0x74, 0x75, 0x76, 0x77}
+ now := uint64(1_700_000_000_000_000_000)
+
+ req := &coltracepb.ExportTraceServiceRequest{
+ ResourceSpans: []*tracepb.ResourceSpans{{
+ Resource: &resourcepb.Resource{Attributes: []*commonpb.KeyValue{strKV("service.name", "worker")}},
+ ScopeSpans: []*tracepb.ScopeSpans{{
+ Spans: []*tracepb.Span{{
+ TraceId: traceId, SpanId: orphanSpanId, ParentSpanId: orphanParentId,
+ Name: "redis GET", Kind: tracepb.Span_SPAN_KIND_INTERNAL,
+ StartTimeUnixNano: now, EndTimeUnixNano: now + 100_000,
+ }},
+ }},
+ }},
+ }
+
+ _, _, spans, _, _, _ := convertTraces(testProjectId, req)
+ if len(spans) != 1 {
+ t.Fatalf("expected 1 span row, got %d", len(spans))
+ }
+ want := otelTraceIDToUUID(traceId)
+ if spans[0].TraceId != want {
+ t.Errorf("expected orphan span.TraceId %s (fallback to OTel trace_id), got %s", want, spans[0].TraceId)
+ }
+}
+
// --- Helpers ---
func makeAttrs(key, val string) []*commonpb.KeyValue {
diff --git a/backend/app/controllers/task.controller.go b/backend/app/controllers/task.controller.go
index 5686dccb..27f12e0d 100644
--- a/backend/app/controllers/task.controller.go
+++ b/backend/app/controllers/task.controller.go
@@ -20,6 +20,8 @@ type TaskSearchRequest struct {
OrderBy string `json:"orderBy"`
SortDirection string `json:"sortDirection"`
Pagination PaginationParams `json:"pagination"`
+ Search string `json:"search"`
+ RootFilter string `json:"rootFilter"`
}
type TaskInstancesRequest struct {
@@ -82,7 +84,7 @@ func (e taskController) FindGroupedByTaskName(c *gin.Context) {
}
span := traceway.StartSpan(c, "loading grouped tasks")
- stats, total, err := repositories.TaskRepository.FindGroupedByTaskName(c, projectId, request.FromDate, request.ToDate, request.Pagination.Page, request.Pagination.PageSize, request.OrderBy, request.SortDirection)
+ stats, total, err := repositories.TaskRepository.FindGroupedByTaskName(c, projectId, request.FromDate, request.ToDate, request.Pagination.Page, request.Pagination.PageSize, request.OrderBy, request.SortDirection, request.Search, request.RootFilter)
span.End()
if err != nil {
c.AbortWithError(500, traceway.NewStackTraceErrorf("error loading stats by name: %w", err))
diff --git a/backend/app/migrations/ch/0058_add_is_root_to_endpoints.up.sql b/backend/app/migrations/ch/0058_add_is_root_to_endpoints.up.sql
new file mode 100644
index 00000000..b266a645
--- /dev/null
+++ b/backend/app/migrations/ch/0058_add_is_root_to_endpoints.up.sql
@@ -0,0 +1 @@
+ALTER TABLE endpoints ADD COLUMN is_root UInt8 DEFAULT 1
diff --git a/backend/app/migrations/ch/0059_add_is_root_to_tasks.up.sql b/backend/app/migrations/ch/0059_add_is_root_to_tasks.up.sql
new file mode 100644
index 00000000..f581b697
--- /dev/null
+++ b/backend/app/migrations/ch/0059_add_is_root_to_tasks.up.sql
@@ -0,0 +1 @@
+ALTER TABLE tasks ADD COLUMN is_root UInt8 DEFAULT 1
diff --git a/backend/app/migrations/ch/0060_add_distributed_trace_id_to_ai_traces.up.sql b/backend/app/migrations/ch/0060_add_distributed_trace_id_to_ai_traces.up.sql
new file mode 100644
index 00000000..fc3fa7e8
--- /dev/null
+++ b/backend/app/migrations/ch/0060_add_distributed_trace_id_to_ai_traces.up.sql
@@ -0,0 +1 @@
+ALTER TABLE ai_traces ADD COLUMN distributed_trace_id Nullable(UUID) DEFAULT NULL
diff --git a/backend/app/migrations/ch/0061_add_distributed_trace_id_index_ai_traces.up.sql b/backend/app/migrations/ch/0061_add_distributed_trace_id_index_ai_traces.up.sql
new file mode 100644
index 00000000..2a61b1bc
--- /dev/null
+++ b/backend/app/migrations/ch/0061_add_distributed_trace_id_index_ai_traces.up.sql
@@ -0,0 +1 @@
+ALTER TABLE ai_traces ADD INDEX idx_distributed_trace_id distributed_trace_id TYPE bloom_filter(0.001) GRANULARITY 1
diff --git a/backend/app/migrations/ch/0062_add_is_root_to_ai_traces.up.sql b/backend/app/migrations/ch/0062_add_is_root_to_ai_traces.up.sql
new file mode 100644
index 00000000..8a651c94
--- /dev/null
+++ b/backend/app/migrations/ch/0062_add_is_root_to_ai_traces.up.sql
@@ -0,0 +1 @@
+ALTER TABLE ai_traces ADD COLUMN is_root UInt8 DEFAULT 1
diff --git a/backend/app/migrations/sqlite_telemetry/0009_add_is_root_and_distributed_trace_id.up.sql b/backend/app/migrations/sqlite_telemetry/0009_add_is_root_and_distributed_trace_id.up.sql
new file mode 100644
index 00000000..10839ebf
--- /dev/null
+++ b/backend/app/migrations/sqlite_telemetry/0009_add_is_root_and_distributed_trace_id.up.sql
@@ -0,0 +1,4 @@
+ALTER TABLE endpoints ADD COLUMN is_root INTEGER NOT NULL DEFAULT 1;
+ALTER TABLE tasks ADD COLUMN is_root INTEGER NOT NULL DEFAULT 1;
+ALTER TABLE ai_traces ADD COLUMN is_root INTEGER NOT NULL DEFAULT 1;
+ALTER TABLE ai_traces ADD COLUMN distributed_trace_id TEXT DEFAULT NULL;
diff --git a/backend/app/models/ai_trace.model.go b/backend/app/models/ai_trace.model.go
index 2edeb40e..ba9cfc29 100644
--- a/backend/app/models/ai_trace.model.go
+++ b/backend/app/models/ai_trace.model.go
@@ -29,8 +29,10 @@ type AiTrace struct {
FinishReason string `json:"finishReason" ch:"finish_reason"`
ServerName string `json:"serverName" ch:"server_name"`
AppVersion string `json:"appVersion" ch:"app_version"`
- StorageKey string `json:"storageKey" ch:"storage_key"`
- Attributes map[string]string `json:"attributes" ch:"attributes"`
+ StorageKey string `json:"storageKey" ch:"storage_key"`
+ Attributes map[string]string `json:"attributes" ch:"attributes"`
+ DistributedTraceId *uuid.UUID `json:"distributedTraceId,omitempty" ch:"distributed_trace_id"`
+ IsRoot bool `json:"isRoot" ch:"is_root"`
}
type AiTraceStats struct {
@@ -44,6 +46,8 @@ type AiTraceStats struct {
AvgInputTokens float64 `json:"avgInputTokens"`
AvgOutputTokens float64 `json:"avgOutputTokens"`
LastSeen time.Time `json:"lastSeen"`
+ HasRoot bool `json:"hasRoot"`
+ HasNonRoot bool `json:"hasNonRoot"`
}
type AiTraceDetailStats struct {
diff --git a/backend/app/models/endpoint.model.go b/backend/app/models/endpoint.model.go
index 5f634f52..48cd94dd 100644
--- a/backend/app/models/endpoint.model.go
+++ b/backend/app/models/endpoint.model.go
@@ -22,6 +22,7 @@ type Endpoint struct {
DistributedTraceId *uuid.UUID `json:"distributedTraceId,omitempty" ch:"distributed_trace_id"`
SpanId *uuid.UUID `json:"spanId,omitempty" ch:"span_id"`
IsStream bool `json:"isStream" ch:"is_stream"`
+ IsRoot bool `json:"isRoot" ch:"is_root"`
}
type EndpointStats struct {
@@ -35,6 +36,8 @@ type EndpointStats struct {
Impact float64 `json:"impact"` // 0-1 impact score
ImpactReason string `json:"impactReason"` // human-readable explanation of the dominant impact factor
IsStream bool `json:"isStream"` // true => latency/Apdex/impact intentionally zero
+ HasRoot bool `json:"hasRoot"`
+ HasNonRoot bool `json:"hasNonRoot"`
}
// EndpointDetailStats contains detailed statistics for a specific endpoint
diff --git a/backend/app/models/task.model.go b/backend/app/models/task.model.go
index 4484d16f..56102556 100644
--- a/backend/app/models/task.model.go
+++ b/backend/app/models/task.model.go
@@ -18,6 +18,7 @@ type Task struct {
ServerName string `json:"serverName" ch:"server_name"`
DistributedTraceId *uuid.UUID `json:"distributedTraceId,omitempty" ch:"distributed_trace_id"`
SpanId *uuid.UUID `json:"spanId,omitempty" ch:"span_id"`
+ IsRoot bool `json:"isRoot" ch:"is_root"`
}
type TaskStats struct {
@@ -27,6 +28,8 @@ type TaskStats struct {
P95Duration time.Duration `json:"p95Duration"`
AvgDuration time.Duration `json:"avgDuration"`
LastSeen time.Time `json:"lastSeen"`
+ HasRoot bool `json:"hasRoot"`
+ HasNonRoot bool `json:"hasNonRoot"`
}
// TaskDetailStats contains detailed statistics for a specific task
diff --git a/backend/app/repositories/ai_trace.repository.go b/backend/app/repositories/ai_trace.repository.go
index edc1341d..031b1188 100644
--- a/backend/app/repositories/ai_trace.repository.go
+++ b/backend/app/repositories/ai_trace.repository.go
@@ -7,6 +7,7 @@ import (
"database/sql"
"encoding/json"
"errors"
+ "strings"
"time"
"github.com/ClickHouse/clickhouse-go/v2"
@@ -19,7 +20,7 @@ type aiTraceRepository struct{}
func (r *aiTraceRepository) InsertAsync(ctx context.Context, lines []models.AiTrace) error {
batch, err := chdb.Conn.PrepareBatch(clickhouse.Context(context.Background(), clickhouse.WithAsync(false)),
- "INSERT INTO ai_traces (id, project_id, recorded_at, duration, status_code, model, response_model, provider, operation, input_tokens, output_tokens, total_tokens, cached_tokens, reasoning_tokens, input_cost, output_cost, total_cost, trace_name, user_id, finish_reason, server_name, app_version, storage_key, attributes)")
+ "INSERT INTO ai_traces (id, project_id, recorded_at, duration, status_code, model, response_model, provider, operation, input_tokens, output_tokens, total_tokens, cached_tokens, reasoning_tokens, input_cost, output_cost, total_cost, trace_name, user_id, finish_reason, server_name, app_version, storage_key, attributes, distributed_trace_id, is_root)")
if err != nil {
return err
}
@@ -30,13 +31,17 @@ func (r *aiTraceRepository) InsertAsync(ctx context.Context, lines []models.AiTr
attributesJSON = string(attributesBytes)
}
}
+ isRoot := uint8(0)
+ if t.IsRoot {
+ isRoot = 1
+ }
if err := batch.Append(
t.Id, t.ProjectId, t.RecordedAt, int64(t.Duration), t.StatusCode,
t.Model, t.ResponseModel, t.Provider, t.Operation,
t.InputTokens, t.OutputTokens, t.TotalTokens, t.CachedTokens, t.ReasoningTokens,
t.InputCost, t.OutputCost, t.TotalCost,
t.TraceName, t.UserId, t.FinishReason, t.ServerName, t.AppVersion,
- t.StorageKey, attributesJSON,
+ t.StorageKey, attributesJSON, t.DistributedTraceId, isRoot,
); err != nil {
return err
}
@@ -44,7 +49,7 @@ func (r *aiTraceRepository) InsertAsync(ctx context.Context, lines []models.AiTr
return batch.Send()
}
-func (r *aiTraceRepository) FindGroupedByTraceName(ctx context.Context, projectId uuid.UUID, fromDate, toDate time.Time, page, pageSize int, orderBy, sortDirection, search string) ([]models.AiTraceStats, int64, error) {
+func (r *aiTraceRepository) FindGroupedByTraceName(ctx context.Context, projectId uuid.UUID, fromDate, toDate time.Time, page, pageSize int, orderBy, sortDirection, search, rootFilter string) ([]models.AiTraceStats, int64, error) {
whereClause := "project_id = ? AND recorded_at >= ? AND recorded_at <= ?"
args := []interface{}{projectId, fromDate, toDate}
@@ -52,6 +57,12 @@ func (r *aiTraceRepository) FindGroupedByTraceName(ctx context.Context, projectI
whereClause += " AND positionCaseInsensitive(trace_name, ?) > 0"
args = append(args, search)
}
+ switch rootFilter {
+ case "root":
+ whereClause += " AND is_root = 1"
+ case "non_root":
+ whereClause += " AND is_root = 0"
+ }
var count uint64
err := chdb.Conn.QueryRow(ctx, "SELECT uniq(trace_name) FROM ai_traces WHERE "+whereClause, args...).Scan(&count)
@@ -90,7 +101,9 @@ func (r *aiTraceRepository) FindGroupedByTraceName(ctx context.Context, projectI
sum(total_cost) as total_cost,
avg(input_tokens) as avg_input_tokens,
avg(output_tokens) as avg_output_tokens,
- max(recorded_at) as last_seen
+ max(recorded_at) as last_seen,
+ max(is_root) as has_root,
+ max(if(is_root = 0, 1, 0)) as has_non_root
FROM ai_traces
WHERE ` + whereClause + `
GROUP BY trace_name
@@ -108,13 +121,16 @@ func (r *aiTraceRepository) FindGroupedByTraceName(ctx context.Context, projectI
for rows.Next() {
var s models.AiTraceStats
var p50, p95, avg float64
+ var hasRoot, hasNonRoot uint8
if err := rows.Scan(&s.TraceName, &s.Count, &p50, &p95, &avg,
- &s.TotalTokens, &s.TotalCost, &s.AvgInputTokens, &s.AvgOutputTokens, &s.LastSeen); err != nil {
+ &s.TotalTokens, &s.TotalCost, &s.AvgInputTokens, &s.AvgOutputTokens, &s.LastSeen, &hasRoot, &hasNonRoot); err != nil {
return nil, 0, err
}
s.P50Duration = time.Duration(p50)
s.P95Duration = time.Duration(p95)
s.AvgDuration = time.Duration(avg)
+ s.HasRoot = hasRoot == 1
+ s.HasNonRoot = hasNonRoot == 1
stats = append(stats, s)
}
@@ -236,13 +252,14 @@ func (r *aiTraceRepository) FindById(ctx context.Context, projectId, traceId uui
input_tokens, output_tokens, total_tokens, cached_tokens, reasoning_tokens,
input_cost, output_cost, total_cost,
trace_name, user_id, finish_reason, server_name, app_version,
- storage_key, attributes
+ storage_key, attributes, distributed_trace_id, is_root
FROM ai_traces
WHERE project_id = ? AND id = ?
LIMIT 1`
var t models.AiTrace
var attributesJSON string
+ var isRoot uint8
err := chdb.Conn.QueryRow(ctx, query, projectId, traceId).Scan(
&t.Id, &t.ProjectId, &t.RecordedAt, &t.Duration, &t.StatusCode,
@@ -250,7 +267,7 @@ func (r *aiTraceRepository) FindById(ctx context.Context, projectId, traceId uui
&t.InputTokens, &t.OutputTokens, &t.TotalTokens, &t.CachedTokens, &t.ReasoningTokens,
&t.InputCost, &t.OutputCost, &t.TotalCost,
&t.TraceName, &t.UserId, &t.FinishReason, &t.ServerName, &t.AppVersion,
- &t.StorageKey, &attributesJSON,
+ &t.StorageKey, &attributesJSON, &t.DistributedTraceId, &isRoot,
)
if err != nil {
if errors.Is(err, sql.ErrNoRows) {
@@ -258,6 +275,7 @@ func (r *aiTraceRepository) FindById(ctx context.Context, projectId, traceId uui
}
return nil, err
}
+ t.IsRoot = isRoot == 1
if attributesJSON != "" && attributesJSON != "{}" {
if err := json.Unmarshal([]byte(attributesJSON), &t.Attributes); err != nil {
@@ -268,4 +286,57 @@ func (r *aiTraceRepository) FindById(ctx context.Context, projectId, traceId uui
return &t, nil
}
+func (r *aiTraceRepository) FindByDistributedTraceId(ctx context.Context, distributedTraceId uuid.UUID, projectIds []uuid.UUID) ([]models.AiTrace, error) {
+ if len(projectIds) == 0 {
+ return nil, nil
+ }
+ placeholders := make([]string, len(projectIds))
+ args := []interface{}{distributedTraceId}
+ for i, pid := range projectIds {
+ placeholders[i] = "?"
+ args = append(args, pid)
+ _ = i
+ }
+ query := `SELECT id, project_id, recorded_at, duration, status_code,
+ model, response_model, provider, operation,
+ input_tokens, output_tokens, total_tokens, cached_tokens, reasoning_tokens,
+ input_cost, output_cost, total_cost,
+ trace_name, user_id, finish_reason, server_name, app_version,
+ storage_key, attributes, distributed_trace_id, is_root
+ FROM ai_traces
+ WHERE distributed_trace_id = ? AND project_id IN (` + strings.Join(placeholders, ",") + `)
+ ORDER BY recorded_at ASC`
+
+ rows, err := chdb.Conn.Query(ctx, query, args...)
+ if err != nil {
+ return nil, err
+ }
+ defer rows.Close()
+
+ var traces []models.AiTrace
+ for rows.Next() {
+ var t models.AiTrace
+ var attributesJSON string
+ var isRoot uint8
+ if err := rows.Scan(
+ &t.Id, &t.ProjectId, &t.RecordedAt, &t.Duration, &t.StatusCode,
+ &t.Model, &t.ResponseModel, &t.Provider, &t.Operation,
+ &t.InputTokens, &t.OutputTokens, &t.TotalTokens, &t.CachedTokens, &t.ReasoningTokens,
+ &t.InputCost, &t.OutputCost, &t.TotalCost,
+ &t.TraceName, &t.UserId, &t.FinishReason, &t.ServerName, &t.AppVersion,
+ &t.StorageKey, &attributesJSON, &t.DistributedTraceId, &isRoot,
+ ); err != nil {
+ return nil, err
+ }
+ t.IsRoot = isRoot == 1
+ if attributesJSON != "" && attributesJSON != "{}" {
+ if err := json.Unmarshal([]byte(attributesJSON), &t.Attributes); err != nil {
+ t.Attributes = nil
+ }
+ }
+ traces = append(traces, t)
+ }
+ return traces, nil
+}
+
var AiTraceRepository = aiTraceRepository{}
diff --git a/backend/app/repositories/ai_trace.repository_sqlite.go b/backend/app/repositories/ai_trace.repository_sqlite.go
index 70677df6..fb5671c8 100644
--- a/backend/app/repositories/ai_trace.repository_sqlite.go
+++ b/backend/app/repositories/ai_trace.repository_sqlite.go
@@ -6,6 +6,7 @@ import (
"context"
"fmt"
"sort"
+ "strings"
"time"
"github.com/google/uuid"
@@ -15,30 +16,32 @@ import (
)
type aiTraceRow struct {
- Id uuid.UUID `lit:"id"`
- ProjectId uuid.UUID `lit:"project_id"`
- RecordedAt SQLiteTime `lit:"recorded_at"`
- Duration int64 `lit:"duration"`
- StatusCode uint8 `lit:"status_code"`
- Model string `lit:"model"`
- ResponseModel string `lit:"response_model"`
- Provider string `lit:"provider"`
- Operation string `lit:"operation"`
- InputTokens int64 `lit:"input_tokens"`
- OutputTokens int64 `lit:"output_tokens"`
- TotalTokens int64 `lit:"total_tokens"`
- CachedTokens int64 `lit:"cached_tokens"`
- ReasoningTokens int64 `lit:"reasoning_tokens"`
- InputCost float64 `lit:"input_cost"`
- OutputCost float64 `lit:"output_cost"`
- TotalCost float64 `lit:"total_cost"`
- TraceName string `lit:"trace_name"`
- UserId string `lit:"user_id"`
- FinishReason string `lit:"finish_reason"`
- ServerName string `lit:"server_name"`
- AppVersion string `lit:"app_version"`
- StorageKey string `lit:"storage_key"`
- Attributes SQLiteJSONMap `lit:"attributes"`
+ Id uuid.UUID `lit:"id"`
+ ProjectId uuid.UUID `lit:"project_id"`
+ RecordedAt SQLiteTime `lit:"recorded_at"`
+ Duration int64 `lit:"duration"`
+ StatusCode uint8 `lit:"status_code"`
+ Model string `lit:"model"`
+ ResponseModel string `lit:"response_model"`
+ Provider string `lit:"provider"`
+ Operation string `lit:"operation"`
+ InputTokens int64 `lit:"input_tokens"`
+ OutputTokens int64 `lit:"output_tokens"`
+ TotalTokens int64 `lit:"total_tokens"`
+ CachedTokens int64 `lit:"cached_tokens"`
+ ReasoningTokens int64 `lit:"reasoning_tokens"`
+ InputCost float64 `lit:"input_cost"`
+ OutputCost float64 `lit:"output_cost"`
+ TotalCost float64 `lit:"total_cost"`
+ TraceName string `lit:"trace_name"`
+ UserId string `lit:"user_id"`
+ FinishReason string `lit:"finish_reason"`
+ ServerName string `lit:"server_name"`
+ AppVersion string `lit:"app_version"`
+ StorageKey string `lit:"storage_key"`
+ Attributes SQLiteJSONMap `lit:"attributes"`
+ DistributedTraceId *uuid.UUID `lit:"distributed_trace_id"`
+ IsRoot bool `lit:"is_root"`
}
type groupedAiTraceRow struct {
@@ -50,6 +53,8 @@ type groupedAiTraceRow struct {
AvgInputTokens float64 `lit:"avg_input_tokens"`
AvgOutputTokens float64 `lit:"avg_output_tokens"`
LastSeen string `lit:"last_seen"`
+ HasRoot bool `lit:"has_root"`
+ HasNonRoot bool `lit:"has_non_root"`
}
type aiTraceDurationRow struct {
@@ -76,58 +81,62 @@ func init() {
func aiTraceToRow(t models.AiTrace) aiTraceRow {
return aiTraceRow{
- Id: t.Id,
- ProjectId: t.ProjectId,
- RecordedAt: NewSQLiteTime(t.RecordedAt),
- Duration: int64(t.Duration),
- StatusCode: t.StatusCode,
- Model: t.Model,
- ResponseModel: t.ResponseModel,
- Provider: t.Provider,
- Operation: t.Operation,
- InputTokens: t.InputTokens,
- OutputTokens: t.OutputTokens,
- TotalTokens: t.TotalTokens,
- CachedTokens: t.CachedTokens,
- ReasoningTokens: t.ReasoningTokens,
- InputCost: t.InputCost,
- OutputCost: t.OutputCost,
- TotalCost: t.TotalCost,
- TraceName: t.TraceName,
- UserId: t.UserId,
- FinishReason: t.FinishReason,
- ServerName: t.ServerName,
- AppVersion: t.AppVersion,
- StorageKey: t.StorageKey,
- Attributes: NewSQLiteJSONMap(t.Attributes),
+ Id: t.Id,
+ ProjectId: t.ProjectId,
+ RecordedAt: NewSQLiteTime(t.RecordedAt),
+ Duration: int64(t.Duration),
+ StatusCode: t.StatusCode,
+ Model: t.Model,
+ ResponseModel: t.ResponseModel,
+ Provider: t.Provider,
+ Operation: t.Operation,
+ InputTokens: t.InputTokens,
+ OutputTokens: t.OutputTokens,
+ TotalTokens: t.TotalTokens,
+ CachedTokens: t.CachedTokens,
+ ReasoningTokens: t.ReasoningTokens,
+ InputCost: t.InputCost,
+ OutputCost: t.OutputCost,
+ TotalCost: t.TotalCost,
+ TraceName: t.TraceName,
+ UserId: t.UserId,
+ FinishReason: t.FinishReason,
+ ServerName: t.ServerName,
+ AppVersion: t.AppVersion,
+ StorageKey: t.StorageKey,
+ Attributes: NewSQLiteJSONMap(t.Attributes),
+ DistributedTraceId: t.DistributedTraceId,
+ IsRoot: t.IsRoot,
}
}
func (r *aiTraceRow) toModel() models.AiTrace {
t := models.AiTrace{
- Id: r.Id,
- ProjectId: r.ProjectId,
- RecordedAt: r.RecordedAt.Time,
- Duration: time.Duration(r.Duration),
- StatusCode: r.StatusCode,
- Model: r.Model,
- ResponseModel: r.ResponseModel,
- Provider: r.Provider,
- Operation: r.Operation,
- InputTokens: r.InputTokens,
- OutputTokens: r.OutputTokens,
- TotalTokens: r.TotalTokens,
- CachedTokens: r.CachedTokens,
- ReasoningTokens: r.ReasoningTokens,
- InputCost: r.InputCost,
- OutputCost: r.OutputCost,
- TotalCost: r.TotalCost,
- TraceName: r.TraceName,
- UserId: r.UserId,
- FinishReason: r.FinishReason,
- ServerName: r.ServerName,
- AppVersion: r.AppVersion,
- StorageKey: r.StorageKey,
+ Id: r.Id,
+ ProjectId: r.ProjectId,
+ RecordedAt: r.RecordedAt.Time,
+ Duration: time.Duration(r.Duration),
+ StatusCode: r.StatusCode,
+ Model: r.Model,
+ ResponseModel: r.ResponseModel,
+ Provider: r.Provider,
+ Operation: r.Operation,
+ InputTokens: r.InputTokens,
+ OutputTokens: r.OutputTokens,
+ TotalTokens: r.TotalTokens,
+ CachedTokens: r.CachedTokens,
+ ReasoningTokens: r.ReasoningTokens,
+ InputCost: r.InputCost,
+ OutputCost: r.OutputCost,
+ TotalCost: r.TotalCost,
+ TraceName: r.TraceName,
+ UserId: r.UserId,
+ FinishReason: r.FinishReason,
+ ServerName: r.ServerName,
+ AppVersion: r.AppVersion,
+ StorageKey: r.StorageKey,
+ DistributedTraceId: r.DistributedTraceId,
+ IsRoot: r.IsRoot,
}
if r.Attributes != nil {
t.Attributes = map[string]string(r.Attributes)
@@ -150,7 +159,7 @@ func (r *aiTraceRepository) InsertAsync(ctx context.Context, lines []models.AiTr
return nil
}
-func (r *aiTraceRepository) FindGroupedByTraceName(ctx context.Context, projectId uuid.UUID, fromDate, toDate time.Time, page, pageSize int, orderBy, sortDirection, search string) ([]models.AiTraceStats, int64, error) {
+func (r *aiTraceRepository) FindGroupedByTraceName(ctx context.Context, projectId uuid.UUID, fromDate, toDate time.Time, page, pageSize int, orderBy, sortDirection, search, rootFilter string) ([]models.AiTraceStats, int64, error) {
params := lit.P{"project_id": projectId, "from": NewSQLiteTime(fromDate), "to": NewSQLiteTime(toDate)}
whereClause := "project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to"
@@ -158,6 +167,7 @@ func (r *aiTraceRepository) FindGroupedByTraceName(ctx context.Context, projectI
whereClause += " AND INSTR(LOWER(trace_name), LOWER(:search)) > 0"
params["search"] = search
}
+ whereClause += rootFilterClause("is_root", rootFilter)
countResult, err := lit.SelectSingleNamed[models.CountResult](db.TelemetryDB,
"SELECT COUNT(DISTINCT trace_name) AS count FROM ai_traces WHERE "+whereClause, params)
@@ -176,7 +186,9 @@ func (r *aiTraceRepository) FindGroupedByTraceName(ctx context.Context, projectI
SUM(total_cost) AS total_cost,
AVG(input_tokens) AS avg_input_tokens,
AVG(output_tokens) AS avg_output_tokens,
- MAX(recorded_at) AS last_seen
+ MAX(recorded_at) AS last_seen,
+ MAX(is_root) AS has_root,
+ MAX(CASE WHEN is_root = 0 THEN 1 ELSE 0 END) AS has_non_root
FROM ai_traces WHERE `+whereClause+`
GROUP BY trace_name`, params)
if err != nil {
@@ -213,6 +225,8 @@ func (r *aiTraceRepository) FindGroupedByTraceName(ctx context.Context, projectI
AvgInputTokens: row.AvgInputTokens,
AvgOutputTokens: row.AvgOutputTokens,
LastSeen: lastSeen,
+ HasRoot: row.HasRoot,
+ HasNonRoot: row.HasNonRoot,
})
}
@@ -286,7 +300,7 @@ func (r *aiTraceRepository) FindByTraceName(ctx context.Context, projectId uuid.
input_tokens, output_tokens, total_tokens, cached_tokens, reasoning_tokens,
input_cost, output_cost, total_cost,
trace_name, user_id, finish_reason, server_name, app_version,
- storage_key, attributes
+ storage_key, attributes, distributed_trace_id, is_root
FROM ai_traces
WHERE project_id = :project_id AND trace_name = :trace_name AND recorded_at >= :from AND recorded_at <= :to
ORDER BY %s %s LIMIT :limit OFFSET :offset`, orderBy, sortDir),
@@ -362,7 +376,7 @@ func (r *aiTraceRepository) FindById(ctx context.Context, projectId, traceId uui
input_tokens, output_tokens, total_tokens, cached_tokens, reasoning_tokens,
input_cost, output_cost, total_cost,
trace_name, user_id, finish_reason, server_name, app_version,
- storage_key, attributes
+ storage_key, attributes, distributed_trace_id, is_root
FROM ai_traces
WHERE project_id = :project_id AND id = :id
LIMIT 1`,
@@ -377,4 +391,52 @@ func (r *aiTraceRepository) FindById(ctx context.Context, projectId, traceId uui
return &result, nil
}
+func (r *aiTraceRepository) FindByDistributedTraceId(ctx context.Context, distributedTraceId uuid.UUID, projectIds []uuid.UUID) ([]models.AiTrace, error) {
+ if len(projectIds) == 0 {
+ return nil, nil
+ }
+ params := lit.P{"trace_id": distributedTraceId}
+ placeholders := make([]string, len(projectIds))
+ for i, pid := range projectIds {
+ key := fmt.Sprintf("pid_%d", i)
+ placeholders[i] = ":" + key
+ params[key] = pid
+ }
+ query := `SELECT id, project_id, recorded_at, duration, status_code,
+ model, response_model, provider, operation,
+ input_tokens, output_tokens, total_tokens, cached_tokens, reasoning_tokens,
+ input_cost, output_cost, total_cost,
+ trace_name, user_id, finish_reason, server_name, app_version,
+ storage_key, attributes, distributed_trace_id, is_root
+ FROM ai_traces WHERE distributed_trace_id = :trace_id AND project_id IN (` + strings.Join(placeholders, ",") + `)
+ ORDER BY recorded_at ASC`
+
+ parsedQuery, args, err := lit.ParseNamedQuery(db.Driver, query, params)
+ if err != nil {
+ return nil, err
+ }
+ sqlRows, err := db.TelemetryDB.QueryContext(ctx, parsedQuery, args...)
+ if err != nil {
+ return nil, err
+ }
+ defer sqlRows.Close()
+
+ var traces []models.AiTrace
+ for sqlRows.Next() {
+ var row aiTraceRow
+ if err := sqlRows.Scan(
+ &row.Id, &row.ProjectId, &row.RecordedAt, &row.Duration, &row.StatusCode,
+ &row.Model, &row.ResponseModel, &row.Provider, &row.Operation,
+ &row.InputTokens, &row.OutputTokens, &row.TotalTokens, &row.CachedTokens, &row.ReasoningTokens,
+ &row.InputCost, &row.OutputCost, &row.TotalCost,
+ &row.TraceName, &row.UserId, &row.FinishReason, &row.ServerName, &row.AppVersion,
+ &row.StorageKey, &row.Attributes, &row.DistributedTraceId, &row.IsRoot,
+ ); err != nil {
+ return nil, err
+ }
+ traces = append(traces, row.toModel())
+ }
+ return traces, nil
+}
+
var AiTraceRepository = aiTraceRepository{}
diff --git a/backend/app/repositories/endpoint.repository.go b/backend/app/repositories/endpoint.repository.go
index 93f1de61..a7b898b5 100644
--- a/backend/app/repositories/endpoint.repository.go
+++ b/backend/app/repositories/endpoint.repository.go
@@ -18,7 +18,7 @@ import (
type endpointRepository struct{}
func (e *endpointRepository) InsertAsync(ctx context.Context, lines []models.Endpoint) error {
- batch, err := chdb.Conn.PrepareBatch(clickhouse.Context(context.Background(), clickhouse.WithAsync(false)), "INSERT INTO endpoints (id, project_id, endpoint, duration, recorded_at, status_code, body_size, client_ip, attributes, app_version, server_name, distributed_trace_id, span_id, is_stream)")
+ batch, err := chdb.Conn.PrepareBatch(clickhouse.Context(context.Background(), clickhouse.WithAsync(false)), "INSERT INTO endpoints (id, project_id, endpoint, duration, recorded_at, status_code, body_size, client_ip, attributes, app_version, server_name, distributed_trace_id, span_id, is_stream, is_root)")
if err != nil {
return err
}
@@ -33,7 +33,11 @@ func (e *endpointRepository) InsertAsync(ctx context.Context, lines []models.End
if t.IsStream {
isStream = 1
}
- if err := batch.Append(t.Id, t.ProjectId, t.Endpoint, int64(t.Duration), t.RecordedAt, t.StatusCode, t.BodySize, t.ClientIP, attributesJSON, t.AppVersion, t.ServerName, t.DistributedTraceId, t.SpanId, isStream); err != nil {
+ isRoot := uint8(0)
+ if t.IsRoot {
+ isRoot = 1
+ }
+ if err := batch.Append(t.Id, t.ProjectId, t.Endpoint, int64(t.Duration), t.RecordedAt, t.StatusCode, t.BodySize, t.ClientIP, attributesJSON, t.AppVersion, t.ServerName, t.DistributedTraceId, t.SpanId, isStream, isRoot); err != nil {
return err
}
}
@@ -91,7 +95,7 @@ func (e *endpointRepository) FindAll(ctx context.Context, projectId uuid.UUID, f
return endpoints, int64(count), nil
}
-func (e *endpointRepository) FindGroupedByEndpoint(ctx context.Context, projectId uuid.UUID, fromDate, toDate time.Time, page, pageSize int, orderBy string, sortDirection string, search string) ([]models.EndpointStats, int64, error) {
+func (e *endpointRepository) FindGroupedByEndpoint(ctx context.Context, projectId uuid.UUID, fromDate, toDate time.Time, page, pageSize int, orderBy string, sortDirection string, search string, rootFilter string) ([]models.EndpointStats, int64, error) {
// Build WHERE clause with optional search filter
// Count query uses bare column names; main query uses e. prefix for LEFT JOIN
whereClause := "project_id = ? AND recorded_at >= ? AND recorded_at <= ?"
@@ -104,6 +108,15 @@ func (e *endpointRepository) FindGroupedByEndpoint(ctx context.Context, projectI
args = append(args, search)
}
+ switch rootFilter {
+ case "root":
+ whereClause += " AND is_root = 1"
+ joinWhereClause += " AND e.is_root = 1"
+ case "non_root":
+ whereClause += " AND is_root = 0"
+ joinWhereClause += " AND e.is_root = 0"
+ }
+
// Count unique endpoints
var count uint64
countQuery := "SELECT uniq(endpoint) FROM endpoints WHERE " + whereClause
@@ -140,7 +153,7 @@ func (e *endpointRepository) FindGroupedByEndpoint(ctx context.Context, projectI
endpoint, total_count as count, p50_duration, p95_duration, p99_duration,
avg_duration, last_seen, offset_ms,
satisfied_count, tolerating_count, bad_count, client_error_count,
- server_error_count, is_stream,
+ server_error_count, is_stream, has_root, has_non_root,
if(is_stream = 1,
-- Streaming endpoints: skip Apdex + P99 (duration-driven, meaningless
-- for connection lifetimes). Substitute server_error_count for
@@ -213,9 +226,11 @@ func (e *endpointRepository) FindGroupedByEndpoint(ctx context.Context, projectI
countIf(duration > (1500000000 + toInt64(offset_ms) * 1000000)
OR status_code >= 500) as bad_count,
countIf(status_code >= 400 AND status_code < 500) as client_error_count,
- countIf(status_code >= 500) as server_error_count
+ countIf(status_code >= 500) as server_error_count,
+ max(is_root) as has_root,
+ max(if(is_root = 0, 1, 0)) as has_non_root
FROM (
- SELECT e.endpoint, e.duration, e.status_code, e.recorded_at, e.is_stream,
+ SELECT e.endpoint, e.duration, e.status_code, e.recorded_at, e.is_stream, e.is_root,
s.offset_ms as offset_ms
FROM endpoints e
LEFT JOIN (SELECT * FROM slow_endpoints FINAL) AS s
@@ -242,13 +257,15 @@ func (e *endpointRepository) FindGroupedByEndpoint(ctx context.Context, projectI
var p50, p95, p99, avg float64
var offsetMs uint32
var satisfiedCount, toleratingCount, badCount, clientErrorCount, serverErrorCount uint64
- var isStream uint8
+ var isStream, hasRoot, hasNonRoot uint8
if err := rows.Scan(&s.Endpoint, &s.Count, &p50, &p95, &p99, &avg, &s.LastSeen,
&offsetMs, &satisfiedCount, &toleratingCount, &badCount, &clientErrorCount,
- &serverErrorCount, &isStream, &s.Impact); err != nil {
+ &serverErrorCount, &isStream, &hasRoot, &hasNonRoot, &s.Impact); err != nil {
return nil, 0, err
}
s.IsStream = isStream == 1
+ s.HasRoot = hasRoot == 1
+ s.HasNonRoot = hasNonRoot == 1
if s.IsStream {
// Latency/Apdex are zero for streams. Impact came from the
// stream-specific branch of the SQL (status-code components only);
@@ -321,17 +338,19 @@ func (e *endpointRepository) FindByEndpoint(ctx context.Context, projectId uuid.
// FindById returns a single endpoint by ID
func (e *endpointRepository) FindById(ctx context.Context, projectId, endpointId uuid.UUID) (*models.Endpoint, error) {
- query := `SELECT id, project_id, endpoint, duration, recorded_at, status_code, body_size, client_ip, attributes, app_version, server_name, distributed_trace_id, span_id
+ query := `SELECT id, project_id, endpoint, duration, recorded_at, status_code, body_size, client_ip, attributes, app_version, server_name, distributed_trace_id, span_id, is_root
FROM endpoints
WHERE project_id = ? AND id = ?
LIMIT 1`
var t models.Endpoint
var attributesJSON string
+ var isRoot uint8
err := chdb.Conn.QueryRow(ctx, query, projectId, endpointId).Scan(
&t.Id, &t.ProjectId, &t.Endpoint, &t.Duration, &t.RecordedAt,
- &t.StatusCode, &t.BodySize, &t.ClientIP, &attributesJSON, &t.AppVersion, &t.ServerName, &t.DistributedTraceId, &t.SpanId)
+ &t.StatusCode, &t.BodySize, &t.ClientIP, &attributesJSON, &t.AppVersion, &t.ServerName, &t.DistributedTraceId, &t.SpanId, &isRoot)
+ t.IsRoot = isRoot == 1
if err != nil {
if errors.Is(err, sql.ErrNoRows) {
diff --git a/backend/app/repositories/endpoint.repository_sqlite.go b/backend/app/repositories/endpoint.repository_sqlite.go
index 8767aba5..6924f8ea 100644
--- a/backend/app/repositories/endpoint.repository_sqlite.go
+++ b/backend/app/repositories/endpoint.repository_sqlite.go
@@ -32,6 +32,7 @@ type endpoint struct {
DistributedTraceId *uuid.UUID `lit:"distributed_trace_id"`
SpanId *uuid.UUID `lit:"span_id"`
IsStream bool `lit:"is_stream"`
+ IsRoot bool `lit:"is_root"`
}
type groupedEndpointRow struct {
@@ -46,6 +47,8 @@ type groupedEndpointRow struct {
ToleratingCount uint64 `lit:"tolerating_count"`
BadCount uint64 `lit:"bad_count"`
IsStream bool `lit:"is_stream"`
+ HasRoot bool `lit:"has_root"`
+ HasNonRoot bool `lit:"has_non_root"`
}
type endpointDurationRow struct {
@@ -106,6 +109,7 @@ func endpointToRow(e models.Endpoint) endpoint {
DistributedTraceId: e.DistributedTraceId,
SpanId: e.SpanId,
IsStream: e.IsStream,
+ IsRoot: e.IsRoot,
}
}
@@ -124,6 +128,7 @@ func (r *endpoint) toModel() models.Endpoint {
DistributedTraceId: r.DistributedTraceId,
SpanId: r.SpanId,
IsStream: r.IsStream,
+ IsRoot: r.IsRoot,
}
if r.Attributes != nil {
e.Attributes = map[string]string(r.Attributes)
@@ -131,6 +136,20 @@ func (r *endpoint) toModel() models.Endpoint {
return e
}
+// rootFilterClause returns a SQL fragment ("", " AND <col> = 1", " AND <col> = 0")
+// to splice into a WHERE clause based on the rootFilter param. Accepts "all" |
+// "root" | "non_root"; defaults to "all" (no filter).
+func rootFilterClause(qualifiedCol, rootFilter string) string {
+ switch rootFilter {
+ case "root":
+ return " AND " + qualifiedCol + " = 1"
+ case "non_root":
+ return " AND " + qualifiedCol + " = 0"
+ default:
+ return ""
+ }
+}
+
type endpointRepository struct{}
func (e *endpointRepository) InsertAsync(ctx context.Context, lines []models.Endpoint) error {
@@ -199,7 +218,7 @@ func (e *endpointRepository) FindAll(ctx context.Context, projectId uuid.UUID, f
return endpoints, count, nil
}
-func (e *endpointRepository) FindGroupedByEndpoint(ctx context.Context, projectId uuid.UUID, fromDate, toDate time.Time, page, pageSize int, orderBy string, sortDirection string, search string) ([]models.EndpointStats, int64, error) {
+func (e *endpointRepository) FindGroupedByEndpoint(ctx context.Context, projectId uuid.UUID, fromDate, toDate time.Time, page, pageSize int, orderBy string, sortDirection string, search string, rootFilter string) ([]models.EndpointStats, int64, error) {
params := lit.P{"project_id": projectId, "from": NewSQLiteTime(fromDate), "to": NewSQLiteTime(toDate)}
whereClause := "e.project_id = :project_id AND e.recorded_at >= :from AND e.recorded_at <= :to"
@@ -207,6 +226,7 @@ func (e *endpointRepository) FindGroupedByEndpoint(ctx context.Context, projectI
whereClause += " AND INSTR(LOWER(e.endpoint), LOWER(:search)) > 0"
params["search"] = search
}
+ whereClause += rootFilterClause("e.is_root", rootFilter)
countQuery := "SELECT COUNT(DISTINCT e.endpoint) AS count FROM endpoints e WHERE " + whereClause
totalResult, err := lit.SelectSingleNamed[models.CountResult](db.TelemetryDB, countQuery, params)
@@ -229,7 +249,9 @@ func (e *endpointRepository) FindGroupedByEndpoint(ctx context.Context, projectI
SUM(CASE WHEN e.duration <= (750000000 + COALESCE(s.offset_ms, 0) * 1000000) AND e.status_code < 500 THEN 1 ELSE 0 END) as satisfied_count,
SUM(CASE WHEN e.duration > (750000000 + COALESCE(s.offset_ms, 0) * 1000000) AND e.duration <= (1500000000 + COALESCE(s.offset_ms, 0) * 1000000) AND e.status_code < 500 THEN 1 ELSE 0 END) as tolerating_count,
SUM(CASE WHEN e.duration > (1500000000 + COALESCE(s.offset_ms, 0) * 1000000) OR e.status_code >= 500 THEN 1 ELSE 0 END) as bad_count,
- MAX(e.is_stream) as is_stream
+ MAX(e.is_stream) as is_stream,
+ MAX(e.is_root) as has_root,
+ MAX(CASE WHEN e.is_root = 0 THEN 1 ELSE 0 END) as has_non_root
FROM endpoints e
LEFT JOIN slow_endpoints s ON e.endpoint = s.endpoint AND e.project_id = s.project_id
WHERE ` + whereClause + `
@@ -251,7 +273,7 @@ func (e *endpointRepository) FindGroupedByEndpoint(ctx context.Context, projectI
var g groupedEndpointRow
if err := sqlRows.Scan(&g.Endpoint, &g.TotalCount, &g.AvgDuration, &g.LastSeen,
&g.OffsetMs, &g.ServerErrorCount, &g.ClientErrorCount,
- &g.SatisfiedCount, &g.ToleratingCount, &g.BadCount, &g.IsStream); err != nil {
+ &g.SatisfiedCount, &g.ToleratingCount, &g.BadCount, &g.IsStream, &g.HasRoot, &g.HasNonRoot); err != nil {
return nil, 0, err
}
groups = append(groups, g)
@@ -274,6 +296,8 @@ func (e *endpointRepository) FindGroupedByEndpoint(ctx context.Context, projectI
Impact: ComputeStreamImpact(g.Endpoint, g.TotalCount, g.ServerErrorCount, g.ClientErrorCount),
ImpactReason: ComputeStreamImpactReason(g.Endpoint, g.TotalCount, g.ServerErrorCount, g.ClientErrorCount),
IsStream: true,
+ HasRoot: g.HasRoot,
+ HasNonRoot: g.HasNonRoot,
})
continue
}
@@ -300,6 +324,8 @@ func (e *endpointRepository) FindGroupedByEndpoint(ctx context.Context, projectI
Impact: impact,
ImpactReason: ComputeImpactReason(g.Endpoint, g.TotalCount, g.SatisfiedCount, g.ToleratingCount,
g.BadCount, g.ClientErrorCount, p99, g.OffsetMs),
+ HasRoot: g.HasRoot,
+ HasNonRoot: g.HasNonRoot,
})
}
diff --git a/backend/app/repositories/endpoint_repository_test.go b/backend/app/repositories/endpoint_repository_test.go
index c532f142..05d63651 100644
--- a/backend/app/repositories/endpoint_repository_test.go
+++ b/backend/app/repositories/endpoint_repository_test.go
@@ -134,7 +134,7 @@ func TestEndpointRepository_FindGroupedByEndpoint(t *testing.T) {
t.Fatalf("InsertAsync failed: %v", err)
}
- stats, total, err := EndpointRepository.FindGroupedByEndpoint(ctx, projectId, now.Add(-time.Hour), now.Add(time.Hour), 1, 10, "count", "desc", "")
+ stats, total, err := EndpointRepository.FindGroupedByEndpoint(ctx, projectId, now.Add(-time.Hour), now.Add(time.Hour), 1, 10, "count", "desc", "", "")
if err != nil {
t.Fatalf("FindGroupedByEndpoint failed: %v", err)
}
@@ -170,7 +170,7 @@ func TestEndpointRepository_FindGroupedByEndpoint_Search(t *testing.T) {
t.Fatalf("InsertAsync failed: %v", err)
}
- stats, total, err := EndpointRepository.FindGroupedByEndpoint(ctx, projectId, now.Add(-time.Hour), now.Add(time.Hour), 1, 10, "count", "desc", "users")
+ stats, total, err := EndpointRepository.FindGroupedByEndpoint(ctx, projectId, now.Add(-time.Hour), now.Add(time.Hour), 1, 10, "count", "desc", "users", "")
if err != nil {
t.Fatalf("FindGroupedByEndpoint with search failed: %v", err)
}
diff --git a/backend/app/repositories/task.repository.go b/backend/app/repositories/task.repository.go
index 990630af..14bc51e8 100644
--- a/backend/app/repositories/task.repository.go
+++ b/backend/app/repositories/task.repository.go
@@ -18,7 +18,7 @@ import (
type taskRepository struct{}
func (e *taskRepository) InsertAsync(ctx context.Context, lines []models.Task) error {
- batch, err := chdb.Conn.PrepareBatch(clickhouse.Context(context.Background(), clickhouse.WithAsync(false)), "INSERT INTO tasks (id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name, distributed_trace_id, span_id)")
+ batch, err := chdb.Conn.PrepareBatch(clickhouse.Context(context.Background(), clickhouse.WithAsync(false)), "INSERT INTO tasks (id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name, distributed_trace_id, span_id, is_root)")
if err != nil {
return err
}
@@ -29,7 +29,11 @@ func (e *taskRepository) InsertAsync(ctx context.Context, lines []models.Task) e
attributesJSON = string(attributesBytes)
}
}
- if err := batch.Append(t.Id, t.ProjectId, t.TaskName, int64(t.Duration), t.RecordedAt, t.ClientIP, attributesJSON, t.AppVersion, t.ServerName, t.DistributedTraceId, t.SpanId); err != nil {
+ isRoot := uint8(0)
+ if t.IsRoot {
+ isRoot = 1
+ }
+ if err := batch.Append(t.Id, t.ProjectId, t.TaskName, int64(t.Duration), t.RecordedAt, t.ClientIP, attributesJSON, t.AppVersion, t.ServerName, t.DistributedTraceId, t.SpanId, isRoot); err != nil {
return err
}
}
@@ -85,10 +89,22 @@ func (e *taskRepository) FindAll(ctx context.Context, projectId uuid.UUID, fromD
return tasks, int64(count), nil
}
-func (e *taskRepository) FindGroupedByTaskName(ctx context.Context, projectId uuid.UUID, fromDate, toDate time.Time, page, pageSize int, orderBy string, sortDirection string) ([]models.TaskStats, int64, error) {
- // Count unique task names
+func (e *taskRepository) FindGroupedByTaskName(ctx context.Context, projectId uuid.UUID, fromDate, toDate time.Time, page, pageSize int, orderBy string, sortDirection string, search string, rootFilter string) ([]models.TaskStats, int64, error) {
+ whereClause := "project_id = ? AND recorded_at >= ? AND recorded_at <= ?"
+ args := []interface{}{projectId, fromDate, toDate}
+ if search != "" {
+ whereClause += " AND positionCaseInsensitive(task_name, ?) > 0"
+ args = append(args, search)
+ }
+ switch rootFilter {
+ case "root":
+ whereClause += " AND is_root = 1"
+ case "non_root":
+ whereClause += " AND is_root = 0"
+ }
+
var count uint64
- err := chdb.Conn.QueryRow(ctx, "SELECT uniq(task_name) FROM tasks WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?", projectId, fromDate, toDate).Scan(&count)
+ err := chdb.Conn.QueryRow(ctx, "SELECT uniq(task_name) FROM tasks WHERE "+whereClause, args...).Scan(&count)
if err != nil {
return nil, 0, err
}
@@ -122,14 +138,17 @@ func (e *taskRepository) FindGroupedByTaskName(ctx context.Context, projectId uu
quantile(0.5)(duration) as p50_duration,
quantile(0.95)(duration) as p95_duration,
avg(duration) as avg_duration,
- max(recorded_at) as last_seen
+ max(recorded_at) as last_seen,
+ max(is_root) as has_root,
+ max(if(is_root = 0, 1, 0)) as has_non_root
FROM tasks
- WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?
+ WHERE ` + whereClause + `
GROUP BY task_name
ORDER BY ` + orderExpr + ` ` + sortDir + `
LIMIT ? OFFSET ?`
- rows, err := chdb.Conn.Query(ctx, query, projectId, fromDate, toDate, pageSize, offset)
+ queryArgs := append(args, pageSize, offset)
+ rows, err := chdb.Conn.Query(ctx, query, queryArgs...)
if err != nil {
return nil, 0, err
}
@@ -139,12 +158,15 @@ func (e *taskRepository) FindGroupedByTaskName(ctx context.Context, projectId uu
for rows.Next() {
var s models.TaskStats
var p50, p95, avg float64
- if err := rows.Scan(&s.TaskName, &s.Count, &p50, &p95, &avg, &s.LastSeen); err != nil {
+ var hasRoot, hasNonRoot uint8
+ if err := rows.Scan(&s.TaskName, &s.Count, &p50, &p95, &avg, &s.LastSeen, &hasRoot, &hasNonRoot); err != nil {
return nil, 0, err
}
s.P50Duration = time.Duration(p50)
s.P95Duration = time.Duration(p95)
s.AvgDuration = time.Duration(avg)
+ s.HasRoot = hasRoot == 1
+ s.HasNonRoot = hasNonRoot == 1
stats = append(stats, s)
}
@@ -202,17 +224,19 @@ func (e *taskRepository) FindByTaskName(ctx context.Context, projectId uuid.UUID
// FindById returns a single task by ID
func (e *taskRepository) FindById(ctx context.Context, projectId, taskId uuid.UUID) (*models.Task, error) {
- query := `SELECT id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name, distributed_trace_id, span_id
+ query := `SELECT id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name, distributed_trace_id, span_id, is_root
FROM tasks
WHERE project_id = ? AND id = ?
LIMIT 1`
var t models.Task
var attributesJSON string
+ var isRoot uint8
err := chdb.Conn.QueryRow(ctx, query, projectId, taskId).Scan(
&t.Id, &t.ProjectId, &t.TaskName, &t.Duration, &t.RecordedAt,
- &t.ClientIP, &attributesJSON, &t.AppVersion, &t.ServerName, &t.DistributedTraceId, &t.SpanId)
+ &t.ClientIP, &attributesJSON, &t.AppVersion, &t.ServerName, &t.DistributedTraceId, &t.SpanId, &isRoot)
+ t.IsRoot = isRoot == 1
if err != nil {
if errors.Is(err, sql.ErrNoRows) {
diff --git a/backend/app/repositories/task.repository_sqlite.go b/backend/app/repositories/task.repository_sqlite.go
index 124a2bb0..eb5bd8bd 100644
--- a/backend/app/repositories/task.repository_sqlite.go
+++ b/backend/app/repositories/task.repository_sqlite.go
@@ -27,6 +27,7 @@ type task struct {
ServerName string `lit:"server_name"`
DistributedTraceId *uuid.UUID `lit:"distributed_trace_id"`
SpanId *uuid.UUID `lit:"span_id"`
+ IsRoot bool `lit:"is_root"`
}
type taskGroupRow struct {
@@ -34,6 +35,8 @@ type taskGroupRow struct {
Count uint64 `lit:"count"`
AvgDuration float64 `lit:"avg_duration"`
LastSeen string `lit:"last_seen"`
+ HasRoot bool `lit:"has_root"`
+ HasNonRoot bool `lit:"has_non_root"`
}
type taskCountStatsRow struct {
@@ -67,6 +70,7 @@ func taskToRow(t models.Task) task {
ServerName: t.ServerName,
DistributedTraceId: t.DistributedTraceId,
SpanId: t.SpanId,
+ IsRoot: t.IsRoot,
}
}
@@ -82,6 +86,7 @@ func (r *task) toModel() models.Task {
ServerName: r.ServerName,
DistributedTraceId: r.DistributedTraceId,
SpanId: r.SpanId,
+ IsRoot: r.IsRoot,
}
if r.Attributes != nil {
t.Attributes = map[string]string(r.Attributes)
@@ -155,11 +160,17 @@ func (e *taskRepository) FindAll(ctx context.Context, projectId uuid.UUID, fromD
return tasks, count, nil
}
-func (e *taskRepository) FindGroupedByTaskName(ctx context.Context, projectId uuid.UUID, fromDate, toDate time.Time, page, pageSize int, orderBy string, sortDirection string) ([]models.TaskStats, int64, error) {
+func (e *taskRepository) FindGroupedByTaskName(ctx context.Context, projectId uuid.UUID, fromDate, toDate time.Time, page, pageSize int, orderBy string, sortDirection string, search string, rootFilter string) ([]models.TaskStats, int64, error) {
params := lit.P{"project_id": projectId, "from": NewSQLiteTime(fromDate), "to": NewSQLiteTime(toDate)}
+ whereClause := "project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to"
+ if search != "" {
+ whereClause += " AND INSTR(LOWER(task_name), LOWER(:search)) > 0"
+ params["search"] = search
+ }
+ whereClause += rootFilterClause("is_root", rootFilter)
totalResult, err := lit.SelectSingleNamed[models.CountResult](db.TelemetryDB,
- "SELECT COUNT(DISTINCT task_name) AS count FROM tasks WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to",
+ "SELECT COUNT(DISTINCT task_name) AS count FROM tasks WHERE "+whereClause,
params)
if err != nil {
return nil, 0, err
@@ -180,10 +191,16 @@ func (e *taskRepository) FindGroupedByTaskName(ctx context.Context, projectId uu
var baseQuery string
groupParams := lit.P{"project_id": projectId, "from": NewSQLiteTime(fromDate), "to": NewSQLiteTime(toDate)}
+ if search != "" {
+ groupParams["search"] = search
+ }
+
+ groupedCols := `task_name, COUNT(*) as count, AVG(duration) as avg_duration, MAX(recorded_at) as last_seen,
+ MAX(is_root) as has_root, MAX(CASE WHEN is_root = 0 THEN 1 ELSE 0 END) as has_non_root`
if needsGoSort {
- baseQuery = `SELECT task_name, COUNT(*) as count, AVG(duration) as avg_duration, MAX(recorded_at) as last_seen
- FROM tasks WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
+ baseQuery = `SELECT ` + groupedCols + `
+ FROM tasks WHERE ` + whereClause + `
GROUP BY task_name`
} else {
orderExpr := map[string]string{"count": "count", "last_seen": "last_seen"}
@@ -191,8 +208,8 @@ func (e *taskRepository) FindGroupedByTaskName(ctx context.Context, projectId uu
if !ok {
expr = "count"
}
- baseQuery = fmt.Sprintf(`SELECT task_name, COUNT(*) as count, AVG(duration) as avg_duration, MAX(recorded_at) as last_seen
- FROM tasks WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
+ baseQuery = fmt.Sprintf(`SELECT `+groupedCols+`
+ FROM tasks WHERE `+whereClause+`
GROUP BY task_name ORDER BY %s %s LIMIT :limit OFFSET :offset`, expr, sortDir)
groupParams["limit"] = pageSize
groupParams["offset"] = offset
@@ -218,6 +235,8 @@ func (e *taskRepository) FindGroupedByTaskName(ctx context.Context, projectId uu
P95Duration: time.Duration(computePercentile(durations, 0.95)),
AvgDuration: time.Duration(g.AvgDuration),
LastSeen: ls,
+ HasRoot: g.HasRoot,
+ HasNonRoot: g.HasNonRoot,
})
}
@@ -307,7 +326,7 @@ func (e *taskRepository) FindByTaskName(ctx context.Context, projectId uuid.UUID
func (e *taskRepository) FindById(ctx context.Context, projectId, taskId uuid.UUID) (*models.Task, error) {
row, err := lit.SelectSingleNamed[task](db.TelemetryDB,
- `SELECT id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name, distributed_trace_id, span_id
+ `SELECT id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name, distributed_trace_id, span_id, is_root
FROM tasks WHERE project_id = :project_id AND id = :id LIMIT 1`,
lit.P{"project_id": projectId, "id": taskId})
if err != nil {
diff --git a/backend/app/repositories/task_repository_test.go b/backend/app/repositories/task_repository_test.go
index f9cf3816..f68e8bdb 100644
--- a/backend/app/repositories/task_repository_test.go
+++ b/backend/app/repositories/task_repository_test.go
@@ -141,7 +141,7 @@ func TestTaskRepository_FindGroupedByTaskName(t *testing.T) {
t.Fatalf("InsertAsync failed: %v", err)
}
- stats, total, err := TaskRepository.FindGroupedByTaskName(ctx, projectId, now.Add(-time.Hour), now.Add(time.Hour), 1, 10, "count", "desc")
+ stats, total, err := TaskRepository.FindGroupedByTaskName(ctx, projectId, now.Add(-time.Hour), now.Add(time.Hour), 1, 10, "count", "desc", "", "")
if err != nil {
t.Fatalf("FindGroupedByTaskName failed: %v", err)
}
@@ -162,6 +162,55 @@ func TestTaskRepository_FindGroupedByTaskName(t *testing.T) {
}
}
+func TestTaskRepository_FindGroupedByTaskName_RootFilterAndSearch(t *testing.T) {
+ setupTestDB(t)
+ ctx := context.Background()
+ projectId := uuid.New()
+ now := truncateMs(time.Now().UTC())
+
+ rootTask := makeTask(projectId, "scheduler.cleanup", 100*time.Millisecond, now)
+ rootTask.IsRoot = true
+ nonRootTask := makeTask(projectId, "queue.process", 200*time.Millisecond, now.Add(time.Minute))
+ nonRootTask.IsRoot = false
+
+ if err := TaskRepository.InsertAsync(ctx, []models.Task{rootTask, nonRootTask}); err != nil {
+ t.Fatalf("InsertAsync failed: %v", err)
+ }
+
+ // rootFilter=root → only the cleanup task
+ stats, _, err := TaskRepository.FindGroupedByTaskName(ctx, projectId, now.Add(-time.Hour), now.Add(time.Hour), 1, 10, "count", "desc", "", "root")
+ if err != nil {
+ t.Fatalf("FindGroupedByTaskName(root) failed: %v", err)
+ }
+ if len(stats) != 1 || stats[0].TaskName != "scheduler.cleanup" {
+ t.Errorf("rootFilter=root: expected 1 stats row 'scheduler.cleanup', got %+v", stats)
+ }
+ if !stats[0].HasRoot || stats[0].HasNonRoot {
+ t.Errorf("expected HasRoot=true, HasNonRoot=false, got %+v", stats[0])
+ }
+
+ // rootFilter=non_root → only the queue.process task
+ stats, _, err = TaskRepository.FindGroupedByTaskName(ctx, projectId, now.Add(-time.Hour), now.Add(time.Hour), 1, 10, "count", "desc", "", "non_root")
+ if err != nil {
+ t.Fatalf("FindGroupedByTaskName(non_root) failed: %v", err)
+ }
+ if len(stats) != 1 || stats[0].TaskName != "queue.process" {
+ t.Errorf("rootFilter=non_root: expected 1 stats row 'queue.process', got %+v", stats)
+ }
+ if stats[0].HasRoot || !stats[0].HasNonRoot {
+ t.Errorf("expected HasRoot=false, HasNonRoot=true, got %+v", stats[0])
+ }
+
+ // search=queue → only matching task
+ stats, _, err = TaskRepository.FindGroupedByTaskName(ctx, projectId, now.Add(-time.Hour), now.Add(time.Hour), 1, 10, "count", "desc", "queue", "")
+ if err != nil {
+ t.Fatalf("FindGroupedByTaskName(search) failed: %v", err)
+ }
+ if len(stats) != 1 || stats[0].TaskName != "queue.process" {
+ t.Errorf("search=queue: expected 1 stats row 'queue.process', got %+v", stats)
+ }
+}
+
func TestTaskRepository_FindByTaskName(t *testing.T) {
setupTestDB(t)
ctx := context.Background()
diff --git a/backend/app/repositories/testhelper_sqlite_test.go b/backend/app/repositories/testhelper_sqlite_test.go
index a77ab8dd..fb99973e 100644
--- a/backend/app/repositories/testhelper_sqlite_test.go
+++ b/backend/app/repositories/testhelper_sqlite_test.go
@@ -67,7 +67,9 @@ CREATE TABLE IF NOT EXISTS endpoints (
app_version TEXT NOT NULL DEFAULT '',
server_name TEXT NOT NULL DEFAULT '',
distributed_trace_id TEXT DEFAULT NULL,
- is_stream INTEGER NOT NULL DEFAULT 0
+ span_id TEXT DEFAULT NULL,
+ is_stream INTEGER NOT NULL DEFAULT 0,
+ is_root INTEGER NOT NULL DEFAULT 1
);
CREATE INDEX IF NOT EXISTS idx_endpoints_project_recorded ON endpoints(project_id, recorded_at);
CREATE INDEX IF NOT EXISTS idx_endpoints_project_endpoint ON endpoints(project_id, endpoint);
@@ -82,10 +84,42 @@ CREATE TABLE IF NOT EXISTS tasks (
attributes TEXT NOT NULL DEFAULT '{}',
app_version TEXT NOT NULL DEFAULT '',
server_name TEXT NOT NULL DEFAULT '',
- distributed_trace_id TEXT DEFAULT NULL
+ distributed_trace_id TEXT DEFAULT NULL,
+ span_id TEXT DEFAULT NULL,
+ is_root INTEGER NOT NULL DEFAULT 1
);
CREATE INDEX IF NOT EXISTS idx_tasks_project_recorded ON tasks(project_id, recorded_at);
+CREATE TABLE IF NOT EXISTS ai_traces (
+ id TEXT NOT NULL,
+ project_id TEXT NOT NULL,
+ recorded_at DATETIME NOT NULL,
+ duration INTEGER NOT NULL DEFAULT 0,
+ status_code INTEGER NOT NULL DEFAULT 0,
+ model TEXT NOT NULL DEFAULT '',
+ response_model TEXT NOT NULL DEFAULT '',
+ provider TEXT NOT NULL DEFAULT '',
+ operation TEXT NOT NULL DEFAULT '',
+ input_tokens INTEGER NOT NULL DEFAULT 0,
+ output_tokens INTEGER NOT NULL DEFAULT 0,
+ total_tokens INTEGER NOT NULL DEFAULT 0,
+ cached_tokens INTEGER NOT NULL DEFAULT 0,
+ reasoning_tokens INTEGER NOT NULL DEFAULT 0,
+ input_cost REAL NOT NULL DEFAULT 0,
+ output_cost REAL NOT NULL DEFAULT 0,
+ total_cost REAL NOT NULL DEFAULT 0,
+ trace_name TEXT NOT NULL DEFAULT '',
+ user_id TEXT NOT NULL DEFAULT '',
+ finish_reason TEXT NOT NULL DEFAULT '',
+ server_name TEXT NOT NULL DEFAULT '',
+ app_version TEXT NOT NULL DEFAULT '',
+ storage_key TEXT NOT NULL DEFAULT '',
+ attributes TEXT NOT NULL DEFAULT '{}',
+ distributed_trace_id TEXT DEFAULT NULL,
+ is_root INTEGER NOT NULL DEFAULT 1
+);
+CREATE INDEX IF NOT EXISTS idx_ai_traces_project_recorded ON ai_traces(project_id, recorded_at);
+
CREATE TABLE IF NOT EXISTS exception_stack_traces (
id TEXT NOT NULL,
project_id TEXT NOT NULL,
@@ -98,7 +132,8 @@ CREATE TABLE IF NOT EXISTS exception_stack_traces (
app_version TEXT NOT NULL DEFAULT '',
server_name TEXT NOT NULL DEFAULT '',
is_message INTEGER NOT NULL DEFAULT 0,
- distributed_trace_id TEXT DEFAULT NULL
+ distributed_trace_id TEXT DEFAULT NULL,
+ session_id TEXT DEFAULT NULL
);
CREATE INDEX IF NOT EXISTS idx_exceptions_project_recorded ON exception_stack_traces(project_id, recorded_at);
CREATE INDEX IF NOT EXISTS idx_exceptions_project_hash ON exception_stack_traces(project_id, exception_hash);
@@ -110,7 +145,8 @@ CREATE TABLE IF NOT EXISTS spans (
name TEXT NOT NULL DEFAULT '',
start_time DATETIME NOT NULL,
duration INTEGER NOT NULL DEFAULT 0,
- recorded_at DATETIME NOT NULL
+ recorded_at DATETIME NOT NULL,
+ parent_span_id TEXT DEFAULT NULL
);
CREATE INDEX IF NOT EXISTS idx_spans_project_trace ON spans(project_id, trace_id);
@@ -128,9 +164,12 @@ CREATE TABLE IF NOT EXISTS session_recordings (
project_id TEXT NOT NULL,
exception_id TEXT NOT NULL,
file_path TEXT NOT NULL DEFAULT '',
- recorded_at DATETIME NOT NULL
+ recorded_at DATETIME NOT NULL,
+ session_id TEXT DEFAULT NULL,
+ segment_index INTEGER NOT NULL DEFAULT 0
);
CREATE INDEX IF NOT EXISTS idx_session_recordings_project_exception ON session_recordings(project_id, exception_id);
+CREATE INDEX IF NOT EXISTS idx_session_recordings_session_id ON session_recordings(session_id);
CREATE TABLE IF NOT EXISTS archived_exceptions (
project_id TEXT NOT NULL,
diff --git a/examples/devtesting-embedded/go.mod b/examples/devtesting-embedded/go.mod
index a6ea6c02..ac745f7d 100644
--- a/examples/devtesting-embedded/go.mod
+++ b/examples/devtesting-embedded/go.mod
@@ -93,7 +93,7 @@ require (
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.43.0 // indirect
go.opentelemetry.io/otel/metric v1.43.0 // indirect
go.opentelemetry.io/proto/otlp v1.10.0 // indirect
- go.tracewayapp.com v1.0.2 // indirect
+ go.tracewayapp.com v1.0.3 // indirect
go.tracewayapp.com/tracewaygin v1.0.2 // indirect
go.yaml.in/yaml/v3 v3.0.4 // indirect
golang.org/x/arch v0.25.0 // indirect
diff --git a/frontend/src/lib/components/distributed-trace/distributed-trace-card.svelte b/frontend/src/lib/components/distributed-trace/distributed-trace-card.svelte
index a0177698..0a4ba3a1 100644
--- a/frontend/src/lib/components/distributed-trace/distributed-trace-card.svelte
+++ b/frontend/src/lib/components/distributed-trace/distributed-trace-card.svelte
@@ -44,6 +44,8 @@
projectsState.selectProject(node.projectId);
if (node.traceType === 'task' && node.task) {
goto(`/tasks/${encodeURIComponent(node.task.taskName)}/${node.task.id}?preset=24h`);
+ } else if (node.traceType === 'ai_trace' && node.aiTrace) {
+ goto(`/ai-traces/${encodeURIComponent(node.aiTrace.traceName)}/${node.aiTrace.id}?preset=24h`);
} else if (node.traceType === 'exception' && node.exception) {
goto(`/issues/${node.exception.exceptionHash}?preset=24h`);
} else if (node.endpoint) {
@@ -93,6 +95,8 @@
<span class="truncate font-mono text-sm">
{#if node.traceType === 'task'}
{node.task?.taskName}
+ {:else if node.traceType === 'ai_trace'}
+ {node.aiTrace?.traceName}
{:else if node.traceType === 'exception'}
{node.exception?.stackTrace.split('\n')[0]}
{:else}
@@ -104,9 +108,16 @@
{node.endpoint.statusCode}
</span>
{/if}
+ {#if node.traceType === 'ai_trace' && node.aiTrace}
+ <Badge variant="secondary" class="shrink-0">{node.aiTrace.provider || node.aiTrace.model || 'AI'}</Badge>
+ {/if}
{#if node.traceType !== 'exception'}
<span class="shrink-0 font-mono text-sm text-muted-foreground">
- {formatDuration(node.traceType === 'task' ? node.task?.duration ?? 0 : node.endpoint?.duration ?? 0)}
+ {formatDuration(
+ node.traceType === 'task' ? node.task?.duration ?? 0
+ : node.traceType === 'ai_trace' ? node.aiTrace?.duration ?? 0
+ : node.endpoint?.duration ?? 0
+ )}
</span>
{/if}
{#if node.exception}
diff --git a/frontend/src/lib/components/ui/non-root-chip/index.ts b/frontend/src/lib/components/ui/non-root-chip/index.ts
new file mode 100644
index 00000000..0bfa965e
--- /dev/null
+++ b/frontend/src/lib/components/ui/non-root-chip/index.ts
@@ -0,0 +1 @@
+export { default as NonRootChip } from './non-root-chip.svelte';
diff --git a/frontend/src/lib/components/ui/non-root-chip/non-root-chip.svelte b/frontend/src/lib/components/ui/non-root-chip/non-root-chip.svelte
new file mode 100644
index 00000000..65d04df8
--- /dev/null
+++ b/frontend/src/lib/components/ui/non-root-chip/non-root-chip.svelte
@@ -0,0 +1,30 @@
+<script lang="ts">
+ import { Badge } from '$lib/components/ui/badge';
+ import * as Tooltip from '$lib/components/ui/tooltip';
+
+ type Props = {
+ mixed?: boolean;
+ };
+
+ let { mixed = false }: Props = $props();
+
+ const label = $derived(mixed ? 'Mixed' : 'Non-root');
+ const tooltip = $derived(
+ mixed
+ ? 'Some runs were triggered by another trace — see the distributed trace view.'
+ : 'Triggered by another trace — see the distributed trace view.'
+ );
+</script>
+
+<Tooltip.Provider>
+ <Tooltip.Root>
+ <Tooltip.Trigger>
+ <Badge variant="secondary" class="ml-2 text-[10px]">
+ {label}
+ </Badge>
+ </Tooltip.Trigger>
+ <Tooltip.Content>
+ {tooltip}
+ </Tooltip.Content>
+ </Tooltip.Root>
+</Tooltip.Provider>
diff --git a/frontend/src/lib/components/ui/root-filter/index.ts b/frontend/src/lib/components/ui/root-filter/index.ts
new file mode 100644
index 00000000..2438e3e2
--- /dev/null
+++ b/frontend/src/lib/components/ui/root-filter/index.ts
@@ -0,0 +1 @@
+export { default as RootFilter } from './root-filter.svelte';
diff --git a/frontend/src/lib/components/ui/root-filter/root-filter.svelte b/frontend/src/lib/components/ui/root-filter/root-filter.svelte
new file mode 100644
index 00000000..745598de
--- /dev/null
+++ b/frontend/src/lib/components/ui/root-filter/root-filter.svelte
@@ -0,0 +1,30 @@
+<script lang="ts">
+ import * as Select from '$lib/components/ui/select';
+
+ type Props = {
+ value?: string;
+ };
+
+ let { value = $bindable('all') }: Props = $props();
+
+ const options = [
+ { value: 'all', label: 'All' },
+ { value: 'root', label: 'Root' },
+ { value: 'non_root', label: 'Non-root' }
+ ];
+
+ const label = $derived(options.find((o) => o.value === value)?.label ?? 'All');
+</script>
+
+<Select.Root type="single" bind:value>
+ <Select.Trigger class="h-9 w-[120px] rounded-none border-r-0 shadow-none">
+ {label}
+ </Select.Trigger>
+ <Select.Content>
+ {#each options as option}
+ <Select.Item value={option.value} label={option.label}>
+ {option.label}
+ </Select.Item>
+ {/each}
+ </Select.Content>
+</Select.Root>
diff --git a/frontend/src/lib/types/distributed-trace.ts b/frontend/src/lib/types/distributed-trace.ts
index 41e8054d..e22d8eae 100644
--- a/frontend/src/lib/types/distributed-trace.ts
+++ b/frontend/src/lib/types/distributed-trace.ts
@@ -1,7 +1,7 @@
export type DistributedTraceNode = {
projectId: string;
projectName: string;
- traceType: 'endpoint' | 'task' | 'exception';
+ traceType: 'endpoint' | 'task' | 'ai_trace' | 'exception';
endpoint?: {
id: string;
endpoint: string;
@@ -15,6 +15,16 @@ export type DistributedTraceNode = {
duration: number;
recordedAt: string;
};
+ aiTrace?: {
+ id: string;
+ traceName: string;
+ model: string;
+ provider: string;
+ duration: number;
+ totalTokens: number;
+ totalCost: number;
+ recordedAt: string;
+ };
spans: any[];
exception?: {
exceptionHash: string;
diff --git a/frontend/src/routes/ai-traces/+page.svelte b/frontend/src/routes/ai-traces/+page.svelte
index d166b4b9..8bfd098f 100644
--- a/frontend/src/routes/ai-traces/+page.svelte
+++ b/frontend/src/routes/ai-traces/+page.svelte
@@ -10,6 +10,8 @@
import { PaginationFooter } from "$lib/components/ui/pagination-footer";
import { TimeRangePicker } from "$lib/components/ui/time-range-picker";
import { SearchBar } from "$lib/components/ui/search-bar";
+ import { RootFilter } from "$lib/components/ui/root-filter";
+ import { NonRootChip } from "$lib/components/ui/non-root-chip";
import { browser } from '$app/environment';
import { CalendarDate } from "@internationalized/date";
import { projectsState } from '$lib/state/projects.svelte';
@@ -45,6 +47,8 @@
avgInputTokens: number;
avgOutputTokens: number;
lastSeen: string;
+ hasRoot: boolean;
+ hasNonRoot: boolean;
};
type SortField = 'count' | 'p50_duration' | 'p95_duration' | 'total_tokens' | 'total_cost' | 'last_seen';
@@ -59,14 +63,15 @@
let total = $state(0);
let totalPages = $state(0);
- // Parse URL params including search
+ // Parse URL params including search + rootFilter
function parseAiTracesUrlParams() {
- if (!browser) return { preset: '24h', from: null, to: null, search: '' };
+ if (!browser) return { preset: '24h', from: null, to: null, search: '', rootFilter: 'all' };
const params = new URLSearchParams(window.location.search);
const timeParams = parseTimeRangeFromUrl(timezone, '24h');
return {
...timeParams,
- search: params.get('search') || ''
+ search: params.get('search') || '',
+ rootFilter: params.get('rootFilter') || 'all'
};
}
@@ -74,8 +79,9 @@
const initialUrlParams = parseAiTracesUrlParams();
const initialRange = getResolvedTimeRange(initialUrlParams, timezone);
- // Search state
+ // Search + rootFilter state
let searchQuery = $state(initialUrlParams.search);
+ let rootFilter = $state(initialUrlParams.rootFilter);
// Date Range State
let selectedPreset = $state<string | null>(initialUrlParams.preset);
@@ -96,6 +102,9 @@
if (searchQuery.trim()) {
params.search = searchQuery.trim();
}
+ if (rootFilter && rootFilter !== 'all') {
+ params.rootFilter = rootFilter;
+ }
updateUrl(params, { pushToHistory });
}
@@ -110,6 +119,7 @@
toDate = dateToCalendarDate(range.to, timezone);
toTime = dateToTimeString(range.to, timezone);
searchQuery = urlParams.search;
+ rootFilter = urlParams.rootFilter;
page = 1;
loadData(false);
@@ -189,7 +199,8 @@
page: page,
pageSize: pageSize
},
- search: searchQuery.trim()
+ search: searchQuery.trim(),
+ rootFilter: rootFilter === 'all' ? '' : rootFilter
};
const response = await api.post('/ai-traces/grouped', requestBody, { projectId: projectsState.currentProjectId ?? undefined });
@@ -269,7 +280,11 @@
bind:value={searchQuery}
onSearch={handleSearch}
disabled={loading}
- />
+ >
+ {#snippet children()}
+ <RootFilter bind:value={rootFilter} />
+ {/snippet}
+ </SearchBar>
<!-- AI Traces Table -->
<div class="rounded-md border overflow-hidden">
@@ -367,6 +382,9 @@
>
<Table.Cell class="font-mono text-sm">
{trace.traceName}
+ {#if trace.hasNonRoot}
+ <NonRootChip mixed={trace.hasRoot && trace.hasNonRoot} />
+ {/if}
</Table.Cell>
<Table.Cell class="tabular-nums">
{formatCount(trace.count)}
diff --git a/frontend/src/routes/endpoints/+page.svelte b/frontend/src/routes/endpoints/+page.svelte
index 5e0e51e6..b65ae43f 100644
--- a/frontend/src/routes/endpoints/+page.svelte
+++ b/frontend/src/routes/endpoints/+page.svelte
@@ -16,6 +16,8 @@
import { PaginationFooter } from "$lib/components/ui/pagination-footer";
import { TimeRangePicker } from "$lib/components/ui/time-range-picker";
import { SearchBar } from "$lib/components/ui/search-bar";
+ import { RootFilter } from "$lib/components/ui/root-filter";
+ import { NonRootChip } from "$lib/components/ui/non-root-chip";
import { browser } from '$app/environment';
import { CalendarDate } from "@internationalized/date";
import { projectsState } from '$lib/state/projects.svelte';
@@ -54,6 +56,8 @@
impact: number;
impactReason: string;
isStream?: boolean;
+ hasRoot: boolean;
+ hasNonRoot: boolean;
};
type SortField = 'count' | 'p50_duration' | 'p95_duration' | 'p99_duration' | 'last_seen' | 'impact';
@@ -93,14 +97,15 @@
let total = $state(0);
let totalPages = $state(0);
- // Parse URL params including search
+ // Parse URL params including search + rootFilter
function parseEndpointsUrlParams() {
- if (!browser) return { preset: '24h', from: null, to: null, search: '' };
+ if (!browser) return { preset: '24h', from: null, to: null, search: '', rootFilter: 'all' };
const params = new URLSearchParams(window.location.search);
const timeParams = parseTimeRangeFromUrl(timezone, '24h');
return {
...timeParams,
- search: params.get('search') || ''
+ search: params.get('search') || '',
+ rootFilter: params.get('rootFilter') || 'all'
};
}
@@ -108,8 +113,9 @@
const initialUrlParams = parseEndpointsUrlParams();
const initialRange = getResolvedTimeRange(initialUrlParams, timezone);
- // Search state
+ // Search + rootFilter state
let searchQuery = $state(initialUrlParams.search);
+ let rootFilter = $state(initialUrlParams.rootFilter);
// Date Range State
let selectedPreset = $state<string | null>(initialUrlParams.preset);
@@ -130,6 +136,9 @@
if (searchQuery.trim()) {
params.search = searchQuery.trim();
}
+ if (rootFilter && rootFilter !== 'all') {
+ params.rootFilter = rootFilter;
+ }
updateUrl(params, { pushToHistory });
}
@@ -144,6 +153,7 @@
toDate = dateToCalendarDate(range.to, timezone);
toTime = dateToTimeString(range.to, timezone);
searchQuery = urlParams.search;
+ rootFilter = urlParams.rootFilter;
page = 1;
loadData(false);
@@ -264,7 +274,8 @@
page: page,
pageSize: pageSize
},
- search: searchQuery.trim()
+ search: searchQuery.trim(),
+ rootFilter: rootFilter === 'all' ? '' : rootFilter
};
const response = await api.post('/endpoints/grouped', requestBody, { projectId: projectsState.currentProjectId ?? undefined });
@@ -402,7 +413,11 @@
bind:value={searchQuery}
onSearch={handleSearch}
disabled={loading}
- />
+ >
+ {#snippet children()}
+ <RootFilter bind:value={rootFilter} />
+ {/snippet}
+ </SearchBar>
<!-- Performance Chart -->
<Card.Root class="pt-2">
@@ -550,6 +565,9 @@
{#if endpoint.isStream}
<Badge variant="outline" class="font-sans text-xs" title="Streaming response (SSE / WebSocket / long-poll). Excluded from latency, Apdex, and impact scoring.">Stream</Badge>
{/if}
+ {#if endpoint.hasNonRoot}
+ <NonRootChip mixed={endpoint.hasRoot && endpoint.hasNonRoot} />
+ {/if}
</span>
</Table.Cell>
<Table.Cell class="tabular-nums">
diff --git a/frontend/src/routes/tasks/+page.svelte b/frontend/src/routes/tasks/+page.svelte
index 6a552338..33cadd93 100644
--- a/frontend/src/routes/tasks/+page.svelte
+++ b/frontend/src/routes/tasks/+page.svelte
@@ -9,7 +9,11 @@
import { TableEmptyState } from "$lib/components/ui/table-empty-state";
import { PaginationFooter } from "$lib/components/ui/pagination-footer";
import { TimeRangePicker } from "$lib/components/ui/time-range-picker";
+ import { SearchBar } from '$lib/components/ui/search-bar';
+ import { RootFilter } from '$lib/components/ui/root-filter';
+ import { NonRootChip } from '$lib/components/ui/non-root-chip';
import { CalendarDate } from "@internationalized/date";
+ import { browser } from '$app/environment';
import { projectsState } from '$lib/state/projects.svelte';
import { createRowClickHandler } from '$lib/utils/navigation';
import { resolve } from '$app/paths';
@@ -39,6 +43,8 @@
p95Duration: number;
avgDuration: number;
lastSeen: string;
+ hasRoot: boolean;
+ hasNonRoot: boolean;
};
type SortField = 'count' | 'p50_duration' | 'p95_duration' | 'last_seen';
@@ -57,6 +63,19 @@
const initialUrlParams = parseTimeRangeFromUrl(timezone);
const initialRange = getResolvedTimeRange(initialUrlParams, timezone);
+ function readSearchAndFilterFromUrl() {
+ if (!browser) return { search: '', rootFilter: 'all' };
+ const params = new URLSearchParams(window.location.search);
+ return {
+ search: params.get('search') || '',
+ rootFilter: params.get('rootFilter') || 'all'
+ };
+ }
+
+ const initialSearchState = readSearchAndFilterFromUrl();
+ let searchQuery = $state(initialSearchState.search);
+ let rootFilter = $state(initialSearchState.rootFilter);
+
// Date Range State
let selectedPreset = $state<string | null>(initialUrlParams.preset);
let fromDate = $state<CalendarDate>(dateToCalendarDate(initialRange.from, timezone));
@@ -64,14 +83,14 @@
let fromTime = $state(dateToTimeString(initialRange.from, timezone));
let toTime = $state(dateToTimeString(initialRange.to, timezone));
- // Update URL with current time range
+ // Update URL with current time range + search + rootFilter
function updateTimeRangeUrl(pushToHistory = true) {
- updateUrl(
- selectedPreset
- ? { preset: selectedPreset }
- : { from: getFromDateTimeUTC(), to: getToDateTimeUTC() },
- { pushToHistory }
- );
+ const params: Record<string, string | undefined> = selectedPreset
+ ? { preset: selectedPreset }
+ : { from: getFromDateTimeUTC(), to: getToDateTimeUTC() };
+ if (searchQuery) params.search = searchQuery;
+ if (rootFilter && rootFilter !== 'all') params.rootFilter = rootFilter;
+ updateUrl(params, { pushToHistory });
}
// Handle browser back/forward navigation
@@ -149,6 +168,8 @@
toDate: getToDateTimeUTC(),
orderBy: orderBy,
sortDirection: sortDirection,
+ search: searchQuery,
+ rootFilter: rootFilter === 'all' ? '' : rootFilter,
pagination: {
page: page,
pageSize: pageSize
@@ -190,6 +211,11 @@
loadData(false);
}
+ function handleSearch() {
+ page = 1;
+ loadData(true);
+ }
+
onMount(() => {
// Add popstate listener for back/forward navigation
window.addEventListener('popstate', handlePopState);
@@ -211,7 +237,12 @@
<PageHeader title="Tasks" />
- <div class="flex flex-col">
+ <div class="flex flex-col gap-2 sm:flex-row sm:items-end">
+ <SearchBar placeholder="Search tasks..." bind:value={searchQuery} onSearch={handleSearch}>
+ {#snippet children()}
+ <RootFilter bind:value={rootFilter} />
+ {/snippet}
+ </SearchBar>
<TimeRangePicker
bind:fromDate
bind:toDate
@@ -292,6 +323,9 @@
>
<Table.Cell class="font-mono text-sm">
{task.taskName}
+ {#if task.hasNonRoot}
+ <NonRootChip mixed={task.hasRoot && task.hasNonRoot} />
+ {/if}
</Table.Cell>
<Table.Cell class="tabular-nums">
{formatCount(task.count)}