The only tool you need to know what is happening and how to fix it.
Go alertingexception-trackerissue-trackinglogsmetricsmonitoringobservability-platformopen-telemetryotelsession-replaytelemetrytraces
progress
dduxf97ed864184f
diff --git a/SPEC_NON_ROOT_TRACES.md b/SPEC_NON_ROOT_TRACES.mdnew file mode 100644index 00000000..1730c97f--- /dev/null+++ b/SPEC_NON_ROOT_TRACES.md@@ -0,0 +1,296 @@+# Non-Root Tasks & AI Traces++## Problem++`backend/app/controllers/otelcontrollers/trace_converter.go:98` classifies a span as an Endpoint, Task, or AiTrace only when `isRoot := len(span.ParentSpanId) == 0`. Three real-world cases violate that assumption:++1. **Queue jobs with propagated trace context.** keepsuit's `QueueInstrumentation::recordJobProcessing` (`vendor/keepsuit/laravel-opentelemetry/src/Instrumentation/QueueInstrumentation.php:106`) — and the equivalents in `opentelemetry-instrumentation-celery`, Symfony Messenger without `messenger_root_spans: true`, etc. — calls `setParent($context)` on the worker's CONSUMER span using the trace context serialized into the job payload by the dispatcher. The consumer span has `parent_span_id = <producer span id>` from the originating request, so `isRoot == false`. It falls through to the generic `spans` table and the Tasks page stays empty.+2. **Child LLM spans.** A request span (root) makes an LLM call as a child span. The child has `gen_ai.*` attributes but a non-empty `parent_span_id`, so it never reaches the `aiTraces = append(...)` branch and the AI Traces page stays empty.+3. **Scheduled console commands.** keepsuit's `ConsoleInstrumentation` (`ConsoleInstrumentation.php:77-79`) emits a root span with kind INTERNAL (the OTel default — no `setSpanKind` call) and the `console.command` attribute. It has no HTTP attrs, so today it falls into the `else { continue }` branch and is dropped. zentigo runs `schedule:run`, `bookings:expire-unconfirmed`, and `bookings:send-reminders` under this instrumentation — all silently dropped.++All three are the same line. The fix promotes spans by their Kind / attributes instead of by their root-ness, and adds enough schema and UI to make the resulting non-root rows distinguishable from real roots.++## Approach++Two new columns on the entity tables, one ingestion change, and one query-time change:++- **`is_root Bool`** on `endpoints` / `tasks` / `ai_traces` — explicit because once `distributed_trace_id` is auto-filled from the OTel trace_id, both the root and its non-root siblings carry the same `distributed_trace_id` and "is_root" is no longer derivable from `distributed_trace_id IS NULL`.+- **`distributed_trace_id Nullable(UUID)`** on `ai_traces` (already exists on the other three) — populated from the OTel trace_id, not a vendor attribute, so it works for any OTLP source out of the box.+- **Converter promotes by Kind/attrs**, not by root-ness. Non-root entities get an id derived from their own `span_id` (not the shared OTel trace_id) so they don't collide with the root entity's id.+- **Spans are re-rooted at ingestion to their nearest enclosing entity within the batch.** `spans.trace_id` (and `exception_stack_traces.trace_id`) is repurposed from "OTel trace_id" to "owning entity id" so each waterfall query stays clean.++### Decisions++- **Drop the `isRoot` gate on _all_ classifications, including endpoints.** Non-root SERVER+HTTP spans (cross-service inbound hops) become endpoint rows with `is_root=false`. The "Non-root" chip applies to endpoints too.+- **Promote console spans to tasks.** Detect via the `console.command` attribute keepsuit attaches. Root INTERNAL spans without HTTP attrs that carry `console.command` become task rows.+- **Keep `rootFilter` on the endpoints API/page** — endpoints can now legitimately be non-root.+- **Tasks search ships in the same PR.** Today the tasks page has no search; this change adds it alongside `rootFilter`.++### Worked example (queue case)++OTel trace_id `X`, spans:+- `A` — SERVER, root, HTTP attrs — request handler+- `B` — PRODUCER, parent=`A` — `send queue`+- `C` — CONSUMER, parent=`B` — `process queue` (arrives in a separate worker OTLP batch)+- `D` — child of `C` — `SELECT …` (arrives in the worker's batch)++Resulting rows:++| | `id` | `distributed_trace_id` | `is_root` |+|---|---|---|---|+| endpoints | `otelTraceIDToUUID(A.trace_id)` = **X** | **X** | true |+| tasks | `otelSpanIDToUUID(C.span_id)` = **Y** | **X** | false |+| spans | `B`'s span_id, `trace_id = X` (owner = endpoint) | — | — |+| spans | `D`'s span_id, `trace_id = Y` (owner = task) | — | — |++`SELECT spans WHERE trace_id = X` returns `B` only. `SELECT spans WHERE trace_id = Y` returns `D` only. `FindByDistributedTraceId(X)` returns both the endpoint and the task.++### Worked example (inline gen_ai case)++`A` is a SERVER root, `L` is a child of `A` with `gen_ai.*` attrs. They arrive in the same batch.++| | `id` | `distributed_trace_id` | `is_root` |+|---|---|---|---|+| endpoints | `X` (from `A.trace_id`) | `X` | true |+| ai_traces | `otelSpanIDToUUID(L.span_id)` = `Z` | `X` | false |++Spans between `A` and `L` (e.g. middleware/DB spans) get `trace_id = X` (owner = endpoint). Anything that turns out to be a descendant of `L` (rare — LLM spans are usually leaves; tool-call children would be the case) gets `trace_id = Z`.++### Worked example (scheduled console command)++`bookings:send-reminders` runs under keepsuit's `ConsoleInstrumentation`. The span has kind INTERNAL, no parent, no HTTP attrs, and attribute `console.command = "bookings:send-reminders"`. Children include the database queries it issues.++| | `id` | `distributed_trace_id` | `is_root` |+|---|---|---|---|+| tasks | `otelTraceIDToUUID(trace_id)` = **X** | **X** | true |+| spans | each DB child, `trace_id = X` (owner = task) | — | — |++---++## Backend++### Migrations++ClickHouse — one statement per file (per project rule). Next free number is 0058:++- `0058_add_is_root_to_endpoints.up.sql`+ ```sql+ ALTER TABLE endpoints ADD COLUMN is_root Bool DEFAULT true+ ```+- `0059_add_is_root_to_tasks.up.sql`+ ```sql+ ALTER TABLE tasks ADD COLUMN is_root Bool DEFAULT true+ ```+- `0060_add_distributed_trace_id_to_ai_traces.up.sql`+ ```sql+ ALTER TABLE ai_traces ADD COLUMN distributed_trace_id Nullable(UUID) DEFAULT NULL+ ```+- `0061_add_distributed_trace_id_index_ai_traces.up.sql`+ ```sql+ ALTER TABLE ai_traces ADD INDEX idx_distributed_trace_id_ai_traces distributed_trace_id TYPE bloom_filter(0.001) GRANULARITY 1+ ```+- `0062_add_is_root_to_ai_traces.up.sql`+ ```sql+ ALTER TABLE ai_traces ADD COLUMN is_root Bool DEFAULT true+ ```++SQLite (`backend/app/migrations/sqlite_telemetry/`, next number is 0009). The SQLite migration runner splits on `;` (see `migrations_sqlite.go:80`), so a single multi-statement file is fine:++- `0009_add_is_root_and_distributed_trace_id.up.sql`+ ```sql+ ALTER TABLE endpoints ADD COLUMN is_root INTEGER NOT NULL DEFAULT 1;+ ALTER TABLE tasks ADD COLUMN is_root INTEGER NOT NULL DEFAULT 1;+ ALTER TABLE ai_traces ADD COLUMN is_root INTEGER NOT NULL DEFAULT 1;+ ALTER TABLE ai_traces ADD COLUMN distributed_trace_id TEXT DEFAULT NULL;+ ```++> Default `true`/`1` so backfill is correct — every existing row was promoted under the old `isRoot` gate, so it was a root span by definition.++### Models++`backend/app/models/endpoint.model.go`, `task.model.go`, `ai_trace.model.go`:++- Add `IsRoot bool \`json:"isRoot" ch:"is_root"\``+- For `AiTrace`, also add `DistributedTraceId *uuid.UUID \`json:"distributedTraceId,omitempty" ch:"distributed_trace_id"\``++Aggregate result models — `TaskStats`, `AiTraceStats`, `EndpointStats` (`models/*.go` and inline in repositories) gain `HasRoot bool` and `HasNonRoot bool` so the grouped list page can show the chip on names that contain at least one non-root run.++### Converter (`backend/app/controllers/otelcontrollers/trace_converter.go`)++Rewrite the per-span classification block (currently lines 92–178). Two passes inside the existing `for _, rs := range req.ResourceSpans` loop.++**Pass 1 — classify each span as one of `endpoint`, `task`, `ai_trace`, or `span`.** Drop the `isRoot` gate from all three entity branches. Rule table:++| Condition | Promoted to |+|---|---|+| `Kind ∈ {SERVER, INTERNAL}` AND `hasHTTPAttributes(attrs)` | endpoint |+| `Kind == CONSUMER` | task |+| `Kind == INTERNAL` AND attribute `console.command` is present AND `isRoot` | task |+| `hasGenAiAttributes(attrs)` | ai_trace |+| otherwise | (handled in pass 2) |++For each promoted entity, compute its id:+- `is_root = len(span.ParentSpanId) == 0`+- if root: `id = otelTraceIDToUUID(span.TraceId)` (preserves today's invariant that the root entity's id equals the OTel trace_id)+- if non-root: `id = otelSpanIDToUUID(span.SpanId)` (distinct from any root entity sharing the same trace)+- `distributed_trace_id = otelTraceIDToUUID(span.TraceId)` for **both** root and non-root (this is the change from "vendor attribute only" to "auto-derived"). The vendor attribute `traceway.distributed_trace_id` still wins if present, for backwards compatibility with manually-linked traces from the Go SDK.++Build `spanIdToEntityId map[string]uuid.UUID` from these promotions (keyed by `string(span.SpanId)`).++> Console rule rationale: keepsuit's `ConsoleInstrumentation.commandStarting` sets `console.command` explicitly (`ConsoleInstrumentation.php:78`). We don't use "any root INTERNAL with no HTTP attrs" because it would scoop up arbitrary manual roots from other instrumentations. `console.command` is a concrete signal.++**Pass 2 — emit span rows for the unpromoted spans, re-rooted to their nearest enclosing entity.**++For each unpromoted span, walk parents via `parentMap` until you find one in `spanIdToEntityId`. That entity's id becomes the span's `trace_id`. If no enclosing entity exists in this batch, fall back to `otelTraceIDToUUID(span.TraceId)` (preserves today's behavior for orphan spans whose parent was emitted from another process and never matched any promoted entity — they remain linkable via `distributed_trace_id` on the cross-trace page).++Skip emitting a span row for any span that was itself promoted to an entity (today, only root entities are skipped; the new logic skips all promoted entities including non-root ones).++**Exceptions.** `exc.TraceId` is set to the owning entity's id (the same value the span's `trace_id` would have gotten), not the raw OTel trace_id. `traceType` is `"endpoint"` / `"task"` / `"ai_trace"` based on which kind owned it. Exceptions on unpromoted spans use the entity walked up the parent chain.++### Repositories++Files to update — every read/write that touches `endpoints`, `tasks`, `ai_traces`, `exception_stack_traces`, or `spans`. Each has a ClickHouse variant (`*.repository.go`) and a SQLite variant (`*.repository_sqlite.go`).++- `endpoint.repository.go` + `endpoint.repository_sqlite.go`:+ - `BatchInsert` — add `is_root` to the INSERT column list and `Append` call.+ - `FindAll`, `FindGroupedByEndpoint`, `FindByEndpoint`, `FindById`, `FindByDistributedTraceId` — add `is_root` to SELECT and Scan.+ - Add `rootFilter` parameter to `FindGroupedByEndpoint` (`all` | `root` | `non_root`); when set to `root` add `AND is_root = true`, when `non_root` add `AND is_root = false`. Grouped projection includes `has_root = max(is_root)` and `has_non_root = max(NOT is_root)` so the list row can show the chip if any run in the group is non-root.+- `task.repository.go` + `task.repository_sqlite.go`:+ - Same as endpoints: `BatchInsert`, all reads, root filter on `FindGroupedByTaskName`, `has_root` / `has_non_root` in the grouped projection.+ - Add `search` parameter (substring match on `task_name` — `positionCaseInsensitive` in ClickHouse, `LIKE` lowercased in SQLite). Currently tasks have no search; this brings them in line with endpoints.+ - Add `FindByDistributedTraceId(ctx, distributedTraceId, projectIds)` — mirrors `EndpointRepository.FindByDistributedTraceId`.+- `ai_trace.repository.go` + `ai_trace.repository_sqlite.go`:+ - `BatchInsert` — add `is_root` and `distributed_trace_id` columns.+ - All reads — add `is_root` and `distributed_trace_id` to SELECT/Scan.+ - Same `rootFilter` parameter + grouped projection as tasks.+ - Add `FindByDistributedTraceId` (mirrors `EndpointRepository.FindByDistributedTraceId`).+- `exception_stack_trace.repository.go` + `_sqlite`:+ - No schema change here — `trace_id` already exists. Only the value being stored changes (owning entity id, not raw OTel trace_id). Existing callers don't need to change.+- `span.repository.go` + `_sqlite`:+ - No schema change. `FindByTraceId(projectId, traceId)` still works — it will now be called with `endpoint.id` or `task.id` or `ai_trace.id` (the owning entity id), all of which are valid values to look up.++### Controllers++- `tasks.controller.go` — `FindGrouped` accepts new request fields `search string` and `rootFilter string`. Forward both to the repository. Emit `hasRoot` / `hasNonRoot` per row in the response.+- `ai_trace.controller.go` — same change as tasks (search already exists, just add `rootFilter` + `hasRoot` / `hasNonRoot`).+- `endpoints.controller.go` — `FindGrouped` accepts new `rootFilter` field. (Search already exists.) Emit `hasRoot` / `hasNonRoot` per row.+- `distributed_trace.controller.go` — add a third call to `repositories.AiTraceRepository.FindByDistributedTraceId` and emit `ai_trace` nodes in the response alongside `endpoint` and `task`. Render in the same `Nodes` list.+- Detail controllers (`endpoint.controller.go`, `task.controller.go`, `ai_trace.controller.go`) — include `isRoot` and `distributedTraceId` in their JSON responses so the detail pages can render the chip and the cross-trace link.++### Tests++`trace_converter_test.go` additions:++1. CONSUMER span with non-empty `ParentSpanId` →+ - one row in `tasks` with `is_root == false`, `id == otelSpanIDToUUID(span.SpanId)`, `distributed_trace_id == otelTraceIDToUUID(span.TraceId)`+ - no row in `endpoints`+ - any child spans in the batch land in `spans` with `trace_id == task.id`+2. Child span with `gen_ai.*` attrs under a SERVER root, both in one batch →+ - root endpoint row with `is_root == true`, `id == distributed_trace_id == otelTraceIDToUUID(trace_id)`+ - ai_trace row with `is_root == false`, `id == otelSpanIDToUUID(child.SpanId)`, same `distributed_trace_id` as the endpoint+ - intermediate spans (if any) get `trace_id == endpoint.id`; descendants of the ai_trace span get `trace_id == ai_trace.id`+3. Root INTERNAL span carrying `console.command` attribute, no HTTP attrs →+ - one row in `tasks` with `is_root == true`, `id == otelTraceIDToUUID(span.TraceId)`+ - child spans get `trace_id == task.id`+4. Non-root SERVER+HTTP span (cross-service inbound hop) →+ - one row in `endpoints` with `is_root == false`, `id == otelSpanIDToUUID(span.SpanId)`+5. Existing test — root SERVER + HTTP → endpoint with `is_root == true` — must still pass unchanged.+6. Exception event on the consumer span → `exc.TraceId == task.id`, `exc.TraceType == "task"`.+7. Orphan span whose parent isn't in the batch and which isn't itself promoted → falls back to `trace_id = otelTraceIDToUUID(span.TraceId)` (the OTel trace_id). Document this fallback in the test name.++Repository tests (`endpoint_repository_test.go`, `task_repository_test.go`, the soon-to-exist `ai_trace_repository_test.go`):+- `is_root` round-trips through insert + select.+- `rootFilter` parameter narrows results correctly (`all` / `root` / `non_root`).+- Grouped queries surface `has_root` and `has_non_root`.+- Task `search` parameter filters by task name substring.++---++## Frontend++### Shared component++New `frontend/src/lib/components/ui/root-filter/root-filter.svelte` — a `Select.Root` matching the existing inline-select styling, dropped via the SearchBar's `children` snippet so it joins the bar visually (mirroring how `issues/+page.svelte` uses `typeOptions`).++Options:+```ts+const rootFilterOptions = [+ { value: 'all', label: 'All' },+ { value: 'root', label: 'Root' },+ { value: 'non_root', label: 'Non-root' }+];+```++URL param `rootFilter` (`all` is the default and is omitted from the URL, matching how `searchType` is handled on the issues page).++### Chip++New `frontend/src/lib/components/ui/non-root-chip/non-root-chip.svelte` — small `Badge variant="secondary"` reading `Non-root` (or `Mixed` when both `hasRoot` and `hasNonRoot` are true), with tooltip "Triggered by another trace — see the distributed trace view." Rendered:+- On Endpoint, Task, and AI Trace list rows where `hasNonRoot && !hasRoot` ("Non-root") or where both flags are true ("Mixed").+- On detail pages (`/endpoints/[endpoint]/[id]`, `/tasks/[task]/[id]`, `/ai-traces/[traceName]/[id]`) next to the title when the specific instance being displayed has `isRoot === false`.++### `frontend/src/routes/tasks/+page.svelte`++Currently has no search. Add:+- `searchQuery` state + URL persistence (`?search=…`), copying the pattern from `ai-traces/+page.svelte` lines 62–112.+- `rootFilter` state + URL persistence (`?rootFilter=root` etc.).+- `<SearchBar placeholder="Search tasks..." bind:value={searchQuery} onSearch={handleSearch}>` wrapping the new `<RootFilter bind:value={rootFilter} />` in its `children` snippet.+- Send `search` and `rootFilter` in the `/tasks/grouped` POST body.+- Render `<NonRootChip mixed={task.hasRoot && task.hasNonRoot} />` next to `task.taskName` when `task.hasNonRoot` is true.++`/tasks/[task]/+page.svelte` (instance list): pass `rootFilter` through to `/tasks/task` and render the chip per-row using each instance's `isRoot`.++### `frontend/src/routes/ai-traces/+page.svelte`++Already has search. Add:+- `rootFilter` state + URL persistence.+- Pass the filter via the SearchBar `children` snippet (same as tasks).+- Send `rootFilter` in the API call.+- Render `<NonRootChip>` next to `trace.traceName` when `trace.hasNonRoot`.++`/ai-traces/[traceName]/+page.svelte` — chip per instance row using `aiTrace.isRoot`.++### `frontend/src/routes/endpoints/+page.svelte`++- `rootFilter` state + URL persistence (endpoints can now be non-root after this change).+- Render `<NonRootChip>` next to the endpoint name when `endpoint.hasNonRoot`.++`/endpoints/[endpoint]/+page.svelte` — chip per instance row using `endpoint.isRoot`.++### Distributed-trace page++Wherever `DistributedTraceResponse.Nodes` are rendered, add a case for `traceType === "ai_trace"` so the new third entity type shows up alongside endpoints and tasks. Reuse the existing node card styling.++### Detail pages — cross-trace link++On `/tasks/[task]/[id]`, `/ai-traces/[traceName]/[id]`, and `/endpoints/[endpoint]/[id]`, when `distributedTraceId` is set and there are sibling entities, show a "View distributed trace" link to `/distributed-traces/<distributedTraceId>`. (Today this link is only meaningful when the vendor attribute was set; after this change it's meaningful for every OTel-sourced trace.)++---++## Verification++1. **Unit/integration tests** — `cd backend && go test ./...` covers converter + repo round-trips on both ClickHouse and SQLite paths.+2. **End-to-end against a real Laravel/keepsuit app (zentigo is the canonical reproduction):**+ - Build backend on this branch, point the app at it via `OTEL_EXPORTER_OTLP_ENDPOINT`.+ - **Queue task (non-root):** `php artisan tinker --execute='\App\Jobs\TestTracewayJob::dispatch();'`, then run a queue worker. Confirm a new row appears on `/tasks` with the **Non-root** chip. Open it and confirm the "View distributed trace" link reaches a page showing both endpoint (dispatcher request) and task (worker) nodes.+ - **Console task (root):** `php artisan bookings:send-reminders`. Confirm a Tasks row with the command name, no chip (root).+ - **HTTP+queue together:** trigger a request that dispatches a notification (e.g. `NewReviewNotification`). On the endpoint detail page confirm the cross-trace link; on the distributed-trace view confirm both nodes.+ - **AI trace (non-root):** easiest is a synthetic OTLP payload (`curl` a hand-rolled JSON with a SERVER root and a child span carrying `gen_ai.system="openai"` etc.). Confirm the AI Traces page shows the child with the chip and that the cross-trace view includes endpoint + ai_trace nodes.+3. **`rootFilter` UI** — toggle the dropdown on `/tasks`, `/ai-traces`, `/endpoints` and confirm URL persistence (`?rootFilter=non_root` etc., `all` omitted from URL).+4. **Backfill sanity** — after migration run, query `SELECT count(), sum(is_root) FROM endpoints / tasks / ai_traces`; both numbers should match (every row defaults to `is_root=1`).++---++## Migration / Backfill Notes++- All existing rows default to `is_root = true` — correct because the old converter only ever inserted root spans.+- `ai_traces.distributed_trace_id` is `NULL` on existing rows. That's fine — the cross-trace lookup degrades gracefully and there's no UI that requires it to be present on historical rows.+- The change to `spans.trace_id` semantics (owning entity id, not OTel trace_id) is forward-only — existing span rows still have OTel trace_id stored there, which equals the endpoint's id for the single-process case (the only case the old converter produced), so existing waterfalls keep working.++---++## Out of Scope++- Cross-service SERVER hops where you'd want one node per hop in the cross-trace view — endpoints now accept non-root SERVER spans, but the UX for showing per-hop endpoint nodes (e.g., a tree view of cross-service requests) is a separate decision.+- Reconciling the legacy `/api/report` distributed-trace flow with the new auto-derivation. The Go SDK's opt-in `distributedTraceId` field on `clientmodels` continues to work unchanged.+- A "show me only the root trace" view at the distributed-trace page — current behavior shows everything, which is what users want for debugging.diff --git a/backend/app/controllers/ai_trace.controller.go b/backend/app/controllers/ai_trace.controller.goindex f141dc67..7fbf5398 100644--- a/backend/app/controllers/ai_trace.controller.go+++ b/backend/app/controllers/ai_trace.controller.go@@ -24,6 +24,7 @@ type AiTraceSearchRequest struct {SortDirection string `json:"sortDirection"`Pagination PaginationParams `json:"pagination"`Search string `json:"search"`+ RootFilter string `json:"rootFilter"`}type AiTraceInstancesRequest struct {@@ -58,7 +59,7 @@ func (a aiTraceController) FindGroupedByTraceName(c *gin.Context) {return}- stats, total, err := repositories.AiTraceRepository.FindGroupedByTraceName(c, projectId, request.FromDate, request.ToDate, request.Pagination.Page, request.Pagination.PageSize, request.OrderBy, request.SortDirection, request.Search)+ stats, total, err := repositories.AiTraceRepository.FindGroupedByTraceName(c, projectId, request.FromDate, request.ToDate, request.Pagination.Page, request.Pagination.PageSize, request.OrderBy, request.SortDirection, request.Search, request.RootFilter)if err != nil {c.AbortWithError(500, traceway.NewStackTraceErrorf("error loading ai trace stats: %w", err))returndiff --git a/backend/app/controllers/distributed_trace.controller.go b/backend/app/controllers/distributed_trace.controller.goindex 472b09bc..43cbfcb4 100644--- a/backend/app/controllers/distributed_trace.controller.go+++ b/backend/app/controllers/distributed_trace.controller.go@@ -23,6 +23,7 @@ type DistributedTraceNode struct {TraceType string `json:"traceType"`Endpoint *models.Endpoint `json:"endpoint,omitempty"`Task *models.Task `json:"task,omitempty"`+ AiTrace *models.AiTrace `json:"aiTrace,omitempty"`Spans []models.Span `json:"spans"`Exception *EndpointExceptionInfo `json:"exception,omitempty"`}@@ -79,6 +80,12 @@ func (d distributedTraceController) GetDistributedTrace(c *gin.Context) {return}+ aiTraces, err := repositories.AiTraceRepository.FindByDistributedTraceId(ctx, distributedTraceId, projectIds)+ if err != nil {+ c.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to query ai traces: %w", err))+ return+ }+exceptions, err := repositories.ExceptionStackTraceRepository.FindByDistributedTraceId(ctx, distributedTraceId, projectIds)if err != nil {c.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to query exceptions: %w", err))@@ -105,6 +112,9 @@ func (d distributedTraceController) GetDistributedTrace(c *gin.Context) {for _, t := range tasks {matchedIds[t.Id] = true}+ for _, a := range aiTraces {+ matchedIds[a.Id] = true+ }var nodes []DistributedTraceNode@@ -132,6 +142,18 @@ func (d distributedTraceController) GetDistributedTrace(c *gin.Context) {nodes = append(nodes, node)}+ for _, a := range aiTraces {+ node := DistributedTraceNode{+ ProjectId: a.ProjectId,+ ProjectName: projectNameMap[a.ProjectId],+ TraceType: "ai_trace",+ AiTrace: &a,+ Spans: []models.Span{},+ Exception: exceptionByTraceId[a.Id],+ }+ nodes = append(nodes, node)+ }+for _, exc := range exceptions {if exc.TraceId != nil && matchedIds[*exc.TraceId] {continuediff --git a/backend/app/controllers/endpoint.controller.go b/backend/app/controllers/endpoint.controller.goindex 5c2bf4d1..7c90d62a 100644--- a/backend/app/controllers/endpoint.controller.go+++ b/backend/app/controllers/endpoint.controller.go@@ -23,6 +23,7 @@ type EndpointSearchRequest struct {SortDirection string `json:"sortDirection"`Pagination PaginationParams `json:"pagination"`Search string `json:"search"`+ RootFilter string `json:"rootFilter"`}type EndpointInstancesRequest struct {@@ -92,7 +93,7 @@ func (e endpointController) FindGroupedByEndpoint(c *gin.Context) {}span := traceway.StartSpan(c, "loading grouped endpoints")- stats, total, err := repositories.EndpointRepository.FindGroupedByEndpoint(c, projectId, request.FromDate, request.ToDate, request.Pagination.Page, request.Pagination.PageSize, request.OrderBy, request.SortDirection, request.Search)+ stats, total, err := repositories.EndpointRepository.FindGroupedByEndpoint(c, projectId, request.FromDate, request.ToDate, request.Pagination.Page, request.Pagination.PageSize, request.OrderBy, request.SortDirection, request.Search, request.RootFilter)span.End()if err != nil {c.AbortWithError(500, traceway.NewStackTraceErrorf("error loading stats: %w", err))diff --git a/backend/app/controllers/otelcontrollers/testdata/node_sign_in.json.golden.json b/backend/app/controllers/otelcontrollers/testdata/node_sign_in.json.golden.jsonindex 40da201c..22ca730f 100644--- a/backend/app/controllers/otelcontrollers/testdata/node_sign_in.json.golden.json+++ b/backend/app/controllers/otelcontrollers/testdata/node_sign_in.json.golden.json@@ -36,7 +36,7 @@"exceptions": [{"stackTrace": "APIError: Invalid email or password",- "traceType": "task"+ "traceType": "endpoint"}],"aiTraceCount": 0,diff --git a/backend/app/controllers/otelcontrollers/trace_converter.go b/backend/app/controllers/otelcontrollers/trace_converter.goindex dc540825..5202a4ba 100644--- a/backend/app/controllers/otelcontrollers/trace_converter.go+++ b/backend/app/controllers/otelcontrollers/trace_converter.go@@ -20,6 +20,27 @@ type aiTraceConversation struct {Content []byte}+type entityKind int++const (+ entityNone entityKind = iota+ entityEndpoint+ entityTask+ entityAiTrace+)++func (k entityKind) traceType() string {+ switch k {+ case entityEndpoint:+ return "endpoint"+ case entityTask:+ return "task"+ case entityAiTrace:+ return "ai_trace"+ }+ return ""+}+func convertTraces(projectId uuid.UUID, req *coltracepb.ExportTraceServiceRequest) (endpoints []models.Endpoint,tasks []models.Task,@@ -41,14 +62,13 @@ func convertTraces(projectId uuid.UUID, req *coltracepb.ExportTraceServiceReques}}- // First pass: collect all spans and build parent→child relationshipstype spanEntry struct {span *tracepb.SpanscopeName string}var allSpans []spanEntry- parentMap := map[string]string{} // childSpanId → parentSpanId- spanById := map[string]*tracepb.Span{} // spanId → span (for trace_id lookup)+ parentMap := map[string]string{}+ spanById := map[string]*tracepb.Span{}for _, ss := range rs.ScopeSpans {for _, span := range ss.Spans {allSpans = append(allSpans, spanEntry{span: span, scopeName: ss.GetScope().GetName()})@@ -59,88 +79,157 @@ func convertTraces(projectId uuid.UUID, req *coltracepb.ExportTraceServiceReques}}- // Build spanToTraceId. Prefer each span's native OTel trace_id so logs on- // the wire (which carry trace_id as hex) can be joined against the- // converted traces via traceIdUuidToHex(endpoint.id). Fall back to walking- // parents + minting a UUID only when the OTel trace_id is missing or- // malformed — an unusual case but worth handling defensively.- spanToTraceId := map[string]uuid.UUID{}- var resolveTraceId func(spanIdStr string) uuid.UUID- resolveTraceId = func(spanIdStr string) uuid.UUID {- if tid, ok := spanToTraceId[spanIdStr]; ok {- return tid+ // Pass 1: classify each span by Kind/attrs and assign an entity id.+ // Roots get id = otelTraceIDToUUID(trace_id); non-roots get id = otelSpanIDToUUID(span_id).+ // distributed_trace_id = otelTraceIDToUUID(trace_id) for both, unless overridden by+ // the vendor `traceway.distributed_trace_id` attribute.+ type promotion struct {+ kind entityKind+ id uuid.UUID+ isRoot bool+ distributedTraceId *uuid.UUID+ }+ spanIdToPromotion := map[string]promotion{}++ for _, entry := range allSpans {+ span := entry.span+ kind := classifySpan(span, spanById)+ if kind == entityNone {+ continue}- if sp, ok := spanById[spanIdStr]; ok && len(sp.TraceId) > 0 {- if tid := otelTraceIDToUUID(sp.TraceId); tid != uuid.Nil {- spanToTraceId[spanIdStr] = tid- return tid++ isRoot := len(span.ParentSpanId) == 0++ var id uuid.UUID+ if isRoot {+ id = otelTraceIDToUUID(span.TraceId)+ if id == uuid.Nil {+ id = otelSpanIDToUUID(span.SpanId)+ }+ } else {+ id = otelSpanIDToUUID(span.SpanId)+ if id == uuid.Nil {+ id = uuid.New()+ }+ }++ dtId := otelTraceIDToUUID(span.TraceId)+ var distributedTraceId *uuid.UUID+ if dtId != uuid.Nil {+ distributedTraceId = &dtId+ }+ if override := getStringAttribute(span.Attributes, "traceway.distributed_trace_id"); override != "" {+ if parsed, err := uuid.Parse(override); err == nil {+ distributedTraceId = &parsed}}- if parentId, hasParent := parentMap[spanIdStr]; hasParent {- tid := resolveTraceId(parentId)- spanToTraceId[spanIdStr] = tid- return tid++ spanIdToPromotion[string(span.SpanId)] = promotion{+ kind: kind,+ id: id,+ isRoot: isRoot,+ distributedTraceId: distributedTraceId,}- tid := uuid.New()- spanToTraceId[spanIdStr] = tid- return tid}- for _, entry := range allSpans {- resolveTraceId(string(entry.span.SpanId))++ // resolveOwner walks parents until it finds a promoted entity. Returns the+ // promotion (so callers know the kind for trace_type) and whether one was+ // found. Cached per span id within this resource batch.+ ownerCache := map[string]*promotion{}+ var resolveOwner func(spanIdStr string) *promotion+ resolveOwner = func(spanIdStr string) *promotion {+ if cached, ok := ownerCache[spanIdStr]; ok {+ return cached+ }+ if p, ok := spanIdToPromotion[spanIdStr]; ok {+ pp := p+ ownerCache[spanIdStr] = &pp+ return &pp+ }+ parentId, hasParent := parentMap[spanIdStr]+ if !hasParent {+ ownerCache[spanIdStr] = nil+ return nil+ }+ owner := resolveOwner(parentId)+ ownerCache[spanIdStr] = owner+ return owner}- // Second pass: process spans and create records+ // Pass 2: emit entity rows + span rows + exceptions.for _, entry := range allSpans {span := entry.spanspanAttrs := span.AttributesallAttrs := extractAttributes(spanAttrs)-- isRoot := len(span.ParentSpanId) == 0- traceId := spanToTraceId[string(span.SpanId)]-- spanId := otelSpanIDToUUID(span.SpanId)startTime := nanoToTime(span.StartTimeUnixNano)endTime := nanoToTime(span.EndTimeUnixNano)duration := endTime.Sub(startTime)- var distributedTraceId *uuid.UUID- if dtid := getStringAttribute(spanAttrs, "traceway.distributed_trace_id"); dtid != "" {- if parsed, err := uuid.Parse(dtid); err == nil {- distributedTraceId = &parsed+ prom, promoted := spanIdToPromotion[string(span.SpanId)]++ // Determine the owning entity for span/exception trace_id.+ var owner *promotion+ if promoted {+ p := prom+ owner = &p+ } else {+ owner = resolveOwner(string(span.SpanId))+ }++ // trace_id for span rows / exceptions: owning entity id when known;+ // otherwise fall back to the OTel trace_id (orphan path — preserves+ // today's behavior for cross-process children whose parent never+ // matched a promoted entity within this batch).+ var ownerId uuid.UUID+ var ownerTraceType string+ if owner != nil {+ ownerId = owner.id+ ownerTraceType = owner.kind.traceType()+ } else {+ ownerId = otelTraceIDToUUID(span.TraceId)+ if ownerId == uuid.Nil {+ ownerId = uuid.New()}}- if isRoot {- rootSpanId := spanId- if (span.Kind == tracepb.Span_SPAN_KIND_SERVER || span.Kind == tracepb.Span_SPAN_KIND_INTERNAL) && hasHTTPAttributes(spanAttrs) {+ if promoted {+ rootSpanId := otelSpanIDToUUID(span.SpanId)+ switch prom.kind {+ case entityEndpoint:ep := buildEndpoint(- traceId, projectId, span, spanAttrs, allAttrs,+ prom.id, projectId, span, spanAttrs, allAttrs,startTime, duration, serverName, appVersion,)- ep.DistributedTraceId = distributedTraceId+ ep.DistributedTraceId = prom.distributedTraceIdep.SpanId = &rootSpanId+ ep.IsRoot = prom.isRootendpoints = append(endpoints, ep)- } else if span.Kind == tracepb.Span_SPAN_KIND_CONSUMER {+ case entityTask:t := buildTask(- traceId, projectId, span, allAttrs,+ prom.id, projectId, span, allAttrs,startTime, endTime, duration, serverName, appVersion,)- t.DistributedTraceId = distributedTraceId+ t.DistributedTraceId = prom.distributedTraceIdt.SpanId = &rootSpanId+ t.IsRoot = prom.isRoottasks = append(tasks, t)- } else if hasGenAiAttributes(spanAttrs) {+ case entityAiTrace:aiTrace := buildAiTrace(- traceId, projectId, span, spanAttrs, allAttrs,+ prom.id, projectId, span, spanAttrs, allAttrs,startTime, duration, serverName, appVersion,)+ aiTrace.DistributedTraceId = prom.distributedTraceId+ aiTrace.IsRoot = prom.isRootaiTraces = append(aiTraces, aiTrace)- if conv := extractConversation(spanAttrs, projectId, traceId); conv != nil {+ if conv := extractConversation(spanAttrs, projectId, prom.id); conv != nil {aiConversations = append(aiConversations, *conv)}- } else {- continue}- } else {+ } else if len(span.ParentSpanId) > 0 {+ // Non-root, unpromoted span → goes to the generic spans table,+ // re-rooted to its nearest enclosing entity (or the OTel+ // trace_id as fallback when the parent chain doesn't reach a+ // promoted span in this batch).spanName := span.Nameif dbQuery := getStringAttribute(spanAttrs, "db.query.text"); dbQuery != "" {spanName = dbQuery@@ -149,8 +238,8 @@ func convertTraces(projectId uuid.UUID, req *coltracepb.ExportTraceServiceReques}spans = append(spans, models.Span{- Id: spanId,- TraceId: traceId,+ Id: otelSpanIDToUUID(span.SpanId),+ TraceId: ownerId,ProjectId: projectId,Name: spanName,StartTime: startTime,@@ -158,20 +247,28 @@ func convertTraces(projectId uuid.UUID, req *coltracepb.ExportTraceServiceRequesRecordedAt: startTime,ParentSpanId: ptrSpanUUID(span.ParentSpanId),})+ } else {+ // Unpromoted root span — match historical behavior and drop+ // it (no entity row, no span row, no exception). Common case:+ // CLIENT-kind roots or non-HTTP SERVER roots from custom+ // instrumentation that we don't have a dedicated page for.+ continue}- traceType := "task"- if isRoot && (span.Kind == tracepb.Span_SPAN_KIND_SERVER || span.Kind == tracepb.Span_SPAN_KIND_INTERNAL) && hasHTTPAttributes(spanAttrs) {- traceType = "endpoint"+ traceType := ownerTraceType+ if traceType == "" {+ traceType = "task"}for _, event := range span.Events {if event.Name == "exception" {exc := buildException(- projectId, traceId, traceType, event,+ projectId, ownerId, traceType, event,allAttrs, serverName, appVersion,)- exc.DistributedTraceId = distributedTraceId+ if owner != nil {+ exc.DistributedTraceId = owner.distributedTraceId+ }exceptions = append(exceptions, exc)}}@@ -180,6 +277,31 @@ func convertTraces(projectId uuid.UUID, req *coltracepb.ExportTraceServiceRequesreturn}+func classifySpan(span *tracepb.Span, spanById map[string]*tracepb.Span) entityKind {+ attrs := span.Attributes+ if (span.Kind == tracepb.Span_SPAN_KIND_SERVER || span.Kind == tracepb.Span_SPAN_KIND_INTERNAL) && hasHTTPAttributes(attrs) {+ if len(span.ParentSpanId) == 0 {+ return entityEndpoint+ }+ if _, parentInBatch := spanById[string(span.ParentSpanId)]; !parentInBatch {+ return entityEndpoint+ }+ }+ if span.Kind == tracepb.Span_SPAN_KIND_CONSUMER {+ return entityTask+ }+ // keepsuit's ConsoleInstrumentation (and equivalents) emits a root INTERNAL+ // span with `console.command` set. Only promote when it is a root span —+ // otherwise we would scoop up arbitrary manual roots from other code paths.+ if span.Kind == tracepb.Span_SPAN_KIND_INTERNAL && len(span.ParentSpanId) == 0 && getStringAttribute(attrs, "console.command") != "" {+ return entityTask+ }+ if hasGenAiAttributes(attrs) {+ return entityAiTrace+ }+ return entityNone+}+func hasHTTPAttributes(attrs []*commonpb.KeyValue) bool {for _, kv := range attrs {switch kv.Key {diff --git a/backend/app/controllers/otelcontrollers/trace_converter_test.go b/backend/app/controllers/otelcontrollers/trace_converter_test.goindex d55f2b32..6d4355bb 100644--- a/backend/app/controllers/otelcontrollers/trace_converter_test.go+++ b/backend/app/controllers/otelcontrollers/trace_converter_test.go@@ -445,6 +445,206 @@ func TestFormatExceptionStackTrace(t *testing.T) {}}+// --- Non-root classification tests ---++func TestConvertTraces_ConsumerNonRoot_BecomesTask(t *testing.T) {+ // Worker batch: CONSUMER span with a parent that lives in the producer's+ // batch (not present in our scope), plus a child DB span.+ traceId := []byte{0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, 0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f}+ producerSpanId := []byte{0xA0, 0xA1, 0xA2, 0xA3, 0xA4, 0xA5, 0xA6, 0xA7}+ consumerSpanId := []byte{0xB0, 0xB1, 0xB2, 0xB3, 0xB4, 0xB5, 0xB6, 0xB7}+ childSpanId := []byte{0xC0, 0xC1, 0xC2, 0xC3, 0xC4, 0xC5, 0xC6, 0xC7}+ now := uint64(1_700_000_000_000_000_000)++ req := &coltracepb.ExportTraceServiceRequest{+ ResourceSpans: []*tracepb.ResourceSpans{{+ Resource: &resourcepb.Resource{Attributes: []*commonpb.KeyValue{strKV("service.name", "worker")}},+ ScopeSpans: []*tracepb.ScopeSpans{{+ Spans: []*tracepb.Span{+ {TraceId: traceId, SpanId: consumerSpanId, ParentSpanId: producerSpanId, Name: "process job", Kind: tracepb.Span_SPAN_KIND_CONSUMER, StartTimeUnixNano: now, EndTimeUnixNano: now + 1_000_000},+ {TraceId: traceId, SpanId: childSpanId, ParentSpanId: consumerSpanId, Name: "SELECT users", Kind: tracepb.Span_SPAN_KIND_INTERNAL, StartTimeUnixNano: now, EndTimeUnixNano: now + 500_000},+ },+ }},+ }},+ }++ endpoints, tasks, spans, _, _, _ := convertTraces(testProjectId, req)++ if len(endpoints) != 0 {+ t.Fatalf("expected 0 endpoints, got %d", len(endpoints))+ }+ if len(tasks) != 1 {+ t.Fatalf("expected 1 task, got %d", len(tasks))+ }+ if tasks[0].IsRoot {+ t.Errorf("expected task.IsRoot == false, got true")+ }+ wantTaskId := otelSpanIDToUUID(consumerSpanId)+ if tasks[0].Id != wantTaskId {+ t.Errorf("expected task.Id %s (from span_id), got %s", wantTaskId, tasks[0].Id)+ }+ wantDtId := otelTraceIDToUUID(traceId)+ if tasks[0].DistributedTraceId == nil || *tasks[0].DistributedTraceId != wantDtId {+ t.Errorf("expected task.DistributedTraceId %s (from trace_id), got %v", wantDtId, tasks[0].DistributedTraceId)+ }++ if len(spans) != 1 {+ t.Fatalf("expected 1 span row, got %d", len(spans))+ }+ if spans[0].TraceId != wantTaskId {+ t.Errorf("expected child span.TraceId == task.Id %s, got %s", wantTaskId, spans[0].TraceId)+ }+}++func TestConvertTraces_ConsoleCommand_BecomesTask(t *testing.T) {+ traceId := []byte{0x20, 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, 0x28, 0x29, 0x2a, 0x2b, 0x2c, 0x2d, 0x2e, 0x2f}+ rootSpanId := []byte{0xD0, 0xD1, 0xD2, 0xD3, 0xD4, 0xD5, 0xD6, 0xD7}+ now := uint64(1_700_000_000_000_000_000)++ req := &coltracepb.ExportTraceServiceRequest{+ ResourceSpans: []*tracepb.ResourceSpans{{+ Resource: &resourcepb.Resource{Attributes: []*commonpb.KeyValue{strKV("service.name", "scheduler")}},+ ScopeSpans: []*tracepb.ScopeSpans{{+ Spans: []*tracepb.Span{{+ TraceId: traceId, SpanId: rootSpanId,+ Name: "bookings:send-reminders", Kind: tracepb.Span_SPAN_KIND_INTERNAL,+ StartTimeUnixNano: now, EndTimeUnixNano: now + 2_000_000,+ Attributes: []*commonpb.KeyValue{strKV("console.command", "bookings:send-reminders")},+ }},+ }},+ }},+ }++ _, tasks, _, _, _, _ := convertTraces(testProjectId, req)+ if len(tasks) != 1 {+ t.Fatalf("expected 1 task, got %d", len(tasks))+ }+ if !tasks[0].IsRoot {+ t.Errorf("expected task.IsRoot == true, got false")+ }+ if tasks[0].Id != otelTraceIDToUUID(traceId) {+ t.Errorf("expected task.Id == otelTraceIDToUUID(trace_id), got %s", tasks[0].Id)+ }+}++func TestConvertTraces_InlineGenAi_BecomesAiTrace(t *testing.T) {+ traceId := []byte{0x30, 0x31, 0x32, 0x33, 0x34, 0x35, 0x36, 0x37, 0x38, 0x39, 0x3a, 0x3b, 0x3c, 0x3d, 0x3e, 0x3f}+ rootSpanId := []byte{0xE0, 0xE1, 0xE2, 0xE3, 0xE4, 0xE5, 0xE6, 0xE7}+ llmSpanId := []byte{0xE8, 0xE9, 0xEA, 0xEB, 0xEC, 0xED, 0xEE, 0xEF}+ now := uint64(1_700_000_000_000_000_000)++ req := &coltracepb.ExportTraceServiceRequest{+ ResourceSpans: []*tracepb.ResourceSpans{{+ Resource: &resourcepb.Resource{Attributes: []*commonpb.KeyValue{strKV("service.name", "api")}},+ ScopeSpans: []*tracepb.ScopeSpans{{+ Spans: []*tracepb.Span{+ {TraceId: traceId, SpanId: rootSpanId, Name: "POST /chat", Kind: tracepb.Span_SPAN_KIND_SERVER, StartTimeUnixNano: now, EndTimeUnixNano: now + 10_000_000,+ Attributes: []*commonpb.KeyValue{strKV("http.request.method", "POST"), strKV("http.route", "/chat")}},+ {TraceId: traceId, SpanId: llmSpanId, ParentSpanId: rootSpanId, Name: "chat openai", Kind: tracepb.Span_SPAN_KIND_INTERNAL, StartTimeUnixNano: now, EndTimeUnixNano: now + 5_000_000,+ Attributes: []*commonpb.KeyValue{strKV("gen_ai.system", "openai"), strKV("gen_ai.request.model", "gpt-4")}},+ },+ }},+ }},+ }++ endpoints, _, _, _, aiTraces, _ := convertTraces(testProjectId, req)+ if len(endpoints) != 1 {+ t.Fatalf("expected 1 endpoint, got %d", len(endpoints))+ }+ if !endpoints[0].IsRoot {+ t.Errorf("expected endpoint.IsRoot == true")+ }+ wantEndpointId := otelTraceIDToUUID(traceId)+ if endpoints[0].Id != wantEndpointId {+ t.Errorf("expected endpoint.Id %s, got %s", wantEndpointId, endpoints[0].Id)+ }++ if len(aiTraces) != 1 {+ t.Fatalf("expected 1 ai_trace, got %d", len(aiTraces))+ }+ if aiTraces[0].IsRoot {+ t.Errorf("expected aiTrace.IsRoot == false")+ }+ wantAiId := otelSpanIDToUUID(llmSpanId)+ if aiTraces[0].Id != wantAiId {+ t.Errorf("expected aiTrace.Id %s, got %s", wantAiId, aiTraces[0].Id)+ }+ if aiTraces[0].DistributedTraceId == nil || *aiTraces[0].DistributedTraceId != wantEndpointId {+ t.Errorf("expected aiTrace.DistributedTraceId %s, got %v", wantEndpointId, aiTraces[0].DistributedTraceId)+ }+}++func TestConvertTraces_ExceptionOnConsumer_TraceTypeIsTask(t *testing.T) {+ traceId := []byte{0x40, 0x41, 0x42, 0x43, 0x44, 0x45, 0x46, 0x47, 0x48, 0x49, 0x4a, 0x4b, 0x4c, 0x4d, 0x4e, 0x4f}+ producerSpanId := []byte{0xF0, 0xF1, 0xF2, 0xF3, 0xF4, 0xF5, 0xF6, 0xF7}+ consumerSpanId := []byte{0xF8, 0xF9, 0xFA, 0xFB, 0xFC, 0xFD, 0xFE, 0xFF}+ now := uint64(1_700_000_000_000_000_000)++ req := &coltracepb.ExportTraceServiceRequest{+ ResourceSpans: []*tracepb.ResourceSpans{{+ Resource: &resourcepb.Resource{Attributes: []*commonpb.KeyValue{strKV("service.name", "worker")}},+ ScopeSpans: []*tracepb.ScopeSpans{{+ Spans: []*tracepb.Span{{+ TraceId: traceId, SpanId: consumerSpanId, ParentSpanId: producerSpanId,+ Name: "process job", Kind: tracepb.Span_SPAN_KIND_CONSUMER,+ StartTimeUnixNano: now, EndTimeUnixNano: now + 1_000_000,+ Events: []*tracepb.Span_Event{{+ Name: "exception",+ TimeUnixNano: now + 500_000,+ Attributes: []*commonpb.KeyValue{+ strKV("exception.type", "RuntimeError"),+ strKV("exception.message", "boom"),+ },+ }},+ }},+ }},+ }},+ }++ _, tasks, _, exceptions, _, _ := convertTraces(testProjectId, req)+ if len(tasks) != 1 || len(exceptions) != 1 {+ t.Fatalf("expected 1 task + 1 exception, got %d / %d", len(tasks), len(exceptions))+ }+ if exceptions[0].TraceType != "task" {+ t.Errorf("expected exception.TraceType == 'task', got %q", exceptions[0].TraceType)+ }+ if exceptions[0].TraceId == nil || *exceptions[0].TraceId != tasks[0].Id {+ t.Errorf("expected exception.TraceId == task.Id %s, got %v", tasks[0].Id, exceptions[0].TraceId)+ }+}++func TestConvertTraces_OrphanSpan_FallsBackToTraceId(t *testing.T) {+ // A non-root span whose parent is not in this batch and the span itself+ // isn't classified as an entity. Should land in the spans table with+ // trace_id == otelTraceIDToUUID(trace_id) (orphan fallback path).+ traceId := []byte{0x50, 0x51, 0x52, 0x53, 0x54, 0x55, 0x56, 0x57, 0x58, 0x59, 0x5a, 0x5b, 0x5c, 0x5d, 0x5e, 0x5f}+ orphanParentId := []byte{0x60, 0x61, 0x62, 0x63, 0x64, 0x65, 0x66, 0x67}+ orphanSpanId := []byte{0x70, 0x71, 0x72, 0x73, 0x74, 0x75, 0x76, 0x77}+ now := uint64(1_700_000_000_000_000_000)++ req := &coltracepb.ExportTraceServiceRequest{+ ResourceSpans: []*tracepb.ResourceSpans{{+ Resource: &resourcepb.Resource{Attributes: []*commonpb.KeyValue{strKV("service.name", "worker")}},+ ScopeSpans: []*tracepb.ScopeSpans{{+ Spans: []*tracepb.Span{{+ TraceId: traceId, SpanId: orphanSpanId, ParentSpanId: orphanParentId,+ Name: "redis GET", Kind: tracepb.Span_SPAN_KIND_INTERNAL,+ StartTimeUnixNano: now, EndTimeUnixNano: now + 100_000,+ }},+ }},+ }},+ }++ _, _, spans, _, _, _ := convertTraces(testProjectId, req)+ if len(spans) != 1 {+ t.Fatalf("expected 1 span row, got %d", len(spans))+ }+ want := otelTraceIDToUUID(traceId)+ if spans[0].TraceId != want {+ t.Errorf("expected orphan span.TraceId %s (fallback to OTel trace_id), got %s", want, spans[0].TraceId)+ }+}+// --- Helpers ---func makeAttrs(key, val string) []*commonpb.KeyValue {diff --git a/backend/app/controllers/task.controller.go b/backend/app/controllers/task.controller.goindex 5686dccb..27f12e0d 100644--- a/backend/app/controllers/task.controller.go+++ b/backend/app/controllers/task.controller.go@@ -20,6 +20,8 @@ type TaskSearchRequest struct {OrderBy string `json:"orderBy"`SortDirection string `json:"sortDirection"`Pagination PaginationParams `json:"pagination"`+ Search string `json:"search"`+ RootFilter string `json:"rootFilter"`}type TaskInstancesRequest struct {@@ -82,7 +84,7 @@ func (e taskController) FindGroupedByTaskName(c *gin.Context) {}span := traceway.StartSpan(c, "loading grouped tasks")- stats, total, err := repositories.TaskRepository.FindGroupedByTaskName(c, projectId, request.FromDate, request.ToDate, request.Pagination.Page, request.Pagination.PageSize, request.OrderBy, request.SortDirection)+ stats, total, err := repositories.TaskRepository.FindGroupedByTaskName(c, projectId, request.FromDate, request.ToDate, request.Pagination.Page, request.Pagination.PageSize, request.OrderBy, request.SortDirection, request.Search, request.RootFilter)span.End()if err != nil {c.AbortWithError(500, traceway.NewStackTraceErrorf("error loading stats by name: %w", err))diff --git a/backend/app/migrations/ch/0058_add_is_root_to_endpoints.up.sql b/backend/app/migrations/ch/0058_add_is_root_to_endpoints.up.sqlnew file mode 100644index 00000000..b266a645--- /dev/null+++ b/backend/app/migrations/ch/0058_add_is_root_to_endpoints.up.sql@@ -0,0 +1 @@+ALTER TABLE endpoints ADD COLUMN is_root UInt8 DEFAULT 1diff --git a/backend/app/migrations/ch/0059_add_is_root_to_tasks.up.sql b/backend/app/migrations/ch/0059_add_is_root_to_tasks.up.sqlnew file mode 100644index 00000000..f581b697--- /dev/null+++ b/backend/app/migrations/ch/0059_add_is_root_to_tasks.up.sql@@ -0,0 +1 @@+ALTER TABLE tasks ADD COLUMN is_root UInt8 DEFAULT 1diff --git a/backend/app/migrations/ch/0060_add_distributed_trace_id_to_ai_traces.up.sql b/backend/app/migrations/ch/0060_add_distributed_trace_id_to_ai_traces.up.sqlnew file mode 100644index 00000000..fc3fa7e8--- /dev/null+++ b/backend/app/migrations/ch/0060_add_distributed_trace_id_to_ai_traces.up.sql@@ -0,0 +1 @@+ALTER TABLE ai_traces ADD COLUMN distributed_trace_id Nullable(UUID) DEFAULT NULLdiff --git a/backend/app/migrations/ch/0061_add_distributed_trace_id_index_ai_traces.up.sql b/backend/app/migrations/ch/0061_add_distributed_trace_id_index_ai_traces.up.sqlnew file mode 100644index 00000000..2a61b1bc--- /dev/null+++ b/backend/app/migrations/ch/0061_add_distributed_trace_id_index_ai_traces.up.sql@@ -0,0 +1 @@+ALTER TABLE ai_traces ADD INDEX idx_distributed_trace_id distributed_trace_id TYPE bloom_filter(0.001) GRANULARITY 1diff --git a/backend/app/migrations/ch/0062_add_is_root_to_ai_traces.up.sql b/backend/app/migrations/ch/0062_add_is_root_to_ai_traces.up.sqlnew file mode 100644index 00000000..8a651c94--- /dev/null+++ b/backend/app/migrations/ch/0062_add_is_root_to_ai_traces.up.sql@@ -0,0 +1 @@+ALTER TABLE ai_traces ADD COLUMN is_root UInt8 DEFAULT 1diff --git a/backend/app/migrations/sqlite_telemetry/0009_add_is_root_and_distributed_trace_id.up.sql b/backend/app/migrations/sqlite_telemetry/0009_add_is_root_and_distributed_trace_id.up.sqlnew file mode 100644index 00000000..10839ebf--- /dev/null+++ b/backend/app/migrations/sqlite_telemetry/0009_add_is_root_and_distributed_trace_id.up.sql@@ -0,0 +1,4 @@+ALTER TABLE endpoints ADD COLUMN is_root INTEGER NOT NULL DEFAULT 1;+ALTER TABLE tasks ADD COLUMN is_root INTEGER NOT NULL DEFAULT 1;+ALTER TABLE ai_traces ADD COLUMN is_root INTEGER NOT NULL DEFAULT 1;+ALTER TABLE ai_traces ADD COLUMN distributed_trace_id TEXT DEFAULT NULL;diff --git a/backend/app/models/ai_trace.model.go b/backend/app/models/ai_trace.model.goindex 2edeb40e..ba9cfc29 100644--- a/backend/app/models/ai_trace.model.go+++ b/backend/app/models/ai_trace.model.go@@ -29,8 +29,10 @@ type AiTrace struct {FinishReason string `json:"finishReason" ch:"finish_reason"`ServerName string `json:"serverName" ch:"server_name"`AppVersion string `json:"appVersion" ch:"app_version"`- StorageKey string `json:"storageKey" ch:"storage_key"`- Attributes map[string]string `json:"attributes" ch:"attributes"`+ StorageKey string `json:"storageKey" ch:"storage_key"`+ Attributes map[string]string `json:"attributes" ch:"attributes"`+ DistributedTraceId *uuid.UUID `json:"distributedTraceId,omitempty" ch:"distributed_trace_id"`+ IsRoot bool `json:"isRoot" ch:"is_root"`}type AiTraceStats struct {@@ -44,6 +46,8 @@ type AiTraceStats struct {AvgInputTokens float64 `json:"avgInputTokens"`AvgOutputTokens float64 `json:"avgOutputTokens"`LastSeen time.Time `json:"lastSeen"`+ HasRoot bool `json:"hasRoot"`+ HasNonRoot bool `json:"hasNonRoot"`}type AiTraceDetailStats struct {diff --git a/backend/app/models/endpoint.model.go b/backend/app/models/endpoint.model.goindex 5f634f52..48cd94dd 100644--- a/backend/app/models/endpoint.model.go+++ b/backend/app/models/endpoint.model.go@@ -22,6 +22,7 @@ type Endpoint struct {DistributedTraceId *uuid.UUID `json:"distributedTraceId,omitempty" ch:"distributed_trace_id"`SpanId *uuid.UUID `json:"spanId,omitempty" ch:"span_id"`IsStream bool `json:"isStream" ch:"is_stream"`+ IsRoot bool `json:"isRoot" ch:"is_root"`}type EndpointStats struct {@@ -35,6 +36,8 @@ type EndpointStats struct {Impact float64 `json:"impact"` // 0-1 impact scoreImpactReason string `json:"impactReason"` // human-readable explanation of the dominant impact factorIsStream bool `json:"isStream"` // true => latency/Apdex/impact intentionally zero+ HasRoot bool `json:"hasRoot"`+ HasNonRoot bool `json:"hasNonRoot"`}// EndpointDetailStats contains detailed statistics for a specific endpointdiff --git a/backend/app/models/task.model.go b/backend/app/models/task.model.goindex 4484d16f..56102556 100644--- a/backend/app/models/task.model.go+++ b/backend/app/models/task.model.go@@ -18,6 +18,7 @@ type Task struct {ServerName string `json:"serverName" ch:"server_name"`DistributedTraceId *uuid.UUID `json:"distributedTraceId,omitempty" ch:"distributed_trace_id"`SpanId *uuid.UUID `json:"spanId,omitempty" ch:"span_id"`+ IsRoot bool `json:"isRoot" ch:"is_root"`}type TaskStats struct {@@ -27,6 +28,8 @@ type TaskStats struct {P95Duration time.Duration `json:"p95Duration"`AvgDuration time.Duration `json:"avgDuration"`LastSeen time.Time `json:"lastSeen"`+ HasRoot bool `json:"hasRoot"`+ HasNonRoot bool `json:"hasNonRoot"`}// TaskDetailStats contains detailed statistics for a specific taskdiff --git a/backend/app/repositories/ai_trace.repository.go b/backend/app/repositories/ai_trace.repository.goindex edc1341d..031b1188 100644--- a/backend/app/repositories/ai_trace.repository.go+++ b/backend/app/repositories/ai_trace.repository.go@@ -7,6 +7,7 @@ import ("database/sql""encoding/json""errors"+ "strings""time""github.com/ClickHouse/clickhouse-go/v2"@@ -19,7 +20,7 @@ type aiTraceRepository struct{}func (r *aiTraceRepository) InsertAsync(ctx context.Context, lines []models.AiTrace) error {batch, err := chdb.Conn.PrepareBatch(clickhouse.Context(context.Background(), clickhouse.WithAsync(false)),- "INSERT INTO ai_traces (id, project_id, recorded_at, duration, status_code, model, response_model, provider, operation, input_tokens, output_tokens, total_tokens, cached_tokens, reasoning_tokens, input_cost, output_cost, total_cost, trace_name, user_id, finish_reason, server_name, app_version, storage_key, attributes)")+ "INSERT INTO ai_traces (id, project_id, recorded_at, duration, status_code, model, response_model, provider, operation, input_tokens, output_tokens, total_tokens, cached_tokens, reasoning_tokens, input_cost, output_cost, total_cost, trace_name, user_id, finish_reason, server_name, app_version, storage_key, attributes, distributed_trace_id, is_root)")if err != nil {return err}@@ -30,13 +31,17 @@ func (r *aiTraceRepository) InsertAsync(ctx context.Context, lines []models.AiTrattributesJSON = string(attributesBytes)}}+ isRoot := uint8(0)+ if t.IsRoot {+ isRoot = 1+ }if err := batch.Append(t.Id, t.ProjectId, t.RecordedAt, int64(t.Duration), t.StatusCode,t.Model, t.ResponseModel, t.Provider, t.Operation,t.InputTokens, t.OutputTokens, t.TotalTokens, t.CachedTokens, t.ReasoningTokens,t.InputCost, t.OutputCost, t.TotalCost,t.TraceName, t.UserId, t.FinishReason, t.ServerName, t.AppVersion,- t.StorageKey, attributesJSON,+ t.StorageKey, attributesJSON, t.DistributedTraceId, isRoot,); err != nil {return err}@@ -44,7 +49,7 @@ func (r *aiTraceRepository) InsertAsync(ctx context.Context, lines []models.AiTrreturn batch.Send()}-func (r *aiTraceRepository) FindGroupedByTraceName(ctx context.Context, projectId uuid.UUID, fromDate, toDate time.Time, page, pageSize int, orderBy, sortDirection, search string) ([]models.AiTraceStats, int64, error) {+func (r *aiTraceRepository) FindGroupedByTraceName(ctx context.Context, projectId uuid.UUID, fromDate, toDate time.Time, page, pageSize int, orderBy, sortDirection, search, rootFilter string) ([]models.AiTraceStats, int64, error) {whereClause := "project_id = ? AND recorded_at >= ? AND recorded_at <= ?"args := []interface{}{projectId, fromDate, toDate}@@ -52,6 +57,12 @@ func (r *aiTraceRepository) FindGroupedByTraceName(ctx context.Context, projectIwhereClause += " AND positionCaseInsensitive(trace_name, ?) > 0"args = append(args, search)}+ switch rootFilter {+ case "root":+ whereClause += " AND is_root = 1"+ case "non_root":+ whereClause += " AND is_root = 0"+ }var count uint64err := chdb.Conn.QueryRow(ctx, "SELECT uniq(trace_name) FROM ai_traces WHERE "+whereClause, args...).Scan(&count)@@ -90,7 +101,9 @@ func (r *aiTraceRepository) FindGroupedByTraceName(ctx context.Context, projectIsum(total_cost) as total_cost,avg(input_tokens) as avg_input_tokens,avg(output_tokens) as avg_output_tokens,- max(recorded_at) as last_seen+ max(recorded_at) as last_seen,+ max(is_root) as has_root,+ max(if(is_root = 0, 1, 0)) as has_non_rootFROM ai_tracesWHERE ` + whereClause + `GROUP BY trace_name@@ -108,13 +121,16 @@ func (r *aiTraceRepository) FindGroupedByTraceName(ctx context.Context, projectIfor rows.Next() {var s models.AiTraceStatsvar p50, p95, avg float64+ var hasRoot, hasNonRoot uint8if err := rows.Scan(&s.TraceName, &s.Count, &p50, &p95, &avg,- &s.TotalTokens, &s.TotalCost, &s.AvgInputTokens, &s.AvgOutputTokens, &s.LastSeen); err != nil {+ &s.TotalTokens, &s.TotalCost, &s.AvgInputTokens, &s.AvgOutputTokens, &s.LastSeen, &hasRoot, &hasNonRoot); err != nil {return nil, 0, err}s.P50Duration = time.Duration(p50)s.P95Duration = time.Duration(p95)s.AvgDuration = time.Duration(avg)+ s.HasRoot = hasRoot == 1+ s.HasNonRoot = hasNonRoot == 1stats = append(stats, s)}@@ -236,13 +252,14 @@ func (r *aiTraceRepository) FindById(ctx context.Context, projectId, traceId uuiinput_tokens, output_tokens, total_tokens, cached_tokens, reasoning_tokens,input_cost, output_cost, total_cost,trace_name, user_id, finish_reason, server_name, app_version,- storage_key, attributes+ storage_key, attributes, distributed_trace_id, is_rootFROM ai_tracesWHERE project_id = ? AND id = ?LIMIT 1`var t models.AiTracevar attributesJSON string+ var isRoot uint8err := chdb.Conn.QueryRow(ctx, query, projectId, traceId).Scan(&t.Id, &t.ProjectId, &t.RecordedAt, &t.Duration, &t.StatusCode,@@ -250,7 +267,7 @@ func (r *aiTraceRepository) FindById(ctx context.Context, projectId, traceId uui&t.InputTokens, &t.OutputTokens, &t.TotalTokens, &t.CachedTokens, &t.ReasoningTokens,&t.InputCost, &t.OutputCost, &t.TotalCost,&t.TraceName, &t.UserId, &t.FinishReason, &t.ServerName, &t.AppVersion,- &t.StorageKey, &attributesJSON,+ &t.StorageKey, &attributesJSON, &t.DistributedTraceId, &isRoot,)if err != nil {if errors.Is(err, sql.ErrNoRows) {@@ -258,6 +275,7 @@ func (r *aiTraceRepository) FindById(ctx context.Context, projectId, traceId uui}return nil, err}+ t.IsRoot = isRoot == 1if attributesJSON != "" && attributesJSON != "{}" {if err := json.Unmarshal([]byte(attributesJSON), &t.Attributes); err != nil {@@ -268,4 +286,57 @@ func (r *aiTraceRepository) FindById(ctx context.Context, projectId, traceId uuireturn &t, nil}+func (r *aiTraceRepository) FindByDistributedTraceId(ctx context.Context, distributedTraceId uuid.UUID, projectIds []uuid.UUID) ([]models.AiTrace, error) {+ if len(projectIds) == 0 {+ return nil, nil+ }+ placeholders := make([]string, len(projectIds))+ args := []interface{}{distributedTraceId}+ for i, pid := range projectIds {+ placeholders[i] = "?"+ args = append(args, pid)+ _ = i+ }+ query := `SELECT id, project_id, recorded_at, duration, status_code,+ model, response_model, provider, operation,+ input_tokens, output_tokens, total_tokens, cached_tokens, reasoning_tokens,+ input_cost, output_cost, total_cost,+ trace_name, user_id, finish_reason, server_name, app_version,+ storage_key, attributes, distributed_trace_id, is_root+ FROM ai_traces+ WHERE distributed_trace_id = ? AND project_id IN (` + strings.Join(placeholders, ",") + `)+ ORDER BY recorded_at ASC`++ rows, err := chdb.Conn.Query(ctx, query, args...)+ if err != nil {+ return nil, err+ }+ defer rows.Close()++ var traces []models.AiTrace+ for rows.Next() {+ var t models.AiTrace+ var attributesJSON string+ var isRoot uint8+ if err := rows.Scan(+ &t.Id, &t.ProjectId, &t.RecordedAt, &t.Duration, &t.StatusCode,+ &t.Model, &t.ResponseModel, &t.Provider, &t.Operation,+ &t.InputTokens, &t.OutputTokens, &t.TotalTokens, &t.CachedTokens, &t.ReasoningTokens,+ &t.InputCost, &t.OutputCost, &t.TotalCost,+ &t.TraceName, &t.UserId, &t.FinishReason, &t.ServerName, &t.AppVersion,+ &t.StorageKey, &attributesJSON, &t.DistributedTraceId, &isRoot,+ ); err != nil {+ return nil, err+ }+ t.IsRoot = isRoot == 1+ if attributesJSON != "" && attributesJSON != "{}" {+ if err := json.Unmarshal([]byte(attributesJSON), &t.Attributes); err != nil {+ t.Attributes = nil+ }+ }+ traces = append(traces, t)+ }+ return traces, nil+}+var AiTraceRepository = aiTraceRepository{}diff --git a/backend/app/repositories/ai_trace.repository_sqlite.go b/backend/app/repositories/ai_trace.repository_sqlite.goindex 70677df6..fb5671c8 100644--- a/backend/app/repositories/ai_trace.repository_sqlite.go+++ b/backend/app/repositories/ai_trace.repository_sqlite.go@@ -6,6 +6,7 @@ import ("context""fmt""sort"+ "strings""time""github.com/google/uuid"@@ -15,30 +16,32 @@ import ()type aiTraceRow struct {- Id uuid.UUID `lit:"id"`- ProjectId uuid.UUID `lit:"project_id"`- RecordedAt SQLiteTime `lit:"recorded_at"`- Duration int64 `lit:"duration"`- StatusCode uint8 `lit:"status_code"`- Model string `lit:"model"`- ResponseModel string `lit:"response_model"`- Provider string `lit:"provider"`- Operation string `lit:"operation"`- InputTokens int64 `lit:"input_tokens"`- OutputTokens int64 `lit:"output_tokens"`- TotalTokens int64 `lit:"total_tokens"`- CachedTokens int64 `lit:"cached_tokens"`- ReasoningTokens int64 `lit:"reasoning_tokens"`- InputCost float64 `lit:"input_cost"`- OutputCost float64 `lit:"output_cost"`- TotalCost float64 `lit:"total_cost"`- TraceName string `lit:"trace_name"`- UserId string `lit:"user_id"`- FinishReason string `lit:"finish_reason"`- ServerName string `lit:"server_name"`- AppVersion string `lit:"app_version"`- StorageKey string `lit:"storage_key"`- Attributes SQLiteJSONMap `lit:"attributes"`+ Id uuid.UUID `lit:"id"`+ ProjectId uuid.UUID `lit:"project_id"`+ RecordedAt SQLiteTime `lit:"recorded_at"`+ Duration int64 `lit:"duration"`+ StatusCode uint8 `lit:"status_code"`+ Model string `lit:"model"`+ ResponseModel string `lit:"response_model"`+ Provider string `lit:"provider"`+ Operation string `lit:"operation"`+ InputTokens int64 `lit:"input_tokens"`+ OutputTokens int64 `lit:"output_tokens"`+ TotalTokens int64 `lit:"total_tokens"`+ CachedTokens int64 `lit:"cached_tokens"`+ ReasoningTokens int64 `lit:"reasoning_tokens"`+ InputCost float64 `lit:"input_cost"`+ OutputCost float64 `lit:"output_cost"`+ TotalCost float64 `lit:"total_cost"`+ TraceName string `lit:"trace_name"`+ UserId string `lit:"user_id"`+ FinishReason string `lit:"finish_reason"`+ ServerName string `lit:"server_name"`+ AppVersion string `lit:"app_version"`+ StorageKey string `lit:"storage_key"`+ Attributes SQLiteJSONMap `lit:"attributes"`+ DistributedTraceId *uuid.UUID `lit:"distributed_trace_id"`+ IsRoot bool `lit:"is_root"`}type groupedAiTraceRow struct {@@ -50,6 +53,8 @@ type groupedAiTraceRow struct {AvgInputTokens float64 `lit:"avg_input_tokens"`AvgOutputTokens float64 `lit:"avg_output_tokens"`LastSeen string `lit:"last_seen"`+ HasRoot bool `lit:"has_root"`+ HasNonRoot bool `lit:"has_non_root"`}type aiTraceDurationRow struct {@@ -76,58 +81,62 @@ func init() {func aiTraceToRow(t models.AiTrace) aiTraceRow {return aiTraceRow{- Id: t.Id,- ProjectId: t.ProjectId,- RecordedAt: NewSQLiteTime(t.RecordedAt),- Duration: int64(t.Duration),- StatusCode: t.StatusCode,- Model: t.Model,- ResponseModel: t.ResponseModel,- Provider: t.Provider,- Operation: t.Operation,- InputTokens: t.InputTokens,- OutputTokens: t.OutputTokens,- TotalTokens: t.TotalTokens,- CachedTokens: t.CachedTokens,- ReasoningTokens: t.ReasoningTokens,- InputCost: t.InputCost,- OutputCost: t.OutputCost,- TotalCost: t.TotalCost,- TraceName: t.TraceName,- UserId: t.UserId,- FinishReason: t.FinishReason,- ServerName: t.ServerName,- AppVersion: t.AppVersion,- StorageKey: t.StorageKey,- Attributes: NewSQLiteJSONMap(t.Attributes),+ Id: t.Id,+ ProjectId: t.ProjectId,+ RecordedAt: NewSQLiteTime(t.RecordedAt),+ Duration: int64(t.Duration),+ StatusCode: t.StatusCode,+ Model: t.Model,+ ResponseModel: t.ResponseModel,+ Provider: t.Provider,+ Operation: t.Operation,+ InputTokens: t.InputTokens,+ OutputTokens: t.OutputTokens,+ TotalTokens: t.TotalTokens,+ CachedTokens: t.CachedTokens,+ ReasoningTokens: t.ReasoningTokens,+ InputCost: t.InputCost,+ OutputCost: t.OutputCost,+ TotalCost: t.TotalCost,+ TraceName: t.TraceName,+ UserId: t.UserId,+ FinishReason: t.FinishReason,+ ServerName: t.ServerName,+ AppVersion: t.AppVersion,+ StorageKey: t.StorageKey,+ Attributes: NewSQLiteJSONMap(t.Attributes),+ DistributedTraceId: t.DistributedTraceId,+ IsRoot: t.IsRoot,}}func (r *aiTraceRow) toModel() models.AiTrace {t := models.AiTrace{- Id: r.Id,- ProjectId: r.ProjectId,- RecordedAt: r.RecordedAt.Time,- Duration: time.Duration(r.Duration),- StatusCode: r.StatusCode,- Model: r.Model,- ResponseModel: r.ResponseModel,- Provider: r.Provider,- Operation: r.Operation,- InputTokens: r.InputTokens,- OutputTokens: r.OutputTokens,- TotalTokens: r.TotalTokens,- CachedTokens: r.CachedTokens,- ReasoningTokens: r.ReasoningTokens,- InputCost: r.InputCost,- OutputCost: r.OutputCost,- TotalCost: r.TotalCost,- TraceName: r.TraceName,- UserId: r.UserId,- FinishReason: r.FinishReason,- ServerName: r.ServerName,- AppVersion: r.AppVersion,- StorageKey: r.StorageKey,+ Id: r.Id,+ ProjectId: r.ProjectId,+ RecordedAt: r.RecordedAt.Time,+ Duration: time.Duration(r.Duration),+ StatusCode: r.StatusCode,+ Model: r.Model,+ ResponseModel: r.ResponseModel,+ Provider: r.Provider,+ Operation: r.Operation,+ InputTokens: r.InputTokens,+ OutputTokens: r.OutputTokens,+ TotalTokens: r.TotalTokens,+ CachedTokens: r.CachedTokens,+ ReasoningTokens: r.ReasoningTokens,+ InputCost: r.InputCost,+ OutputCost: r.OutputCost,+ TotalCost: r.TotalCost,+ TraceName: r.TraceName,+ UserId: r.UserId,+ FinishReason: r.FinishReason,+ ServerName: r.ServerName,+ AppVersion: r.AppVersion,+ StorageKey: r.StorageKey,+ DistributedTraceId: r.DistributedTraceId,+ IsRoot: r.IsRoot,}if r.Attributes != nil {t.Attributes = map[string]string(r.Attributes)@@ -150,7 +159,7 @@ func (r *aiTraceRepository) InsertAsync(ctx context.Context, lines []models.AiTrreturn nil}-func (r *aiTraceRepository) FindGroupedByTraceName(ctx context.Context, projectId uuid.UUID, fromDate, toDate time.Time, page, pageSize int, orderBy, sortDirection, search string) ([]models.AiTraceStats, int64, error) {+func (r *aiTraceRepository) FindGroupedByTraceName(ctx context.Context, projectId uuid.UUID, fromDate, toDate time.Time, page, pageSize int, orderBy, sortDirection, search, rootFilter string) ([]models.AiTraceStats, int64, error) {params := lit.P{"project_id": projectId, "from": NewSQLiteTime(fromDate), "to": NewSQLiteTime(toDate)}whereClause := "project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to"@@ -158,6 +167,7 @@ func (r *aiTraceRepository) FindGroupedByTraceName(ctx context.Context, projectIwhereClause += " AND INSTR(LOWER(trace_name), LOWER(:search)) > 0"params["search"] = search}+ whereClause += rootFilterClause("is_root", rootFilter)countResult, err := lit.SelectSingleNamed[models.CountResult](db.TelemetryDB,"SELECT COUNT(DISTINCT trace_name) AS count FROM ai_traces WHERE "+whereClause, params)@@ -176,7 +186,9 @@ func (r *aiTraceRepository) FindGroupedByTraceName(ctx context.Context, projectISUM(total_cost) AS total_cost,AVG(input_tokens) AS avg_input_tokens,AVG(output_tokens) AS avg_output_tokens,- MAX(recorded_at) AS last_seen+ MAX(recorded_at) AS last_seen,+ MAX(is_root) AS has_root,+ MAX(CASE WHEN is_root = 0 THEN 1 ELSE 0 END) AS has_non_rootFROM ai_traces WHERE `+whereClause+`GROUP BY trace_name`, params)if err != nil {@@ -213,6 +225,8 @@ func (r *aiTraceRepository) FindGroupedByTraceName(ctx context.Context, projectIAvgInputTokens: row.AvgInputTokens,AvgOutputTokens: row.AvgOutputTokens,LastSeen: lastSeen,+ HasRoot: row.HasRoot,+ HasNonRoot: row.HasNonRoot,})}@@ -286,7 +300,7 @@ func (r *aiTraceRepository) FindByTraceName(ctx context.Context, projectId uuid.input_tokens, output_tokens, total_tokens, cached_tokens, reasoning_tokens,input_cost, output_cost, total_cost,trace_name, user_id, finish_reason, server_name, app_version,- storage_key, attributes+ storage_key, attributes, distributed_trace_id, is_rootFROM ai_tracesWHERE project_id = :project_id AND trace_name = :trace_name AND recorded_at >= :from AND recorded_at <= :toORDER BY %s %s LIMIT :limit OFFSET :offset`, orderBy, sortDir),@@ -362,7 +376,7 @@ func (r *aiTraceRepository) FindById(ctx context.Context, projectId, traceId uuiinput_tokens, output_tokens, total_tokens, cached_tokens, reasoning_tokens,input_cost, output_cost, total_cost,trace_name, user_id, finish_reason, server_name, app_version,- storage_key, attributes+ storage_key, attributes, distributed_trace_id, is_rootFROM ai_tracesWHERE project_id = :project_id AND id = :idLIMIT 1`,@@ -377,4 +391,52 @@ func (r *aiTraceRepository) FindById(ctx context.Context, projectId, traceId uuireturn &result, nil}+func (r *aiTraceRepository) FindByDistributedTraceId(ctx context.Context, distributedTraceId uuid.UUID, projectIds []uuid.UUID) ([]models.AiTrace, error) {+ if len(projectIds) == 0 {+ return nil, nil+ }+ params := lit.P{"trace_id": distributedTraceId}+ placeholders := make([]string, len(projectIds))+ for i, pid := range projectIds {+ key := fmt.Sprintf("pid_%d", i)+ placeholders[i] = ":" + key+ params[key] = pid+ }+ query := `SELECT id, project_id, recorded_at, duration, status_code,+ model, response_model, provider, operation,+ input_tokens, output_tokens, total_tokens, cached_tokens, reasoning_tokens,+ input_cost, output_cost, total_cost,+ trace_name, user_id, finish_reason, server_name, app_version,+ storage_key, attributes, distributed_trace_id, is_root+ FROM ai_traces WHERE distributed_trace_id = :trace_id AND project_id IN (` + strings.Join(placeholders, ",") + `)+ ORDER BY recorded_at ASC`++ parsedQuery, args, err := lit.ParseNamedQuery(db.Driver, query, params)+ if err != nil {+ return nil, err+ }+ sqlRows, err := db.TelemetryDB.QueryContext(ctx, parsedQuery, args...)+ if err != nil {+ return nil, err+ }+ defer sqlRows.Close()++ var traces []models.AiTrace+ for sqlRows.Next() {+ var row aiTraceRow+ if err := sqlRows.Scan(+ &row.Id, &row.ProjectId, &row.RecordedAt, &row.Duration, &row.StatusCode,+ &row.Model, &row.ResponseModel, &row.Provider, &row.Operation,+ &row.InputTokens, &row.OutputTokens, &row.TotalTokens, &row.CachedTokens, &row.ReasoningTokens,+ &row.InputCost, &row.OutputCost, &row.TotalCost,+ &row.TraceName, &row.UserId, &row.FinishReason, &row.ServerName, &row.AppVersion,+ &row.StorageKey, &row.Attributes, &row.DistributedTraceId, &row.IsRoot,+ ); err != nil {+ return nil, err+ }+ traces = append(traces, row.toModel())+ }+ return traces, nil+}+var AiTraceRepository = aiTraceRepository{}diff --git a/backend/app/repositories/endpoint.repository.go b/backend/app/repositories/endpoint.repository.goindex 93f1de61..a7b898b5 100644--- a/backend/app/repositories/endpoint.repository.go+++ b/backend/app/repositories/endpoint.repository.go@@ -18,7 +18,7 @@ import (type endpointRepository struct{}func (e *endpointRepository) InsertAsync(ctx context.Context, lines []models.Endpoint) error {- batch, err := chdb.Conn.PrepareBatch(clickhouse.Context(context.Background(), clickhouse.WithAsync(false)), "INSERT INTO endpoints (id, project_id, endpoint, duration, recorded_at, status_code, body_size, client_ip, attributes, app_version, server_name, distributed_trace_id, span_id, is_stream)")+ batch, err := chdb.Conn.PrepareBatch(clickhouse.Context(context.Background(), clickhouse.WithAsync(false)), "INSERT INTO endpoints (id, project_id, endpoint, duration, recorded_at, status_code, body_size, client_ip, attributes, app_version, server_name, distributed_trace_id, span_id, is_stream, is_root)")if err != nil {return err}@@ -33,7 +33,11 @@ func (e *endpointRepository) InsertAsync(ctx context.Context, lines []models.Endif t.IsStream {isStream = 1}- if err := batch.Append(t.Id, t.ProjectId, t.Endpoint, int64(t.Duration), t.RecordedAt, t.StatusCode, t.BodySize, t.ClientIP, attributesJSON, t.AppVersion, t.ServerName, t.DistributedTraceId, t.SpanId, isStream); err != nil {+ isRoot := uint8(0)+ if t.IsRoot {+ isRoot = 1+ }+ if err := batch.Append(t.Id, t.ProjectId, t.Endpoint, int64(t.Duration), t.RecordedAt, t.StatusCode, t.BodySize, t.ClientIP, attributesJSON, t.AppVersion, t.ServerName, t.DistributedTraceId, t.SpanId, isStream, isRoot); err != nil {return err}}@@ -91,7 +95,7 @@ func (e *endpointRepository) FindAll(ctx context.Context, projectId uuid.UUID, freturn endpoints, int64(count), nil}-func (e *endpointRepository) FindGroupedByEndpoint(ctx context.Context, projectId uuid.UUID, fromDate, toDate time.Time, page, pageSize int, orderBy string, sortDirection string, search string) ([]models.EndpointStats, int64, error) {+func (e *endpointRepository) FindGroupedByEndpoint(ctx context.Context, projectId uuid.UUID, fromDate, toDate time.Time, page, pageSize int, orderBy string, sortDirection string, search string, rootFilter string) ([]models.EndpointStats, int64, error) {// Build WHERE clause with optional search filter// Count query uses bare column names; main query uses e. prefix for LEFT JOINwhereClause := "project_id = ? AND recorded_at >= ? AND recorded_at <= ?"@@ -104,6 +108,15 @@ func (e *endpointRepository) FindGroupedByEndpoint(ctx context.Context, projectIargs = append(args, search)}+ switch rootFilter {+ case "root":+ whereClause += " AND is_root = 1"+ joinWhereClause += " AND e.is_root = 1"+ case "non_root":+ whereClause += " AND is_root = 0"+ joinWhereClause += " AND e.is_root = 0"+ }+// Count unique endpointsvar count uint64countQuery := "SELECT uniq(endpoint) FROM endpoints WHERE " + whereClause@@ -140,7 +153,7 @@ func (e *endpointRepository) FindGroupedByEndpoint(ctx context.Context, projectIendpoint, total_count as count, p50_duration, p95_duration, p99_duration,avg_duration, last_seen, offset_ms,satisfied_count, tolerating_count, bad_count, client_error_count,- server_error_count, is_stream,+ server_error_count, is_stream, has_root, has_non_root,if(is_stream = 1,-- Streaming endpoints: skip Apdex + P99 (duration-driven, meaningless-- for connection lifetimes). Substitute server_error_count for@@ -213,9 +226,11 @@ func (e *endpointRepository) FindGroupedByEndpoint(ctx context.Context, projectIcountIf(duration > (1500000000 + toInt64(offset_ms) * 1000000)OR status_code >= 500) as bad_count,countIf(status_code >= 400 AND status_code < 500) as client_error_count,- countIf(status_code >= 500) as server_error_count+ countIf(status_code >= 500) as server_error_count,+ max(is_root) as has_root,+ max(if(is_root = 0, 1, 0)) as has_non_rootFROM (- SELECT e.endpoint, e.duration, e.status_code, e.recorded_at, e.is_stream,+ SELECT e.endpoint, e.duration, e.status_code, e.recorded_at, e.is_stream, e.is_root,s.offset_ms as offset_msFROM endpoints eLEFT JOIN (SELECT * FROM slow_endpoints FINAL) AS s@@ -242,13 +257,15 @@ func (e *endpointRepository) FindGroupedByEndpoint(ctx context.Context, projectIvar p50, p95, p99, avg float64var offsetMs uint32var satisfiedCount, toleratingCount, badCount, clientErrorCount, serverErrorCount uint64- var isStream uint8+ var isStream, hasRoot, hasNonRoot uint8if err := rows.Scan(&s.Endpoint, &s.Count, &p50, &p95, &p99, &avg, &s.LastSeen,&offsetMs, &satisfiedCount, &toleratingCount, &badCount, &clientErrorCount,- &serverErrorCount, &isStream, &s.Impact); err != nil {+ &serverErrorCount, &isStream, &hasRoot, &hasNonRoot, &s.Impact); err != nil {return nil, 0, err}s.IsStream = isStream == 1+ s.HasRoot = hasRoot == 1+ s.HasNonRoot = hasNonRoot == 1if s.IsStream {// Latency/Apdex are zero for streams. Impact came from the// stream-specific branch of the SQL (status-code components only);@@ -321,17 +338,19 @@ func (e *endpointRepository) FindByEndpoint(ctx context.Context, projectId uuid.// FindById returns a single endpoint by IDfunc (e *endpointRepository) FindById(ctx context.Context, projectId, endpointId uuid.UUID) (*models.Endpoint, error) {- query := `SELECT id, project_id, endpoint, duration, recorded_at, status_code, body_size, client_ip, attributes, app_version, server_name, distributed_trace_id, span_id+ query := `SELECT id, project_id, endpoint, duration, recorded_at, status_code, body_size, client_ip, attributes, app_version, server_name, distributed_trace_id, span_id, is_rootFROM endpointsWHERE project_id = ? AND id = ?LIMIT 1`var t models.Endpointvar attributesJSON string+ var isRoot uint8err := chdb.Conn.QueryRow(ctx, query, projectId, endpointId).Scan(&t.Id, &t.ProjectId, &t.Endpoint, &t.Duration, &t.RecordedAt,- &t.StatusCode, &t.BodySize, &t.ClientIP, &attributesJSON, &t.AppVersion, &t.ServerName, &t.DistributedTraceId, &t.SpanId)+ &t.StatusCode, &t.BodySize, &t.ClientIP, &attributesJSON, &t.AppVersion, &t.ServerName, &t.DistributedTraceId, &t.SpanId, &isRoot)+ t.IsRoot = isRoot == 1if err != nil {if errors.Is(err, sql.ErrNoRows) {diff --git a/backend/app/repositories/endpoint.repository_sqlite.go b/backend/app/repositories/endpoint.repository_sqlite.goindex 8767aba5..6924f8ea 100644--- a/backend/app/repositories/endpoint.repository_sqlite.go+++ b/backend/app/repositories/endpoint.repository_sqlite.go@@ -32,6 +32,7 @@ type endpoint struct {DistributedTraceId *uuid.UUID `lit:"distributed_trace_id"`SpanId *uuid.UUID `lit:"span_id"`IsStream bool `lit:"is_stream"`+ IsRoot bool `lit:"is_root"`}type groupedEndpointRow struct {@@ -46,6 +47,8 @@ type groupedEndpointRow struct {ToleratingCount uint64 `lit:"tolerating_count"`BadCount uint64 `lit:"bad_count"`IsStream bool `lit:"is_stream"`+ HasRoot bool `lit:"has_root"`+ HasNonRoot bool `lit:"has_non_root"`}type endpointDurationRow struct {@@ -106,6 +109,7 @@ func endpointToRow(e models.Endpoint) endpoint {DistributedTraceId: e.DistributedTraceId,SpanId: e.SpanId,IsStream: e.IsStream,+ IsRoot: e.IsRoot,}}@@ -124,6 +128,7 @@ func (r *endpoint) toModel() models.Endpoint {DistributedTraceId: r.DistributedTraceId,SpanId: r.SpanId,IsStream: r.IsStream,+ IsRoot: r.IsRoot,}if r.Attributes != nil {e.Attributes = map[string]string(r.Attributes)@@ -131,6 +136,20 @@ func (r *endpoint) toModel() models.Endpoint {return e}+// rootFilterClause returns a SQL fragment ("", " AND <col> = 1", " AND <col> = 0")+// to splice into a WHERE clause based on the rootFilter param. Accepts "all" |+// "root" | "non_root"; defaults to "all" (no filter).+func rootFilterClause(qualifiedCol, rootFilter string) string {+ switch rootFilter {+ case "root":+ return " AND " + qualifiedCol + " = 1"+ case "non_root":+ return " AND " + qualifiedCol + " = 0"+ default:+ return ""+ }+}+type endpointRepository struct{}func (e *endpointRepository) InsertAsync(ctx context.Context, lines []models.Endpoint) error {@@ -199,7 +218,7 @@ func (e *endpointRepository) FindAll(ctx context.Context, projectId uuid.UUID, freturn endpoints, count, nil}-func (e *endpointRepository) FindGroupedByEndpoint(ctx context.Context, projectId uuid.UUID, fromDate, toDate time.Time, page, pageSize int, orderBy string, sortDirection string, search string) ([]models.EndpointStats, int64, error) {+func (e *endpointRepository) FindGroupedByEndpoint(ctx context.Context, projectId uuid.UUID, fromDate, toDate time.Time, page, pageSize int, orderBy string, sortDirection string, search string, rootFilter string) ([]models.EndpointStats, int64, error) {params := lit.P{"project_id": projectId, "from": NewSQLiteTime(fromDate), "to": NewSQLiteTime(toDate)}whereClause := "e.project_id = :project_id AND e.recorded_at >= :from AND e.recorded_at <= :to"@@ -207,6 +226,7 @@ func (e *endpointRepository) FindGroupedByEndpoint(ctx context.Context, projectIwhereClause += " AND INSTR(LOWER(e.endpoint), LOWER(:search)) > 0"params["search"] = search}+ whereClause += rootFilterClause("e.is_root", rootFilter)countQuery := "SELECT COUNT(DISTINCT e.endpoint) AS count FROM endpoints e WHERE " + whereClausetotalResult, err := lit.SelectSingleNamed[models.CountResult](db.TelemetryDB, countQuery, params)@@ -229,7 +249,9 @@ func (e *endpointRepository) FindGroupedByEndpoint(ctx context.Context, projectISUM(CASE WHEN e.duration <= (750000000 + COALESCE(s.offset_ms, 0) * 1000000) AND e.status_code < 500 THEN 1 ELSE 0 END) as satisfied_count,SUM(CASE WHEN e.duration > (750000000 + COALESCE(s.offset_ms, 0) * 1000000) AND e.duration <= (1500000000 + COALESCE(s.offset_ms, 0) * 1000000) AND e.status_code < 500 THEN 1 ELSE 0 END) as tolerating_count,SUM(CASE WHEN e.duration > (1500000000 + COALESCE(s.offset_ms, 0) * 1000000) OR e.status_code >= 500 THEN 1 ELSE 0 END) as bad_count,- MAX(e.is_stream) as is_stream+ MAX(e.is_stream) as is_stream,+ MAX(e.is_root) as has_root,+ MAX(CASE WHEN e.is_root = 0 THEN 1 ELSE 0 END) as has_non_rootFROM endpoints eLEFT JOIN slow_endpoints s ON e.endpoint = s.endpoint AND e.project_id = s.project_idWHERE ` + whereClause + `@@ -251,7 +273,7 @@ func (e *endpointRepository) FindGroupedByEndpoint(ctx context.Context, projectIvar g groupedEndpointRowif err := sqlRows.Scan(&g.Endpoint, &g.TotalCount, &g.AvgDuration, &g.LastSeen,&g.OffsetMs, &g.ServerErrorCount, &g.ClientErrorCount,- &g.SatisfiedCount, &g.ToleratingCount, &g.BadCount, &g.IsStream); err != nil {+ &g.SatisfiedCount, &g.ToleratingCount, &g.BadCount, &g.IsStream, &g.HasRoot, &g.HasNonRoot); err != nil {return nil, 0, err}groups = append(groups, g)@@ -274,6 +296,8 @@ func (e *endpointRepository) FindGroupedByEndpoint(ctx context.Context, projectIImpact: ComputeStreamImpact(g.Endpoint, g.TotalCount, g.ServerErrorCount, g.ClientErrorCount),ImpactReason: ComputeStreamImpactReason(g.Endpoint, g.TotalCount, g.ServerErrorCount, g.ClientErrorCount),IsStream: true,+ HasRoot: g.HasRoot,+ HasNonRoot: g.HasNonRoot,})continue}@@ -300,6 +324,8 @@ func (e *endpointRepository) FindGroupedByEndpoint(ctx context.Context, projectIImpact: impact,ImpactReason: ComputeImpactReason(g.Endpoint, g.TotalCount, g.SatisfiedCount, g.ToleratingCount,g.BadCount, g.ClientErrorCount, p99, g.OffsetMs),+ HasRoot: g.HasRoot,+ HasNonRoot: g.HasNonRoot,})}diff --git a/backend/app/repositories/endpoint_repository_test.go b/backend/app/repositories/endpoint_repository_test.goindex c532f142..05d63651 100644--- a/backend/app/repositories/endpoint_repository_test.go+++ b/backend/app/repositories/endpoint_repository_test.go@@ -134,7 +134,7 @@ func TestEndpointRepository_FindGroupedByEndpoint(t *testing.T) {t.Fatalf("InsertAsync failed: %v", err)}- stats, total, err := EndpointRepository.FindGroupedByEndpoint(ctx, projectId, now.Add(-time.Hour), now.Add(time.Hour), 1, 10, "count", "desc", "")+ stats, total, err := EndpointRepository.FindGroupedByEndpoint(ctx, projectId, now.Add(-time.Hour), now.Add(time.Hour), 1, 10, "count", "desc", "", "")if err != nil {t.Fatalf("FindGroupedByEndpoint failed: %v", err)}@@ -170,7 +170,7 @@ func TestEndpointRepository_FindGroupedByEndpoint_Search(t *testing.T) {t.Fatalf("InsertAsync failed: %v", err)}- stats, total, err := EndpointRepository.FindGroupedByEndpoint(ctx, projectId, now.Add(-time.Hour), now.Add(time.Hour), 1, 10, "count", "desc", "users")+ stats, total, err := EndpointRepository.FindGroupedByEndpoint(ctx, projectId, now.Add(-time.Hour), now.Add(time.Hour), 1, 10, "count", "desc", "users", "")if err != nil {t.Fatalf("FindGroupedByEndpoint with search failed: %v", err)}diff --git a/backend/app/repositories/task.repository.go b/backend/app/repositories/task.repository.goindex 990630af..14bc51e8 100644--- a/backend/app/repositories/task.repository.go+++ b/backend/app/repositories/task.repository.go@@ -18,7 +18,7 @@ import (type taskRepository struct{}func (e *taskRepository) InsertAsync(ctx context.Context, lines []models.Task) error {- batch, err := chdb.Conn.PrepareBatch(clickhouse.Context(context.Background(), clickhouse.WithAsync(false)), "INSERT INTO tasks (id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name, distributed_trace_id, span_id)")+ batch, err := chdb.Conn.PrepareBatch(clickhouse.Context(context.Background(), clickhouse.WithAsync(false)), "INSERT INTO tasks (id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name, distributed_trace_id, span_id, is_root)")if err != nil {return err}@@ -29,7 +29,11 @@ func (e *taskRepository) InsertAsync(ctx context.Context, lines []models.Task) eattributesJSON = string(attributesBytes)}}- if err := batch.Append(t.Id, t.ProjectId, t.TaskName, int64(t.Duration), t.RecordedAt, t.ClientIP, attributesJSON, t.AppVersion, t.ServerName, t.DistributedTraceId, t.SpanId); err != nil {+ isRoot := uint8(0)+ if t.IsRoot {+ isRoot = 1+ }+ if err := batch.Append(t.Id, t.ProjectId, t.TaskName, int64(t.Duration), t.RecordedAt, t.ClientIP, attributesJSON, t.AppVersion, t.ServerName, t.DistributedTraceId, t.SpanId, isRoot); err != nil {return err}}@@ -85,10 +89,22 @@ func (e *taskRepository) FindAll(ctx context.Context, projectId uuid.UUID, fromDreturn tasks, int64(count), nil}-func (e *taskRepository) FindGroupedByTaskName(ctx context.Context, projectId uuid.UUID, fromDate, toDate time.Time, page, pageSize int, orderBy string, sortDirection string) ([]models.TaskStats, int64, error) {- // Count unique task names+func (e *taskRepository) FindGroupedByTaskName(ctx context.Context, projectId uuid.UUID, fromDate, toDate time.Time, page, pageSize int, orderBy string, sortDirection string, search string, rootFilter string) ([]models.TaskStats, int64, error) {+ whereClause := "project_id = ? AND recorded_at >= ? AND recorded_at <= ?"+ args := []interface{}{projectId, fromDate, toDate}+ if search != "" {+ whereClause += " AND positionCaseInsensitive(task_name, ?) > 0"+ args = append(args, search)+ }+ switch rootFilter {+ case "root":+ whereClause += " AND is_root = 1"+ case "non_root":+ whereClause += " AND is_root = 0"+ }+var count uint64- err := chdb.Conn.QueryRow(ctx, "SELECT uniq(task_name) FROM tasks WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?", projectId, fromDate, toDate).Scan(&count)+ err := chdb.Conn.QueryRow(ctx, "SELECT uniq(task_name) FROM tasks WHERE "+whereClause, args...).Scan(&count)if err != nil {return nil, 0, err}@@ -122,14 +138,17 @@ func (e *taskRepository) FindGroupedByTaskName(ctx context.Context, projectId uuquantile(0.5)(duration) as p50_duration,quantile(0.95)(duration) as p95_duration,avg(duration) as avg_duration,- max(recorded_at) as last_seen+ max(recorded_at) as last_seen,+ max(is_root) as has_root,+ max(if(is_root = 0, 1, 0)) as has_non_rootFROM tasks- WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?+ WHERE ` + whereClause + `GROUP BY task_nameORDER BY ` + orderExpr + ` ` + sortDir + `LIMIT ? OFFSET ?`- rows, err := chdb.Conn.Query(ctx, query, projectId, fromDate, toDate, pageSize, offset)+ queryArgs := append(args, pageSize, offset)+ rows, err := chdb.Conn.Query(ctx, query, queryArgs...)if err != nil {return nil, 0, err}@@ -139,12 +158,15 @@ func (e *taskRepository) FindGroupedByTaskName(ctx context.Context, projectId uufor rows.Next() {var s models.TaskStatsvar p50, p95, avg float64- if err := rows.Scan(&s.TaskName, &s.Count, &p50, &p95, &avg, &s.LastSeen); err != nil {+ var hasRoot, hasNonRoot uint8+ if err := rows.Scan(&s.TaskName, &s.Count, &p50, &p95, &avg, &s.LastSeen, &hasRoot, &hasNonRoot); err != nil {return nil, 0, err}s.P50Duration = time.Duration(p50)s.P95Duration = time.Duration(p95)s.AvgDuration = time.Duration(avg)+ s.HasRoot = hasRoot == 1+ s.HasNonRoot = hasNonRoot == 1stats = append(stats, s)}@@ -202,17 +224,19 @@ func (e *taskRepository) FindByTaskName(ctx context.Context, projectId uuid.UUID// FindById returns a single task by IDfunc (e *taskRepository) FindById(ctx context.Context, projectId, taskId uuid.UUID) (*models.Task, error) {- query := `SELECT id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name, distributed_trace_id, span_id+ query := `SELECT id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name, distributed_trace_id, span_id, is_rootFROM tasksWHERE project_id = ? AND id = ?LIMIT 1`var t models.Taskvar attributesJSON string+ var isRoot uint8err := chdb.Conn.QueryRow(ctx, query, projectId, taskId).Scan(&t.Id, &t.ProjectId, &t.TaskName, &t.Duration, &t.RecordedAt,- &t.ClientIP, &attributesJSON, &t.AppVersion, &t.ServerName, &t.DistributedTraceId, &t.SpanId)+ &t.ClientIP, &attributesJSON, &t.AppVersion, &t.ServerName, &t.DistributedTraceId, &t.SpanId, &isRoot)+ t.IsRoot = isRoot == 1if err != nil {if errors.Is(err, sql.ErrNoRows) {diff --git a/backend/app/repositories/task.repository_sqlite.go b/backend/app/repositories/task.repository_sqlite.goindex 124a2bb0..eb5bd8bd 100644--- a/backend/app/repositories/task.repository_sqlite.go+++ b/backend/app/repositories/task.repository_sqlite.go@@ -27,6 +27,7 @@ type task struct {ServerName string `lit:"server_name"`DistributedTraceId *uuid.UUID `lit:"distributed_trace_id"`SpanId *uuid.UUID `lit:"span_id"`+ IsRoot bool `lit:"is_root"`}type taskGroupRow struct {@@ -34,6 +35,8 @@ type taskGroupRow struct {Count uint64 `lit:"count"`AvgDuration float64 `lit:"avg_duration"`LastSeen string `lit:"last_seen"`+ HasRoot bool `lit:"has_root"`+ HasNonRoot bool `lit:"has_non_root"`}type taskCountStatsRow struct {@@ -67,6 +70,7 @@ func taskToRow(t models.Task) task {ServerName: t.ServerName,DistributedTraceId: t.DistributedTraceId,SpanId: t.SpanId,+ IsRoot: t.IsRoot,}}@@ -82,6 +86,7 @@ func (r *task) toModel() models.Task {ServerName: r.ServerName,DistributedTraceId: r.DistributedTraceId,SpanId: r.SpanId,+ IsRoot: r.IsRoot,}if r.Attributes != nil {t.Attributes = map[string]string(r.Attributes)@@ -155,11 +160,17 @@ func (e *taskRepository) FindAll(ctx context.Context, projectId uuid.UUID, fromDreturn tasks, count, nil}-func (e *taskRepository) FindGroupedByTaskName(ctx context.Context, projectId uuid.UUID, fromDate, toDate time.Time, page, pageSize int, orderBy string, sortDirection string) ([]models.TaskStats, int64, error) {+func (e *taskRepository) FindGroupedByTaskName(ctx context.Context, projectId uuid.UUID, fromDate, toDate time.Time, page, pageSize int, orderBy string, sortDirection string, search string, rootFilter string) ([]models.TaskStats, int64, error) {params := lit.P{"project_id": projectId, "from": NewSQLiteTime(fromDate), "to": NewSQLiteTime(toDate)}+ whereClause := "project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to"+ if search != "" {+ whereClause += " AND INSTR(LOWER(task_name), LOWER(:search)) > 0"+ params["search"] = search+ }+ whereClause += rootFilterClause("is_root", rootFilter)totalResult, err := lit.SelectSingleNamed[models.CountResult](db.TelemetryDB,- "SELECT COUNT(DISTINCT task_name) AS count FROM tasks WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to",+ "SELECT COUNT(DISTINCT task_name) AS count FROM tasks WHERE "+whereClause,params)if err != nil {return nil, 0, err@@ -180,10 +191,16 @@ func (e *taskRepository) FindGroupedByTaskName(ctx context.Context, projectId uuvar baseQuery stringgroupParams := lit.P{"project_id": projectId, "from": NewSQLiteTime(fromDate), "to": NewSQLiteTime(toDate)}+ if search != "" {+ groupParams["search"] = search+ }++ groupedCols := `task_name, COUNT(*) as count, AVG(duration) as avg_duration, MAX(recorded_at) as last_seen,+ MAX(is_root) as has_root, MAX(CASE WHEN is_root = 0 THEN 1 ELSE 0 END) as has_non_root`if needsGoSort {- baseQuery = `SELECT task_name, COUNT(*) as count, AVG(duration) as avg_duration, MAX(recorded_at) as last_seen- FROM tasks WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to+ baseQuery = `SELECT ` + groupedCols + `+ FROM tasks WHERE ` + whereClause + `GROUP BY task_name`} else {orderExpr := map[string]string{"count": "count", "last_seen": "last_seen"}@@ -191,8 +208,8 @@ func (e *taskRepository) FindGroupedByTaskName(ctx context.Context, projectId uuif !ok {expr = "count"}- baseQuery = fmt.Sprintf(`SELECT task_name, COUNT(*) as count, AVG(duration) as avg_duration, MAX(recorded_at) as last_seen- FROM tasks WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to+ baseQuery = fmt.Sprintf(`SELECT `+groupedCols+`+ FROM tasks WHERE `+whereClause+`GROUP BY task_name ORDER BY %s %s LIMIT :limit OFFSET :offset`, expr, sortDir)groupParams["limit"] = pageSizegroupParams["offset"] = offset@@ -218,6 +235,8 @@ func (e *taskRepository) FindGroupedByTaskName(ctx context.Context, projectId uuP95Duration: time.Duration(computePercentile(durations, 0.95)),AvgDuration: time.Duration(g.AvgDuration),LastSeen: ls,+ HasRoot: g.HasRoot,+ HasNonRoot: g.HasNonRoot,})}@@ -307,7 +326,7 @@ func (e *taskRepository) FindByTaskName(ctx context.Context, projectId uuid.UUIDfunc (e *taskRepository) FindById(ctx context.Context, projectId, taskId uuid.UUID) (*models.Task, error) {row, err := lit.SelectSingleNamed[task](db.TelemetryDB,- `SELECT id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name, distributed_trace_id, span_id+ `SELECT id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name, distributed_trace_id, span_id, is_rootFROM tasks WHERE project_id = :project_id AND id = :id LIMIT 1`,lit.P{"project_id": projectId, "id": taskId})if err != nil {diff --git a/backend/app/repositories/task_repository_test.go b/backend/app/repositories/task_repository_test.goindex f9cf3816..f68e8bdb 100644--- a/backend/app/repositories/task_repository_test.go+++ b/backend/app/repositories/task_repository_test.go@@ -141,7 +141,7 @@ func TestTaskRepository_FindGroupedByTaskName(t *testing.T) {t.Fatalf("InsertAsync failed: %v", err)}- stats, total, err := TaskRepository.FindGroupedByTaskName(ctx, projectId, now.Add(-time.Hour), now.Add(time.Hour), 1, 10, "count", "desc")+ stats, total, err := TaskRepository.FindGroupedByTaskName(ctx, projectId, now.Add(-time.Hour), now.Add(time.Hour), 1, 10, "count", "desc", "", "")if err != nil {t.Fatalf("FindGroupedByTaskName failed: %v", err)}@@ -162,6 +162,55 @@ func TestTaskRepository_FindGroupedByTaskName(t *testing.T) {}}+func TestTaskRepository_FindGroupedByTaskName_RootFilterAndSearch(t *testing.T) {+ setupTestDB(t)+ ctx := context.Background()+ projectId := uuid.New()+ now := truncateMs(time.Now().UTC())++ rootTask := makeTask(projectId, "scheduler.cleanup", 100*time.Millisecond, now)+ rootTask.IsRoot = true+ nonRootTask := makeTask(projectId, "queue.process", 200*time.Millisecond, now.Add(time.Minute))+ nonRootTask.IsRoot = false++ if err := TaskRepository.InsertAsync(ctx, []models.Task{rootTask, nonRootTask}); err != nil {+ t.Fatalf("InsertAsync failed: %v", err)+ }++ // rootFilter=root → only the cleanup task+ stats, _, err := TaskRepository.FindGroupedByTaskName(ctx, projectId, now.Add(-time.Hour), now.Add(time.Hour), 1, 10, "count", "desc", "", "root")+ if err != nil {+ t.Fatalf("FindGroupedByTaskName(root) failed: %v", err)+ }+ if len(stats) != 1 || stats[0].TaskName != "scheduler.cleanup" {+ t.Errorf("rootFilter=root: expected 1 stats row 'scheduler.cleanup', got %+v", stats)+ }+ if !stats[0].HasRoot || stats[0].HasNonRoot {+ t.Errorf("expected HasRoot=true, HasNonRoot=false, got %+v", stats[0])+ }++ // rootFilter=non_root → only the queue.process task+ stats, _, err = TaskRepository.FindGroupedByTaskName(ctx, projectId, now.Add(-time.Hour), now.Add(time.Hour), 1, 10, "count", "desc", "", "non_root")+ if err != nil {+ t.Fatalf("FindGroupedByTaskName(non_root) failed: %v", err)+ }+ if len(stats) != 1 || stats[0].TaskName != "queue.process" {+ t.Errorf("rootFilter=non_root: expected 1 stats row 'queue.process', got %+v", stats)+ }+ if stats[0].HasRoot || !stats[0].HasNonRoot {+ t.Errorf("expected HasRoot=false, HasNonRoot=true, got %+v", stats[0])+ }++ // search=queue → only matching task+ stats, _, err = TaskRepository.FindGroupedByTaskName(ctx, projectId, now.Add(-time.Hour), now.Add(time.Hour), 1, 10, "count", "desc", "queue", "")+ if err != nil {+ t.Fatalf("FindGroupedByTaskName(search) failed: %v", err)+ }+ if len(stats) != 1 || stats[0].TaskName != "queue.process" {+ t.Errorf("search=queue: expected 1 stats row 'queue.process', got %+v", stats)+ }+}+func TestTaskRepository_FindByTaskName(t *testing.T) {setupTestDB(t)ctx := context.Background()diff --git a/backend/app/repositories/testhelper_sqlite_test.go b/backend/app/repositories/testhelper_sqlite_test.goindex a77ab8dd..fb99973e 100644--- a/backend/app/repositories/testhelper_sqlite_test.go+++ b/backend/app/repositories/testhelper_sqlite_test.go@@ -67,7 +67,9 @@ CREATE TABLE IF NOT EXISTS endpoints (app_version TEXT NOT NULL DEFAULT '',server_name TEXT NOT NULL DEFAULT '',distributed_trace_id TEXT DEFAULT NULL,- is_stream INTEGER NOT NULL DEFAULT 0+ span_id TEXT DEFAULT NULL,+ is_stream INTEGER NOT NULL DEFAULT 0,+ is_root INTEGER NOT NULL DEFAULT 1);CREATE INDEX IF NOT EXISTS idx_endpoints_project_recorded ON endpoints(project_id, recorded_at);CREATE INDEX IF NOT EXISTS idx_endpoints_project_endpoint ON endpoints(project_id, endpoint);@@ -82,10 +84,42 @@ CREATE TABLE IF NOT EXISTS tasks (attributes TEXT NOT NULL DEFAULT '{}',app_version TEXT NOT NULL DEFAULT '',server_name TEXT NOT NULL DEFAULT '',- distributed_trace_id TEXT DEFAULT NULL+ distributed_trace_id TEXT DEFAULT NULL,+ span_id TEXT DEFAULT NULL,+ is_root INTEGER NOT NULL DEFAULT 1);CREATE INDEX IF NOT EXISTS idx_tasks_project_recorded ON tasks(project_id, recorded_at);+CREATE TABLE IF NOT EXISTS ai_traces (+ id TEXT NOT NULL,+ project_id TEXT NOT NULL,+ recorded_at DATETIME NOT NULL,+ duration INTEGER NOT NULL DEFAULT 0,+ status_code INTEGER NOT NULL DEFAULT 0,+ model TEXT NOT NULL DEFAULT '',+ response_model TEXT NOT NULL DEFAULT '',+ provider TEXT NOT NULL DEFAULT '',+ operation TEXT NOT NULL DEFAULT '',+ input_tokens INTEGER NOT NULL DEFAULT 0,+ output_tokens INTEGER NOT NULL DEFAULT 0,+ total_tokens INTEGER NOT NULL DEFAULT 0,+ cached_tokens INTEGER NOT NULL DEFAULT 0,+ reasoning_tokens INTEGER NOT NULL DEFAULT 0,+ input_cost REAL NOT NULL DEFAULT 0,+ output_cost REAL NOT NULL DEFAULT 0,+ total_cost REAL NOT NULL DEFAULT 0,+ trace_name TEXT NOT NULL DEFAULT '',+ user_id TEXT NOT NULL DEFAULT '',+ finish_reason TEXT NOT NULL DEFAULT '',+ server_name TEXT NOT NULL DEFAULT '',+ app_version TEXT NOT NULL DEFAULT '',+ storage_key TEXT NOT NULL DEFAULT '',+ attributes TEXT NOT NULL DEFAULT '{}',+ distributed_trace_id TEXT DEFAULT NULL,+ is_root INTEGER NOT NULL DEFAULT 1+);+CREATE INDEX IF NOT EXISTS idx_ai_traces_project_recorded ON ai_traces(project_id, recorded_at);+CREATE TABLE IF NOT EXISTS exception_stack_traces (id TEXT NOT NULL,project_id TEXT NOT NULL,@@ -98,7 +132,8 @@ CREATE TABLE IF NOT EXISTS exception_stack_traces (app_version TEXT NOT NULL DEFAULT '',server_name TEXT NOT NULL DEFAULT '',is_message INTEGER NOT NULL DEFAULT 0,- distributed_trace_id TEXT DEFAULT NULL+ distributed_trace_id TEXT DEFAULT NULL,+ session_id TEXT DEFAULT NULL);CREATE INDEX IF NOT EXISTS idx_exceptions_project_recorded ON exception_stack_traces(project_id, recorded_at);CREATE INDEX IF NOT EXISTS idx_exceptions_project_hash ON exception_stack_traces(project_id, exception_hash);@@ -110,7 +145,8 @@ CREATE TABLE IF NOT EXISTS spans (name TEXT NOT NULL DEFAULT '',start_time DATETIME NOT NULL,duration INTEGER NOT NULL DEFAULT 0,- recorded_at DATETIME NOT NULL+ recorded_at DATETIME NOT NULL,+ parent_span_id TEXT DEFAULT NULL);CREATE INDEX IF NOT EXISTS idx_spans_project_trace ON spans(project_id, trace_id);@@ -128,9 +164,12 @@ CREATE TABLE IF NOT EXISTS session_recordings (project_id TEXT NOT NULL,exception_id TEXT NOT NULL,file_path TEXT NOT NULL DEFAULT '',- recorded_at DATETIME NOT NULL+ recorded_at DATETIME NOT NULL,+ session_id TEXT DEFAULT NULL,+ segment_index INTEGER NOT NULL DEFAULT 0);CREATE INDEX IF NOT EXISTS idx_session_recordings_project_exception ON session_recordings(project_id, exception_id);+CREATE INDEX IF NOT EXISTS idx_session_recordings_session_id ON session_recordings(session_id);CREATE TABLE IF NOT EXISTS archived_exceptions (project_id TEXT NOT NULL,diff --git a/examples/devtesting-embedded/go.mod b/examples/devtesting-embedded/go.modindex a6ea6c02..ac745f7d 100644--- a/examples/devtesting-embedded/go.mod+++ b/examples/devtesting-embedded/go.mod@@ -93,7 +93,7 @@ require (go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.43.0 // indirectgo.opentelemetry.io/otel/metric v1.43.0 // indirectgo.opentelemetry.io/proto/otlp v1.10.0 // indirect- go.tracewayapp.com v1.0.2 // indirect+ go.tracewayapp.com v1.0.3 // indirectgo.tracewayapp.com/tracewaygin v1.0.2 // indirectgo.yaml.in/yaml/v3 v3.0.4 // indirectgolang.org/x/arch v0.25.0 // indirectdiff --git a/frontend/src/lib/components/distributed-trace/distributed-trace-card.svelte b/frontend/src/lib/components/distributed-trace/distributed-trace-card.svelteindex a0177698..0a4ba3a1 100644--- a/frontend/src/lib/components/distributed-trace/distributed-trace-card.svelte+++ b/frontend/src/lib/components/distributed-trace/distributed-trace-card.svelte@@ -44,6 +44,8 @@projectsState.selectProject(node.projectId);if (node.traceType === 'task' && node.task) {goto(`/tasks/${encodeURIComponent(node.task.taskName)}/${node.task.id}?preset=24h`);+ } else if (node.traceType === 'ai_trace' && node.aiTrace) {+ goto(`/ai-traces/${encodeURIComponent(node.aiTrace.traceName)}/${node.aiTrace.id}?preset=24h`);} else if (node.traceType === 'exception' && node.exception) {goto(`/issues/${node.exception.exceptionHash}?preset=24h`);} else if (node.endpoint) {@@ -93,6 +95,8 @@<span class="truncate font-mono text-sm">{#if node.traceType === 'task'}{node.task?.taskName}+ {:else if node.traceType === 'ai_trace'}+ {node.aiTrace?.traceName}{:else if node.traceType === 'exception'}{node.exception?.stackTrace.split('\n')[0]}{:else}@@ -104,9 +108,16 @@{node.endpoint.statusCode}</span>{/if}+ {#if node.traceType === 'ai_trace' && node.aiTrace}+ <Badge variant="secondary" class="shrink-0">{node.aiTrace.provider || node.aiTrace.model || 'AI'}</Badge>+ {/if}{#if node.traceType !== 'exception'}<span class="shrink-0 font-mono text-sm text-muted-foreground">- {formatDuration(node.traceType === 'task' ? node.task?.duration ?? 0 : node.endpoint?.duration ?? 0)}+ {formatDuration(+ node.traceType === 'task' ? node.task?.duration ?? 0+ : node.traceType === 'ai_trace' ? node.aiTrace?.duration ?? 0+ : node.endpoint?.duration ?? 0+ )}</span>{/if}{#if node.exception}diff --git a/frontend/src/lib/components/ui/non-root-chip/index.ts b/frontend/src/lib/components/ui/non-root-chip/index.tsnew file mode 100644index 00000000..0bfa965e--- /dev/null+++ b/frontend/src/lib/components/ui/non-root-chip/index.ts@@ -0,0 +1 @@+export { default as NonRootChip } from './non-root-chip.svelte';diff --git a/frontend/src/lib/components/ui/non-root-chip/non-root-chip.svelte b/frontend/src/lib/components/ui/non-root-chip/non-root-chip.sveltenew file mode 100644index 00000000..65d04df8--- /dev/null+++ b/frontend/src/lib/components/ui/non-root-chip/non-root-chip.svelte@@ -0,0 +1,30 @@+<script lang="ts">+ import { Badge } from '$lib/components/ui/badge';+ import * as Tooltip from '$lib/components/ui/tooltip';++ type Props = {+ mixed?: boolean;+ };++ let { mixed = false }: Props = $props();++ const label = $derived(mixed ? 'Mixed' : 'Non-root');+ const tooltip = $derived(+ mixed+ ? 'Some runs were triggered by another trace — see the distributed trace view.'+ : 'Triggered by another trace — see the distributed trace view.'+ );+</script>++<Tooltip.Provider>+ <Tooltip.Root>+ <Tooltip.Trigger>+ <Badge variant="secondary" class="ml-2 text-[10px]">+ {label}+ </Badge>+ </Tooltip.Trigger>+ <Tooltip.Content>+ {tooltip}+ </Tooltip.Content>+ </Tooltip.Root>+</Tooltip.Provider>diff --git a/frontend/src/lib/components/ui/root-filter/index.ts b/frontend/src/lib/components/ui/root-filter/index.tsnew file mode 100644index 00000000..2438e3e2--- /dev/null+++ b/frontend/src/lib/components/ui/root-filter/index.ts@@ -0,0 +1 @@+export { default as RootFilter } from './root-filter.svelte';diff --git a/frontend/src/lib/components/ui/root-filter/root-filter.svelte b/frontend/src/lib/components/ui/root-filter/root-filter.sveltenew file mode 100644index 00000000..745598de--- /dev/null+++ b/frontend/src/lib/components/ui/root-filter/root-filter.svelte@@ -0,0 +1,30 @@+<script lang="ts">+ import * as Select from '$lib/components/ui/select';++ type Props = {+ value?: string;+ };++ let { value = $bindable('all') }: Props = $props();++ const options = [+ { value: 'all', label: 'All' },+ { value: 'root', label: 'Root' },+ { value: 'non_root', label: 'Non-root' }+ ];++ const label = $derived(options.find((o) => o.value === value)?.label ?? 'All');+</script>++<Select.Root type="single" bind:value>+ <Select.Trigger class="h-9 w-[120px] rounded-none border-r-0 shadow-none">+ {label}+ </Select.Trigger>+ <Select.Content>+ {#each options as option}+ <Select.Item value={option.value} label={option.label}>+ {option.label}+ </Select.Item>+ {/each}+ </Select.Content>+</Select.Root>diff --git a/frontend/src/lib/types/distributed-trace.ts b/frontend/src/lib/types/distributed-trace.tsindex 41e8054d..e22d8eae 100644--- a/frontend/src/lib/types/distributed-trace.ts+++ b/frontend/src/lib/types/distributed-trace.ts@@ -1,7 +1,7 @@export type DistributedTraceNode = {projectId: string;projectName: string;- traceType: 'endpoint' | 'task' | 'exception';+ traceType: 'endpoint' | 'task' | 'ai_trace' | 'exception';endpoint?: {id: string;endpoint: string;@@ -15,6 +15,16 @@ export type DistributedTraceNode = {duration: number;recordedAt: string;};+ aiTrace?: {+ id: string;+ traceName: string;+ model: string;+ provider: string;+ duration: number;+ totalTokens: number;+ totalCost: number;+ recordedAt: string;+ };spans: any[];exception?: {exceptionHash: string;diff --git a/frontend/src/routes/ai-traces/+page.svelte b/frontend/src/routes/ai-traces/+page.svelteindex d166b4b9..8bfd098f 100644--- a/frontend/src/routes/ai-traces/+page.svelte+++ b/frontend/src/routes/ai-traces/+page.svelte@@ -10,6 +10,8 @@import { PaginationFooter } from "$lib/components/ui/pagination-footer";import { TimeRangePicker } from "$lib/components/ui/time-range-picker";import { SearchBar } from "$lib/components/ui/search-bar";+ import { RootFilter } from "$lib/components/ui/root-filter";+ import { NonRootChip } from "$lib/components/ui/non-root-chip";import { browser } from '$app/environment';import { CalendarDate } from "@internationalized/date";import { projectsState } from '$lib/state/projects.svelte';@@ -45,6 +47,8 @@avgInputTokens: number;avgOutputTokens: number;lastSeen: string;+ hasRoot: boolean;+ hasNonRoot: boolean;};type SortField = 'count' | 'p50_duration' | 'p95_duration' | 'total_tokens' | 'total_cost' | 'last_seen';@@ -59,14 +63,15 @@let total = $state(0);let totalPages = $state(0);- // Parse URL params including search+ // Parse URL params including search + rootFilterfunction parseAiTracesUrlParams() {- if (!browser) return { preset: '24h', from: null, to: null, search: '' };+ if (!browser) return { preset: '24h', from: null, to: null, search: '', rootFilter: 'all' };const params = new URLSearchParams(window.location.search);const timeParams = parseTimeRangeFromUrl(timezone, '24h');return {...timeParams,- search: params.get('search') || ''+ search: params.get('search') || '',+ rootFilter: params.get('rootFilter') || 'all'};}@@ -74,8 +79,9 @@const initialUrlParams = parseAiTracesUrlParams();const initialRange = getResolvedTimeRange(initialUrlParams, timezone);- // Search state+ // Search + rootFilter statelet searchQuery = $state(initialUrlParams.search);+ let rootFilter = $state(initialUrlParams.rootFilter);// Date Range Statelet selectedPreset = $state<string | null>(initialUrlParams.preset);@@ -96,6 +102,9 @@if (searchQuery.trim()) {params.search = searchQuery.trim();}+ if (rootFilter && rootFilter !== 'all') {+ params.rootFilter = rootFilter;+ }updateUrl(params, { pushToHistory });}@@ -110,6 +119,7 @@toDate = dateToCalendarDate(range.to, timezone);toTime = dateToTimeString(range.to, timezone);searchQuery = urlParams.search;+ rootFilter = urlParams.rootFilter;page = 1;loadData(false);@@ -189,7 +199,8 @@page: page,pageSize: pageSize},- search: searchQuery.trim()+ search: searchQuery.trim(),+ rootFilter: rootFilter === 'all' ? '' : rootFilter};const response = await api.post('/ai-traces/grouped', requestBody, { projectId: projectsState.currentProjectId ?? undefined });@@ -269,7 +280,11 @@bind:value={searchQuery}onSearch={handleSearch}disabled={loading}- />+ >+ {#snippet children()}+ <RootFilter bind:value={rootFilter} />+ {/snippet}+ </SearchBar><!-- AI Traces Table --><div class="rounded-md border overflow-hidden">@@ -367,6 +382,9 @@><Table.Cell class="font-mono text-sm">{trace.traceName}+ {#if trace.hasNonRoot}+ <NonRootChip mixed={trace.hasRoot && trace.hasNonRoot} />+ {/if}</Table.Cell><Table.Cell class="tabular-nums">{formatCount(trace.count)}diff --git a/frontend/src/routes/endpoints/+page.svelte b/frontend/src/routes/endpoints/+page.svelteindex 5e0e51e6..b65ae43f 100644--- a/frontend/src/routes/endpoints/+page.svelte+++ b/frontend/src/routes/endpoints/+page.svelte@@ -16,6 +16,8 @@import { PaginationFooter } from "$lib/components/ui/pagination-footer";import { TimeRangePicker } from "$lib/components/ui/time-range-picker";import { SearchBar } from "$lib/components/ui/search-bar";+ import { RootFilter } from "$lib/components/ui/root-filter";+ import { NonRootChip } from "$lib/components/ui/non-root-chip";import { browser } from '$app/environment';import { CalendarDate } from "@internationalized/date";import { projectsState } from '$lib/state/projects.svelte';@@ -54,6 +56,8 @@impact: number;impactReason: string;isStream?: boolean;+ hasRoot: boolean;+ hasNonRoot: boolean;};type SortField = 'count' | 'p50_duration' | 'p95_duration' | 'p99_duration' | 'last_seen' | 'impact';@@ -93,14 +97,15 @@let total = $state(0);let totalPages = $state(0);- // Parse URL params including search+ // Parse URL params including search + rootFilterfunction parseEndpointsUrlParams() {- if (!browser) return { preset: '24h', from: null, to: null, search: '' };+ if (!browser) return { preset: '24h', from: null, to: null, search: '', rootFilter: 'all' };const params = new URLSearchParams(window.location.search);const timeParams = parseTimeRangeFromUrl(timezone, '24h');return {...timeParams,- search: params.get('search') || ''+ search: params.get('search') || '',+ rootFilter: params.get('rootFilter') || 'all'};}@@ -108,8 +113,9 @@const initialUrlParams = parseEndpointsUrlParams();const initialRange = getResolvedTimeRange(initialUrlParams, timezone);- // Search state+ // Search + rootFilter statelet searchQuery = $state(initialUrlParams.search);+ let rootFilter = $state(initialUrlParams.rootFilter);// Date Range Statelet selectedPreset = $state<string | null>(initialUrlParams.preset);@@ -130,6 +136,9 @@if (searchQuery.trim()) {params.search = searchQuery.trim();}+ if (rootFilter && rootFilter !== 'all') {+ params.rootFilter = rootFilter;+ }updateUrl(params, { pushToHistory });}@@ -144,6 +153,7 @@toDate = dateToCalendarDate(range.to, timezone);toTime = dateToTimeString(range.to, timezone);searchQuery = urlParams.search;+ rootFilter = urlParams.rootFilter;page = 1;loadData(false);@@ -264,7 +274,8 @@page: page,pageSize: pageSize},- search: searchQuery.trim()+ search: searchQuery.trim(),+ rootFilter: rootFilter === 'all' ? '' : rootFilter};const response = await api.post('/endpoints/grouped', requestBody, { projectId: projectsState.currentProjectId ?? undefined });@@ -402,7 +413,11 @@bind:value={searchQuery}onSearch={handleSearch}disabled={loading}- />+ >+ {#snippet children()}+ <RootFilter bind:value={rootFilter} />+ {/snippet}+ </SearchBar><!-- Performance Chart --><Card.Root class="pt-2">@@ -550,6 +565,9 @@{#if endpoint.isStream}<Badge variant="outline" class="font-sans text-xs" title="Streaming response (SSE / WebSocket / long-poll). Excluded from latency, Apdex, and impact scoring.">Stream</Badge>{/if}+ {#if endpoint.hasNonRoot}+ <NonRootChip mixed={endpoint.hasRoot && endpoint.hasNonRoot} />+ {/if}</span></Table.Cell><Table.Cell class="tabular-nums">diff --git a/frontend/src/routes/tasks/+page.svelte b/frontend/src/routes/tasks/+page.svelteindex 6a552338..33cadd93 100644--- a/frontend/src/routes/tasks/+page.svelte+++ b/frontend/src/routes/tasks/+page.svelte@@ -9,7 +9,11 @@import { TableEmptyState } from "$lib/components/ui/table-empty-state";import { PaginationFooter } from "$lib/components/ui/pagination-footer";import { TimeRangePicker } from "$lib/components/ui/time-range-picker";+ import { SearchBar } from '$lib/components/ui/search-bar';+ import { RootFilter } from '$lib/components/ui/root-filter';+ import { NonRootChip } from '$lib/components/ui/non-root-chip';import { CalendarDate } from "@internationalized/date";+ import { browser } from '$app/environment';import { projectsState } from '$lib/state/projects.svelte';import { createRowClickHandler } from '$lib/utils/navigation';import { resolve } from '$app/paths';@@ -39,6 +43,8 @@p95Duration: number;avgDuration: number;lastSeen: string;+ hasRoot: boolean;+ hasNonRoot: boolean;};type SortField = 'count' | 'p50_duration' | 'p95_duration' | 'last_seen';@@ -57,6 +63,19 @@const initialUrlParams = parseTimeRangeFromUrl(timezone);const initialRange = getResolvedTimeRange(initialUrlParams, timezone);+ function readSearchAndFilterFromUrl() {+ if (!browser) return { search: '', rootFilter: 'all' };+ const params = new URLSearchParams(window.location.search);+ return {+ search: params.get('search') || '',+ rootFilter: params.get('rootFilter') || 'all'+ };+ }++ const initialSearchState = readSearchAndFilterFromUrl();+ let searchQuery = $state(initialSearchState.search);+ let rootFilter = $state(initialSearchState.rootFilter);+// Date Range Statelet selectedPreset = $state<string | null>(initialUrlParams.preset);let fromDate = $state<CalendarDate>(dateToCalendarDate(initialRange.from, timezone));@@ -64,14 +83,14 @@let fromTime = $state(dateToTimeString(initialRange.from, timezone));let toTime = $state(dateToTimeString(initialRange.to, timezone));- // Update URL with current time range+ // Update URL with current time range + search + rootFilterfunction updateTimeRangeUrl(pushToHistory = true) {- updateUrl(- selectedPreset- ? { preset: selectedPreset }- : { from: getFromDateTimeUTC(), to: getToDateTimeUTC() },- { pushToHistory }- );+ const params: Record<string, string | undefined> = selectedPreset+ ? { preset: selectedPreset }+ : { from: getFromDateTimeUTC(), to: getToDateTimeUTC() };+ if (searchQuery) params.search = searchQuery;+ if (rootFilter && rootFilter !== 'all') params.rootFilter = rootFilter;+ updateUrl(params, { pushToHistory });}// Handle browser back/forward navigation@@ -149,6 +168,8 @@toDate: getToDateTimeUTC(),orderBy: orderBy,sortDirection: sortDirection,+ search: searchQuery,+ rootFilter: rootFilter === 'all' ? '' : rootFilter,pagination: {page: page,pageSize: pageSize@@ -190,6 +211,11 @@loadData(false);}+ function handleSearch() {+ page = 1;+ loadData(true);+ }+onMount(() => {// Add popstate listener for back/forward navigationwindow.addEventListener('popstate', handlePopState);@@ -211,7 +237,12 @@<PageHeader title="Tasks" />- <div class="flex flex-col">+ <div class="flex flex-col gap-2 sm:flex-row sm:items-end">+ <SearchBar placeholder="Search tasks..." bind:value={searchQuery} onSearch={handleSearch}>+ {#snippet children()}+ <RootFilter bind:value={rootFilter} />+ {/snippet}+ </SearchBar><TimeRangePickerbind:fromDatebind:toDate@@ -292,6 +323,9 @@><Table.Cell class="font-mono text-sm">{task.taskName}+ {#if task.hasNonRoot}+ <NonRootChip mixed={task.hasRoot && task.hasNonRoot} />+ {/if}</Table.Cell><Table.Cell class="tabular-nums">{formatCount(task.count)}